Dump MongoDB daily and upload on S3 with AWS CLI


Dump MongoDB daily and upload on S3 with AWS CLI

Install AWS CLI on Ubuntu

$ sudo apt-get update
$ sudo apt-get install awscli

AWS Credentials

To configure AWS Credentials

$ aws configure

Then you can check ~/.aws/credentials and ~/.aws/config to see the content.

Credentials file

$ cat ~/.aws/credentials

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

Config file

$ cat ~/.aws/config

[default]
output = json
region = eu-central-1

Backup files

First, we need to create a folder and within add two files

Got to the root

$ cd

Create a new folders bin and logs

$ mkdir bin logs

Add a new file to backup the database to S3

$ vi bin/db-backups.sh

And add the content

backup_name=~/db_backups-`date +%Y-%m-%d-%H%M`
mongodump --host localhost --port 27017 --authenticationDatabase admin -u ADMIN_USER -p YOUR_PASSWORD --out $backup_name
tar czf $backup_name.tar.gz $backup_name
aws s3 cp $backup_name.tar.gz s3://YOUR_PATH_HERE
rm -rf $backup_name
rm $backup_name.tar.gz

Export multiple databases with mongodump

But what happens when you need to export multiple databases. The script above it works without any issue, but the real deal goes out when you need to exclude collections with multiple export.

Here is the script

DB_BACKUP_PATH=~/db_backups-`date +%Y-%m-%d-%H%M`

MONGO_HOST=localhost
MONGO_PORT=27017
MONGO_USER=<ADMIN_USER>
MONGO_PASSWORD=<YOUR_PASSWORD>

MONGO_AUTH_STRING="--host $MONGO_HOST --port $MONGO_PORT --authenticationDatabase admin -u $MONGO_USER -p $MONGO_PASSWORD"

dbs=`mongo $MONGO_AUTH_STRING --quiet --eval "db.getMongo().getDBNames()" | tr -d '"' | tr -d '",' | tr -d "[" | tr -d "]"`
for db in $dbs; do
    echo "Dumping database: $db"

    db_export_string="--db=${db}"
    if [ $db == "DATABASE_NAME" ]; then # add the name of your database
        db_export_string+=" --excludeCollection=COLLECTION_NAME" # add the collection name that you want to be excluded
    fi
    mongodump $MONGO_AUTH_STRING $db_export_string --out $DB_BACKUP_PATH
done

tar czf $DB_BACKUP_PATH.tar.gz $DB_BACKUP_PATH
aws s3 cp $backup_name.tar.gz s3://YOUR_PATH_HERE
rm -rf $DB_BACKUP_PATH
rm $DB_BACKUP_PATH.tar.gz

The second file needs to upload the logs to S3

$ vi bin/log-backup.sh 

Add the content

root_folder=/home/ubuntu
backup_name=~/log_backups-`date +%Y-%m-%d-%H%M`
tar czf $backup_name.tar.gz $root_folder/logs
aws s3 cp $backup_name.tar.gz s3://YOUR_PATH_HERE
for f in $root_folder/logs/*.log; do :> $f; done
rm $backup_name.tar.gz

The last step add the following script in crontab

$ crontab -e

At the end of the file add

0 0 * * * /bin/bash ~/bin/db-backups.sh >> ~/logs/db_backups.log 2>&1
0 0 * * * /bin/bash ~/bin/log-backup.sh >> ~/logs/log_backup.log 2>&1

Newsletter


Related Posts

Get your Jenkins Passwords from secrets

How to get your Jenkins passwords from your secrets credentials. Follow this simple tutoriale and find your password or ssh private key.

How to launch a 6 figures startup with 0$ investment

Are you curious how to launch a startup in the fast and effective way? Here's not the answer, but you can understand how to achieve it

How to cold email

How to cold email - a practical guide about how to use cold email in your activity as a CEO of a software agency or startup owner.

Linux disk space alerts on Discord

Linux disk space alerts. Check your space and be responsible with your project. Take care of your space and don't crack the project.

Git clone private repo in docker

A simple how to guide to use git clone of a private repository inside docker container. With vuejs / nodejs app.

How to detect the browser language in javascript

How to detect the browser language in javascript in just a line of code. The method is working on all major browser

You don't need a macbook pro to be a pro

You don't need a macbook pro to be a pro! You need a laptop and internet connection, and a brain, but you already have it

Free templates for startup

Free HTML templates for startup or a new project. A complete list with free resources to build your next startup's website and gain the traction to the sky.

What I learned to never do in a presentation meeting

What I learned to never do in a presentation meeting. I had a very bad meeting experience with a gateway provider.

Startup graveyard is a series of killed projects that made millions in revenue per year

Startup graveyard is a series of killed projects that made millions in revenue per year. Just watch their graveyard and do it your own. What better validation do you expect.