Dump MongoDB daily and upload on S3 with AWS CLI


Dump MongoDB daily and upload on S3 with AWS CLI

Install AWS CLI on Ubuntu

$ sudo apt-get update
$ sudo apt-get install awscli

AWS Credentials

To configure AWS Credentials

$ aws configure

Then you can check ~/.aws/credentials and ~/.aws/config to see the content.

Credentials file

$ cat ~/.aws/credentials

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

Config file

$ cat ~/.aws/config

[default]
output = json
region = eu-central-1

Backup files

First, we need to create a folder and within add two files

Got to the root

$ cd

Create a new folders bin and logs

$ mkdir bin logs

Add a new file to backup the database to S3

$ vi bin/db-backups.sh

And add the content

backup_name=~/db_backups-`date +%Y-%m-%d-%H%M`
mongodump --host localhost --port 27017 --authenticationDatabase admin -u ADMIN_USER -p YOUR_PASSWORD --out $backup_name
tar czf $backup_name.tar.gz $backup_name
aws s3 cp $backup_name.tar.gz s3://YOUR_PATH_HERE
rm -rf $backup_name
rm $backup_name.tar.gz

Export multiple databases with mongodump

But what happens when you need to export multiple databases. The script above it works without any issue, but the real deal goes out when you need to exclude collections with multiple export.

Here is the script

DB_BACKUP_PATH=~/db_backups-`date +%Y-%m-%d-%H%M`

MONGO_HOST=localhost
MONGO_PORT=27017
MONGO_USER=<ADMIN_USER>
MONGO_PASSWORD=<YOUR_PASSWORD>

MONGO_AUTH_STRING="--host $MONGO_HOST --port $MONGO_PORT --authenticationDatabase admin -u $MONGO_USER -p $MONGO_PASSWORD"

dbs=`mongo $MONGO_AUTH_STRING --quiet --eval "db.getMongo().getDBNames()" | tr -d '"' | tr -d '",' | tr -d "[" | tr -d "]"`
for db in $dbs; do
    echo "Dumping database: $db"

    db_export_string="--db=${db}"
    if [ $db == "DATABASE_NAME" ]; then # add the name of your database
        db_export_string+=" --excludeCollection=COLLECTION_NAME" # add the collection name that you want to be excluded
    fi
    mongodump $MONGO_AUTH_STRING $db_export_string --out $DB_BACKUP_PATH
done

tar czf $DB_BACKUP_PATH.tar.gz $DB_BACKUP_PATH
aws s3 cp $backup_name.tar.gz s3://YOUR_PATH_HERE
rm -rf $DB_BACKUP_PATH
rm $DB_BACKUP_PATH.tar.gz

The second file needs to upload the logs to S3

$ vi bin/log-backup.sh 

Add the content

root_folder=/home/ubuntu
backup_name=~/log_backups-`date +%Y-%m-%d-%H%M`
tar czf $backup_name.tar.gz $root_folder/logs
aws s3 cp $backup_name.tar.gz s3://YOUR_PATH_HERE
for f in $root_folder/logs/*.log; do :> $f; done
rm $backup_name.tar.gz

The last step add the following script in crontab

$ crontab -e

At the end of the file add

0 0 * * * /bin/bash ~/bin/db-backups.sh >> ~/logs/db_backups.log 2>&1
0 0 * * * /bin/bash ~/bin/log-backup.sh >> ~/logs/log_backup.log 2>&1

Newsletter


Related Posts

Discover the Best Free AI Art Tools for Your Next Masterpiece

Explore a curated collection of the finest free AI art tools, designed to help you bring your artistic vision to life.

SaaS vs. Software Agency: Choosing the Right Business Model for Your Startup

Discover the key differences between SaaS and Software Agencies to make informed business decisions and stay ahead in the competitive market.

Top ChatGPT Prompts for Technical People and Startup Founders

Discover the ultimate list of ChatGPT prompts designed specifically for technical people and startup founders. Enhance your AI interactions and boost productivity with these powerful prompts. Revolutionize the way you work and stay ahead of the competition. Read now!

Chatbots for your website

Chatbots for your website

Discover the Power of Whatsapp-web.js for Safe and Easy Automation

Automate WhatsApp with ease using Whatsapp-web.js, a NodeJS client library that connects through the official WhatsApp Web app, reducing ban risks. Perfect for user or business accounts.

Revving Up Success: How to Validate Your Car Alert App Idea

Discover the keys to success in the car alert app market. Learn how to validate your business idea, ensuring your app sends timely alerts for document expirations and service checks.

Navigating the Food Delivery Fleet Management Landscape with ManagerLivrari

Discover Manager Livrari: Your all-in-one solution for streamlining food delivery fleet management. Automate calculations, generate weekly reports, and optimize your operations with ease.

Tools for Analyzing Your Competition

Before knowing where your business stands, you should look at the market and your competition. You need to know what marketing tactics and strategies are working for your audience, what traffic the competition has, and how they position.

Free Online Photo Editor with Cool Effects | Edit Images Easily

Transform your images with our free online photo editor with cool effects. Add filters, adjust colors, and apply creative enhancements. Edit your photos easily and make them stand out.

How to find the size of a table in SQL?

Learn how to determine the size of a SQL table with this informative guide. Discover efficient methods for measuring table size.