Giter VIP home page Giter VIP logo

mongo-s3-backup's Introduction

mongo-s3-backup

A Docker container to backup a MongoDB deployment to S3 and list or restore those backups.

Usage

Backup

The simplest backup you can do is by only providing the S3 config and a MONGO_HOST value. MONGO_HOST can be in any form that is supported by the mongodump or mongorestore --host flag. See more detail on that in the docs. Specifically, note that you can provide a single Mongo host or a comma-separated list of replica set hosts. For a replica set, be sure to supply the replica set name before the hosts. The formats are as follows:

# single host
export MONGO_HOST="mymongo.com:27017"

# replica set (where the replica set name is "rs0")
export MONGO_HOST="rs0/one.mymongo.com:27017,two.mymongo.com:27017,three.mymongo.com:27017"

Now run a backup...

docker run --rm \
  -e AWS_ACCESS_KEY_ID=<key> \
  -e AWS_SECRET_ACCESS_KEY=<secret> \
  -e S3_BUCKET=<bucket> \
  -e MONGO_HOST=$MONGO_HOST \
  reactioncommerce/mongo-s3-backup

To link to a running Mongo container (named mongo) and run a backup, just ensure the link resolves to the name mongo inside the container. You can do that with --link flag. The format is --link your-mongo-name:mongo

docker run --rm \
  --link your-mongo-name:mongo \
  -e AWS_ACCESS_KEY_ID=<key> \
  -e AWS_SECRET_ACCESS_KEY=<secret> \
  -e S3_BUCKET=<bucket> \
  reactioncommerce/mongo-s3-backup

You can optionally provide any supported mongodump flags with the $MONGODUMP_FLAGS variable. For example, to provide a specific database:

docker run --rm \
  -e AWS_ACCESS_KEY_ID=<key> \
  -e AWS_SECRET_ACCESS_KEY=<secret> \
  -e S3_BUCKET=<bucket> \
  -e MONGO_HOST=$MONGO_HOST \
  -e MONGODUMP_FLAGS="--db mydatabase" \
  reactioncommerce/mongo-s3-backup

Or any amount of additional flags...

docker run --rm \
  -e AWS_ACCESS_KEY_ID=<key> \
  -e AWS_SECRET_ACCESS_KEY=<secret> \
  -e S3_BUCKET=<bucket> \
  -e MONGO_HOST=$MONGO_HOST \
  -e MONGODUMP_FLAGS="--db mydatabase --username <myuser> --password <pass123> --oplog" \
  reactioncommerce/mongo-s3-backup

Custom backup names

The file names of backups can be customized with the following environment variables:

  • FILE_PREFIX add a file name prefix to a backup's name. Default: blank.
  • DATE_FORMAT accepts a Unix date format. Default: %Y%m%d_%H%M%S

To create a file with a name like mydb.2018-01-07_13-10-43.tar.gz you would do this:

docker run --rm \
  -e FILE_PREFIX=mydb. \
  -e DATE_FORMAT=%Y-%m-%d_%H-%M-%S \
  -e AWS_ACCESS_KEY_ID=<key> \
  -e AWS_SECRET_ACCESS_KEY=<secret> \
  -e S3_BUCKET=<bucket> \
  -e MONGO_HOST=$MONGO_HOST \
  reactioncommerce/mongo-s3-backup

List

To list the backups on S3:

docker run --rm \
  -e AWS_ACCESS_KEY_ID=<key> \
  -e AWS_SECRET_ACCESS_KEY=<secret> \
  -e S3_BUCKET=<bucket> \
  reactioncommerce/mongo-s3-backup list

Restore

To restore a specific backup, provide the name of the backup within the S3 bucket:

docker run --rm \
  -e AWS_ACCESS_KEY_ID=<key> \
  -e AWS_SECRET_ACCESS_KEY=<secret> \
  -e S3_BUCKET=<bucket> \
  -e MONGO_HOST=$MONGO_HOST \
  reactioncommerce/mongo-s3-backup restore <file>

You can optionally provide any supported mongorestore flags with the $MONGORESTORE_FLAGS variable. For example, to a backup that came from a replica set dump that used the --oplog flag, you can replay the oplog for the restore like this:

docker run --rm \
  -e AWS_ACCESS_KEY_ID=<key> \
  -e AWS_SECRET_ACCESS_KEY=<secret> \
  -e S3_BUCKET=<bucket> \
  -e MONGO_HOST=$MONGO_HOST \
  -e MONGORESTORE_FLAGS="--oplogReplay" \
  reactioncommerce/mongo-s3-backup restore <file>

See the mongorestore docs for all available flags.

Restore Latest

To restore the latest backup on S3:

docker run --rm \
  -e AWS_ACCESS_KEY_ID=<key> \
  -e AWS_SECRET_ACCESS_KEY=<secret> \
  -e S3_BUCKET=<bucket> \
  -e MONGO_HOST=$MONGO_HOST \
  reactioncommerce/mongo-s3-backup restore latest

A sort is used to determine the latest backup. If a FILE_PREFIX is defined, this will filter the bucket list results by the FILE_PREFIX that was originally used. If you are using a custom DATE_FORMAT, you will need to set that variable as well to ensure the sort order will still list the correct date order.

mongo-s3-backup's People

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mongo-s3-backup's Issues

mongo-s3-backup never returns back

This call never returns mongoBackup.dumpToS3(config, function (err, data) {
console.log('E', err)
console.log('D', data)
});

Any Solution ?

Unsatisfiable constraints (python) and AWS CLI v2

This repo uses alpine:latest and, as a result, those without a cached image will grab the latest version fo Alpine Linux. However, upon running the included Dockerfile fails on the latest version of Alpine with:

ERROR: unsatisfiable constraints:
  python (missing):
    required by: world[python]

Replacing python with python3 and symlinking the binary may be a possible. But it seems this becomes a slippery slope as one soon learns AWS CLI no longer uses py-pip for installation: neither CLI v1 or the newer v2.

Attempting to install AWS CLI v2 using the new documented method (after adding curl and sudo dependencies) leads to an unintelligible error ostensibly as a result of a missing glibc installation:

./aws/install: line 78: /aws/dist/aws2: not found

A related issue was opened against the CLI in which a contributor suggests Alpine isn't supported nor is support of alpine likely. The contributor does suggest using Ubuntu instead of Alpine but this Dockerfile was written to use APK and a micro-container approach and so it's not clear what the best course of action is for backing up Reaction Commerce's MongoDB.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.