georgin-lau / aws-s3-glacier-restore Goto Github PK

Utility script to restore files on AWS S3 that have GLACIER storage class.

• Calculates exact costs before restore
• Multithreaded API requests significantly reduce restore time
• Can check if all files for given prefix are already restored or not.

aws-s3-glacier-restore is developed in Python and uses pip.

The easiest way to install/upgrade s3select is to use pip in a virtualenv:

$ pip install -U aws-s3-glacier-restore

or, if you are not installing in a virtualenv, to install/upgrade globally:

$ sudo pip install -U aws-s3-glacier-restore

or for your user:

$ pip install --user -U aws-s3-glacier-restore

aws-s3-glacier-restore uses the same authentication and endpoint configuration as aws-cli. If aws command is working on your machine, there is no need for any additional configuration.

First get some help:

$ aws-s3-glacier-restore -h
usage: aws-s3-glacier-restore [-h] [-p PREFIX] [-i INPUT_FILE]
                              [-d DAYS_TO_KEEP] [-D DESTINATION_BUCKET]
                              [-t THREADS] [-s] [--profile PROFILE]

Utility script to restore files on AWS S3 that have GLACIER storage class

optional arguments:
  -h, --help            show this help message and exit
  -p PREFIX, --prefix PREFIX
                        S3 prefix URL to restore
  -i INPUT_FILE, --input_file INPUT_FILE
                        Input file containing all s3 paths to restore
  -d DAYS_TO_KEEP, --days_to_keep DAYS_TO_KEEP
                        How many days to keep restored files
  -D DESTINATION_BUCKET, --destination_bucket DESTINATION_BUCKET
                        Restore to this bucket instead of to original bucket
                        while preserving same path structure as in original
                        bucket. This is useful if you don't know for how long
                        you'll need restored files. Once you don't need them
                        you can delete them from destination bucket.
  -t THREADS, --threads THREADS
                        Number of threads to use. Default=40
  -s, --status_print    Just print status of files and how many of them are in
                        glacier and how many of them are restored already
  --profile PROFILE     Use a specific AWS profile from your credential file.

Then full run might look like this:

$ aws-s3-glacier-restore -p s3://test-restore-bucket/backup2/image_1503789115/part-r-0003 -d 2

Getting a listing of the files to restore... Done

About to restore 18.72GB in 10 files

Restore will cost us:
1) Expedited tier: $0.66
2) Standard tier:  $0.19
3) Bulk tier:      $0.05
Keeping files on S3 will cost: 0.02 per day
 
Press number in front of an option you wish or any other key to exit: 3
Starting restore using Bulk tier... Restoring backup2/image_1503789115/part-r-00030.gz
Restoring backup2/image_1503789115/part-r-00031.gz
Restoring backup2/image_1503789115/part-r-00032.gz
Restoring backup2/image_1503789115/part-r-00034.gz
Restoring backup2/image_1503789115/part-r-00036.gz
Restoring backup2/image_1503789115/part-r-00038.gz
Restoring backup2/image_1503789115/part-r-00033.gz
Restoring backup2/image_1503789115/part-r-00037.gz
Restoring backup2/image_1503789115/part-r-00035.gz
Restoring backup2/image_1503789115/part-r-00039.gz
Done
Time elapsed: 2s

Parameter --days_to_keep (-d) specifies how long files will be accessible as regular files over S3. In case you have finished with files earlier than expected or want to extend the amount of days they are kept available, you can run the same command, but this time with modified "-d" parameter. If you specify "-d 1" this will signal that files should be archived back as soon as possible (usually happens within one day):

$ aws-s3-glacier-restore -p s3://test-restore-bucket/backup2/image_1503789115/part-r-0003 -d 0

If you want to check restore status of files you can use -s switch:

$ aws-s3-glacier-restore -p s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-0009 -s
Getting a listing of the files... Done

Object s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-00091 is being restored
Object s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-00092 is in Glacier and not being restored
Object s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-00093 is in Glacier and not being restored
Object s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-00095 is in Glacier and not being restored
Object s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-00094 is in Glacier and not being restored
Object s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-00090 is restored until Fri, 19 Oct 2018 00:00:00 GMT
Object s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-00098 is restored until Fri, 19 Oct 2018 00:00:00 GMT
Object s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-00096 is restored until Mon, 24 Sep 2018 00:00:00 GMT
Object s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-00099 is in Glacier and not being restored
Object s3://test-restore-bucket/archives/cars/GlacierImageArchive_2017_10-00000001-r-00097 is in Glacier and not being restored
Restored count: 3/10

Distributed under the MIT license. See LICENSE for more information.