leeroybrun / glacier-vault-remove Goto Github PK

Remove all archives stored inside an Amazon Glacier vault, even if you have a huge number of them.

Python 97.63% Dockerfile 2.37%

glacier-vault-remove's Introduction

glacier-vault-remove

This tool can help you remove an Amazon Glacier vault, even if it's not empty.

It will remove all archives contained inside the vault, and then remove the vault itself.

It is intended to work well with vaults containing a lot of archives. I developed it because my vault contained more than 500'000 archives, and all other softwares crashed when I tried to remove all of them.

Install

Docker

If you have docker installed, you can easily run the script using the leeroyb/glacier-vault-remove image available on Docker Hub:

docker run -v /ABSOLUTE_LOCAL_PATH_TO/credentials.json:/app/credentials.json -d leeroyb/glacier-vault-remove <region-name> [<vault-name>|LIST] [DEBUG] [NUM_PROCESSES] [<job_id>|LIST|NEW|LATEST]

Make sure you use the full absolute path to credentials.json (see below), relative paths do not work here.

Without Docker

You can clone or download this repository and install the dependencies (boto3) by calling the install script :

python setup.py install

Configure

Then create a credentials.json file in the same directory as removeVault.py (or anywhere on your filesystem if using Docker) :

{
	"AWSAccessKeyId": "YOURACCESSKEY",
	"AWSSecretKey":   "YOURSECRETKEY"
}

Use

You can then use the script like this :

python removeVault.py <region-name> [<vault-name>|LIST] [DEBUG] [NUM_PROCESSES] [<job_id>|LIST|NEW|LATEST]

Example :

python removeVault.py eu-west-1 my_vault

Multiple processes

If you want to perform the removal using multiple processes (4 processes here) :

python removeVault.py <region-name> <vault-name> 4

List vaults

If you don't know the vault name, you can generate a list like this:

python removeVault.py <region-name> LIST

Inventory retrieval jobs

A vault inventory is necessary to remove all the archives inside of it. By default, we will look for an existing inventory and use it. If there is none, we will initiate a new one.

You can change this behavior using the job_id argument, with the following values:

LATEST (default): use the latest inventory available, or initiate a new one if there is none
NEW: force initiating a new inventory retrieval job
LIST: list the inventory retrieval jobs availables
<job_id>: give a specific job ID to use

For example to force a new inventory:

python removeVault.py <region-name> <vault-name> NEW

If you want to use 4 processes to remove the archives AND force a new inventory:

python removeVault.py <region-name> <vault-name> 4 NEW

Debug

By default, only the INFO messages will be printed to console. You can add a third argument "DEBUG" to the removeVault.py script if you want to show all messages.

Example :

python removeVault.py <region-name> <vault-name> DEBUG

Building the Docker image

If for whatever reason you don't want to use the leeroyb/glacier-vault-remove image on Docker Hub, you can also build the image yourself from this repository.

Make sure you have docker installed
Clone or download this repository
Build the image:

docker build -t glacier-vault-remove .

Create a credentials.json as described above
Run the tool in the docker container:

docker run -v /path/to/credentials.json:/app/credentials.json glacier-vault-remove <region> <vault|LIST> [DEBUG] [NUM_PROCESSES] [<job_id>|LIST|NEW|LATEST]

Make sure you use the full absolute path to credentials.json, relative paths do not work here.

Licence

(The MIT License)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

glacier-vault-remove's People

Contributors

Stargazers

Watchers

glacier-vault-remove's Issues

A vault is still there after removing of all its archives

Hi Leeroy. Thanks for the project, really helpful. Please advise, the vault is still there after a couple hours after the script successfully removed all the archives. Many thanks for considering my request.

Resetting dropped connection

Hello,

I started the script, but it doesn't seem to do so much. I can list all vaults, it doesn't seem to delete any archive.

$ python removeVault.py eu-west-1 myVault
10:37:14 - INFO : Running with 1 processes
10:37:14 - INFO : Found credentials in environment variables.
10:37:14 - INFO : Starting new HTTPS connection (1): sts.amazonaws.com
10:37:15 - INFO : Working on AccountID: 405759245276
10:37:15 - INFO : Connecting to Amazon Glacier...
10:37:15 - INFO : Getting selected vault... [myVault]
10:37:15 - INFO : Starting new HTTPS connection (1): glacier.eu-west-1.amazonaws.com
10:37:15 - INFO : Working on ARN arn:aws:glacier:eu-west-1:405759245276:vaults/myVault
10:37:15 - INFO : Getting jobs list...
10:37:15 - INFO : No existing job found, initiate inventory retrieval...
10:37:15 - INFO : Starting new HTTPS connection (1): glacier.eu-west-1.amazonaws.com
10:37:15 - INFO : Job ID : xxxx
10:37:15 - INFO : Job Creation Date: 2018-07-05T08:37:15.741Z
10:37:15 - INFO : Inventory not ready, sleep for 10 mins...
10:47:15 - INFO : Resetting dropped connection: glacier.eu-west-1.amazonaws.com
10:47:15 - INFO : Inventory not ready, sleep for 10 mins...
10:57:15 - INFO : Resetting dropped connection: glacier.eu-west-1.amazonaws.com
10:57:16 - INFO : Inventory not ready, sleep for 10 mins...
11:07:16 - INFO : Resetting dropped connection: glacier.eu-west-1.amazonaws.com
11:07:17 - INFO : Inventory not ready, sleep for 10 mins...
11:17:17 - INFO : Resetting dropped connection: glacier.eu-west-1.amazonaws.com
11:17:17 - INFO : Inventory not ready, sleep for 10 mins...
11:27:17 - INFO : Resetting dropped connection: glacier.eu-west-1.amazonaws.com
11:27:17 - INFO : Inventory not ready, sleep for 10 mins...

Is this normal behaviour?

Killed?

Hi there,

Just spun up an Amazon ec2 instance, installed and then got this. It gets to the point of removing archives and then outputs "Killed" and drops me back to the shell. Any thoughts?

Inventory not ready, sleep for 10 mins...

HI
I repeatedly receive Inventory not ready, sleep for 10 mins..., what is the solution?
Thanks

Issues when installing on macOS

I can't seem to install the tool on macOS 10.14 (Mojave). Here's the error I get :

(…)
Installed /Library/Python/2.7/site-packages/GlacierVaultRemove-0.1-py2.7.egg
Processing dependencies for GlacierVaultRemove==0.1
Searching for urllib3<1.26,>=1.20
Reading https://pypi.python.org/simple/urllib3/
Best match: urllib3 1.25.7
Downloading https://files.pythonhosted.org/packages/ad/fc/54d62fa4fc6e675678f9519e677dfc29b8964278d75333cf142892caf015/urllib3-1.25.7.tar.gz#sha256=f3c5fd51747d450d4dcf6f923c81f78f811aab8205fda64b0aba34a4e48b0745
Processing urllib3-1.25.7.tar.gz
Writing /tmp/easy_install-Gj3a2G/urllib3-1.25.7/setup.cfg
Running urllib3-1.25.7/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Gj3a2G/urllib3-1.25.7/egg-dist-tmp-CB4YAZ
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'
warnings.warn(msg)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
error: Setup script exited with error in urllib3 setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.

Can you help ?

ResourceNotFoundException

python removeVault.py us-east-1 LIST produces the following output:

16:38:21 - INFO : Running with 1 processes
16:38:21 - INFO : Found credentials in environment variables.
16:38:23 - INFO : Working on AccountID: 999999999999
16:38:23 - INFO : Connecting to Amazon Glacier...
16:38:23 - INFO : Getting list of vaults...
16:38:25 - INFO : some_vault_name
16:38:25 - INFO : some_vault_name_mapping
16:38:25 - INFO : the_vault_i_want

So I issue this command:

python removeVault.py eu-west-1 the_vault_i_want 4

And I get the following error:

Exception "ResourceNotFoundException" occured with message "An error occurred (ResourceNotFoundException) when calling the DescribeVault operation: Vault not found for ARN: arn:aws:glacier:eu-west-1:058211890657:vaults/Photos"

What am I doing wrong? I get the same results when I substitute the_vault_i_want with some_vault_name. The full DEBUG output follows.

16:47:45 - INFO : Logging level set to DEBUG.
16:47:45 - INFO : Running with 1 processes
16:47:45 - DEBUG : Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
16:47:45 - DEBUG : Changing event name from before-call.apigateway to before-call.api-gateway
16:47:45 - DEBUG : Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
16:47:45 - DEBUG : Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
16:47:45 - DEBUG : Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
16:47:45 - DEBUG : Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
16:47:45 - DEBUG : Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section
16:47:45 - DEBUG : Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
16:47:45 - DEBUG : Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section
16:47:45 - DEBUG : Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
16:47:45 - DEBUG : Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section
16:47:45 - DEBUG : Looking for credentials via: env
16:47:45 - INFO : Found credentials in environment variables.
16:47:45 - DEBUG : Loading JSON file: C:\Users\hargrove\AppData\Local\Programs\Python\Python36-32\lib\site-packages\botocore-1.12.130-py3.6.egg\botocore\data\endpoints.json
16:47:45 - DEBUG : Event choose-service-name: calling handler <function handle_service_name_alias at 0x03C7AF18>
16:47:46 - DEBUG : Loading JSON file: C:\Users\hargrove\AppData\Local\Programs\Python\Python36-32\lib\site-packages\botocore-1.12.130-py3.6.egg\botocore\data\sts\2011-06-15\service-2.json
16:47:46 - DEBUG : Event creating-client-class.sts: calling handler <function add_generate_presigned_url at 0x03B14ED0>
16:47:46 - DEBUG : The s3 config key is not a dictionary type, ignoring its value of: None
16:47:46 - DEBUG : Setting sts timeout as (60, 60)
16:47:46 - DEBUG : Loading JSON file: C:\Users\hargrove\AppData\Local\Programs\Python\Python36-32\lib\site-packages\botocore-1.12.130-py3.6.egg\botocore\data_retry.json
16:47:46 - DEBUG : Registering retry handlers for service: sts
16:47:46 - DEBUG : Event before-parameter-build.sts.GetCallerIdentity: calling handler <function generate_idempotent_uuid at 0x03C8BAE0>
16:47:46 - DEBUG : Event before-call.sts.GetCallerIdentity: calling handler <function inject_api_version_header_if_needed at 0x03C918E8>
16:47:46 - DEBUG : Making request for OperationModel(name=GetCallerIdentity) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'Boto3/1.9.130 Python/3.6.4 Windows/10 Botocore/1.12.130'}, 'body': {'Action': 'GetCallerIdentity', 'Version': '2011-06-15'}, 'url': 'https://sts.amazonaws.com/', 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x03EE84D0>, 'has_streaming_input': False, 'auth_type': None}}
16:47:46 - DEBUG : Event request-created.sts.GetCallerIdentity: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x03EE8410>>
16:47:46 - DEBUG : Event choose-signer.sts.GetCallerIdentity: calling handler <function set_operation_specific_signer at 0x03C8BA50>
16:47:46 - DEBUG : Calculating signature using v4 auth.
16:47:46 - DEBUG : CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:sts.amazonaws.com
x-amz-date:20190405T214746Z

content-type;host;x-amz-date
ab821ae955788b0e33ebd34c208442ccfc2d406e2edc5e7a39bd6458fbb4f843
16:47:46 - DEBUG : StringToSign:
AWS4-HMAC-SHA256
20190405T214746Z
20190405/us-east-1/sts/aws4_request
801a6ca33577d66d1064199d4edfb8f74e7428d2a252966c5a8011be5d35e697
16:47:46 - DEBUG : Signature:
fb979e2c0f091946253011a1edc9e80aacc000335cac479c9e95064688a49331
16:47:46 - DEBUG : Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'Boto3/1.9.130 Python/3.6.4 Windows/10 Botocore/1.12.130', 'X-Amz-Date': b'20190405T214746Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAI27BC4DWAXEILKPQ/20190405/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=fb979e2c0f091946253011a1edc9e80aacc000335cac479c9e95064688a49331', 'Content-Length': '43'}>
16:47:46 - DEBUG : Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
16:47:46 - DEBUG : Starting new HTTPS connection (1): sts.amazonaws.com:443
16:47:47 - DEBUG : https://sts.amazonaws.com:443 "POST / HTTP/1.1" 200 387
16:47:47 - DEBUG : Response headers: {'x-amzn-RequestId': '73c8d2e0-57ec-11e9-9deb-b93098d96c28', 'Content-Type': 'text/xml', 'Content-Length': '387', 'Date': 'Fri, 05 Apr 2019 21:47:47 GMT'}
16:47:47 - DEBUG : Response body:
b'\n \n arn:aws:iam::058211890657:root\n 058211890657\n 058211890657\n \n \n 73c8d2e0-57ec-11e9-9deb-b93098d96c28\n \n\n'
16:47:47 - DEBUG : Event needs-retry.sts.GetCallerIdentity: calling handler <botocore.retryhandler.RetryHandler object at 0x03EE86D0>
16:47:47 - DEBUG : No retry needed.
16:47:47 - INFO : Working on AccountID: 058211890657
16:47:47 - INFO : Connecting to Amazon Glacier...
16:47:47 - DEBUG : Event choose-service-name: calling handler <function handle_service_name_alias at 0x03C7AF18>
16:47:47 - DEBUG : Loading JSON file: C:\Users\hargrove\AppData\Local\Programs\Python\Python36-32\lib\site-packages\botocore-1.12.130-py3.6.egg\botocore\data\glacier\2012-06-01\service-2.json
16:47:47 - DEBUG : Event creating-client-class.glacier: calling handler <function add_generate_presigned_url at 0x03B14ED0>
16:47:47 - DEBUG : The s3 config key is not a dictionary type, ignoring its value of: None
16:47:47 - DEBUG : Setting glacier timeout as (60, 60)
16:47:47 - DEBUG : Registering retry handlers for service: glacier
16:47:47 - INFO : Getting selected vault... [the_vault_i_want]
16:47:47 - DEBUG : Event before-parameter-build.glacier.DescribeVault: calling handler <function inject_account_id at 0x03C914B0>
16:47:47 - DEBUG : Event before-parameter-build.glacier.DescribeVault: calling handler <function generate_idempotent_uuid at 0x03C8BAE0>
16:47:47 - DEBUG : Event before-call.glacier.DescribeVault: calling handler <function add_glacier_version at 0x03C914F8>
16:47:47 - DEBUG : Event before-call.glacier.DescribeVault: calling handler <function inject_api_version_header_if_needed at 0x03C918E8>
16:47:47 - DEBUG : Making request for OperationModel(name=DescribeVault) with params: {'url_path': '/-/vaults/the_vault_i_want', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.9.130 Python/3.6.4 Windows/10 Botocore/1.12.130', 'x-amz-glacier-version': '2012-06-01'}, 'body': b'', 'url': 'https://glacier.eu-west-1.amazonaws.com/-/vaults/the_vault_i_want', 'context': {'client_region': 'eu-west-1', 'client_config': <botocore.config.Config object at 0x045C4170>, 'has_streaming_input': False, 'auth_type': None}}
16:47:47 - DEBUG : Event request-created.glacier.DescribeVault: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x045C4110>>
16:47:47 - DEBUG : Event choose-signer.glacier.DescribeVault: calling handler <function set_operation_specific_signer at 0x03C8BA50>
16:47:47 - DEBUG : Calculating signature using v4 auth.
16:47:47 - DEBUG : CanonicalRequest:
GET
/-/vaults/the_vault_i_want

host:glacier.eu-west-1.amazonaws.com
x-amz-date:20190405T214747Z
x-amz-glacier-version:2012-06-01

host;x-amz-date;x-amz-glacier-version
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
16:47:47 - DEBUG : StringToSign:
AWS4-HMAC-SHA256
20190405T214747Z
20190405/eu-west-1/glacier/aws4_request
b5d496b49748bf116a47814d922eed3129d76080c8a6ac70192669a3292e5faa
16:47:47 - DEBUG : Signature:
0c53981ba328115e9d892e8af82b3c6d3b049267bcb06ebcd3ad0f4d99c6e3f1
16:47:47 - DEBUG : Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://glacier.eu-west-1.amazonaws.com/-/vaults/the_vault_i_want, headers={'User-Agent': b'Boto3/1.9.130 Python/3.6.4 Windows/10 Botocore/1.12.130', 'x-amz-glacier-version': b'2012-06-01', 'X-Amz-Date': b'20190405T214747Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAI27BC4DWAXEILKPQ/20190405/eu-west-1/glacier/aws4_request, SignedHeaders=host;x-amz-date;x-amz-glacier-version, Signature=0c53981ba328115e9d892e8af82b3c6d3b049267bcb06ebcd3ad0f4d99c6e3f1'}>
16:47:47 - DEBUG : Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
16:47:47 - DEBUG : Starting new HTTPS connection (1): glacier.eu-west-1.amazonaws.com:443
16:47:49 - DEBUG : https://glacier.eu-west-1.amazonaws.com:443 "GET /-/vaults/the_vault_i_want HTTP/1.1" 404 142
16:47:49 - DEBUG : Response headers: {'x-amzn-RequestId': 'Qc5bvSHT9xbxNQ5GHJXFD0SXLt_nMRvjeF-evo5VwshsMgk', 'Content-Type': 'application/json', 'Content-Length': '142', 'Date': 'Fri, 05 Apr 2019 21:47:48 GMT'}
16:47:49 - DEBUG : Response body:
b'{"code":"ResourceNotFoundException","message":"Vault not found for ARN: arn:aws:glacier:eu-west-1:058211890657:vaults/the_vault_i_want","type":"Client"}'
16:47:49 - DEBUG : Event needs-retry.glacier.DescribeVault: calling handler <botocore.retryhandler.RetryHandler object at 0x045C4590>
16:47:49 - DEBUG : No retry needed.
16:47:49 - ERROR : Exception "ResourceNotFoundException" occured with message "An error occurred (ResourceNotFoundException) when calling the DescribeVault operation: Vault not found for ARN: arn:aws:glacier:eu-west-1:058211890657:vaults/the_vault_i_want"

Wrong format output of get_job_output

I try to face against something strange, in your code you in the output of "get_job_ouput", the variable "body" should contain a string parsable to Json, but for me if I print this value: job_output['body'].read().decode('utf-8')
I got something to csv format, I try to take a look at the boto3 documentation and I don't find any solution to change the output format for the moment :/
I try to use initiate_job function to specify the Format variable to JSON, but I don't understand how this function works...
So if anyone got the same problem and can help me :D

Deletes not working?

I have 411 items at 1.8 TB. The script takes about 5 hours, then this:

23:23:56 - INFO : 12385 Remove archive number 410 of 411, ID : XXX-XXX
23:23:56 - INFO : Removing vault...
23:23:56 - INFO : Resetting dropped connection: glacier.us-east-1.amazonaws.com
23:23:56 - ERROR : Exception "InvalidParameterValueException" occured with message "An error occurred (InvalidParameterValueException) when calling the DeleteVault operation: Vault not empty or recently written to: arn:aws:glacier:us-east-1:XXX:vaults/xxx"
23:23:56 - ERROR : We cant remove the vault now. Please wait some time and try again. You can also remove it from the AWS console, now that all archives have been removed.

The count starts a 0, so 410 of 411 is actually correct. I have "waited some time" (about 24 hours) as instructed, but I still cannot delete my vault on the AWS console. In fact, the vault is still showing that it was last updated in 2015 and still has 411 archives. Any ideas on this?

Access via Synology?

Hi, sorry if this isn't the correct spot for it, i'm trying to get this up & running!

I have a Synology dS918+ which I installed Docker on.

This is an upgrade for me from a previous Synology, a DS212. Mostly used for photo & video storage.

Anyway, I run an Amazon Glacier Backup via Synology. When I switched NAS devices, it created an entirely new S3 Vault. I'm unable to delete my old vault (from the DS212), because of an error saying, "Vault not empty or recently written to "

After some googling, I found a solution your solution!

I have successfully setup the Container within my DS918+, but I can't seem to get it to run at all.

I created the requested file (credentials.json) and put them into my /Docker shared folder on the NAS.

But whenever I start the container, it just shuts down immediately.

https://imgur.com/vwMPZkR

I'm not sure what I'm doing wrong, it's probably something super obvious, but i've been googling and trying things for almost 2 hours and made no progress. My guess is maybe it's shutting down because credentials aren't connecting because I didn't put them in the right place? The Docker shared folder where I placed them on my NAS is otherwise empty. I can't run any scripts because the container just shuts down a second after I start it.

Edit: I ended up getting this running via Ubuntu in VMWare Workstation instead of Docker. That worked fine, although after it ran I still don't seem to be able to delete the vault; seems like it may just take some additional time.

Trackback Error

I always get the following error message:

INFO : Removing 76480 archives... please be patient, this may take some time...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\runpy.py", line 263, in run_path
Traceback (most recent call last):
  File "C:\...\Documents\GitHub\glacier-vault-remove\removeVault.py", line 174, in <module>
    pkg_name=pkg_name, script_name=fname)
    p.start()
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\runpy.py", line 96, in _run_module_code
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\process.py", line 105, in start
    mod_name, mod_spec, pkg_name, script_name)
    self._popen = self._Popen(self)
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\runpy.py", line 85, in _run_code
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\context.py", line 223, in _Popen
    exec(code, run_globals)
  File "C:\...\Documents\GitHub\glacier-vault-remove\removeVault.py", line 174, in <module>
    p.start()
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\process.py", line 105, in start
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\context.py", line 322, in _Popen
    self._popen = self._Popen(self)
    return Popen(process_obj)
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\context.py", line 322, in _Popen
    reduction.dump(process_obj, to_child)
    return Popen(process_obj)
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\reduction.py", line 60, in dump
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    ForkingPickler(file, protocol).dump(obj)
    _check_not_importing_main()
BrokenPipeError: [Errno 32] Broken pipe
  File "C:\...\AppData\Local\Programs\Python\Python36-32\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

Inventory not ready, sleep for 10 mins

I get this error message:

21:04:18 - INFO : Inventory not ready, sleep for 10 mins...

In a loop, even after waiting for 10 minutes several time, there is no progress.
Whta's the issue in this case?

Thanks

Docker image in docker registry.

Hi! Thank you very much for this software, I have seen that it even has a Dockerfile. I have a suggestion, maybe you can include an image in the docker hub so people don't need to construct in his own machine, install dep, etc.

Have you think about it? :D

Error: Too Many Open Files

Hi. I get the error "Too many open files" using Ubuntu Server in a VirtualBox environment.

File "removeVault.py", line 81, in <module>

Thank you for writing this tool. I'm trying to remove a couple old crufty glacier archives. After trying to write my own boto utility I stumbled on yours. It is throwing the following error however. Any ideas what I can do to fix this?

INFO : Logging level set to DEBUG.
INFO : Connecting to Amazon Glacier...
INFO : Getting selected vault...
INFO : Getting jobs list...
INFO : Found existing inventory retrieval job...
INFO : Inventory retrieved, parsing data...
Traceback (most recent call last):
File "removeVault.py", line 81, in
inventory = json.loads(job.get_output().read())
File "/Library/Python/2.7/site-packages/boto/glacier/job.py", line 89, in get_output
byte_range)
File "/Library/Python/2.7/site-packages/boto/glacier/layer1.py", line 740, in get_job_output
response_headers=response_headers)
File "/Library/Python/2.7/site-packages/boto/glacier/layer1.py", line 116, in make_request
return GlacierResponse(response, response_headers)
File "/Library/Python/2.7/site-packages/boto/glacier/response.py", line 40, in init
body = json.loads(http_response.read().decode('utf-8'))
File "/Library/Python/2.7/site-packages/boto/connection.py", line 410, in read
self._cached_response = http_client.HTTPResponse.read(self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 551, in read
s = self._safe_read(self.length)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 658, in _safe_read
chunk = self.fp.read(min(amt, MAXAMOUNT))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 241, in recv
return self.read(buflen)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 160, in read
return self._sslobj.read(len)
ssl.SSLError: The read operation timed out

HTTPS connection fails

I've created a new Access and secret key but i repeatedly get the error security token included in the request is invalid. I've double and triple checked these and they are correct,

Exception "gaierror" with "Errno -3"

After around 900 succesful removals I end up getting an exception gaierror with message [Errno -3] Try again. Restarting Docker fixes this, but I start at 0 again.

Full error:

19:44:19 - ERROR : Exception "gaierror" occured with message "[Errno -3] Try again"

UnexpectedHTTPResponseError

I'm attempting to remove a Vault with 115 Archives in it. When I run the command, I'm getting an error (private info removed below):

codydh@dev:~/glacier-vault-remove# python ./removeVault.py us-east-1 vaultname
INFO : Connecting to Amazon Glacier...
INFO : Getting selected vault...
INFO : Getting jobs list...
INFO : Found existing inventory retrieval job...
INFO : Inventory retrieved, parsing data...
INFO : Removing archives... please be patient, this may take some time...
INFO : Removing vault...
Traceback (most recent call last):
  File "./removeVault.py", line 98, in <module>
    vault.delete()
  File "/usr/local/lib/python2.7/dist-packages/boto-2.15.0-py2.7.egg/boto/glacier/vault.py", line 71, in delete
    self.layer1.delete_vault(self.name)
  File "/usr/local/lib/python2.7/dist-packages/boto-2.15.0-py2.7.egg/boto/glacier/layer1.py", line 189, in delete_vault
    return self.make_request('DELETE', uri, ok_responses=(204,))
  File "/usr/local/lib/python2.7/dist-packages/boto-2.15.0-py2.7.egg/boto/glacier/layer1.py", line 84, in make_request
    raise UnexpectedHTTPResponseError(ok_responses, response)
boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (400, code=InvalidParameterValueException, message=Vault not empty or recently written to: arn:aws:glacier:us-east-1:449410151148:vaults/vaultname)
codydh@dev:~/glacier-vault-remove#

even less memory usage

Hey guys, any interest on reducing even more the memory usage of this thing ? As of now this script receives the json string from the inventory retrieval job as a whole, loads all of it in memory. I had a glacier vault with hundreds of thousands of archives, a >1GB job output. Very impossible to do even on a machine with a lot of ram.
I wrote a small change to glacier-vault-remove that uses a queue for the multiprocessing part, and a json streaming mechanisms from the job_output (using the range parameter) so that we are not reading the whole job output all at once.
https://github.com/rienafairefr/glacier-vault-remove/tree/jsonstream2
Is it something that you guys are interested in pulling in here ?

Can`t start deleting

Hi!
Thank you very much fo the script! I cant start deleting with the code python .\removeVault.py eu-west-1 my_vault. Ubuntu 16.04. Perhaps, it is not an issue, maybe I did something wrong. But when I removed .\ , deleting started. Now it works with python removeVault.py eu-west-1 my_vault. Maybe it could be useful for somebody, cause I tried to start 4 hours)))). Thank you!

Run on Synology docker

Hi, sorry if this isn't the correct spot for it, i'm trying to get this up & running!

I have a Synology dS918+ which I installed Docker on.

This is an upgrade for me from a previous Synology, a DS212. Mostly used for photo & video storage.

After some googling, I found a solution your solution!

I have successfully setup the Container within my DS918+, but I can't seem to get it to run at all.

I created the requested file (credentials.json) and put them into my /Docker shared folder on the NAS.

But whenever I start the container, it just shuts down immediately.

https://imgur.com/vwMPZkR

I have same issue as above. Anyone can advise?

Errno8 : ERROR : Exception "gaierror" occured with message

Hello,

When I try to delete archives on my Glacier vault, I have the following error : ERROR : Exception "gaierror" occured with message "[Errno 8] nodename nor servname provided, or not known".
How can I solve it ?

Regards,

ERROR : [Errno 185090050] _ssl.c:344: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib

Thanks for you lib! I have been paying for Glacier vaults I don't want for months.

I am getting the following error when trying to delete my vaults. It always works on 2nd retry after 2 mins.
https://gist.github.com/jeberly/4f92dd5829b13eb95bb2

Thanks in advance.

in case it matters, boto version 2.38.0, Python 2.7.6

Dockerfile installing incorrect version of boto

Running the container as described in the readme produces following error

$: docker run -v $(pwd)/credentials.json:/app/credentials.json glacier-test eu-central-1 LIST
Traceback (most recent call last):
  File "/app/removeVault.py", line 10, in <module>
    import boto3
ImportError: No module named boto3

Dockerfile installs an earlier version of boto than the one being used by the python scripts. Expected version is boto3.

Proposing installing pip, then installing boto3 via pip.

Missing option for forcing new inventory retrieval

Hello,

Maybe there would be cool to have ability for forcing new inventory retrieval if something fail or so, like this happened me yesterday. If you want I can provide you PR.
If you do not have time for maintaining this repository, you could move ownership to me and I would maintain it, provide auto pip packaging and also auto docker building if you want.

Error deleting vault with around 7m files

Hi getting an error when removing a vault with lots of files (circa 7m)... any help much appreciated.

Traceback (most recent call last):
File "./removeVault.py", line 92, in
inventory = json.loads(job.get_output().read())
File "/usr/local/lib/python2.7/dist-packages/boto-2.38.0-py2.7.egg/boto/glacier/job.py", line 89, in get_output
byte_range)
File "/usr/local/lib/python2.7/dist-packages/boto-2.38.0-py2.7.egg/boto/glacier/layer1.py", line 740, in get_job_output
response_headers=response_headers)
File "/usr/local/lib/python2.7/dist-packages/boto-2.38.0-py2.7.egg/boto/glacier/layer1.py", line 116, in make_request
return GlacierResponse(response, response_headers)
File "/usr/local/lib/python2.7/dist-packages/boto-2.38.0-py2.7.egg/boto/glacier/response.py", line 41, in init
body = json.loads(http_response.read().decode('utf-8'))
File "/usr/local/lib/python2.7/dist-packages/boto-2.38.0-py2.7.egg/boto/connection.py", line 410, in read
self._cached_response = http_client.HTTPResponse.read(self)
File "/usr/lib/python2.7/httplib.py", line 551, in read
s = self._safe_read(self.length)
File "/usr/lib/python2.7/httplib.py", line 660, in _safe_read
raise IncompleteRead(''.join(s), amt)
httplib.IncompleteRead: IncompleteRead(155886763 bytes read, 2697175103 more expected)

Error when running removeVault

Hi,

Thanks so much for this script. I have a vault with ~750,000 archives and I need to delete them all. I've installed the script and configured it but it is failing early with the following output.

root@domU-12-31-39-09-86-67:/tmp/glacier/glacier-vault-remove# python ./removeVault.py us-east-1 DiskStation_0011321CBE91_1 DEBUG
INFO : Logging level set to DEBUG.
INFO : Connecting to Amazon Glacier...
INFO : Getting selected vault...
INFO : Getting jobs list...
INFO : Found existing inventory retrieval job...
INFO : Inventory retrieved, parsing data...
Traceback (most recent call last):
File "./removeVault.py", line 80, in
inventory = json.loads(job.get_output().read())
File "/usr/lib/python2.7/json/init.py", line 326, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")

I really hope I can get this script working! Every other option I've tried to delete this vault's contents has failed so far. Thank you.

boto 'UnexpectedHTTPResponseError'

It raised following error message:

$ python removeVault.py ap-northeast-1 LIST DEBUG
INFO : Logging level set to DEBUG.
INFO : Connecting to Amazon Glacier...
INFO : Getting list of vaults...
ERROR : <class 'boto.glacier.exceptions.UnexpectedHTTPResponseError'>

I am using OS X El Capitan, and my boto is:

$pip freeze | grep boto
boto==2.38.0
botocore==1.1.1

leeroybrun / glacier-vault-remove Goto Github PK

glacier-vault-remove's Introduction

glacier-vault-remove

Install

Docker

Without Docker

Configure

Use

Multiple processes

List vaults

Inventory retrieval jobs

Debug

Building the Docker image

Licence

glacier-vault-remove's People

Contributors

Stargazers

Watchers

Forkers

glacier-vault-remove's Issues

Recommend Projects

Recommend Topics

Recommend Org