Giter VIP home page Giter VIP logo

wajihcz / s3nukem Goto Github PK

View Code? Open in Web Editor NEW

This project forked from lathanh/s3nukem

0.0 2.0 0.0 122 KB

A single-file Ruby script to delete Amazon S3 an bucket with many objects (millions) quickly by using multiple threads to retrieve and delete the individual objects.

Home Page: http://robertlathanh.com/2010/07/s3nukem-delete-large-amazon-s3-buckets/

License: Apache License 2.0

Ruby 100.00%

s3nukem's Introduction

S3NUKEM

s3nukem is a slightly improved version of s3nuke, a Ruby script by Steve Eley that relatively quickly deletes an Amazon Web Services (AWS) Simple Storage Service (S3) bucket with many objects (millions) by using multiple threads to retrieve and delete the individual objects.

In my use case, s3cmd, which deletes with a single thread, deleted objects at a rate of about 1,800/minute (2.5 million / day). s3nukem, with 10 delete threads deleted objects at a rate of about 9,000/minute (13 million / day). My task of deleting 99 million objects went from 40 days to 7.6 days. More threads and Ruby 1.9 (I was using 1.8.5) would have probably completed the job even more quickly.

Improvements

  • The key retrieval thread will pause when the queue contains 1000 * thread_count items. The original script's queue would grow unabated, eating up memory unnecessarily.

  • All output is automatically flushed, which ensures you can keep an eye on progress in real-time

  • Added the number of seconds elapsed since the start of the script to output so you can calculate the rate at which items are being deleted.

Installation

You'll need:

  • Ruby Ruby 1.9 should work faster because of the native thread implementation (on the other hand, network/S3 latency may be your biggest bottleneck).

  • right_aws gem; dmarkow's version if you're running Ruby >= 1.9

      # Ruby < 1.9
      sudo gem install right_aws
    
      # Ruby >= 1.9
      sudo gem install dmarkow-right_aws --source http://gems.github.com
    

Download and make executable; e.g.,

# download
wget https://raw.github.com/lathanh/s3nukem/master/s3nukem
# or
curl -O https://raw.github.com/lathanh/s3nukem/master/s3nukem

# make executable
chmod 755 s3nukem

Obvious Warning

This script is intended to delete all of the items in an S3 bucket very quickly. You will not be prompted to ask you if you're sure. There is no undo.

Do not taunt Happy Fun Script.

License

This script is released under the Apache License, Version 2.0. I really don't care what you do with it, so long as "sue me" is not on the agenda.

Credits

Original script by Steve Eley.

Improvements by Robert LaThanh.

s3nukem's People

Contributors

lathanh avatar nimster avatar sfeley avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.