Giter VIP home page Giter VIP logo

dynamodb-copy-table's Introduction

dynamodb-copy-table-python3

A simple python 3 script to copy dynamodb table


Requirements

  • Python 3.x
  • boto (pip install boto)

Usage

A simple usage example:

$ python dynamodb-copy-table.py src_table dst_table

The following environment variables can be used:

Variable Purpose
AWS_DEFAULT_REGION Select the region (the default region is us-west-2)
DISABLE_CREATION Disable the creation of a new table (Useful if the table already exists)
DISABLE_DATACOPY Disable the copying of data from source table to destination table
$ AWS_DEFAULT_REGION=us-east-1 DISABLE_CREATION=yes DISABLE_DATACOPY=yes \
python dynamodb-copy-table.py src_table dst_table

Docker Image

The docker image is available as techgaun/dynamodb-copy-table:latest in the official docker hub that you can pull from.

Usage:

# pull image down
docker pull techgaun/dynamodb-copy-table:latest

# invoke help
$ docker run --rm -it techgaun/dynamodb-copy-table:latest
Usage: dynamodb-copy-table.py <source_table_name> <destination_table_name>

# invoke copy
docker run -e AWS_ACCESS_KEY_ID=abc -e AWS_SECRET_ACCESS_KEY=def --rm -it techgaun/dynamodb-copy-table:latest src dest

References

dynamodb-copy-table's People

Contributors

c-py avatar kiddo3 avatar techgaun avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

dynamodb-copy-table's Issues

Tried using your script but it hangs on a dynamodb-local

Tried using your script but it hangs on a dynamodb-local . Here is the modified script I was trying to use.
https://github.com/djangofan/dynamodb-copy-table

Here is how you can setup the local dynamodb:

  1. docker run -p 8000:8000 --name dynamodb-local --restart unless-stopped -d dwmkerr/dynamodb -sharedDb
  2. aws dynamodb create-table --table-name Music --attribute-definitions AttributeName=Artist,AttributeType=S AttributeName=SongTitle,AttributeType=S --key-schema AttributeName=Artist,KeyType=HASH AttributeName=SongTitle,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1
  3. aws dynamodb list-tables --endpoint-url http://192.168.99.100:8000
  4. then , try to copy it with this command:
    python dynamodb-copy-table.py Music copyof-Music true

Doesn't copy indexes

Indexes are an important part of a table schema, are a far more tedious part of table creation than the table itself. This would be much more useful if it copied indexes

this doesnt appear to handle throughput / throttling...

@jakelodwick @techgaun

given a source table of x read throughput and a dest table of <x write throughput, this utility will result in throttled writes, which means the throttled data will not be copied to dest.

The converse of this could also exist: a throttled read resulting in nothing returned, which then never gets written.

Am I observing this correctly, or am I missing something?

If so, I may be able to do a PR For something to fix it. Let me know.

Booleans converted to Numbers

Hey there. Thanks a lot for making this repo, very helpful. Wish Amazon provided an easy way to do this!

Noticed an issue though: seems like Booleans are converted to Numbers.

Here's an entry in my source table:
image

...and now the destination table:
image

Note that the public and the tiles[0].trigger.sync parameters have changed from true to 1. Fortunately, false becomes 0 in other cases, so we can probably guess what's happening.

Should be easy enough fix manually on my end with a DynamoDB query, but thought you should be aware!

AttributeError: 'NoneType' object has no attribute 'endpoint'

I use the docker command provided in the Readme. I keep getting the following error:

Traceback (most recent call last):
  File "dynamodb-copy-table.py", line 22, in <module>
    ddbc = DynamoDBConnection()
  File "/usr/local/lib/python3.6/site-packages/boto/dynamodb2/layer1.py", line 173, in __init__
    kwargs['host'] = region.endpoint
AttributeError: 'NoneType' object has no attribute 'endpoint'

I have initialized all the required environment variables.

Datatype of Number columns changed to String

Source table:

Partition key
scheduleId (Number)
Sort key
timestamp (Number)

Command:

docker run --rm -v $HOME/.aws:/root/.aws -it techgaun/dynamodb-copy-table:latest source dest

Destination table:

Partition key
scheduleId (String)
Sort key
timestamp (String)

Docker image

Hi there,

I have used this script several times and it is working just fine.
Though, I think it would be nice to have a Dockerfile and ability to run this script in container.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.