Giter VIP home page Giter VIP logo

redis-dump's Introduction

Redis::Dump - v0.6.0

Redis to JSON and back again. Dump and load Redis databases to/from JSON files.

Installation

One of:

  1. Gemfile: gem 'redis-dump', '~> 0.4.0'
  2. Install manually: gem install redis-dump
  3. Clone with git: git clone [email protected]:delano/redis-dump.git

Usage

There are two executables: redis-dump and redis-load.

  $ redis-dump
  $ redis-dump -u 127.0.0.1:6379 > db_full.json
  $ redis-dump -u 127.0.0.1:6379 -d 15 > db_db15.json

  $ < db_full.json redis-load
  $ < db_db15.json redis-load -d 15
  # OR
  $ cat db_full | redis-load
  $ cat db_db15.json | redis-load -d 15

  # You can specify the redis URI via an environment variable
  $ export REDIS_URI=127.0.0.1:6379
  $ redis-dump

  # If your instance uses a password (such as on RedisToGo), you
  # can specify the Redis URL as such:
  # :<password>@<domain>:<port>
  # Note the leading colon is important for specifying no username.
  $ redis-dump -u :[email protected]:9055

Output format

All redis datatypes are output to a simple JSON object. All objects have the following 5 fields:

  • db (Integer)
  • key (String)
  • ttl (Integer): The amount of time in seconds that the key will live. If no expire is set, it's -1.
  • type (String), one of: string, list, set, zset, hash, none.
  • value (String): A JSON-encoded string. For keys of type list, set, zset, and hash, the data is given a specific structure (see below).

Here are examples of each datatype:

{"db":0,"key":"hashkey","ttl":-1,"type":"hash","value":{"field_a":"value_a","field_b":"value_b","field_c":"value_c"},"size":42}
{"db":0,"key":"listkey","ttl":-1,"type":"list","value":["value_0","value_1","value_2","value_0","value_1","value_2"],"size":42}
{"db":0,"key":"setkey","ttl":-1,"type":"set","value":["value_2","value_0","value_1","value_3"],"size":28}
{"db":0,"key":"zsetkey","ttl":-1,"type":"zset","value":[["value_0","100"],["value_1","100"],["value_2","200"],["value_3","300"],["value_4","400"]],"size":50}
{"db":0,"key":"stringkey","ttl":79,"type":"string","value":"stringvalue","size":11}

Important notes

About TTLs

One of the purposes of redis-dump is the ability to restore the database to a known state. When you restore a redis database from a redis-dump file, the expires are reset to their values at the time the dump was created. This is different from restoring from Redis' native .rdb or .aof files (expires are stored relative to the actual time they were set).

Output directly to an encrypted file

For most sensitive data, you should consider encrypting the data directly without writing first to a temp file. You can do this using the power of gpg and file descriptors. Here are a couple examples:

  # Encrypt the data (interactive)
  $ redis-dump -u 127.0.0.1:6379 -d 15 | gpg --force-mdc -v -c > path/2/backup-db1

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/delano/redis-dump.

License

The gem is available as open source under the terms of the MIT License.

redis-dump's People

Contributors

bryanrite avatar delano avatar tit avatar vakuum avatar wuhuaxu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

redis-dump's Issues

Unable to use redis-dump for data export

  1. I have renamed the redis keys command, but redis-dump seems to only use the keys command. Is there a solution?

2, the second requirement is that I only want to export the data of hgetall *, is there a solution?

Error: uninitialized class variable @@schemes

Just running redis-dump generates this error for me:

/home/vagrant/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/uri-redis-0.4.2/lib/uri/redis.rb:93:in `<module:URI>': uninitialized class variable @@schemes in URI (NameError)
Did you mean?  scheme_list
        from /home/vagrant/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/uri-redis-0.4.2/lib/uri/redis.rb:4:in `<top (required)>'
        from <internal:/home/vagrant/.rbenv/versions/3.1.3/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/home/vagrant/.rbenv/versions/3.1.3/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /home/vagrant/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/redis-dump-0.4.0/lib/redis/dump.rb:6:in `<top (required)>'
        from <internal:/home/vagrant/.rbenv/versions/3.1.3/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:/home/vagrant/.rbenv/versions/3.1.3/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from /home/vagrant/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/redis-dump-0.4.0/bin/redis-dump:18:in `<top (required)>'
        from /home/vagrant/.rbenv/versions/3.1.3/bin/redis-dump:25:in `load'
        from /home/vagrant/.rbenv/versions/3.1.3/bin/redis-dump:25:in `<main>'

on Ubuntu 20.04, Ruby 3.1.3

Gem path set incorrectly for executables in 0.6.0

$ gem install redis-dump:0.6.0
Fetching redis-dump-0.6.0.gem
Successfully installed redis-dump-0.6.0
Parsing documentation for redis-dump-0.6.0
Installing ri documentation for redis-dump-0.6.0
Done installing documentation for redis-dump after 0 seconds
1 gem installed
~/Projects/opensource/delano/tryouts delano/update-deps                                                                               3.1.4 21:05:58
venv3.11 3.1.4 ❯ redis-dump -V         
rbenv: redis-dump: command not found

ERROR (Yajl::ParseError): lexical error: invalid char in json text.

when use redis-dump, something went wrong.

os:windows
the error value is \x03\x01000AAABfgSYbkk+7S9B94lIjJDrTQhJDSW\xB9,when export a json file ,is \u0003\u0001000AAABfgSYblkDQSRDVLxJv7DxWrXA3B+? .so it is wrong when load.

ERROR (Yajl::ParseError): lexical error: invalid char in json text.
gSYbkk+7S9B94lIjJDrTQhJDSW?,"\u0003\u0001000AAABfgSYblkDQSRD
(right here) ------^

Support redis stack

stack

 redis-dump -a 'xxx' -u prod_redis-stack_master:6379 > stak-full.json

get error: ERROR (TypeError): Unknown type: ReJSON-RL

Issues with serialized data and non-UTF8 encoding

I store hashes in Redis by serializing them with the help of MessagePack.

The serialized objects typically look like like some variation of this: \x93\x01\x02\x03

The problem is it completely breaks the usability of redis-dump because Yajl throws a parse error when I try to re-load the file:

ERROR (Yajl::ParseError): lexical error: invalid bytes in UTF8 string.
          421,"type":"zset","value":[["??vid?13289295403089412158?tid?
                     (right here) ------^

Any suggestions on how to approach this issue?

Password access

Hi, is this possible to add password access option for redis-dump?

Password contains '/' gives error.

Thanks for the good utility.

When password contains / character, it gives below error.

ERROR (URI::InvalidURIError): the scheme redis does not accept registry part: :Nwb8BFqt0fbP (or bad hostname?)

I want to fix by myself but I'm new to Ruby. sorry.

wrong number of arguments (1 for 2)

Hi,

I got the following error when running redis-dump:

# redis-dump -u 192.168.5.208:35535 > redisdb_35535_`date +%F`.json
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:294: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:428: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:435: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:442: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:449: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:519: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:635: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:667: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:674: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:695: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:702: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:709: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:723: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:730: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:737: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:800: warning: parenthesize argument(s) for future version
/usr/lib/ruby/gems/1.8/gems/redis-2.2.1/lib/redis.rb:1131: warning: parenthesize argument(s) for future version
ERROR (ArgumentError): wrong number of arguments (1 for 2)

ruby version 1.8.5
gem env:

RubyGems Environment:
  - RUBYGEMS VERSION: 1.3.1
  - RUBY VERSION: 1.8.5 (2006-08-25) [x86_64-linux]
  - INSTALLATION DIRECTORY: /usr/lib/ruby/gems/1.8
  - RUBY EXECUTABLE: /usr/bin/ruby
  - EXECUTABLE DIRECTORY: /usr/bin
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /usr/lib/ruby/gems/1.8
     - /root/.gem/ruby/1.8
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :benchmark => false
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - http://gems.rubyforge.org/

undefined method "select!"

I try run reddis-dump and response next error
ERROR (NoMethodError): undefined method `select!' for #Array:0x102c17818

mac, ruby 1.8.x, redis 2.6.x

`redis-load` has some problems executing `sadd`

hi, when I execute the command cat [*.json] | redis-load -u [host]/[db_index] -a [passwd],
"sadd" seems problematic. See details from the image below
I'm new to ruby, can you help me with this problem? or tell me why output these
image

Error on alpine

ps: unrecognized option: p
BusyBox v1.25.1 (2016-10-26 16:15:20 GMT) multi-call binary.

Usage: ps [-o COL1,COL2=HEADER]

Show list of processes

	-o COL1,COL2=HEADER	Select columns for display

TimeoutError

ERROR (Redis::TimeoutError): Waited 1.0 seconds

Redis Server

redis_version:7.0.12

Can you support the latest version?

ERR SELECT is not allowed in cluster mode

./bin/redis-dump -u 172.16.10.xx:6379 > db_full.json
ERR SELECT is not allowed in cluster mode

redis version 4.0.9
Redis cluster can not use this tool to get data ?

stack level too deep (SystemStackError)

When dumping a large redis database (1gb+ in size), redis-dump raises an exception after dumping about 262mb of data.

This is the last few lines of output (running with -D):

...
1320554699.0341: dump[-2199019514904509058, {"db"=>0, "key"=>"customer+43573945", "ttl"=>-1, "type"=>"set", "value"=>["1"], "size"=>1}]
1320554699.0341: customer+43573945 (set): 1B
/Users/paul/.rvm/gems/ruby-1.9.2-p180/bin/redis-dump:19: stack level too deep (SystemStackError)

mkmf.rb can't find header files for ruby

# gem install redis-dump
Fetching: yajl-ruby-1.3.1.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing redis-dump:
        ERROR: Failed to build gem native extension.

    current directory: /var/lib/gems/2.3.0/gems/yajl-ruby-1.3.1/ext/yajl
/usr/bin/ruby2.3 -r ./siteconf20180302-17663-4bv1w.rb extconf.rb
mkmf.rb can't find header files for ruby at /usr/lib/ruby/include/ruby.h

extconf failed, exit code 1

Gem files will remain installed in /var/lib/gems/2.3.0/gems/yajl-ruby-1.3.1 for inspection.
Results logged to /var/lib/gems/2.3.0/extensions/x86_64-linux/2.3.0/yajl-ruby-1.3.1/gem_make.out

select another database for import

I dump database 5 from server and I want to import to database 3 but command

cat db5.json | ./redis-load -d 3
not work. I need to replace in file "db":5, to "db":3, for successfully import.
It needed to fix on use -d 3 to ignore database in file

support filtering keys

Support the following params:
--include-keys (regex)
--exclude-keys (regex)

By default it includes all keys. If there are any include-exclude params given, they will be executed in chain, and the key will be included based on the output.

For example

redis-dump .... --include-keys foo-.* --exclude-keys foo-bar-.* would include bar, foo-2, but not foo-bar-2

redis-dump .... --exclude-keys .* . --include-keys foo-[0-9]+ would exclude bar, foo-bar-2, but include foo-2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.