Script to copy data by keys pattern from one redis instance to another.

go-redis-migrate copy <source> <destination> --pattern="prefix:*"

Source, destination - can be provided as just <host>:<port> or in Redis URL format: redis://[:<password>@]<host>:<port>[/<dbIndex>] Pattern - can be glob-style pattern supported by Redis SCAN command.

Other flags:

  --report int           Report current status every N seconds (default 1)
  --scanCount int        COUNT parameter for redis SCAN command (default 100)
  --exportRoutines int   Number of parallel export goroutines (default 30)
  --pushRoutines int     Number of parallel push goroutines (default 30)

Download the binary for your platform from the releases page.

There are 3 main stages of copying keys:

1. Scanning keys in source
2. Dumping values and TTLs from source
3. Restoring values and TTLs in destination

Scanning is performed with a single goroutine, scanned keys are sent to keys channel (type is chan string). From keys channel N export goroutines are consuming keys and perform DUMP and PTTL for them as a pipeline command. Results are combined in KeyDump structure and transferred to channel of this type. Another M push goroutines are consuming from the channel and perform RESTORE command on the destination instance.

To guarantee that all keys are exported and restored sync.WaitingGroup is used. To monitor current status there is a separate goroutine that outputs value of atomic counters (for each stage) every K seconds.

Performed on a laptop with redis instances, running in docker.

Source database: 453967 keys. Keys to copy: 10000 keys.

Source database: 453967 keys. Keys to copy: 367610 keys.

• An article about how it works: https://medium.com/amboss/copy-redis-keys-in-minutes-with-golang-3c06f3cd3af8