Giter VIP home page Giter VIP logo

dynamo-backup-streams's People

Stargazers

avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

pmcclory axel-springer-kugawana jverneuer

dynamo-backup-streams's Issues

Add support to use more capacity for backup/restore

Hey Stefan,

I have a feature request I would love to discuss with you.

We have steady user traffic on the dynamodb and the capacity percentage option in combination with back pressure/retries helps a lot in not throttling the requests from the user. But that solution does not always scale and you need to be careful setting fitting auto-scaling and capacity percentage values.

Example:

  • Table Provisioned Read Capacity: 20
  • Avg. User Consumed Read Capacity: 10
  • Capacity Percentage: 50%, Auto-Scaling Target Utilization: 70%
  • Backup will consume 10 reads --> 👍
  • Auto Scaling will kick in step by step to let's say 100
  • Users still consume 10 read, Backup will consume 50 reads --> 60% usage, auto-scaling will scale down or at least not up any more

Do you have the same "problem"? How do you handle it?

What about another approach:

  • get the provisioned capacity
  • Everything above that can be used by the backup
  • After reaching some speed, keep a safety distance to the max capacity.

Thinking about that, I guess there is still a potential dead zone where backup has no capacity and/or auto-scaling is not kicking in.

Just some starting point for a discussion. The current implementation is working for our use case.
Cheers, Marcel

restoreStream's end event is emitted to early

Hey @SaltwaterC,

fist of all: Your lib is the only one handling retries, back pressure and capacity percentage in a good way! Awesome work! Thank you! 👍

One error I run into is at the end of the restore process:

...
progress: 8834288 / 8834288 bytes - 100%; read items: 172653 - restored items: 170123; max write capacity: 180 - used 184 by 10 threads
Restored items: 170281
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 170923; max write capacity: 180 - used 183 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 171730; max write capacity: 180 - used 180 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 172535; max write capacity: 180 - used 185 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 172784; max write capacity: 180 - used 161 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 172784; max write capacity: 180 - used 161 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 172784; max write capacity: 180 - used 161 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 172784; max write capacity: 180 - used 161 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 172784; max write capacity: 180 - used 161 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 172784; max write capacity: 180 - used 161 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 172784; max write capacity: 180 - used 161 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 172784; max write capacity: 180 - used 161 by 10 threads
progress: 8834288 / 8834288 bytes - 100%; read items: 172802 - restored items: 172784; max write capacity: 180 - used 161 by 10 threads
restoreStream: Error: Fail: too many retries when writing records

Code is basically:

 ...
   .pipe(gunzip)
    .on('error', handleError(`gunzip`))
    .pipe(restoreStream)
    .on('error', handleError(`restoreStream`))
    .on('end', () => {
      logger.info('Restored items: %d', restoreStream.processedLines);
    });

So as you see

  1. the end event on the restoreStream is triggered to early.
  2. I'm not sure why the too many retries error happens. I cannot see any throttles on the AWS metrics.

Do you have any idea?

Thx
Marcel

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.