Giter VIP home page Giter VIP logo

node-fast-ratelimit's Introduction

node-fast-ratelimit

Test and Build Build and Release NPM Downloads Gitter Buy Me A Coffee

Fast and efficient in-memory rate-limit, used to alleviate most common DOS attacks.

This rate-limiter was designed to be as generic as possible, usable in any NodeJS project environment, regardless of whether you're using a framework or just vanilla code. It does not require any dependencies, making it lightweight to install and use.

🇫🇷 Crafted in Lannion, France.

Who uses it?

Crisp Doctrine Anchor.Chat WeStudents

👋 You use fast-ratelimit and you want to be listed there? Contact me.

How to install?

Include fast-ratelimit in your package.json dependencies.

Alternatively, you can run npm install fast-ratelimit --save.

TypeScript users can install type definitions by running npm install --save-dev @types/fast-ratelimit. Note that this is a third-party package, ie. not maintained by myself.

How to use?

The fast-ratelimit API is pretty simple, here are some keywords used in the docs:

  • ratelimiter: ratelimiter instance, which plays the role of limits storage
  • namespace: the master ratelimit storage namespace (eg: set namespace to the user client IP, or user username)

You can create as many ratelimiter instances as you need in your application. This is great if you need to rate-limit IPs on specific zones (eg: for a chat application, you don't want the message send rate limit to affect the message composing notification rate limit).

Here's how to proceed (we take the example of rate-limiting messages sending in a chat app):

1. Create the rate-limiter

The rate-limiter can be instanciated as such:

var FastRateLimit = require("fast-ratelimit").FastRateLimit;

var messageLimiter = new FastRateLimit({
  threshold : 20, // available tokens over timespan
  ttl       : 60  // time-to-live value of token bucket (in seconds)
});

This limiter will allow 20 messages to be sent every minute per namespace. An user can send a maximum number of 20 messages in a 1 minute timespan, with a token counter reset every minute for a given namespace.

The reset scheduling is done per-namespace; eg: if namespace user_1 sends 1 message at 11:00:32am, he will have 19 messages remaining from 11:00:32am to 11:01:32am. Hence, his limiter will reset at 11:01:32am, and won't scheduler any more reset until he consumes another token.

2. Check by consuming a token

On the message send portion of our application code, we would add a call to the ratelimiter instance.

2.1. Consume token with asynchronous API (Promise catch/reject)

// This would be dynamic in your application, based on user session data, or user IP
namespace = "user_1";

// Check if user is allowed to send message
messageLimiter.consume(namespace)
  .then(() => {
    // Consumed a token
    // Send message
    message.send();
  })
  .catch(() => {
    // No more token for namespace in current timespan
    // Silently discard message
  });

2.2. Consume token with synchronous API (boolean test)

// This would be dynamic in your application, based on user session data, or user IP
namespace = "user_1";

// Check if user is allowed to send message
if (messageLimiter.consumeSync(namespace) === true) {
  // Consumed a token
  // Send message
  message.send();
} else {
  // consumeSync returned false since there is no more tokens available
  // Silently discard message
}

3. Check without consuming a token

In some instances, like password brute forcing prevention, you may want to check without consuming a token and consume only when password validation fails.

3.1. Check whether there are remaining tokens with asynchronous API (Promise catch/reject)

limiter.hasToken(request.ip).then(() => {
  return authenticate(request.login, request.password)
})
  .then(
    () => {
      // User is authenticated
    },

    () => {
      // User is not authenticated
      // Consume a token and reject promise
      return limiter.consume(request.ip)
        .then(() => Promise.reject())
    }
  )
  .catch(() => {
    // Either invalid authentication or too many invalid login
    return response.unauthorized();
  })

3.2. Check whether there are remaining tokens with synchronous API (boolean test)

if (!limiter.hasTokenSync(request.ip)) {
  throw new Error("Too many invalid login");
}

const is_authenticated = authenticateSync(request.login, request.password);

if (!is_authenticated) {
  limiter.consumeSync(request.ip);

  throw new Error("Invalid login/password");
}

Notes on performance

This module is used extensively on edge WebSocket servers, handling thousands of connections every second with multiple rate limit lists on the top of each other. Everything works smoothly, I/O doesn't block and RAM didn't move that much with the rate-limiting module enabled.

On one core of a 2,3 GHz 8-Core Intel Core i9, the parallel asynchronous processing of 100,000 namespaces in the same limiter take an average of 160 ms, which is fine (1.6 microseconds per operation).

Why not using existing similar modules?

I was looking for an efficient, yet simple, DOS-prevention technique that wouldn't hurt performance and consume tons of memory. All proper modules I found were relying on Redis as the keystore for limits, which is definitely not great if you want to keep away from DOS attacks: using such a module under DOS conditions would subsequently DOS Redis since 1 (or more) Redis queries are made per limit check (1 attacker request = 1 limit check). Attacks should definitely not be allieviated this way, although a Redis-based solution would be perfect to limit abusing users.

This module keeps all limits in-memory, which is much better for our attack-prevention concern. The only downside: since the limits database isn't shared, limits are per-process. This means that you should only use this module to prevent hard-attacks at any level of your infrastructure. This works pretty well for micro-service infrastructures, which is what we're using it in.

node-fast-ratelimit's People

Contributors

jorgenvatle avatar karonl avatar valeriansaliou avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

node-fast-ratelimit's Issues

Warnings

Getting the following warnings:

...\node_modules\hashtable-patch-valeriansaliou\src\hashtable.h(6): warning C4067: unexpected tokens following preprocessor directive - expected a newline (compiling source file ..\src\hashtable.cpp) [...\node_modules\hashtable-patch-valeriansaliou\build\native.vcxproj]

...\node_modules\hashtable-patch-valeriansaliou\src\v8_value_hasher.h(7): warning C4067: unexpected tokens following preprocessor directive - expected a newline (compiling source file ..\src\hashtable.cpp) [...\node_modules\hashtable-patch-valeriansaliou\build\native.vcxproj]

..\src\hashtable.cpp(268): warning C4244: 'argument': conversion from 'double' to 'float', possible loss of data [...\node_modules\hashtable-patch-valeriansaliou\build\native.vcxproj]

Several compile warnings

node-fast-ratelimit emits sevral errors when being compiled, for example:

../node_modules/nan/nan_implementation_12_inl.h:340:28: warning: 'New' is deprecated [-Wdeprecated-declarations] return v8::StringObject::New(value).As<v8::StringObject>();

../node_modules/nan/nan.h:1066:44: warning: 'ToString' is deprecated [-Wdeprecated-declarations] v8::Local<v8::String> string = from->ToString();

../node_modules/nan/nan.h:1080:27: warning: 'WriteUtf8' is deprecated [-Wdeprecated-declarations] length_ = string->WriteUtf8(str_, static_cast<int>(len), 0, flags);

9 warnings generated. SOLINK_MODULE(target) Release/native.node

Maybe its time that @valeriansaliou had a chat with the maintainer of the hashtable module?

Compile error on Travis CI: missing binary operator before token "("

This is the error you get if you try to npm install fast-ratelimit on Travis

npm ci stdout:

> [email protected] install /home/travis/build/xyz/node_modules/hashtable-patch-valeriansaliou
> node-gyp configure build
make: Entering directory `/home/travis/build/xyz/node_modules/hashtable-patch-valeriansaliou/build'
  CXX(target) Release/obj.target/native/src/hashtable.o
make: Leaving directory `/home/travis/build/xyz/node_modules/hashtable-patch-valeriansaliou/build'
lerna ERR! npm ci stderr:
In file included from ../src/hashtable.cpp:1:0:
../src/hashtable.h:6:64: error: missing binary operator before token "("
 #if defined __APPLE__ && defined __has_include && __has_include(<tr1/unordered_map>)
                                                                ^
In file included from ../src/hashtable.h:15:0,
                 from ../src/hashtable.cpp:1:
../src/v8_value_hasher.h:7:64: error: missing binary operator before token "("
 #if defined __APPLE__ && defined __has_include && __has_include(<tr1/unordered_map>)
                                                                ^
In file included from ../src/hashtable.h:15:0,
                 from ../src/hashtable.cpp:1:
../src/v8_value_hasher.h: In member function ‘bool v8_value_equal_to::operator()(CopyablePersistent*, CopyablePersistent*) const’:
../src/v8_value_hasher.h:43:24: warning: ‘bool v8::Value::Equals(v8::Local<v8::Value>) const’ is deprecated (declared at /home/travis/.node-gyp/10.15.3/include/node/v8.h:2481): Use maybe version [-Wdeprecated-declarations]
         if (a->Equals(b)) {          /* same as JS == */
                        ^
../src/hashtable.cpp: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE HashTable::Constructor(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/hashtable.cpp:50:43: warning: ‘int32_t v8::Value::Int32Value() const’ is deprecated (declared at /home/travis/.node-gyp/10.15.3/include/node/v8.h:2478): Use maybe version [-Wdeprecated-declarations]
         int buckets = info[0]->Int32Value();
                                           ^
../src/hashtable.cpp: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE HashTable::Rehash(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/hashtable.cpp:231:42: warning: ‘int32_t v8::Value::Int32Value() const’ is deprecated (declared at /home/travis/.node-gyp/10.15.3/include/node/v8.h:2478): Use maybe version [-Wdeprecated-declarations]
     size_t buckets = info[0]->Int32Value();
                                          ^
../src/hashtable.cpp: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE HashTable::Reserve(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/hashtable.cpp:249:43: warning: ‘int32_t v8::Value::Int32Value() const’ is deprecated (declared at /home/travis/.node-gyp/10.15.3/include/node/v8.h:2478): Use maybe version [-Wdeprecated-declarations]
     size_t elements = info[0]->Int32Value();
                                           ^
../src/hashtable.cpp: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE HashTable::MaxLoadFactor(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/hashtable.cpp:266:39: warning: ‘double v8::Value::NumberValue() const’ is deprecated (declared at /home/travis/.node-gyp/10.15.3/include/node/v8.h:2475): Use maybe version [-Wdeprecated-declarations]
         factor = info[0]->NumberValue();
                                       ^
../src/hashtable.cpp: In static member function ‘static Nan::NAN_METHOD_RETURN_TYPE HashTable::ForEach(Nan::NAN_METHOD_ARGS_TYPE)’:
../src/hashtable.cpp:289:33: warning: ‘v8::Local<v8::Object> v8::Value::ToObject() const’ is deprecated (declared at /home/travis/.node-gyp/10.15.3/include/node/v8.h:10046): Use maybe version [-Wdeprecated-declarations]
         ctx = info[1]->ToObject();
                                 ^
make: *** [Release/obj.target/native/src/hashtable.o] Error 1
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/home/travis/.nvm/versions/node/v10.15.3/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:189:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
gyp ERR! System Linux 4.4.0-101-generic
gyp ERR! command "/home/travis/.nvm/versions/node/v10.15.3/bin/node" "/home/travis/.nvm/versions/node/v10.15.3/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "build"
gyp ERR! cwd /home/travis/build/xyz/node_modules/hashtable-patch-valeriansaliou
gyp ERR! node -v v10.15.3
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp configure build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script.

PYTHON is required on package.json include

Problem and How to Reproduce?

First of all I have Python 3.7.2 installed and PYTHON variable set (mapped to python.exe) on my environment

I include "fast-ratelimit": "^2.2.0" as a dependency in my package.json

Then execute npm install command. However it cannot install correctly, printing these;

gyp ERR! configure error
gyp ERR! stack Error: Command failed: C:\Program Files\python\python.exe -c import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack   File "<string>", line 1
gyp ERR! stack     import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack                                ^
gyp ERR! stack SyntaxError: invalid syntax
gyp ERR! stack
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:294:12)
gyp ERR! stack     at ChildProcess.emit (events.js:182:13)
gyp ERR! stack     at maybeClose (internal/child_process.js:962:16)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
gyp ERR! System Windows_NT 10.0.17763
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "configure" "build"
gyp ERR! cwd D:\Visual Studio Code Workspace\workspace-commissions\exponent-socket-server\node_modules\hashtable-patch-valeriansaliou
gyp ERR! node -v v10.15.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm WARN [email protected] No description
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp configure build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\xgen_\AppData\Roaming\npm-cache\_logs\2019-02-18T14_36_11_427Z-debug.log

Check without consume

I have a use case which I wonder if it would be accepted for this library, as a PR.

Basically, I would like to use this as ratelimit against password bruteforce.

I have a route which is protected by API key. What I want is that, if the the client is properly authenticated, then the request doesn't count in the ratelimit. If the key is wrong, it counts.

Implementation for such scheme right now is:

const valid = validateApiKey()
if (!valid) {
  if (limiter.consumeSync('user')) {
    throw new Error('Invalid key')
  } else {
    throw new Error('Brute force attempt')
  }
}

Problem is that this doesn't block the bruteforce. Even if there is no more token, attacker can continue to bruteforce and he will get access to the API when he gets the right key.

What I need is a function which can check that there are available tokens, without consuming them.

if( !limiter.hasTokensSync('user')){
  throw new Error('Brute force attempt')
}
const valid = validateApiKey()
if (!valid) {
  if (limiter.consumeSync('user')) {
    throw new Error('Invalid key')
  } else {
    throw new Error('Brute force attempt')
  }
}

Would you accept a PR to support this use case?

Error on windows npm install

C:\Users\Marat\Desktop\drawify\node_modules\hashtable-patch-valeriansaliou>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\....\node_modules\node-gyp\bin\node-gyp.js" configure build ) else (node "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" configure build )
gyp ERR! configure error
gyp ERR! stack Error: Command failed: C:\Python37\python.EXE -c import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack File "", line 1
gyp ERR! stack import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack ^
gyp ERR! stack SyntaxError: invalid syntax
gyp ERR! stack
gyp ERR! stack at ChildProcess.exithandler (child_process.js:289:12)
gyp ERR! stack at ChildProcess.emit (events.js:182:13)
gyp ERR! stack at maybeClose (internal/child_process.js:962:16)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "configure" "build"
gyp ERR! cwd C:\Users\Marat\Desktop\drawify\node_modules\hashtable-patch-valeriansaliou
gyp ERR! node -v v10.11.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
npm WARN enoent ENOENT: no such file or directory, open 'C:\Users\Marat\Desktop\drawify\package.json'
npm WARN drawify No description
npm WARN drawify No repository field.
npm WARN drawify No README data
npm WARN drawify No license field.

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: node-gyp configure build
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\Marat\AppData\Roaming\npm-cache_logs\2018-11-02T12_06_17_338Z-debug.log

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.