sanofi-iadc / whispr Goto Github PK

View Code? Open in Web Editor NEW

87.0 87.0 8.0 77.21 MB

Open source event, comment and alert processing hub created by Sanofi IADC

License: MIT License

Dockerfile 0.63% JavaScript 6.94% TypeScript 86.39% Makefile 4.78% Shell 1.25%

whispr's People

Stargazers

Watchers

Forkers

stjordanis bainoit alexisno mestrak laurence-liu-923 jatin510 kapseliboi bonfim-sanofi

whispr's Issues

Authentication from mutliple JWT issuers

Is your feature request related to a problem? Please describe.

As Whispr is designed to be a hub that multiple clients to connect to, it would be great to offer authentication which allows direct passthrough and validation of JWTs for those clients to avoid the need for an additional authentication against whispr.

Describe the solution you'd like

Clients within a trusted environment should be able to log in with their existing JWTs.
Security config should be parameterised and not hard coded.
All queries/mutations/endpoints should be secured (possibly with a global guard).

Describe alternatives you've considered

Option x

Use standard passport-jwt which allows for only a single JWT issuer or secret.
This option is not selected because it would either force additional authentication to whispr, or limit each whispr instance to only one client.

Option y

Use an extended version of passport-jwt implemented in @MeStrak/passport-multi-jwt which allows configuration using an array of passport-jwt configurations.
Manage configuration through an env var, so that no code modification is required to configure an additional JWT issuer or secret.
Use of guards to be reviewed - a global guard might be possible, but we should take into account future requirements of restrictions at Application ID level (see additional context)

Additional context

This feature covers authentication only, not authorisation. As a next step we should look into authorisation at application ID level to protect the data further.

Use MongoDB query language for filter on webhooks

Is your feature request related to a problem? Please describe.

The filter features of whispr are not consistent. I am not able to apply the same filter to the webhooks which I can apply to the whisps query. The webhook mutation does not support the MongoDB query language.

Example filter which is working on the whisps query but cannot be added to a new webhook:

"filter": {
    "attribute1":  { "$exists": true },
    "data.attribute2": { "$eq": 3 }
}

The same problem could occur for subscriptions aswell.

There are actually two problems to solve:

save a query to be reused by webhooks
apply the query on a JavaScript Object on runtime

Describe the solution you'd like

A consistent behaviour of all filter possibilities in whispr. The MongoDB query language should be used for queries, subscriptions and webhooks.

A possible solution for both problems would be the ucast ecosystem.
With using mongo2js we could:

save a query with its Mongo Query AST representation
apply the query

Describe alternatives you've considered

Option 1

Use mingo. This package is capable of in memory comparism. Saving the query could be achieved with mongodb-js features by transforming the query to JavaScript objects with mongodb-language-model.parse or mongodb-query-parser

Use MongoDB query language for filter on webhooks

Is your feature request related to a problem? Please describe.

Example filter which is working on the whisps query but cannot be added to a new webhook:

"filter": {
    "attribute1":  { "$exists": true },
    "data.attribute2": { "$eq": 3 }
}

The same problem could occur for subscriptions aswell.

There are actually two problems to solve:

save a query to be reused by webhooks
apply the query on a JavaScript Object on runtime

Describe the solution you'd like

A consistent behaviour of all filter possibilities in whispr. The MongoDB query language should be used for queries, subscriptions and webhooks.

A possible solution for both problems would be the ucast ecosystem.
With using mongo2js we could:

save a query with its Mongo Query AST representation
apply the query

Describe alternatives you've considered

Option 1

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

chore(dependency maintenance): update dependency eslint-config-airbnb-base to v15

Detected dependencies

docker-compose

docker-compose.dev.rep.yml

docker-compose.dev.yml

docker-compose.rep.yml

docker-compose.yml

dockerfile

.gitpod.Dockerfile

Dockerfile

node 16.14.0-alpine

node 16.14.0-alpine

Dockerfile.aws

github-actions

.github/workflows/ci.yml

actions/checkout v2

actions/setup-node v2

actions/cache v2

actions/checkout v2

actions/setup-node v2

actions/cache v2

azure/docker-login v1

ubuntu 20.04

ubuntu 20.04

.github/workflows/code-ql-analysis.yml

actions/checkout v3

github/codeql-action v2

github/codeql-action v2

github/codeql-action v2

.github/workflows/docker-hub-description.yml

actions/checkout v2

peter-evans/dockerhub-description v2

ubuntu 20.04

.github/workflows/gh-pages.yml

actions/checkout v2

actions/setup-node v2

actions/cache v2

peaceiris/actions-gh-pages v3

ubuntu 20.04

.github/workflows/k6-load-test.yml

actions/checkout v2

actions/setup-node v2

actions/cache v2

ubuntu 20.04

npm

docs/package.json

@vuepress/theme-default 2.0.0-beta.45

vuepress 2.0.0-beta.45

package.json

@apollo/gateway 0.44.1

@instana/collector 1.140.1

@sanofi-iadc/passport-multi-jwt 1.0.0

@nestjs/apollo 10.0.11

@nestjs/axios 0.0.7

@nestjs/common 8.2.6

@nestjs/config 1.2.1

@nestjs/core 8.2.6

@nestjs/graphql 10.0.11

@nestjs/jwt 8.0.0

@nestjs/mongoose 9.0.3

@nestjs/passport 8.2.1

@nestjs/platform-fastify 8.2.6

@nestjs/platform-socket.io 8.4.5

@nestjs/terminus 8.0.6

@nestjs/websockets 8.2.6

@typegoose/typegoose 9.7.1

@types/express 4.17.13

@types/graphql 14.5.0

amazon-cognito-identity-js 5.2.8

apollo-server-core 3.5.0

apollo-server-fastify 3.5.0

aws-sdk 2.1137.0

class-transformer 0.3.1

class-validator 0.13.2

cross-fetch 3.1.5

dotenv 10.0.0

eslint-plugin-graphql 4.0.0

express ^4.18.1

fastify 3.24.1

fastify-compress 3.7.0

flatted 3.2.5

graphql 15.8.0

graphql-redis-subscriptions 2.4.2

graphql-subscriptions 2.0.0

graphql-tag-pluck 0.8.7

graphql-tools 8.2.9

graphql-type-json 0.3.2

graphql-upload 13.0.0

haiku-random 1.0.0

https-proxy-agent 5.0.1

ioredis 4.28.5

joi 17.6.0

jwks-rsa 2.1.2

lodash 4.17.21

mongoose 6.2.11

mongoose-cast-aggregation 0.2.1

nock 13.2.4

parse-duration 1.0.2

passport 0.5.3

passport-jwt 4.0.0

reflect-metadata 0.1.13

request 2.88.2

rimraf 3.0.2

rxjs 7.5.5

superagent 6.1.0

superagent-promise 1.1.0

superagent-proxy 3.0.0

tunnel 0.0.6

@commitlint/cli 15.0.0

@commitlint/config-conventional 15.0.0

@nestjs/cli 8.2.5

@nestjs/platform-express 8.4.5

@nestjs/schematics 8.0.11

@nestjs/testing 8.2.6

@semantic-release/changelog 6.0.1

@semantic-release/exec 6.0.3

@semantic-release/git 10.0.1

@semantic-release/github 8.0.4

@types/jest 27.5.1

@types/node 16.11.36

@types/socket.io 3.0.1

@types/supertest 2.0.12

@types/tunnel 0.0.3

@typescript-eslint/eslint-plugin 5.25.0

@typescript-eslint/parser 5.25.0

apollo-server-express 3.6.8

cz-conventional-changelog 3.3.0

eslint ^7.32.0

eslint-config-airbnb-base 14.2.1

eslint-config-prettier 8.5.0

eslint-config-standard 16.0.3

eslint-import-resolver-typescript 2.7.1

eslint-plugin-import 2.26.0

eslint-plugin-node 11.1.0

eslint-plugin-prettier 3.4.1

eslint-plugin-promise 5.2.0

eslint-plugin-standard 4.1.0

husky 7.0.4

jest 27.5.1

jsonwebtoken ^8.2.0

mock-jwks 1.0.3

mongodb-memory-server 8.5.2

prettier 2.6.2

prettier-eslint-cli 5.0.1

semantic-release 19.0.2

supertest 6.2.3

ts-jest 27.1.4

ts-loader 9.3.0

ts-morph 13.0.3

ts-node 10.7.0

tsconfig-paths 3.14.1

typescript 4.6.4

webpack 5.72.1

minimist 1.2.6

node-fetch 2.6.7

cross-fetch 3.1.5

node-forge 1.3.1

glob-parent 5.1.2

nvm

.nvmrc

node 16.15.0

Check this box to trigger a request for Renovate to run again on this repository

Linting configured to allow 55 warnings

Describe the bug
When we first published Whispr, it 55 lint warnings were allowed for the CI job to pass. Configured in package.json with the following:
"lint:es": "eslint --max-warnings=55 --fix --ext .js,.ts src"

All warnings are still there and we should start to correct them and reduce the number of allowed warnings.

To Reproduce
Run the linter, or check warnings in CI from a pull request. E.g. https://github.com/Sanofi-IADC/whispr/actions/runs/374172150.

Expected behavior
We should start to clean these up (most are simple missing return types) and reduce the number of allowed warnings.

Screenshots
n/a

Desktop (please complete the following information):
n/a

Smartphone (please complete the following information):
n/a

Additional context
n/a

Add rate limiting to whispr

Is your feature request related to a problem? Please describe.

Creating this feature request following the rate limiter discussion with @nikola-kovacevic and @Babbafett in #349 to separate the two topics.

Currently there is no rate limiting functionality in whispr, meaning that a single client can send a very high number of requests to the API and (depending on the infrastructure/auto scaling setup) potentially cause a slow down for all users.

Describe the solution you'd like

We should implement a rate limiting solution, and I'm opening this issue to discuss the topic as there are a couple of potential solutions.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

@nestjs/throttler

NestJS has a native package: https://docs.nestjs.com/security/rate-limiting. I did a short test of it. I had to add the following additional code within the useFactory section of AppModule because we're using Fastify and GQL:

context: ({ request, reply }) => {
          return { req: request, res: reply }

Apart from that I just followed the docs and it works, but the options are limited to the following:

ttl	the number of seconds that each request will last in storage
limit	the maximum number of requests within the TTL limit
ignoreUserAgents	an array of regular expressions of user-agents to ignore when it comes to throttling requests
storage	the storage setting for how to keep track of the requests

It means that we could very quickly and easily add a solid rate limiting feature, but we would do so knowing that we cannot easily implement request rules by client or other advanced logic.

NestJS Rate Limiter

Another rate limiter: https://github.com/ozkanonur/nestjs-rate-limiter. The feature set seems to be more complete, but it's not an official NestJS package. I haven't tested this one, but I think that we should try it out while deciding what we implement.

In particlular I like the fact that it has a points system (https://github.com/ozkanonur/nestjs-rate-limiter#-points), and a feature to smooth out request bursts (https://github.com/ozkanonur/nestjs-rate-limiter#-execevenly).

However, what I did not look into was whether we could achieve something similar using a combination of NestJS throttler and Bull which could put requests into a queue.

Additional context

n/a

Add bulk requests and grouping to the whisp count query

Is your feature request related to a problem? Please describe

Currently the whispsCount query takes a single mongo format query, and returns the count of documents which match that query. In cases where an application needs to get the count of whisps in bulk, a separate request is required for each query, resulting in a high number of requests from the browser (e.g. one for every row of a table).

Describe the solution you'd like

Extend whispsCount to allow bulk requests, and grouping by specific criteria in the mongo query
Stop using the mongo countDocuments() function, and use db.collection.aggregate instead which will allow us to have a grouping functionality (explained more in detail below)
Grouping criteria would be optional - not necessary if the consuming app wants a total document count based on the matched query

Currently the query runs the mongo command countDocuments() (https://docs.mongodb.com/manual/reference/method/db.collection.countDocuments/). countDocuments is actually just a wrapper for the following command:

db.collection.aggregate([
--
{ $match: <query> },
{ $group: { _id: null, n: { $sum: 1 } } }
])

We can easily replace countDocuments() with that command to allow us to group, and therefore manage bulk requests.

For example this query:

 db.whisps.aggregate([
    { $match: {"applicationID": "myapp"} },
    { $group: { _id: { criteria1: "$data.thing1", criteria2: "$data.thing2" }, count: { $sum: 1 } } }

Would return something like this, which satisfies the requirements:

{ "_id" : { "criteria1" : "abc", "criteria2" : "123" }, "count" : 3 }
{ "_id" : { "criteria1" : "def", "criteria2" : "123" }, "count" : 5 }
{ "_id" : { "criteria1" : "def", "criteria2" : "345" }, "count" : 7 }
{ "_id" : { "criteria1" : "xyz", "criteria2" : "999" }, "count" : 9 }

In that example the match criteria is very simple, but it could also be a more complex query with multiple things to match.

BREAKING CHANGE(S)

If we use exactly the same function for a simple query with no grouping (like the implementation today), the response would contain an object with an _id field, instead of just the number of records.
This query will also never return a record count of 0, because for the grouping functionality to work there must be at least one record matched
We could manage both of those by using the existing logic if no grouping criteria is specified, I'm just not sure if the change is critical enough to require both of these to be maintained (added to open points for discussion)

Describe alternatives you've considered

Consuming app provides an array of queries, with an ID for each query, whispr executes individual `documentCount` queries but returns a single response

Not an optimal solution as we would be sending many separate queries to the database

Use a mongo aggregate function so that we can add grouping functionality

(this is the proposed solution described in detail above)

Allows flexible grouping on a single field, or an object
Would potentially introduce a breaking change for single ungrouped queries, depending on how we manage it

 db.whisps.aggregate([
    { $match: {"applicationID": "myapp"} },
    { $group: { _id: { criteria1: "$data.thing1", criteria2: "$data.thing2" }, count: { $sum: 1 } } }

Additional context

Open points for discussion:

whispsCount - should it be called whispCount instead? - if it should then it allows us to introduce this feature without a breaking change (for now) as we could add the new query and decommission the old one later
Should we accept an array of queries to be matched, or have the consuming app specify a single query to match in mongo format? (either we construct the query with an OR for every item in the array, or the consuming app does that for us)
Recommended maximum number of queries in the bulk request?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

sanofi-iadc / whispr Goto Github PK

whispr's People

Stargazers

Watchers

Forkers

whispr's Issues

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Option x

Option y

Additional context

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Option 1

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Option 1

Rate-Limited

Open

Ignored or Blocked

Detected dependencies

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

@nestjs/throttler

NestJS Rate Limiter

Additional context

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Consuming app provides an array of queries, with an ID for each query, whispr executes individual documentCount queries but returns a single response

Use a mongo aggregate function so that we can add grouping functionality

Additional context

Recommend Projects

Recommend Topics

Recommend Org

Consuming app provides an array of queries, with an ID for each query, whispr executes individual `documentCount` queries but returns a single response