sanofi-iadc / whispr Goto Github PK
View Code? Open in Web Editor NEWOpen source event, comment and alert processing hub created by Sanofi IADC
License: MIT License
Open source event, comment and alert processing hub created by Sanofi IADC
License: MIT License
As Whispr is designed to be a hub that multiple clients to connect to, it would be great to offer authentication which allows direct passthrough and validation of JWTs for those clients to avoid the need for an additional authentication against whispr.
This feature covers authentication only, not authorisation. As a next step we should look into authorisation at application ID level to protect the data further.
The filter features of whispr are not consistent. I am not able to apply the same filter to the webhooks which I can apply to the whisps query. The webhook mutation does not support the MongoDB query language.
Example filter which is working on the whisps query but cannot be added to a new webhook:
"filter": {
"attribute1": { "$exists": true },
"data.attribute2": { "$eq": 3 }
}
The same problem could occur for subscriptions aswell.
There are actually two problems to solve:
A consistent behaviour of all filter possibilities in whispr. The MongoDB query language should be used for queries, subscriptions and webhooks.
A possible solution for both problems would be the ucast ecosystem.
With using mongo2js we could:
Use mingo. This package is capable of in memory comparism. Saving the query could be achieved with mongodb-js features by transforming the query to JavaScript objects with mongodb-language-model.parse or mongodb-query-parser
The filter features of whispr are not consistent. I am not able to apply the same filter to the webhooks which I can apply to the whisps query. The webhook mutation does not support the MongoDB query language.
Example filter which is working on the whisps query but cannot be added to a new webhook:
"filter": {
"attribute1": { "$exists": true },
"data.attribute2": { "$eq": 3 }
}
The same problem could occur for subscriptions aswell.
There are actually two problems to solve:
A consistent behaviour of all filter possibilities in whispr. The MongoDB query language should be used for queries, subscriptions and webhooks.
A possible solution for both problems would be the ucast ecosystem.
With using mongo2js we could:
Use mingo. This package is capable of in memory comparism. Saving the query could be achieved with mongodb-js features by transforming the query to JavaScript objects with mongodb-language-model.parse or mongodb-query-parser
There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.
Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
@semantic-release/changelog
, @semantic-release/github
)@vuepress/theme-default
, vuepress
)@nestjs/passport
, @nestjs/platform-express
, @nestjs/platform-socket.io
, @nestjs/terminus
)node
, @types/node
)@typescript-eslint/eslint-plugin
, @typescript-eslint/parser
)@commitlint/cli
, @commitlint/config-conventional
)node
, @types/node
)supertest
, @types/supertest
)@types/jest
, jest
, ts-jest
)@typescript-eslint/eslint-plugin
, @typescript-eslint/parser
)@nestjs/common
, @nestjs/passport
, @nestjs/platform-express
, @nestjs/platform-fastify
, @nestjs/platform-socket.io
, @nestjs/schematics
, @nestjs/terminus
, @nestjs/testing
, @nestjs/websockets
)These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
@apollo/gateway
, apollo-server-core
, apollo-server-express
, apollo-server-fastify
)@nestjs/common
, @nestjs/platform-fastify
, @nestjs/terminus
, @nestjs/testing
, @nestjs/websockets
)These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.
docker-compose.dev.rep.yml
docker-compose.dev.yml
docker-compose.rep.yml
docker-compose.yml
.gitpod.Dockerfile
Dockerfile
node 16.14.0-alpine
node 16.14.0-alpine
Dockerfile.aws
.github/workflows/ci.yml
actions/checkout v2
actions/setup-node v2
actions/cache v2
actions/checkout v2
actions/setup-node v2
actions/cache v2
azure/docker-login v1
ubuntu 20.04
ubuntu 20.04
.github/workflows/code-ql-analysis.yml
actions/checkout v3
github/codeql-action v2
github/codeql-action v2
github/codeql-action v2
.github/workflows/docker-hub-description.yml
actions/checkout v2
peter-evans/dockerhub-description v2
ubuntu 20.04
.github/workflows/gh-pages.yml
actions/checkout v2
actions/setup-node v2
actions/cache v2
peaceiris/actions-gh-pages v3
ubuntu 20.04
.github/workflows/k6-load-test.yml
actions/checkout v2
actions/setup-node v2
actions/cache v2
ubuntu 20.04
docs/package.json
@vuepress/theme-default 2.0.0-beta.45
vuepress 2.0.0-beta.45
package.json
@apollo/gateway 0.44.1
@instana/collector 1.140.1
@sanofi-iadc/passport-multi-jwt 1.0.0
@nestjs/apollo 10.0.11
@nestjs/axios 0.0.7
@nestjs/common 8.2.6
@nestjs/config 1.2.1
@nestjs/core 8.2.6
@nestjs/graphql 10.0.11
@nestjs/jwt 8.0.0
@nestjs/mongoose 9.0.3
@nestjs/passport 8.2.1
@nestjs/platform-fastify 8.2.6
@nestjs/platform-socket.io 8.4.5
@nestjs/terminus 8.0.6
@nestjs/websockets 8.2.6
@typegoose/typegoose 9.7.1
@types/express 4.17.13
@types/graphql 14.5.0
amazon-cognito-identity-js 5.2.8
apollo-server-core 3.5.0
apollo-server-fastify 3.5.0
aws-sdk 2.1137.0
class-transformer 0.3.1
class-validator 0.13.2
cross-fetch 3.1.5
dotenv 10.0.0
eslint-plugin-graphql 4.0.0
express ^4.18.1
fastify 3.24.1
fastify-compress 3.7.0
flatted 3.2.5
graphql 15.8.0
graphql-redis-subscriptions 2.4.2
graphql-subscriptions 2.0.0
graphql-tag-pluck 0.8.7
graphql-tools 8.2.9
graphql-type-json 0.3.2
graphql-upload 13.0.0
haiku-random 1.0.0
https-proxy-agent 5.0.1
ioredis 4.28.5
joi 17.6.0
jwks-rsa 2.1.2
lodash 4.17.21
mongoose 6.2.11
mongoose-cast-aggregation 0.2.1
nock 13.2.4
parse-duration 1.0.2
passport 0.5.3
passport-jwt 4.0.0
reflect-metadata 0.1.13
request 2.88.2
rimraf 3.0.2
rxjs 7.5.5
superagent 6.1.0
superagent-promise 1.1.0
superagent-proxy 3.0.0
tunnel 0.0.6
@commitlint/cli 15.0.0
@commitlint/config-conventional 15.0.0
@nestjs/cli 8.2.5
@nestjs/platform-express 8.4.5
@nestjs/schematics 8.0.11
@nestjs/testing 8.2.6
@semantic-release/changelog 6.0.1
@semantic-release/exec 6.0.3
@semantic-release/git 10.0.1
@semantic-release/github 8.0.4
@types/jest 27.5.1
@types/node 16.11.36
@types/socket.io 3.0.1
@types/supertest 2.0.12
@types/tunnel 0.0.3
@typescript-eslint/eslint-plugin 5.25.0
@typescript-eslint/parser 5.25.0
apollo-server-express 3.6.8
cz-conventional-changelog 3.3.0
eslint ^7.32.0
eslint-config-airbnb-base 14.2.1
eslint-config-prettier 8.5.0
eslint-config-standard 16.0.3
eslint-import-resolver-typescript 2.7.1
eslint-plugin-import 2.26.0
eslint-plugin-node 11.1.0
eslint-plugin-prettier 3.4.1
eslint-plugin-promise 5.2.0
eslint-plugin-standard 4.1.0
husky 7.0.4
jest 27.5.1
jsonwebtoken ^8.2.0
mock-jwks 1.0.3
mongodb-memory-server 8.5.2
prettier 2.6.2
prettier-eslint-cli 5.0.1
semantic-release 19.0.2
supertest 6.2.3
ts-jest 27.1.4
ts-loader 9.3.0
ts-morph 13.0.3
ts-node 10.7.0
tsconfig-paths 3.14.1
typescript 4.6.4
webpack 5.72.1
minimist 1.2.6
node-fetch 2.6.7
cross-fetch 3.1.5
node-forge 1.3.1
glob-parent 5.1.2
.nvmrc
node 16.15.0
Describe the bug
When we first published Whispr, it 55 lint warnings were allowed for the CI job to pass. Configured in package.json with the following:
"lint:es": "eslint --max-warnings=55 --fix --ext .js,.ts src"
All warnings are still there and we should start to correct them and reduce the number of allowed warnings.
To Reproduce
Run the linter, or check warnings in CI from a pull request. E.g. https://github.com/Sanofi-IADC/whispr/actions/runs/374172150.
Expected behavior
We should start to clean these up (most are simple missing return types) and reduce the number of allowed warnings.
Screenshots
n/a
Desktop (please complete the following information):
n/a
Smartphone (please complete the following information):
n/a
Additional context
n/a
Creating this feature request following the rate limiter discussion with @nikola-kovacevic and @Babbafett in #349 to separate the two topics.
Currently there is no rate limiting functionality in whispr, meaning that a single client can send a very high number of requests to the API and (depending on the infrastructure/auto scaling setup) potentially cause a slow down for all users.
We should implement a rate limiting solution, and I'm opening this issue to discuss the topic as there are a couple of potential solutions.
A clear and concise description of any alternative solutions or features you've considered.
NestJS has a native package: https://docs.nestjs.com/security/rate-limiting. I did a short test of it. I had to add the following additional code within the useFactory section of AppModule because we're using Fastify and GQL:
context: ({ request, reply }) => {
return { req: request, res: reply }
Apart from that I just followed the docs and it works, but the options are limited to the following:
ttl | the number of seconds that each request will last in storage |
---|---|
limit | the maximum number of requests within the TTL limit |
ignoreUserAgents | an array of regular expressions of user-agents to ignore when it comes to throttling requests |
storage | the storage setting for how to keep track of the requests |
It means that we could very quickly and easily add a solid rate limiting feature, but we would do so knowing that we cannot easily implement request rules by client or other advanced logic.
Another rate limiter: https://github.com/ozkanonur/nestjs-rate-limiter. The feature set seems to be more complete, but it's not an official NestJS package. I haven't tested this one, but I think that we should try it out while deciding what we implement.
In particlular I like the fact that it has a points system (https://github.com/ozkanonur/nestjs-rate-limiter#-points), and a feature to smooth out request bursts (https://github.com/ozkanonur/nestjs-rate-limiter#-execevenly).
However, what I did not look into was whether we could achieve something similar using a combination of NestJS throttler and Bull which could put requests into a queue.
n/a
Currently the whispsCount
query takes a single mongo format query, and returns the count of documents which match that query. In cases where an application needs to get the count of whisps in bulk, a separate request is required for each query, resulting in a high number of requests from the browser (e.g. one for every row of a table).
whispsCount
to allow bulk requests, and grouping by specific criteria in the mongo querycountDocuments()
function, and use db.collection.aggregate instead which will allow us to have a grouping functionality (explained more in detail below)Currently the query runs the mongo command countDocuments()
(https://docs.mongodb.com/manual/reference/method/db.collection.countDocuments/). countDocuments
is actually just a wrapper for the following command:
db.collection.aggregate([
--
{ $match: <query> },
{ $group: { _id: null, n: { $sum: 1 } } }
])
We can easily replace countDocuments()
with that command to allow us to group, and therefore manage bulk requests.
For example this query:
db.whisps.aggregate([
{ $match: {"applicationID": "myapp"} },
{ $group: { _id: { criteria1: "$data.thing1", criteria2: "$data.thing2" }, count: { $sum: 1 } } }
Would return something like this, which satisfies the requirements:
{ "_id" : { "criteria1" : "abc", "criteria2" : "123" }, "count" : 3 }
{ "_id" : { "criteria1" : "def", "criteria2" : "123" }, "count" : 5 }
{ "_id" : { "criteria1" : "def", "criteria2" : "345" }, "count" : 7 }
{ "_id" : { "criteria1" : "xyz", "criteria2" : "999" }, "count" : 9 }
In that example the match criteria is very simple, but it could also be a more complex query with multiple things to match.
BREAKING CHANGE(S)
documentCount
queries but returns a single response(this is the proposed solution described in detail above)
db.whisps.aggregate([
{ $match: {"applicationID": "myapp"} },
{ $group: { _id: { criteria1: "$data.thing1", criteria2: "$data.thing2" }, count: { $sum: 1 } } }
Open points for discussion:
whispsCount
- should it be called whispCount
instead? - if it should then it allows us to introduce this feature without a breaking change (for now) as we could add the new query and decommission the old one laterA declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.