bitsplitter / startupdb Goto Github PK

Chalk library output on the console is hard to read when analysing raw log files from CI/CI pipeline.
I would prefer plain log files.

When a malformed url is passed, an exception is thrown that is not caught anywhere crashing the request and leaving the requester without a response.
Better would be to try Decoding an return a 400 error when it fails

When I call updateObjects without a payload, I get the following error:

STARTUPDB Error TypeError: Cannot read properties of undefined (reading 'id')
at addIdsToItemsThatHaveNone (/Users/jeroen/LatestCollectionServer/node_modules/startupdb/dist/server.js:117:19)
at dbUpdateObjects (/Users/jeroen/LatestCollectionServer/node_modules/startupdb/dist/server.js:569:5)
at async /Users/jeroen/LatestCollectionServer/dist/hooks/beforeGetCampaign.js:24:5
at async processMethod (/Users/jeroen/LatestCollectionServer/node_modules/startupdb/dist/server.js:692:24)
at async /Users/jeroen/LatestCollectionServer/node_modules/startupdb/dist/server.js:772:20

As a developer, I would like a more relevant error that helps me pinpoint the issue quicker.

Please add the following two DBcommands:

1. Clear the whole cache memory.
2. Clear a specific resource out of the cache memory.

When a collection is being loaded (async) startupDB can kick in for another request and decides to remove the new collection since that has no lastAccessed timestamp yet which implies that it is the oldest collection around making it eligible for GC.

This combination does return a checkpoint but the data object within it is a filtered array instead of a filtered object.

Due to the inaccuracy in memory footprint estimation, GC can draw the wrong conclusion about available memory and evict all collections, even the ones that are in the process of being read form disk.

As a developer, I would like to use the DELETE method in a similar way as the GET method.
So it should support the id and filter parameters just as the GET method does.

It should DELETE a single object matching the id or it should DELETE all objects matching the expression in the filter.

The calculateMemoryFootprint function unpredictably crashes. It's unable to access startupDB[collection].data.

In order to make startupDB more generic, allow to configure its memory management.

DB_CACHE_OVERHEAD_RATIO Determines the predicted memory footprint of JSON objects. The default value of 2 seems to work well in existing deployments but in more specific use-cases, another ratio might work better, for example if there are very deeply nested objects that have higher memory overhead.

DB_CACHE_FRACTION Determines the fraction of memory used for the DB cache. If the NodeJS-Express API that embeds StartupDB needs a larger fraction of available memory allocated to NodeJS via the —-max-old-space-size parameter, this fraction needs to be configurable.

While testing the RAM usage of the startupDB, I've got in some how a negative RAM usage which is not possible to be. So please fix this bug.

usedBytesInMemory: -157573965

When accessing a resource that is not in memory yet, with multiple requests simultaneously, different errors are thrown, all boiling down to the resource not (completely) loaded in the in-memory cache.

Currently, the PATCH method only supports jsonpatch. I'd like to be able to merge my update with an existing document, only passing the new or updated properties instead of having to replace the entire document with a PUT.

If the PATCH body does not contain a patch array, apply Object.assign to it to merge the source and target objects.

usedBytesInMemory is updated with a wrong number when a collection loads into memory.
As a result, the garbage collection decides to evict collections for no good reason.

When an operation is performed on a collection that is among the least recently used collection, the GC may decide to delete it causing the application to fail because it expects the data to be available.

We need to 'touch' the collection before the actual transaction is written to keep the GC from interfering.

When using the HEAD method to get the latest update information on a collection, the x-last-oplog-id header will return a wrong id when there are oplog files with different filename lengths.
Oplog files are named .json, when retrieving all files from the oplog directory, at least on Windows, files are return sorted, that is, alphabetically. That does not give us the most recent file when not all filenames are the same length.