chxdeng / mabain Goto Github PK

When running in async mode, mabain can handle insertions concurrently with resource collection (or defragmentation). Need to extend this for removing kv pairs.

The current docker build does not run unit-tests. We need to add that in.
Please assign this to me.

When using disk storage for high volume of insertions, the mabain insertion performance starts to degrade when there are a few hundreds of millions of key-value pairs. This is because the system/OS cannot cache all the index sections. And the disk reading is blocking new insertions.

The solution is use multi-level tries similar to log-structured merge tree (see https://link.springer.com/article/10.1007%2Fs002360050048).

I know you don't claim support for OSX, but I thought I would capture the output of the compilation errors anyway. Note, I've hit some of the same errors attempting to build within an alpine Linux Docker container. Maybe it's just a C++ version dependency? If so, maybe just call out that dependency in the README and close this Issue as resolved.

Attached: Potentially useful software versions, Compiler Errors.

sw_versions.txt
compile_errors.txt

One possible solution is to use snappy for compress the data part.

Currently, mabain DB handles are not thread safe. This means each thread need to open its own DB handle. It is desirable to make the DB handle thread-safe so that one handle is necessary per process.

I've found that build misses support of macOS build out-of-the-box.
Let's add macOS support, it requires quiet few changes, at least for build of the library itself.

Currently, mabain only supports multi-thread DB insertion. Need to extend this to multiple processes use case.

Currently, mabain will discard part or all of the new insertions during resource collection (in async mode) if the process died expectedly before resource collection is completed.
Mabain should be able to recover all those unfinished insertions/deletions.

Temp files may be created during resource collection. These files should be removed after resource collection is done.

One example of the use case: find out all IP addresses within certain range.

If writer dies in the middle of updating db and does not clear the edge offset in header, reader may keeps getting retry return if reading the same edge.

When building mabain for the first time, I found that the machine I was building on did not have the GNU Readline development package installed. It was an easy workaround, but the README file should call out the dependencies.

It should also call out what version of g++ is required, glibc, etc..

TL;DR - The README.md should list dependencies.

Because OSX (w/ default HFS+ settings) does not enforce case sensitivity in the filesystem, the BUILD file ends up matching the build target in the Makefile and you always get:

> make build
make: 'build' is up to date.

If you rename the BUILD file it actually builds (see another issue I'm about to open there)

Attached: potentially useful version info
typescript.txt

It would be desirable to have a configurable option to run in an in-memory-only mode, i.e., no flushing of the database to disk (no notion of a disk storage at all for it). Obviously, this would lose that persistence, but there may be some configurations where that's not needed (and may even be some drawbacks to doing so).