Manic is a asynchronous REST server that allows super fast querying of large memory mapped files, giving O(1) lookup times on the order of 5-10 microseconds (local server response around 1ms), using standard desktop hardware.
Manic runs on top of Sanic, a super fast, Flask-like async platform for Python
Even more, Manic allows you to write protect the file in memory at the page level, and check a previously calculated SHA256 of the text file, against your own SHA256, before it is loaded into a memory map, providing hella fast lookups in the single digit microsecond range on your laptop.
Manic was extracted from an Ethereum oracle project I am working on, and made open source under an MIT license. The functionality has been enhanced beyond the original requirements for the oracle. This repo is still a work in progress, but Manic is fully working and functional for your fast lookup use cases, and can load files in sizes up to the memory capacity you have on hand.
At the moment, Manic will only ingest and index delimited text files.
- clone the repo
[email protected]:robinagist/Manic-Mapped-Memory-Server.git
- high recommended that you run Manic in virtualenv with at least Python 3.5
- install Sanic
pip install sanic
- install pysha3
pip install pysha3
- install xxhash
pip install xxhash
- Manic is already configured to index two example files in the
/manic/memfiles
directory - configure the
BASE_PATH
inconfig.py
to point to your localmanic
root - from the
manic
directory, start the server: 'python server.py` which will immediately index the files - the server listens on port
5216
by default - the query string on the URL for the server will be of the format:
idx=<index_name>&st=<search term>
Index_name is the column you wish to perform a lookup. Column names are index names.
Manic comes with two example files: a short list of poker hands and scores, and a subset of the FCC database for ham radio licenses. It is configured to run right out of the box with those.
try curl "http://0.0.0.0:5216/f?idx=call&st=KC1IVY"
from a terminal, to do a callsign lookup. you should get one result
or try curl "http://0.0.0.0:5216/f?idx=cat&st=D"
and you should get around 250 results
Each column name is also the index name. To do a lookup on a column -- say for an FCC callsign -- use the call
column and the search term st
is an FCC assigned callsign, hopefully appearing in the file. The columns are defined in the configuration file config.json
for the file that is being indexed. These are located in the ~/memfiles
subdirectory.
More documentation later.