neo4reo / aleph Goto Github PK

Truth cannot penetrate a closed mind. If all places in the universe are in the Aleph, then all stars, all lamps, all sources of light are in it, too.

—The Aleph, Jorge Luis Borges

Aleph is a tool for indexing large amounts of both unstructured (PDF, Word, HTML) and structured (CSV, XLS, SQL) data for easy browsing and search. It is built with investigative reporting as a primary use case. Aleph allows cross-referencing mentions of well-known entities (such as people and companies) against watchlists, e.g. from prior research or public datasets.

Here's some key features:

• Web-based UI for search across large document and data sets.
• Watchlist editor for making custom sets of entities to be tracked.
• Equal support for structured (i.e. tabular) and unstructured (i.e. textual) sources.
• Importers include a local filesystem traverser, web crawlers and a SQL query importer.
• Document entity tagger (regular expressions-based, and optionally using NLP).
• Support for OCR, unpacking Zip/RAR/Tarballs, language and encoding detection.
• Entity watchlist importers for OpenNames and Investigative Dashboard.
• OAuth authorization and access control on a per-source and per-watchlist basis.
• Excel export for search result sets.

Documentation

The documentation for Aleph is available at aleph.readthedocs.io. Feel free to edit the source files in the docs folder and send pull requests for improvements.

To build the documentation, please install the dependencies first and run make docs:

(host)$ docker-compose run app bash
(app) $ pip install -r requirements-docs.txt
(app) $ make docs

Now you can browse the documentation locally at http://lvh.me:8000/docs/_build/html/:

(host)$ make docs-web

Support

Aleph is used by multiple organisations, including Code for Africa, OCCRP and OpenOil. For coordination, the following mailing list exists: aleph-search

If you find any errors or issues using Aleph please file an issue on Github or contact the mailing list.