This repository contains the main components of the Common Search backend.

Your help is welcome! We have a complete guide on how to contribute.

This repository has 4 components:

• cosrlib: Python code for parsing, analyzing and indexing documents
• spark: Spark jobs using cosrlib.
• urlserver: A service for getting metadata about URLs from static databases
• explainer: A web service for explaining and debugging results, hosted at explain.commonsearch.org

Here is how they fit in our general architecture:

A complete guide available in INSTALL.md.

Make sure to start the services (make start_services) before trying any tests.

Inside Docker, you can run our full test suite easily:

make test

Alternatively, you can run it from outside Docker with:

make docker_test

You may also want to run only part of the tests, for instance all which do not use Elasticsearch:

py.test tests/ -v -m "not elasticsearch"

If you want to evaluate the speed of a component, for instance HTML parsing, you can repeat the tests N times and output a Python profile:

py.test tests/cosrlibtests/document/html/ -v --repeat 50 --profile

spark-submit spark/jobs/index.py --source commoncrawl:limit=1 --plugin plugins.filter.Homepages:index=1 --profile

After this, if you have a cosr-front instance connected to the same Elasticsearch service, you will see the results!

Common Search supports the insertion of user-provided plugins in the indexation pipeline. Some are included by default, for instance:

spark-submit spark/jobs/index.py --source url:https://about.commonsearch.org/ --plugin plugins.filter.All:index=0 --plugin 'plugins.grep.Words:words=common search,path=/tmp/grep_result'

See the plugins/ directory for more examples.

The explainer allows you to debug results easily. Just run:

make docker_explainer

Then open http://192.168.99.100:9703 in your browser (Assuming 192.168.99.100 is the IP of your Docker host)