scheduler's Introduction

scheduler

Scheduler is a Go written program designed to orchestrate resource parsing

features

use scalable messaging protocol (nats)

how it work

The Scheduler process connect to a nats server (specified by env variable NATS_URI) and set-up a subscriber for message with tag doneSubject
When an URL is received the scheduler will apply list of crawling rules to determinate if resource is to be crawled
If resource should be crawled to scheduler will sent the url to nats with subject todoSubject for the crawlers

crawling rules

Here is the rules that determinate if crawling is to be done:

Url has not been already crawled

scheduler's People

Contributors

Stargazers

Watchers

scheduler's Issues

Complete readme

Fix duplicate persist because of latency

Because it take times for the persister to save content It is possible that the same version of a website is schedule for crawling even if it has been already crawled. This is because the API does not know that the website has been crawled before the persister has saved it.

This should be fix to remote the consistency check done in the persister

Refactor to use elasticsearch

Update go version

Remove now useless .onion check

Now done in crawler since: trandoshan-io/crawler#7

Add travis ci support

Remove forbidden extensions check

Currently the scheduler prevent the crawler from parsing unwanted resources by doing a suffix comparison between the URL and a list of forbidden extensions (taken from database)

This check is partially useless and it would be great to remove it when trandoshan-io/crawler#5 is merged.

(Don't forget to to appropriate change in trandoshan-parent / API)