Salt Crawler is a distributed web crawler based on the Salt remote execution framework. It is currently only for demonstration purposes, but it is likely to expand to a more powerful web crawling engine.
y0j / salt-crawler Goto Github PK
View Code? Open in Web Editor NEWThis project forked from techhat/salt-crawler
A Salt-based web spidering engine
License: Other