ashooner / supercrawler Goto Github PK
View Code? Open in Web Editor NEWThis project forked from brendonboshell/supercrawler
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
License: Apache License 2.0