brendonboshell / supercrawler Goto Github PK
View Code? Open in Web Editor NEWA web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
License: Apache License 2.0