sreenigsd / arachnid Goto Github PK
View Code? Open in Web Editor NEWArachnid is a scalable crawler framework. It covers the whole lifecycle of a crawler: downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler.
License: MIT License