shsms / fetchbot Goto Github PK
View Code? Open in Web Editor NEWThis project forked from puerkitobio/fetchbot
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
License: BSD 3-Clause "New" or "Revised" License