lequytra / smoogle Goto Github PK

Requirements:

• A class to crawl all pages from a seed file.
• Retrieve hyperlinks from crawled pages, compared it against crawled pages (store the url of crawled pages in a txt file)
• Automatically crawl more pages until you want to stop :">
• Each webpage's content is stored in a seperate txt file, named by their id (id.txt). All contents should be store in some folder.
• As web pages are crawled, update the web graph.
• Remember to write the webb graph into a Pickle file after crawling.