kevincobain2000 / email_extractor Goto Github PK

I will like this script to result able to save to file.
Also if possible to save interval by internal not until it crawl all urls Eg. if we are to crawl 40,000 pages website. it will crawl all before it grep one email to file

It would be nice to have a way of excluding a list of URLs or base domains such that the crawler, if it comes to one of these domains, doesn't end up clicking through it and continuing the crawl.

This could be useful for sites that you know are not useful or valid for the search.

I found that an tag on a page like:

<a href="Journal" rel="history">Journal</a>

Is not being traversed?

creating separate issue from #1

@sirolele

Also if possible to save interval by internal not until it crawl all urls Eg. if we are to crawl 40,000 pages website. it will crawl all before it grep one email to file

This might be a nice option if you only expect n number of emails at a given site.

For example:

• email(at)domain(dot)com
• email[at]domain[dot]com
• first(dot)last(at)domain(dot)com

If the full text is being parsed for email matching then I think it would be good to match and then reform the email based on substitution.