-i docs doesn't ignore subdomains containing "docs" about cariddi HOT 3 CLOSED

Thank you so much @cyb3rjerry for your contribution, really appreciated.

Actually you are right, it should not happen.
Investigating a bit it seems like that 99% of the URL that should not pass are actually ignored, but a small part of them still get printed on cli. This is because the method used to print URLs on CLI is:

c.OnResponse(func(r *colly.Response) {
		fmt.Println(r.Request.URL.String())
...

This prints also the ignored URLs in case of a long chain of redirects where the last call is made by a ignored URL.
Imagine something like this:

• Link1 redirects to Link2
• Link2 redirects to Link3
• ...
• LinkToIgnore redirects to LinkN

This behavior makes cariddi printing LinkToIgnore.
I don't have a solution for this, if you want to propose something I'm all ears.

For now you can rely on output files, I'm quite sure they don't include to-be-ignored URLs.

from cariddi.

Sounds good! I'll thinker on this and get back to you at some point :)

Thanks again for the great tool btw!

from cariddi.

Closed by #82

from cariddi.