Comments (3)
Thank you so much @cyb3rjerry for your contribution, really appreciated.
Actually you are right, it should not happen.
Investigating a bit it seems like that 99% of the URL that should not pass are actually ignored, but a small part of them still get printed on cli. This is because the method used to print URLs on CLI is:
c.OnResponse(func(r *colly.Response) {
fmt.Println(r.Request.URL.String())
...
This prints also the ignored URLs in case of a long chain of redirects where the last call is made by a ignored URL.
Imagine something like this:
- Link1 redirects to Link2
- Link2 redirects to Link3
- ...
- LinkToIgnore redirects to LinkN
This behavior makes cariddi printing LinkToIgnore.
I don't have a solution for this, if you want to propose something I'm all ears.
For now you can rely on output files, I'm quite sure they don't include to-be-ignored URLs.
from cariddi.
Sounds good! I'll thinker on this and get back to you at some point :)
Thanks again for the great tool btw!
from cariddi.
Closed by #82
from cariddi.
Related Issues (20)
- Refactor visitHTMLLink() HOT 7
- Improve Regex to avoid false positives HOT 2
- Problem with Crawling POST Parameters
- Add JSON functionality and console reporting of secrets, errors HOT 16
- version and banner as global constant HOT 1
- Multiple info in the same URL HOT 1
- JSON lines aggregate results HOT 4
- Add JSON file output
- Example Regex File HOT 11
- Scan Only HOT 6
- Store raw HTTP responses HOT 7
- Proxy settings not honored HOT 5
- Document input flags in code HOT 1
- Raw requests improvements HOT 4
- Pass a custom directory for raw responses: `-srd /path/to/responses/dir` (similar to `httpx` / `katana`)
- Add `stored_response_path` to JSON lines output to show where the request / response raw file was stored
- Second ctrl+c should force quit the program HOT 4
- Wrap the New() params into a single object HOT 1
- Wrap the results in a struct HOT 1
- Fix code scanning alert - Incomplete regular expression for hostnames HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cariddi.