Comments (7)
aaah okok now I got it ahaha (I thought you was referring to this with the word 'context': https://github.com/gocolly/colly/blob/947eeead97b39d46ce2c89b06164c78b39d25759/response.go#L36).
Anyway, yes I know. I have tried in initial releases to implement the current behavior using that struct, but without success. I have encountered difficulties with -intensive mode, ignoring URLs... but of course if you want to use them it's super okay. I mean, they are built for these types of usage :)
from cariddi.
Sounds good! Will try using Colly OOTB config and if it doesn't work I'll revert back to passing our custom struct
from cariddi.
Actually the implementation of -intensive, ignoring URLs etc.. should be done in CreateColly passing the Scan struct and playing with the fields and methods of the collector object.
We should also have clear test cases to test URLs ignoring and intensive mode as it's really easy to mess with them
from cariddi.
@edoardottt Quick question before I get too deep into this one, is there a reason why we don't use the e.Request.Visit(e.Attr("..."))
colly method instead of making our own version of it?
After looking at the code, I notice that most of the cases are covered by it.
- It supports URL ignore
- It supports checking if the URL has already been visited
- Using the e.Request.Visit method would be a lot more efficient
from cariddi.
Maybe I'm not understanding what you mean...This is the implementation of Request.Visit
:
// Visit continues Collector's collecting job by creating a
// request and preserves the Context of the previous request.
// Visit also calls the previously provided callbacks
func (r *Request) Visit(URL string) error {
return r.collector.scrape(r.AbsoluteURL(URL), "GET", r.Depth+1, nil, r.Ctx, nil, true)
}
ref: https://github.com/gocolly/colly/blob/947eeead97b39d46ce2c89b06164c78b39d25759/request.go#L119
from cariddi.
Yes but it uses the context that's defined at the collector's creation (URLs to ignore for example)
All of the "ignoring checks" that are manually there here:
Lines 381 to 383 in 4d59028
Are already taken into account via the context if you use
e.Request.Visit(e...)
We're currently doing manually something Colly does by default by using e.Request.Visit
For example, we don't need to check whether the URL that's currently being scanned is "allowed" to be scanned or not as colly already does this by using the context values
In essence what I'm trying to say is that we don't need to check if a URL is to be scrapped or not, if it should scrape a given subdomain, if it should be ignored or not, ... This can all be set at the Collectors creation and colly will then (without us having to code it) make sure all the conditions are respected before continuing to a page. e.Request.Visit
by default makes sure it's not out of bounds
from cariddi.
Souldn't be too crazy to do via DisallowedURLFilters which can take a Regex
from cariddi.
Related Issues (20)
- Multiple info in the same URL HOT 1
- JSON lines aggregate results HOT 4
- Add JSON file output
- Example Regex File HOT 11
- Scan Only HOT 6
- Store raw HTTP responses HOT 7
- Proxy settings not honored HOT 5
- Document input flags in code HOT 1
- Raw requests improvements HOT 4
- Pass a custom directory for raw responses: `-srd /path/to/responses/dir` (similar to `httpx` / `katana`)
- Add `stored_response_path` to JSON lines output to show where the request / response raw file was stored
- Depth Setting for More Comprehensive Crawling HOT 5
- Bug hunting
- Proxy issue HOT 6
- User-Agent not set HOT 1
- Consider making binary releases HOT 1
- feat: use go-flags HOT 2
- Add trufflehog for secret detection HOT 5
- Add -secret-extension-filter
- Add rate limit option HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cariddi.