Crawler does not crawl all links of paginated forum list about ache HOT 4 CLOSED

I think you should probably not be using the NonRandomLinkSelector. This is poorly documented, but it still exists in the codebase only for legacy reasons. Can you try another selector such as the default TopkLinkSelector or the MaximizeWebsitesLinkSelector if you are crawling more than one website?

from ache.

Thanks for the quick reply. I did try all of these type of link selectors:

• TopkLinkSelector
• RandomLinkSelector
• NonRandomLinkSelector
• MaximizeWebsitesLinkSelector

As expected, it resulted in a different order of when a page was crawled, but did not change the total amount of crawled pages. I also tried not using Tor.

from ache.

What if you also set the link classifier to something like the following?

link_storage.link_classifier.type: MaxDepthLinkClassifier
link_storage.link_classifier.max_depth: 2147483647

The link classifier (documented here: https://ache.readthedocs.io/en/latest/crawling-strategies.html#link-classifiers) is the class actually assigns the scores and defines the crawling order. The MaxDepthLinkClassifier will crawl pages in the same order that they are discovered up to a maximum depth.

Another option could be the LinkClassifierRandom.

link_storage.link_classifier.type: LinkClassifierRandom

from ache.

Setting this actually did the trick and the crawler downloaded all expected pages:

link_storage.link_classifier.type: MaxDepthLinkClassifier
link_storage.link_classifier.max_depth: 2147483647

Thanks a lot!

from ache.