Crawlergo not working about crawlergo HOT 3 CLOSED

选择规模较小的网站可正常爬取，大规模网站则出现上述报错

from crawlergo.

或许，你只需要设置超时参数，并使用一个网络状况良好的代理即可。

from crawlergo.

I set up a server on AWS with 8 CPUs and 32 GB of RAM to retest.
The internet download speed is 500 mbit. I made the query as follows.

./crawlergo -c /usr/bin/chromium-browser -t 2 --tab-run-timeout=60s --wait-dom-content-loaded-timeout=60s https://www.amazon.com/

Result: navigate timeout.

It first crawling, but when the timeout period is up it gives a "navigate timeout" error.
The timeout is also written in the picture you shared. If there were more to the picture, I'm sure we would see more timeout errors.

Your suggested solution didn't solve the problem, but thank you for your interest.

from crawlergo.