How can I use this to get the number of times a site is crawled by the wayback? about wayback-machine-scraper HOT 2 CLOSED

You can do this by querying the CDX server directly. For example,

URL=panda.com
curl "http://web.archive.org/cdx/search/cdx?url=${URL}&fl=statuscode" | grep 200 | wc -l

will tell you that panda.com has been scraped with a 200 response code 589 times. The fl=statuscode querystrying is telling the server to just return the response status codes for crawls, the grep 200 is removing status codes other than 200, and wc -l counts the number of lines/crawls.

from wayback-machine-scraper.

Thanks, I did it in windows though, in Java with a for loop to count how many times :)

from wayback-machine-scraper.