Scrape TTL? about sketchy HOT 3 CLOSED

That's a great idea, will add this to a feature request.

from sketchy.

This could be achieved with redis. Just set a TTL configuration option in https://github.com/Netflix/sketchy/blob/master/config-default.py , then for every URL scraped set that URL in redis as the key and the file path as the value.

Then, for every request, check redis for the existence of that URL through redis.get(url), if it returns None, scrape, if not, return the value from redis.

from sketchy.

Closing. If someone wants to add this feature, please reopen.

from sketchy.