S3 storage backend for Scrapy's HTTP cache middleware

This extension relies on Scrapy cache mechanism, which should be enabled and configured. For more information see Scrapy documentation on this topic.

These two settings should be enough:

HTTPCACHE_ENABLED = True
HTTPCACHE_STORAGE = 'scrapy_s3_http_cache.S3CacheStorage'

The settings below can be defined as any other Scrapy settings, as described on Scrapy docs.

S3CACHE_URI: the URI where the cache should be stored.

Eg: 's3://aws_key:aws_secret@bucket/%(name)s-%(time)s'

%(name)s will be replaced by the Spider name attribute and %(time)s with the timestamp in iso format of the spider finish time.

S3CACHE_DONT_RETRIEVE: Do not retrieve responses from the cache, only for storage. Default: False.

HTTPCACHE_GZIP: Compress the cached responses using gzip. Default: False.