Thumbor report successful delete while in practice the delete fails.

issue discovered in thumbor_aws.storages.s3_storage:is_expired. stack trace:

2014-04-06 01:26:23 thumbor:ERROR ERROR: Traceback (most recent call last):
File "/home/deploy/thumbor/local/lib/python2.7/site-packages/tornado/web.py", line 1192, in _stack_context_handle_exception
raise_exc_info((type, value, traceback))
File "/home/deploy/thumbor/local/lib/python2.7/site-packages/tornado/web.py", line 1375, in wrapper
result = method(self, _args, *_kwargs)
File "/home/deploy/thumbor/local/lib/python2.7/site-packages/thumbor/handlers/imaging.py", line 85, in get
return self.execute_image_operations()
File "/home/deploy/thumbor/local/lib/python2.7/site-packages/thumbor/handlers/init.py", line 69, in execute_image_operations
self.filters_runner.apply_filters(thumbor.filters.PHASE_PRE_LOAD, self.get_image)
File "/home/deploy/thumbor/local/lib/python2.7/site-packages/thumbor/filters/init.py", line 80, in apply_filters
callback()
File "/home/deploy/thumbor/local/lib/python2.7/site-packages/thumbor/handlers/init.py", line 91, in get_image
self._fetch(self.context.request.image_url, self.context.request.extension, callback)
File "/home/deploy/thumbor/local/lib/python2.7/site-packages/thumbor/handlers/init.py", line 265, in _fetch
buffer = storage.get(url)
File "/home/deploy/thumbor/src/thumbor-aws/thumbor_aws/storages/s3_storage.py", line 94, in get
if not file_key or self.is_expired(file_abspath):
File "/home/deploy/thumbor/src/thumbor-aws/thumbor_aws/storages/s3_storage.py", line 130, in is_expired
timediff = datetime.now() - self.utc_to_local(parse_ts(key.last_modified))
AttributeError: 'str' object has no attribute 'last_modified'

the error is on line 94 in storages/s3_storage.py

this
if not file_key or self.is_expired(file_abspath):
should be
if not file_key or self.is_expired(file_key):

have you tried using cloudfront with this package? feels like boto does support it, so I'm wondering how easy it would be to get the loader class to serve assets from cloudfront instead (say if there is a key value pair in the thumbor.conf file)

Hi, great project... but I can't seem to get it working. I'm not really familiar with Python or this stack, so some docs here would be great.

Right now I followed some directions that I found on this blog post: http://www.dadoune.com/blog/best-thumbnailing-solution-set-up-thumbor-on-aws/

Still, no matter what I do, I run into a S3ResponseError: S3ResponseError: 403 Forbidden. I'm using an IAM role that has wide open admin level permissions, and I've given my bucket read access to everyone. Still, I'm hitting 403s. I feel like it's probably a basic mistake on my end with configuration or something.

Docs would be super helpful, thanks :)

in thumbor_aws.storages.s3_storage:utc_to_local

this:
def utc_to_local(utc_dt):
should be:
def utc_to_local(self,utc_dt):

The code handling the new AWS_ENABLE_HTTP_LOADER property falls over if the property is not defined. Also, this property doesn't seem to be documented in the README. Ideally this would default to False if not defined.

The problem was introduced in the following commit: 12ef2ef

Workaround:

Add AWS_ENABLE_HTTP_LOADER=False to the Thumbor config

Stacktrace:

2015-01-29 18:12:47 thumbor:ERROR ERROR: Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/tornado/web.py", line 1320, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "/usr/lib64/python2.6/site-packages/tornado/web.py", line 1503, in wrapper
    result = method(self, *args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/thumbor/handlers/imaging.py", line 100, in get
    self.check_image(kw)
  File "/usr/lib64/python2.6/site-packages/thumbor/handlers/imaging.py", line 96, in check_image
    return self.execute_image_operations()
  File "/usr/lib64/python2.6/site-packages/thumbor/handlers/__init__.py", line 89, in execute_image_operations
    self.filters_runner.apply_filters(thumbor.filters.PHASE_PRE_LOAD, self.get_image)
  File "/usr/lib64/python2.6/site-packages/thumbor/filters/__init__.py", line 81, in apply_filters
    callback()
  File "/usr/lib64/python2.6/site-packages/thumbor/handlers/__init__.py", line 114, in get_image
    self._fetch(self.context.request.image_url, self.context.request.extension, callback)
  File "/usr/lib64/python2.6/site-packages/thumbor/handlers/__init__.py", line 377, in _fetch
    self.context.modules.loader.load(self.context, url, handle_loader_loaded)
  File "/usr/lib/python2.6/site-packages/thumbor_aws-1-py2.6.egg/thumbor_aws/loaders/s3_loader.py", line 34, in load
    if (context.config.AWS_ENABLE_HTTP_LOADER and
  File "/usr/lib/python2.6/site-packages/derpconf/config.py", line 200, in __getattr__
    raise AttributeError(name)
AttributeError: AWS_ENABLE_HTTP_LOADER

It was working for me up until 4.12.2 but doesn't work after upgrading to version 5.

Hello,

Has this module been used successfully in a production environment?

Also I'm having some issues with safe (signed) URLs. Here's the debug log:

Wed, 18 Sep 2013 14:42:10 GMT
/thumbor/thumbor/storage/a7f21387d6cefb109f67822411f7263792878e2e
2013-09-18 09:42:10 thumbor:ERROR ERROR: Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/tornado/web.py", line 1055, in _stack_context_handle_exception
raise_exc_info((type, value, traceback))
File "/usr/lib/python2.6/site-packages/tornado/web.py", line 1178, in wrapper
result = method(self, _args, *_kwargs)
File "/usr/lib64/python2.6/site-packages/thumbor/handlers/imaging.py", line 92, in get
return self.execute_image_operations()
File "/usr/lib64/python2.6/site-packages/thumbor/handlers/init.py", line 63, in execute_image_operations
result = self.context.modules.result_storage.get()
File "/usr/lib64/python2.6/site-packages/thumbor/utils.py", line 33, in wrapper
return fn(_args, *_kwargs)
TypeError: get() takes exactly 2 arguments (1 given)

2013-09-18 09:42:10 tornado.access:ERROR 500 GET /Ko58RlIHR72cOpQfXkp1KukyafU=/50x50/smart/http%3A%2F%2Fthumbor.policymic.com%3A8888%2Fimage%2Feef304fa724c484a88620a4595460bf7%2Fimage.jpg (98.116.242.17) 25.15ms

Note that signed requests were working as expected prior to switching to the S3 storage engine.

Thanks

As discussed with Bladrak I am opening this issue in respects of the S3 loader not working properly.

Please see the below config and logs.

S3_LOADER_BUCKET = 'my-bucket-xxx'
S3_ALLOWED_BUCKETS = ['my-bucket-xxx']
AWS_ENABLE_HTTP_LOADER=True

2015-09-28 13:51:13 thumbor:DEBUG STATSD: storage.miss:1|c
2015-09-28 13:51:13 tornado.access:WARNING 404 GET /unsafe/fit-in/525x650/filters:watermark(/my-bucket-xxx/watermark.png,0,0,0)/my-bucket-xxx/APAP131C78CE.jpg (127.0.0.1) 5.05ms

Hello @willtrking,

Given you don't seem to have the time to maintain this repository, we forked it under the https://github.com/thumbor-community organisation. I hope you don't mind, but some of us needed to update this repository for the 5.0 version of thumbor.

If you wish to discuss this with us, feel free to contact either @masom or myself, we'll be more than happy to give you any information you may need. You will of course always be credited for your work.

For the contributors, feel free to re-submit your Pull Requests and/or issues to the fork here: https://github.com/thumbor-community/aws ; we'll gladly review them and merge them.

UPDATE
Ah, I missed the pull request in async S3 HTTP with botornado #14
Please merge! :)

ORIGINAL POST
When comparing the loader and storage classes in this plugin with the standard http loader for example (https://github.com/thumbor/thumbor/blob/master/thumbor/loaders/http_loader.py) then you'll see that thumbor_aws is blocking the whole runloop when fetching files from s3.
This is a bad practice when working with tornado and defeats the purpose of thumbor, which usually is very fast.
A possible solution would be to use tornado-botocore (https://github.com/nanvel/tornado-botocore) instead, which allows htto requests through the standard tornado.httpclient.

E.g. compare https://github.com/thumbor/thumbor/blob/master/thumbor/loaders/http_loader.py#L87
client.fetch(req, callback=partial(return_contents, url=url, callback=callback, context=context)) (proper callback usage)
with https://github.com/willtrking/thumbor_aws/blob/master/thumbor_aws/loaders/s3_loader.py#L58
return callback(file_key.read()) (blocks the runloop)