lericson / simples3 Goto Github PK
View Code? Open in Web Editor NEWSimple, quick Amazon AWS S3 interface in Python
License: BSD 2-Clause "Simplified" License
Simple, quick Amazon AWS S3 interface in Python
License: BSD 2-Clause "Simplified" License
S3Bucket.listdir()
method raises UnboundLocalError
. It seems to occur when its result list is empty, but I have no idea about its detail. What is next_marker
attribute for?
Traceback (most recent call last):
File "/.../app.py", line 134, in list_sizes
keys = list(self.bucket.listdir(prefix=prefix + '/'))
File "/home/styleshare/lib/python2.7/site-packages/simples3/bucket.py", line 395, in listdir
for item in listing:
File "/home/styleshare/lib/python2.7/site-packages/simples3/bucket.py", line 189, in __iter__
self.next_marker = item[0]
UnboundLocalError: local variable 'item' referenced before assignment
Simple S3 is liiting the output to 1000 file when using listdir.
http://stackoverflow.com/questions/40331681/python-simples3-listdir-limit-the-output-to-1000-files
Any way / plan to make it support more than 1000 - or an unlimited number of entries?
I am trying to set some far future expiration on my files in a bucket on S3, using this python code:
future = 60*60*24*666
future_date = datetime.now() + timedelta(seconds=future)
future_date = future_date.strftime("%a, %d %b %Y %H:%M:%S GMT")
# and further down in file:
f = open(src)
s3.put(key, f.read())
s3.copy("%s/%s" % (bucket, key), key, metadata={
"Cache-control": "max-age=%d" % future,
"Expires": future_date})
However, when I investigate the files in firebug they do not have neither Expires och Cache-control. Instead there are prefixed versions of the metadata.
x-amz-meta-cache-control max-age=57542400
x-amz-meta-expires Thu, 15 May 2014 01:03:25 GMT
I would highly appreciate to set metadata and skip the "-x-amz-" prefixes.
This not an issue but a small requirement.
There is no return type for put_file() method, hence could not decide whether the object is copied to S3 properly or not.
S3Request
seems to have a problem with handling arguments because of using ;
as args separator(https://github.com/lericson/simples3/blob/master/simples3/bucket.py#L138).
It works right with &
instead of ;
, so I carefully guess S3 may does not support ;
separator.
Did I do anything wrong?
One should be able to create buckets. Suggested API for this is either:
from simples3 import S3Bucket
b = S3Bucket("foo", ...)
b.create(acl="public-read")
or class-style (a bit better IMO):
from simples3 import S3Bucket
b = S3Bucket.create_new("foo", ..., acl="public-read")
There are other suggestions as well, including module-level functions.
The only egress access out of my network is through a proxy server that is tightly controlled. How can I use a proxy with this? Or is this a feature that could be added?
Traceback (most recent call last):
File "./s3sync_olaf.sh", line 317, in <module>
main()
File "./s3sync_olaf.sh", line 301, in main
trim_bucket(bucket)
File "./s3sync_olaf.sh", line 116, in trim_bucket
del s3b[key]
File "/usr/lib/pymodules/python2.5/simples3/bucket.py", line 157, in __delitem__
def __delitem__(self, name): return self.delete(name)
File "/usr/lib/pymodules/python2.5/simples3/bucket.py", line 296, in delete
resp = self.make_request("DELETE", key=key)
File "/usr/lib/pymodules/python2.5/simples3/bucket.py", line 259, in make_request
raise exc_cls.from_urllib(e, key=key)
simples3.bucket.S3Error: HTTP error (code=204, key='space/ourproject-2011-04-24.tar.gz', filename='http://s3.amazonaws.com/Assembla-space-ourproject/space/ourproject-2011-04-24.tar.gz')
I use the S3Bucket.listdir
API call a lot. It gives a list of files. I iterate through it and check the filedate if it is an old file I run del s3b[key]
. That command is on a separate line, and is the line triggering the exception.
Basically:
for (k, modify, etag, size) in b.listdir(...):
if predicate(modify):
del b[k]
Hi,
When you issue 2 put_file request with different file size, the header content-length is stuck at the size of the first file, causing a 400 return code from server due to the invalid content-length.
To reproduce the issue :
f_path = 'tmp'
with open(f_path, 'wb') as f:
f.write("toto")
s3.put_file(f_path, f_path)
with open(f_path, 'wb') as f:
f.write("toto2")
s3.put_file(f_path, f_path)
Tested against RADOS GATEWAY.
The root cause is due to the fact that is header param is empty when calling put_file, a default dict is assigned in the function definition. https://github.com/lericson/simples3/blob/master/simples3/streaming.py#L34
As a result, every subsequent call will get the same object reference (as the dict is in the class scope and not function scope).
Content-length will always be set after the first call, as a result it is not updated
https://github.com/lericson/simples3/blob/master/simples3/streaming.py#L58
Here is a fix :
class StreamingMixin(object):
def put_file(self, key, fp, acl=None, metadata={}, progress=None,
size=None, mimetype=None, transformer=None, headers=None):
"""Put file-like object or filename *fp* on S3 as *key*.
*fp* must have a read method that takes a buffer size, and must behave
correctly with regards to seeking and telling.
*size* can be specified as a size hint. Otherwise the size is figured
out via ``os.fstat``, and requires that *fp* have a functioning
``fileno()`` method.
*progress* is a callback that might look like ``p(current, total,
last_read)``. ``current`` is the current position, ``total`` is the
size, and ``last_read`` is how much was last read. ``last_read`` is
zero on EOF.
"""
if not headers:
headers = {}
do_close = False
The current implementation of RFC 822 formatting uses just time.strftime()
function, and the formatter for it, defined in simples3.util.rfc822_fmt
, is '%a, %d %b %Y %H:%M:%S GMT'
. According to the documentation of time.strftime()
:
%a
: Locale’s abbreviated weekday name.
%b
: Locale’s abbreviated month name.
As a result, the current RFC 822 formatting is broken in non-US locale (e.g. ko_KR
, ja_JP
). You can reproduce it easily:
>>> import time
>>> t = time.gmtime()
>>> time.strftime('%a, %d %b %Y %H:%M:%S GMT', t)
'Wed, 30 Nov 2011 08:29:14 GMT'
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'ko_KR')
'ko_KR'
>>> time.strftime('%a, %d %b %Y %H:%M:%S GMT', t)
'\xec\x88\x98, 30 11 2011 08:29:14 GMT'
>>> print _
수, 30 11 2011 08:29:14 GMT
>>> locale.setlocale(locale.LC_ALL, 'ja_JP')
'ja_JP'
>>> time.strftime('%a, %d %b %Y %H:%M:%S GMT', t)
'\xe6\xb0\xb4, 30 11 2011 08:29:14 GMT'
>>> print _
水, 30 11 2011 08:29:14 GMT
This bug also produces incorrect signatures (used for AWS authentication).
If your python is compiled without ssl support, simples3 will start to cry:
python2.5 setup.py build -b build-2.5
Traceback (most recent call last):
File "setup.py", line 8, in <module>
import simples3
File "simples3/__init__.py", line 124, in <module>
from .bucket import S3File, S3Bucket, S3Error, KeyNotFound
File "simples3/bucket.py", line 66, in <module>
class StreamHTTPSHandler(urllib2.HTTPSHandler):
AttributeError: 'module' object has no attribute 'HTTPSHandler'
IMO ssl should be optional and not required
When trying to use simples3
to connect to a bucket hosted in eu-west-1
, I get the following error:
The bucket you are attempting to access must be addressed using the specified endpoint.
fp = ProgressCallingFile(fp, progress)
...
TypeError: __init__() takes exactly 4 arguments (3 given)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.