smore-inc / tinys3 Goto Github PK

No merged pull requests for a year, no responses to issues. If it's abandoned, are you looking for maintainers?

Really enjoy your lib which is very helpful, however, I have encountered issue, which to be honest not sure if it's actually caused by your lib since I'm not a Python dev, so please forgive my ignorance.

Basically the problem happens when I try to upload a file larger than a certain size 1.X GB, and the stacktrace will looks like this:

Traceback (most recent call last):
File "s3util.py", line 64, in
upload_resp = conn.upload('/live_archive/' + local_file_name, f, push_bucket_name)
File "/usr/local/lib/python2.6/dist-packages/tinys3/connection.py", line 152, in upload
return self.run(r)
File "/usr/local/lib/python2.6/dist-packages/tinys3/connection.py", line 233, in run
return self._handle_request(request)
File "/usr/local/lib/python2.6/dist-packages/tinys3/connection.py", line 255, in _handle_request
return request.run()
File "/usr/local/lib/python2.6/dist-packages/tinys3/request_factory.py", line 178, in run
auth=self.auth)
File "/usr/local/lib/python2.6/dist-packages/requests/api.py", line 99, in put
return request('put', url, data=data, *_kwargs)
File "/usr/local/lib/python2.6/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, *_kwargs)
File "/usr/local/lib/python2.6/dist-packages/requests/sessions.py", line 416, in request
data = data or {},
OverflowError: long int too large to convert to int

Is it because I'm using your lib in a wrong way, or is it a limitation of python 2.6 or any other constraint of your lib?

Will be appreciate any help.
YD

I had to manually install futures pip install futures before tinys3 would run. Might want to add a check for that into the setup

I ran into an issue where importing tinys3 breaks the mimetypes package on linux. Specifically this line prevents the system's mimetype database from being loaded, breaking detection of the application/json mime type among other things. The workaround for this issue should probably only be used on windows machines.

I'm uploading a file to S3 using such as below. But when I look at the object in S3 after the upload, the Expiry Date = None.
t = datetime.timedelta(weeks=5)
s3_response = conn.upload("input/%s" % filename, file_obj, bucket='mybucket', expires=t)

Here is the file in S3:

Bucket: mybucket
Folder: input
Name: file1.mp4
Link: This is a public link
Size: 57461769
Last Modified: Thu Nov 05 09:30:08 GMT-500 2015
ETag: 896378299cd48c0490d1f4bb6c111111
Expiry Date: None
Expiration Rule: N/A

Thanks for the lib. Works like a charm.

It would be awesome/helpful if the URLs were configurable. It seems S3 may be come a popular API.

I updated it to support DreamObjects: https://github.com/cbess/tinys3

Hello. I can read data from an s3 key using boto and the aws tool but I cannot make it work with tinys3. Any suggestions?

[pete@sand ~]$ python -c "import tinys3; tinys3.Connection("$AWS_ACCESS_KEY_ID", "$AWS_SECRET_ACCESS_KEY", tls=False, endpoint='s3-us-west-1.amazonaws.com').get('abc.txt', 'xcp-test')"
Traceback (most recent call last):
File "", line 1, in
File "build/bdist.linux-x86_64/egg/tinys3/connection.py", line 73, in get
File "build/bdist.linux-x86_64/egg/tinys3/connection.py", line 233, in run
File "build/bdist.linux-x86_64/egg/tinys3/connection.py", line 255, in _handle_request
File "build/bdist.linux-x86_64/egg/tinys3/request_factory.py", line 57, in run
File "/usr/local/lib/python2.7/site-packages/requests-2.9.0-py2.7.egg/requests/models.py", line 837, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://s3-us-west-1.amazonaws.com/xcp-test/abc.txt

A short script with boto.s3.connection.S3Connection was able to read the data.

Using the aws client tool works also; when run with debug it shows that the endpoint is indeed us-west-1:
aws --debug s3 ls xcp-demo
[...]
2015-12-19 17:26:13,046 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-bucket-region': 'us-west-1', 'x-amz-id-2': 'bvoaji7VxAeQDaAA4MgzjXdCya/lI3cu8cf1Kdvk0UpJ34e47NT/1Bg7cFCHMFPSp4tWJACJwc8=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': '6726CFD4436DD0B2', 'date': 'Sun, 20 Dec 2015 01:26:13 GMT', 'content-type': 'application/xml'}
[...]
2015-12-19 16:10:11 16 abc.txt

Hopefully something simple? Thanks!

This library works very well, but it has not been maintained for years. There are many quality pull requests that could benefit the library, but no one has reviewed or approved them.

Three steps
conn = tinys3.Connection("xxxx", "yyyy", default_bucket="zzzz", endpoint="pqr")
f = open('x.txt','w')
conn.upload("x.txt", f)

I can see a timeout for Pool but is it possible to specify a timeout to connection.upload()?

Under the security principle of least privilege, uploads should be uploaded private by default. This would be a backward-incompatible change, but would help future users from inadvertently making private keys public and could also help current users who didn't notice that their files are exposed.

Thanks for your work on a useful library.

I have upload code that works with tinys3 0.1.11 but fails with 0.1.12. The difference appears to be the destination URL tinys3 uses to upload a document. It's possible this is the same problem as issue #53, but I get a different error message.

Revision 308e3b9 changed the type of URL generated by S3Request.bucket_url. In 0.1.11, a document I upload is sent to http://s3.amazonaws.com/{bucket}/{key}. This works fine. In 0.1.12, the URL used is http://{bucket}.s3.amazonaws.com/{key}, and I get a 403 error from S3:

  File "/home/leonardr/.local/lib/python2.7/site-packages/tinys3/pool.py", line 84, in as_completed
    yield r.result()
  File "/home/leonardr/.local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/home/leonardr/.local/lib/python2.7/site-packages/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/leonardr/.local/lib/python2.7/site-packages/tinys3/request_factory.py", line 348, in run
    r.raise_for_status()
  File "/home/leonardr/.local/lib/python2.7/site-packages/requests/models.py", line 893, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 403 Client Error: Forbidden for url: {url}

In revision 308e3b9 the reason given for changing the URL format is "Come back of the virtual-hosting URL style as the old one actually isn't working with a recent bucket". I would guess each URL style works on some buckets and fails on others.

Please let me know if you need any more information about my S3 configuration.

Some of our uploads throw:

ConnectionError(MaxRetryError(HTTPConnectionPool(host='s3.amazonaws.com';, port=80): Max retries exceeded with url

but when checking the contents of the bucket it looks like the file has uploaded fine?

I'm trying to load text to an s3 file using this library and StringIO (python 3)

import io

f = StringIO('my text')
connection = get my s3 connection here
connection.upload(s3filename,f,bucket)

But I'm getting an error...

....
 File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tinys3/connection.py", line 152, in upload
    return self.run(r)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tinys3/connection.py", line 233, in run
    return self._handle_request(request)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tinys3/connection.py", line 255, in _handle_request
    return request.run()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/tinys3/request_factory.py", line 178, in run
    auth=self.auth)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/api.py", line 99, in put
    return request('put', url, data=data, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/adapters.py", line 327, in send
    timeout=timeout
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1090, in request
    self._send_request(method, url, body, headers)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 1086, in endheaders
    self._send_output(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 928, in _send_output
    self.send(message_body)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/http/client.py", line 887, in send
    self.sock.sendall(datablock)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/ssl.py", line 718, in sendall
    v = self.send(data[count:])
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/ssl.py", line 679, in send
    v = self._sslobj.write(data)
TypeError: 'str' does not support the buffer interface

Thoughts?

I'm seeing exactly same error as: boto/boto#2836

I was trying out this library and I kept getting this error:
Response [301]

Then I checked out the code and saw this(request_factory.py, line 33):
return "%s://s3.amazonaws.com/%s/%s" % (protocol, bucket, key.lstrip('/'))

Amazon now gives a 301(Moved Permanently) response when we use the old bucket urls(without the region info). You'll have to use urls like this:
https://s3-us-west-2.amazonaws.com/foo-bucket/bar.png

Do update the code when you get the time! :D

One of my machines was on requests version 0.12.1 and the basic examples wouldn't run until I updated it to the latest 2.2.1. Might want to put a minimum version number in the requirements file.

Looks broken...

 conn = tinys3.Connection(aws_access_key,aws_secret_key,tls=True,endpoint='s3-ap-south-1.amazonaws.com')
 f = open("sample.txt",'rb')
print conn.upload("somefolder",f,"my-bucket")

The endpoint resolved to https://my-bucket.s3-ap-south-1.amazonaws.com/somefolder instead of https://s3-ap-south-1.amazonaws.com/my-bucket/somefolder.

When trying to upload to a bucket that has . in its name, this happens:

requests.exceptions.SSLError: hostname 'some.bucket.name.s3-eu-west-1.amazonaws.com' doesn't match either of 's3-eu-west-1.amazonaws.com', '*.s3-eu-west-1.amazonaws.com', 's3.eu-west-1.amazonaws.com', '*.s3.eu-west-1.amazonaws.com', 's3.dualstack.eu-west-1.amazonaws.com', '*.s3.dualstack.eu-west-1.amazonaws.com', '*.s3.amazonaws.com'

TLS is enabled. The requests version is 2.11.1.

As far as I know, it's correct that requests is complaining here, as a wildcard certificate doesn't cover an arbitrary depth of subdomains. The root cause here are the dots in the bucket name.

It would be good if one could configure tinys3 to not validate the server certificate. See http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

Hi,
I'm currently porting an application using boto to Python 3 and I'd like to use tinys3 instead of boto for it (as boto isn't ready for python 3).

The problem is that it makes heavy use of S3 multipart uploads and tinys3 doesn't seem like it's going to feature it in the near future.

So I'm going to fork the repo and try to implement multipart uploads myself (I need it ASAP). If that works well I may do a pull request. Just so you know. :)

Please include more detail in your documentation. Specifically, you mention that tinys3 can be used to get files from S3, but it's not shown in your documentation anywhere. I had to dig through the code to find it. This a great library, but the documentation is really lacking.

folks, after installing tinys3, i get the following when trying to test via ipthon

python -v 
Python 2.6.9 (unknown, Oct 29 2013, 19:58:13)

In [1]: import tinys3
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-c594c9b5da2f> in <module>()
----> 1 import tinys3

/usr/lib/python2.6/site-packages/tinys3-0.1.9-py2.6.egg/tinys3/__init__.py in <module>()
      2
      3 from .connection import Connection
----> 4 from .pool import Pool
      5
      6 # Backward comparability with versions prior to 0.1.7

/usr/lib/python2.6/site-packages/tinys3-0.1.9-py2.6.egg/tinys3/pool.py in <module>()
      2 from .connection import Base
      3
----> 4 from concurrent.futures import ThreadPoolExecutor, Future, as_completed, wait, TimeoutError
      5
      6

ImportError: No module named concurrent.futures

Maybe something that's planned.

Would it be possible to include a software license in both the repository and setup.py.

When doing and upload it's trying to reach s3.amazon.com/BUCKET_NAME, when it should be BUCKET_NAME.s3.amazon.com.

Tested on python 3.3

In my code I have the two lines below. My code has been working fine with versions <= 0.1.11 The code fails on the upload with version 0.1.12 The error raised is below.

c = tinys3.Connection(...)
c.upload(...)

conn.upload(key, uploadFile, bucketpath)
File "/usr/lib/python2.7/site-packages/tinys3-0.1.12-py2.7.egg/tinys3/connection.py", line 171, in upload
return self.run(r)
File "/usr/lib/python2.7/site-packages/tinys3-0.1.12-py2.7.egg/tinys3/connection.py", line 262, in run
return self._handle_request(request)
File "/usr/lib/python2.7/site-packages/tinys3-0.1.12-py2.7.egg/tinys3/connection.py", line 356, in _handle_request
return request.run()
File "/usr/lib/python2.7/site-packages/tinys3-0.1.12-py2.7.egg/tinys3/request_factory.py", line 346, in run
auth=self.auth)
File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/api.py", line 123, in put
return request('put', url, data=data, **kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/sessions.py", line 475, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/sessions.py", line 596, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/adapters.py", line 473, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

I've been fighting with Tinys3 for a couple of days, trying to upload a file to S3 using python 3.3. The code fragment that I'm trying to use follows:

rsConn = tinys3.Connection(
'ABCDEFGHI',
'a6G9GDT34j',
default_bucket='myBucket,
tls=True,
endpoint='s3-us-west-2.amazonaws.com')

f = open('C:/results/ISYCODE.csv.gz','rb')
rsConn.upload('ISYCODE.csv.gz', f, close=True)

When this code is executed, there is about a 10 second pause and then I get the following message:

The debugged program raised the exception
requests.exceptions.ConnectionError
"HTTPSConnectionPool(host='s3-us-west-2.amazonaws.com',port=443):
Max retries exceeded with url: /myBucket/ISYCODE.csv.gz (Caused by <class
'TimeoutError'>:[WinError 10060] A connection attempt failed because the
connected party did not properly respond after a period of time or
established connection failed because connected host has failed to respons)"
File: C;\xxxxx\yyyyyy\apps\python33\lib\site-packages\requests-2.1.0-
py3.3.egg\requests\adapters.py, Line: 372

I've tried different variations including adding a headers clause to the upload statement. Nothing seems to work. Any assistance would be much appreciated. BTW, I am able to upload files to the S3 bucket using the Amazon web interface.

Can this be used as a Django storage, so I could have FileField fields save files automatically on S3? I'm currently banging my head against the wall because of the lack of Python 3 support in django-storages, boto, and Amazon's shitty S3 package that isn't even on PyPI.

But this package is awesome. This teeny tiny package is great. Can it... Can it also... Be a Django storage and sprinkle fairy dust all over my FileFields?

Relatively self explanatory, would love to be able to take 2 set of keys and copy between S3 buckets in 2 different aws accounts.

Hi guys,

I understand that the get feature is only on GitHub, wasn't released yet to PyPI. Can you please get a release out so I could use it on an official version?

Last version supported seems to be Py3.3; it would be nice if there was support for at least Py3.6

See #45 :(

Please reply to this issue if you are taking over maintainership with the new project URL.

import tinys3

conn = tinys3.Connection(S3_ACCESS_KEY,S3_SECRET_KEY,tls=True)

f = open('some_file.zip','rb')
conn.upload('some_file.zip',f,'my_bucket')

The bucket resides in the eu-west-1 region. The code above completes successfully, but here's the HTTP communication:

PUT /my-backet/some_file.zip HTTP/1.1

Host: s3.amazonaws.com
Date: Wed, 30 Apr 2014 10:36:02 GMT
Accept: */*
Content-Length: 4151
Authorization: [...]
x-amz-acl: public-read
User-Agent: python-requests/2.2.1 CPython/3.4.0 Linux/3.14.1-1-ARCH
Content-Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Accept-Encoding: gzip, deflate, compress

HTTP/1.1 301 Moved Permanently

x-amz-request-id: [...]
x-amz-id-2: [...]
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Wed, 30 Apr 2014 10:35:50 GMT
Connection: close
Server: AmazonS3

1d3
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><RequestId>[...]</RequestId><Bucket>my-bucket</Bucket><HostId>[...]</HostId><Endpoint>my-bucket.s3.amazonaws.com</Endpoint></Error>
0

I noticed raise_for_status() is called, but I suppose 301 is not considered an error, thus no exception is raised.

I know I should provide the endpoint to tinys3 but I believe the code should check explicitly for status >= 300 and raise accordingly in order to avoid silently failing requests.

@shlomiatar, what do you think?

steps to reproduce:

run first example in the documentation with a non-existent bucket name, mismatching region name, or other incorrect specification. conn.upload() throws a stack trace :

   File "upload_to_s3.py", line 12, in <module>
    print(conn.upload(filename,f,bucketname))
  File "/path/connection.py", line 152, in upload
    return self.run(r)
  File "/path//connection.py", line 233, in run
    return self._handle_request(request)
  File "/path/connection.py", line 255, in _handle_request
    return request.run()
  File "/path/request_factory.py", line 180, in run
    r.raise_for_status()
  File "/path/models.py", line 851, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request

Either amazon is not throwing a more specific error message, or it is but it's not being passed along.