Can't put contents to s3 bucket outside of US standard region with file object about botocore HOT 3 CLOSED

I'm assuming that the bucket you were trying to write to was a newly created bucket?

In that case, you can receive a temporary redirect (307) when you try to access the bucket because the DNS records that have to be created to resolve the bucket using virtual-hosted addressing have not yet propagated throughout the network.

This 307 redirect should, however, contain a Location header. Are you sure there was no location header present?

from botocore.

Did more digging into this. You do get a location header with the region specific endpoint (bucket.s3region.amazonaws.com), but the problem is that the fileobj has been exhausted so .read() returns nothing. When you send the request to the new location it waits for content-length bytes, but eventually s3 closes the socket on their end.

Does requests provide anything to rewind a stream? Is this something we could do before requests retries the request?

from botocore.

I spent more time researching this and have a few observations:

• The ideal way to handle this would be through an Expect: 100-continue header. From the research I've done it seems like httplib does not properly support this. It will handle a 100 continue response but it already sends the body. I also double checked that requests/urllib3 does not support 100 continue.
• The requests library has a hook when a request is received, this includes redirect responses. We could add a hook that .seek(0)'s the body on redirects, otherwise it raises an error if it's not seekable. The only downside here is that we're potentially sending part of the body twice.

The reason I'd like to get this implemented is for better memory utilization in this branch of the CLI: https://github.com/jamesls/aws-cli/compare/memory-utilization

This branch is ready to go, except that by switching from bytearrays to file objects, we run into this issue here.

from botocore.