The difference between the request time and the current time is too large when uploading to s3 bucket about s3-resource HOT 12 CLOSED

@vito yes, it's synced, we ran the following commands in worker:

service ntp stop
ntpdate -s 0.amazon.pool.ntp.org
service ntp start

We found the problem only occurred when trying to upload a "big" file onto S3, for examples:

391.49 MB / 391.49 MB [============================] 100.00 % 404.16 KB/s 16m31serror running command: RequestTimeTooSkewed: The difference between the request time and the current time is too large.

There took 16m31s, I suspect if there is a bug in s3 resource or in S3 API, seems it compared the time of starting the upload to the end time, it took more than 15 mins. Is it possible? Thanks.

from s3-resource.

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #119803371 S3 uploads are prone to RequestTimeTooSkewed

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

from s3-resource.

Is your worker's clock in sync? (ntp?)

from s3-resource.

@vito Do you think if it might be a bug in S3 resource type when transferring a "bigger" package, spending more than 15 mins? Thanks.

from s3-resource.

Could be. That's a fairly common failure mode for authenticated uploads.

from s3-resource.

This looks like a tricky problem to avoid, but one thing that may help is parallel, chunked uploads. We have code in place to do this, but I think the progress rendering got in the way:

https://github.com/concourse/s3-resource/blob/master/s3client.go#L155

We wrap Body with a progress-emitting io.Reader. The s3manager performs parallel uploads by trying to cast Body to io.Seeker and io.ReaderAt, which this progress-wrapped reader doesn't support. This the upload happens in series, increasing the likelihood for a single long-running request that may hit this error.

I'll prioritize a fix for this; we should probably wrap the body with an object that implements the methods that allow it to still be uploaded in parallel.

from s3-resource.

Thanks, @vito . we are willing to try with the new fix, so please let us know when it's available. Thanks. @jianqiu @maximilien

from s3-resource.

Hello again!

All stories related to this issue have been accepted, so I'm going to automatically close this issue.

At the time of writing, the following stories have been accepted:

• #119803371 S3 uploads are prone to RequestTimeTooSkewed

If you feel there is still more to be done, or if you have any questions, leave a comment and we'll reopen if necessary!

from s3-resource.

Guys please help, here at htis url you may see https://www.pivotaltracker.com/n/projects/1059262/stories/119803371/comments/137771301

"zachgersh made a PR to update the aws sdk, and now uploading large files works even if they take half an hour to upload:"

How do I can get this file/script/solution?

from s3-resource.

Can you add link to PR? Not listed on the Tracker story. Thx

from s3-resource.

Running sudo ntpdate ntp.ubuntu.com on the server fixed this for me.

from s3-resource.

@vito yes, it's synced, we ran the following commands in worker:

service ntp stop
ntpdate -s 0.amazon.pool.ntp.org
service ntp start

We found the problem only occurred when trying to upload a "big" file onto S3, for examples:

391.49 MB / 391.49 MB [============================] 100.00 % 404.16 KB/s 16m31serror running command: RequestTimeTooSkewed: The difference between the request time and the current time is too large.

There took 16m31s, I suspect if there is a bug in s3 resource or in S3 API, seems it compared the time of starting the upload to the end time, it took more than 15 mins. Is it possible? Thanks.

Awsome

@vito yes, it's synced, we ran the following commands in worker:

service ntp stop
ntpdate -s 0.amazon.pool.ntp.org
service ntp start

We found the problem only occurred when trying to upload a "big" file onto S3, for examples:

391.49 MB / 391.49 MB [============================] 100.00 % 404.16 KB/s 16m31serror running command: RequestTimeTooSkewed: The difference between the request time and the current time is too large.

There took 16m31s, I suspect if there is a bug in s3 resource or in S3 API, seems it compared the time of starting the upload to the end time, it took more than 15 mins. Is it possible? Thanks.

Awesome man, this works like boom :)

from s3-resource.