Comments (7)
pko7bc
commented on June 23, 2024
1
Good news - it turned out that it works fine with latest!
I tested it only with 3.1 image so it means that you already fixed it after the release.
In that case I'm waiting for the new release 🤞 😄
from localstack.
pko7bc
commented on June 23, 2024
1
Unfortunately I made a mistake while testing this last time with latest tag - the error still exists when using Localstack with Java. I've just checked the 3.2 and the newest latest.
I have to also change my previous statement about ApacheHttpClient - it doesn't work. The only difference is that ApacheHttpClient is slightly "smarter" and retries automatically. But in general it also fails at execution no. 1001 but then recovers after 1-2 retries.
Logs for the failed 1001 execution for ApacheHttpClient (similar logs for 2002, 3003, ....):
Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=http, host=127.0.0.1, port=32795, encodedPath=/bucket/1001-test, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
http-outgoing-0 >> "PUT /bucket/1001-test HTTP/1.1[\r][\n]"
http-outgoing-0 >> "Host: 127.0.0.1:32795[\r][\n]"
http-outgoing-0 >> "amz-sdk-invocation-id: 830942b4-c975-9a0b-3a2a-d9d8c5a2079d[\r][\n]"
http-outgoing-0 >> "amz-sdk-request: attempt=1; max=4[\r][\n]"
http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=test/20240228/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=dac9a437a473bb1c16bfd54830873ef45a8e8da87a06685db6ebdda73d3d4b36[\r][\n]"
http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
http-outgoing-0 >> "Expect: 100-continue[\r][\n]"
http-outgoing-0 >> "User-Agent: aws-sdk-java/2.20.128 Linux/5.15.0-97-generic OpenJDK_64-Bit_Server_VM/17.0.8+7-jvmci-23.0-b15 Java/17.0.8 vendor/GraalVM_Community io/sync http/Apache cfg/retry-mode/legacy[\r][\n]"
http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
http-outgoing-0 >> "X-Amz-Date: 20240228T221928Z[\r][\n]"
http-outgoing-0 >> "x-amz-decoded-content-length: 9[\r][\n]"
http-outgoing-0 >> "Content-Length: 181[\r][\n]"
http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
http-outgoing-0 >> "[\r][\n]"
http-outgoing-0 << "end of stream"
http-outgoing-0 >> "9;chunk-signature=7a0204981a4ec730c006bea18f3d39303d17e92fb88a2e96727ea3f618f60950[\r][\n]"
http-outgoing-0 >> "1001-test[\r][\n]"
http-outgoing-0 >> "0;chunk-signature=9c26f4a53be22945026b8fb16a20d9a315e8d80110d9dd86bdc079d91b5a502d[\r][\n]"
http-outgoing-0 >> "[\r][\n]"
http-outgoing-0 << "end of stream"
Retryable error detected. Will retry in 35ms. Request attempt number 2: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: The target server failed to respond
Retrying Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=http, host=127.0.0.1, port=32795, encodedPath=/bucket/1001-test, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
http-outgoing-1 >> "PUT /bucket/1001-test HTTP/1.1[\r][\n]"
http-outgoing-1 >> "Host: 127.0.0.1:32795[\r][\n]"
http-outgoing-1 >> "amz-sdk-invocation-id: 830942b4-c975-9a0b-3a2a-d9d8c5a2079d[\r][\n]"
http-outgoing-1 >> "amz-sdk-request: attempt=2; max=4[\r][\n]"
http-outgoing-1 >> "Authorization: AWS4-HMAC-SHA256 Credential=test/20240228/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=053b10594abb30de21d945a5a25ae5c1aa04a04ca742d031b16e78cb634be83e[\r][\n]"
http-outgoing-1 >> "Content-Type: application/octet-stream[\r][\n]"
http-outgoing-1 >> "Expect: 100-continue[\r][\n]"
http-outgoing-1 >> "User-Agent: aws-sdk-java/2.20.128 Linux/5.15.0-97-generic OpenJDK_64-Bit_Server_VM/17.0.8+7-jvmci-23.0-b15 Java/17.0.8 vendor/GraalVM_Community io/sync http/Apache cfg/retry-mode/legacy[\r][\n]"
http-outgoing-1 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
http-outgoing-1 >> "X-Amz-Date: 20240228T221928Z[\r][\n]"
http-outgoing-1 >> "x-amz-decoded-content-length: 9[\r][\n]"
http-outgoing-1 >> "Content-Length: 181[\r][\n]"
http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
http-outgoing-1 >> "[\r][\n]"
http-outgoing-1 << "HTTP/1.1 100 Continue[\r][\n]"
http-outgoing-1 << "date: Wed, 28 Feb 2024 22:19:28 GMT[\r][\n]"
http-outgoing-1 << "server: hypercorn-h11[\r][\n]"
http-outgoing-1 << "[\r][\n]"
http-outgoing-1 >> "9;chunk-signature=163e8a372588b38ce0a7b1090bca5ca39617e285bb2398068664adb6a04b0a33[\r][\n]"
http-outgoing-1 >> "1001-test[\r][\n]"
http-outgoing-1 >> "0;chunk-signature=efec4c921d8873da111a82bf1ec309868b986d6e1f912deb3c9732852de28576[\r][\n]"
http-outgoing-1 >> "[\r][\n]"
http-outgoing-1 << "HTTP/1.1 200 [\r][\n]"
http-outgoing-1 << "Content-Type: application/xml[\r][\n]"
http-outgoing-1 << "ETag: "474822292f50999db74bade2bd9d689c"[\r][\n]"
http-outgoing-1 << "x-amz-server-side-encryption: AES256[\r][\n]"
http-outgoing-1 << "x-amz-request-id: 224c99c5-5208-4ef1-b9b1-674a2a7fe432[\r][\n]"
http-outgoing-1 << "x-amz-id-2: s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=[\r][\n]"
http-outgoing-1 << "Content-Length: 0[\r][\n]"
http-outgoing-1 << "date: Wed, 28 Feb 2024 22:19:28 GMT[\r][\n]"
http-outgoing-1 << "server: hypercorn-h11[\r][\n]"
http-outgoing-1 << "[\r][\n]"
Received successful response: 200, Request ID: 224c99c5-5208-4ef1-b9b1-674a2a7fe432, Extended Request ID: s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=
As you can see there are two failed requests with error "end of stream". And again docker logs don't contain anything about this error.
from localstack.
bentsku
commented on June 23, 2024
Hello @pko7bc and thanks for your report!
Could you please share LocalStack logs when this issue happens?
Could you precise if those requests are sent sequentially or in parallel? If in parallel, this could be an issue in our handling of requests. If sequential, how long between the 1st and last request sent?
Thanks!
from localstack.
pko7bc
commented on June 23, 2024
Requests are sent sequentially and it takes 4-4.5 sec to invoke it 1001 times.
Java api docs for the putObject method: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/s3/S3Client.html#putObject(java.util.function.Consumer,software.amazon.awssdk.core.sync.RequestBody)
There are no logs for the failed request:
2024-02-01T13:01:26.207 DEBUG --- [ asgi_gw_0] rolo.gateway.wsgi : PUT 127.0.0.1:32802/bucket/teste1b1ed31-60d8-43f0-8749-2e679f03d290
2024-02-01T13:01:26.208 DEBUG --- [ asgi_gw_0] l.aws.protocol.serializer : No accept header given. Using request's Content-Type (application/octet-stream) as preferred response Content-Type.
2024-02-01T13:01:26.208 INFO --- [ asgi_gw_0] localstack.request.aws : AWS s3.PutObject => 200; 000000000000/eu-central-1; PutObjectRequest({'ACL': None, 'Bucket': 'bucket', 'CacheControl': None, 'ContentDisposition': None, 'ContentEncoding': None, 'ContentLanguage': None, 'ContentLength': 209, 'ContentMD5': None, 'ContentType': 'application/octet-stream', 'ChecksumAlgorithm': None, 'ChecksumCRC32': None, 'ChecksumCRC32C': None, 'ChecksumSHA1': None, 'ChecksumSHA256': None, 'Expires': None, 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWriteACP': None, 'Key': 'teste1b1ed31-60d8-43f0-8749-2e679f03d290', 'Metadata': {}, 'ServerSideEncryption': None, 'StorageClass': None, 'WebsiteRedirectLocation': None, 'SSECustomerAlgorithm': None, 'SSECustomerKey': None, 'SSECustomerKeyMD5': None, 'SSEKMSKeyId': None, 'SSEKMSEncryptionContext': None, 'BucketKeyEnabled': None, 'RequestPayer': None, 'Tagging': None, 'ObjectLockMode': None, 'ObjectLockRetainUntilDate': None, 'ObjectLockLegalHoldStatus': None, 'ExpectedBucketOwner': None, 'Body': <_io.BufferedReader>}, headers={'amz-sdk-invocation-id': '34a10a9a-a918-5ede-7fa6-4ba3571d656d', 'amz-sdk-request': 'attempt=1; max=4', 'Authorization': 'AWS4-HMAC-SHA256 Credential=dummy/20240201/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=68c0a850557baa99e1ab272eca5928c735c043cd05283cce823ed25854f29ac5', 'Content-Type': 'application/octet-stream', 'Expect': '100-continue', 'User-Agent': 'aws-sdk-java/2.20.128 Linux/5.15.0-91-generic OpenJDK_64-Bit_Server_VM/17.0.8+7-jvmci-23.0-b15 Java/17.0.8 vendor/GraalVM_Community io/sync http/UrlConnection cfg/retry-mode/legacy', 'x-amz-content-sha256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'X-Amz-Date': '20240201T130126Z', 'x-amz-decoded-content-length': '36', 'Accept': '*/*', 'Host': '127.0.0.1:32802', 'Connection': 'keep-alive', 'Content-Length': '209', 'x-moto-account-id': '000000000000'}); PutObjectOutput({'ETag': '"d58e342476c0b2f8845d6d9e49cfbc1a"', 'ServerSideEncryption': 'AES256'}, headers={'Content-Type': 'application/xml', 'ETag': '"d58e342476c0b2f8845d6d9e49cfbc1a"', 'x-amz-server-side-encryption': 'AES256', 'x-amz-request-id': 'fdc555f7-65d8-49ad-9ca1-7616cf82ed6b', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234='})
2024-02-01T13:01:26.210 DEBUG --- [ asgi_gw_0] rolo.gateway.wsgi : PUT 127.0.0.1:32802/bucket/test776b00f8-56c2-42e1-aaa5-1f12104b3eae
2024-02-01T13:01:26.211 DEBUG --- [ asgi_gw_0] l.aws.protocol.serializer : No accept header given. Using request's Content-Type (application/octet-stream) as preferred response Content-Type.
2024-02-01T13:01:26.211 INFO --- [ asgi_gw_0] localstack.request.aws : AWS s3.PutObject => 200; 000000000000/eu-central-1; PutObjectRequest({'ACL': None, 'Bucket': 'bucket', 'CacheControl': None, 'ContentDisposition': None, 'ContentEncoding': None, 'ContentLanguage': None, 'ContentLength': 209, 'ContentMD5': None, 'ContentType': 'application/octet-stream', 'ChecksumAlgorithm': None, 'ChecksumCRC32': None, 'ChecksumCRC32C': None, 'ChecksumSHA1': None, 'ChecksumSHA256': None, 'Expires': None, 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWriteACP': None, 'Key': 'test776b00f8-56c2-42e1-aaa5-1f12104b3eae', 'Metadata': {}, 'ServerSideEncryption': None, 'StorageClass': None, 'WebsiteRedirectLocation': None, 'SSECustomerAlgorithm': None, 'SSECustomerKey': None, 'SSECustomerKeyMD5': None, 'SSEKMSKeyId': None, 'SSEKMSEncryptionContext': None, 'BucketKeyEnabled': None, 'RequestPayer': None, 'Tagging': None, 'ObjectLockMode': None, 'ObjectLockRetainUntilDate': None, 'ObjectLockLegalHoldStatus': None, 'ExpectedBucketOwner': None, 'Body': <_io.BufferedReader>}, headers={'amz-sdk-invocation-id': '50b43947-fb7c-6d4c-b8b0-e34524648ca7', 'amz-sdk-request': 'attempt=1; max=4', 'Authorization': 'AWS4-HMAC-SHA256 Credential=dummy/20240201/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=847a73eac9f7aca02218fde1498cb604930eb09d9ddd28d601c9f6367829fe53', 'Content-Type': 'application/octet-stream', 'Expect': '100-continue', 'User-Agent': 'aws-sdk-java/2.20.128 Linux/5.15.0-91-generic OpenJDK_64-Bit_Server_VM/17.0.8+7-jvmci-23.0-b15 Java/17.0.8 vendor/GraalVM_Community io/sync http/UrlConnection cfg/retry-mode/legacy', 'x-amz-content-sha256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'X-Amz-Date': '20240201T130126Z', 'x-amz-decoded-content-length': '36', 'Accept': '*/*', 'Host': '127.0.0.1:32802', 'Connection': 'keep-alive', 'Content-Length': '209', 'x-moto-account-id': '000000000000'}); PutObjectOutput({'ETag': '"1a8ce4a2856c447a6b1fca0d3a47244f"', 'ServerSideEncryption': 'AES256'}, headers={'Content-Type': 'application/xml', 'ETag': '"1a8ce4a2856c447a6b1fca0d3a47244f"', 'x-amz-server-side-encryption': 'AES256', 'x-amz-request-id': 'e7106881-a48c-40c5-bd1f-89ab32ee5d1c', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234='})
2024-02-01T13:01:27.003 DEBUG --- [ asgi_gw_0] rolo.gateway.wsgi : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-01T13:01:41.305 DEBUG --- [ asgi_gw_0] rolo.gateway.wsgi : GET localhost.localstack.cloud:4566/_localstack/health
------- paused on the error
------- here should be logs for the failed request but only for the prev and the next request are present
2024-02-01T13:01:55.641 DEBUG --- [ asgi_gw_0] rolo.gateway.wsgi : GET localhost.localstack.cloud:4566/_localstack/health
2024-02-01T13:02:08.068 DEBUG --- [ asgi_gw_0] rolo.gateway.wsgi : PUT 127.0.0.1:32802/bucket/teste6773c08-380f-45d1-b6bc-7ed40da35513
2024-02-01T13:02:08.069 DEBUG --- [ asgi_gw_0] l.aws.protocol.serializer : No accept header given. Using request's Content-Type (application/octet-stream) as preferred response Content-Type.
2024-02-01T13:02:08.069 INFO --- [ asgi_gw_0] localstack.request.aws : AWS s3.PutObject => 200; 000000000000/eu-central-1; PutObjectRequest({'ACL': None, 'Bucket': 'bucket', 'CacheControl': None, 'ContentDisposition': None, 'ContentEncoding': None, 'ContentLanguage': None, 'ContentLength': 209, 'ContentMD5': None, 'ContentType': 'application/octet-stream', 'ChecksumAlgorithm': None, 'ChecksumCRC32': None, 'ChecksumCRC32C': None, 'ChecksumSHA1': None, 'ChecksumSHA256': None, 'Expires': None, 'GrantFullControl': None, 'GrantRead': None, 'GrantReadACP': None, 'GrantWriteACP': None, 'Key': 'teste6773c08-380f-45d1-b6bc-7ed40da35513', 'Metadata': {}, 'ServerSideEncryption': None, 'StorageClass': None, 'WebsiteRedirectLocation': None, 'SSECustomerAlgorithm': None, 'SSECustomerKey': None, 'SSECustomerKeyMD5': None, 'SSEKMSKeyId': None, 'SSEKMSEncryptionContext': None, 'BucketKeyEnabled': None, 'RequestPayer': None, 'Tagging': None, 'ObjectLockMode': None, 'ObjectLockRetainUntilDate': None, 'ObjectLockLegalHoldStatus': None, 'ExpectedBucketOwner': None, 'Body': <_io.BufferedReader>}, headers={'amz-sdk-invocation-id': '3ee710b9-4af7-ffb0-240b-28379c394170', 'amz-sdk-request': 'attempt=1; max=4', 'Authorization': 'AWS4-HMAC-SHA256 Credential=dummy/20240201/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=adc75666bd2de1285adbdaa8c165633e256bb1b5dc5d29b90d1e3093d9fd3404', 'Content-Type': 'application/octet-stream', 'Expect': '100-continue', 'User-Agent': 'aws-sdk-java/2.20.128 Linux/5.15.0-91-generic OpenJDK_64-Bit_Server_VM/17.0.8+7-jvmci-23.0-b15 Java/17.0.8 vendor/GraalVM_Community io/sync http/UrlConnection cfg/retry-mode/legacy', 'x-amz-content-sha256': 'STREAMING-AWS4-HMAC-SHA256-PAYLOAD', 'X-Amz-Date': '20240201T130208Z', 'x-amz-decoded-content-length': '36', 'Accept': '*/*', 'Host': '127.0.0.1:32802', 'Connection': 'keep-alive', 'Content-Length': '209', 'x-moto-account-id': '000000000000'}); PutObjectOutput({'ETag': '"0fb4c8e3276428a8a822687db5a1b6f3"', 'ServerSideEncryption': 'AES256'}, headers={'Content-Type': 'application/xml', 'ETag': '"0fb4c8e3276428a8a822687db5a1b6f3"', 'x-amz-server-side-encryption': 'AES256', 'x-amz-request-id': '4d372014-a385-4ea1-aadd-9271836f8b36', 'x-amz-id-2': 's9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234='})
I think it's worth mentioning that it worked fine before upgrading to 3.1 from version 2.2
from localstack.
bentsku
commented on June 23, 2024
Thanks for the data!
To be sure the issue does not originate from our S3 implementation itself (the 1000 arbitrary limit does not look like something we would have set in S3, but a more global one), could you try running the same test with PROVIDER_OVERRIDE_S3=legacy_v2 and see if the issue is still there?
It is also very weird that they are no logs at all for the failed request.
(I'm sorry I'm asking for you to run it directly, we do have an integration test putting 1010 objects in our test suite and this has been running without flakes for a while, even locally.)
2024-02-01T22:26:09.044 DEBUG --- [ asgi_gw_0] rolo.gateway.wsgi : PUT s3.localhost.localstack.cloud:4566/test-bucket-1a544da8/test-key-0
2024-02-01T22:26:09.047 INFO --- [ asgi_gw_0] localstack.request.aws : AWS s3.PutObject => 200
...
2024-02-01T22:26:11.500 DEBUG --- [ asgi_gw_0] rolo.gateway.wsgi : PUT s3.localhost.localstack.cloud:4566/test-bucket-1a544da8/test-key-1009
2024-02-01T22:26:11.501 INFO --- [ asgi_gw_0] localstack.request.aws : AWS s3.PutObject => 200
from localstack.
pko7bc
commented on June 23, 2024
It works with 2.2 but doesn't with 3.1 and PROVIDER_OVERRIDE_S3=legacy_v2 doesn't help.
I did some additional tests and it works with python.
I found out that it also works in Java but with different HTTP client.
When I use software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient it fails but after changing to software.amazon.awssdk.http.apache.ApacheHttpClient it works fine.
Client config:
var s3Client = S3Client.builder()
//.httpClientBuilder(UrlConnectionHttpClient.builder())
.httpClientBuilder(ApacheHttpClient.builder())
.endpointOverride(URI.create(endpoint))
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create("dummy", "dummy")
)
)
.build();I managed to get some logs for both clients - maybe it will help somehow.
UrlConnectionHttpClient:
2024-02-02T10:04:20.918 DEBUG [so.am.aw.request] Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=http, host=127.0.0.1, port=33202, encodedPath=/bucket/test3b204e22-f6f0-4f91-8aab-e70ed29063f6, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
2024-02-02T10:04:20.919 TRACE [su.ne.ww.pr.ht.HttpURLConnection] ProxySelector Request for http://127.0.0.1:33202/bucket/test3b204e22-f6f0-4f91-8aab-e70ed29063f6
2024-02-02T10:04:20.919 TRACE [su.ne.ww.pr.ht.HttpURLConnection] cached HttpClient was idle for 2
2024-02-02T10:04:20.919 TRACE [su.ne.ww.pr.ht.HttpURLConnection] KeepAlive stream retrieved from the cache, sun.net.www.http.HttpClient(http://127.0.0.1:33202/bucket/teste3ee3af1-cb05-47de-b2bd-e31e09c5a7ff)
2024-02-02T10:04:20.919 TRACE [su.ne.ww.pr.ht.HttpURLConnection] Proxy used: DIRECT
2024-02-02T10:04:20.919 DEBUG [su.ne.ww.pr.ht.HttpURLConnection] sun.net.www.MessageHeader@6fafbdac14 pairs: {PUT /bucket/test3b204e22-f6f0-4f91-8aab-e70ed29063f6 HTTP/1.1: null}{amz-sdk-invocation-id: ca3bbe6a-e579-6efd-b855-ccf99bb378f2}{amz-sdk-request: attempt=1; max=4}{Authorization: AWS4-HMAC-SHA256 Credential=dummy/20240202/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=bd7d15df131cf6dbfa56fdbd35f3a465cdb14256d04eab8834ee6db8f97f4a67}{Content-Type: application/octet-stream}{Expect: 100-continue}{User-Agent: aws-sdk-java/2.20.128 Linux/5.15.0-91-generic OpenJDK_64-Bit_Server_VM/17.0.8+7-jvmci-23.0-b15 Java/17.0.8 vendor/GraalVM_Community io/sync http/UrlConnection cfg/retry-mode/legacy}{x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD}{X-Amz-Date: 20240202T090420Z}{x-amz-decoded-content-length: 36}{Accept: */*}{Host: 127.0.0.1:33202}{Connection: keep-alive}{Content-Length: 209}
// for the failed request the following lines are missing
2024-02-02T10:04:20.921 DEBUG [su.ne.ww.pr.ht.HttpURLConnection] sun.net.www.MessageHeader@c7c07ff9 pairs: {null: HTTP/1.1 200}{Content-Type: application/xml}{ETag: "2408d7a077af4628de1c05c9d6b3a2a3"}{x-amz-server-side-encryption: AES256}{x-amz-request-id: 58b36a28-d34a-42ea-9d61-6c44d59d390e}{x-amz-id-2: s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=}{Content-Length: 0}{date: Fri, 02 Feb 2024 09:04:20 GMT}{server: hypercorn-h11}
2024-02-02T10:04:20.922 DEBUG [so.am.aw.request] Received successful response: 200, Request ID: 58b36a28-d34a-42ea-9d61-6c44d59d390e, Extended Request ID: s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=
ApacheHttpClient:
2024-02-02T10:10:53.522 DEBUG [so.am.aw.request] Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=http, host=127.0.0.1, port=33204, encodedPath=/bucket/test20fc9439-fc42-4aec-bf62-655b27cf5b5c, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
2024-02-02T10:10:53.538 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "PUT /bucket/test20fc9439-fc42-4aec-bf62-655b27cf5b5c HTTP/1.1[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "Host: 127.0.0.1:33204[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "amz-sdk-invocation-id: 86004e60-0215-e057-7c58-4a5cf86b8ccb[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "amz-sdk-request: attempt=1; max=4[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "Authorization: AWS4-HMAC-SHA256 Credential=dummy/20240202/eu-central-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=095cafb5c7c608d21b8436217d34f0b0263daed85b25ad0b02fd73aa132e3995[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "Content-Type: application/octet-stream[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "Expect: 100-continue[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "User-Agent: aws-sdk-java/2.20.128 Linux/5.15.0-91-generic OpenJDK_64-Bit_Server_VM/17.0.8+7-jvmci-23.0-b15 Java/17.0.8 vendor/GraalVM_Community io/sync http/Apache cfg/retry-mode/legacy[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "X-Amz-Date: 20240202T091053Z[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "x-amz-decoded-content-length: 36[\r][\n]"
2024-02-02T10:10:53.539 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "Content-Length: 209[\r][\n]"
2024-02-02T10:10:53.540 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
2024-02-02T10:10:53.540 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "[\r][\n]"
2024-02-02T10:10:53.541 DEBUG [or.ap.ht.wire] http-outgoing-0 << "HTTP/1.1 100 Continue[\r][\n]"
2024-02-02T10:10:53.541 DEBUG [or.ap.ht.wire] http-outgoing-0 << "date: Fri, 02 Feb 2024 09:10:53 GMT[\r][\n]"
2024-02-02T10:10:53.541 DEBUG [or.ap.ht.wire] http-outgoing-0 << "server: hypercorn-h11[\r][\n]"
2024-02-02T10:10:53.541 DEBUG [or.ap.ht.wire] http-outgoing-0 << "[\r][\n]"
2024-02-02T10:10:53.543 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "24;chunk-signature=2274bc9dccf744c32ce99710050047dc41305a308688d1708487dfea366adc88[\r][\n]"
2024-02-02T10:10:53.543 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "90a8dffa-79f3-4000-81c0-b8a3eea243f2[\r][\n]"
2024-02-02T10:10:53.543 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "0;chunk-signature=449539fe0c96c66c22b9611889ac62f31a27c0a0422f189988038bd9f9723414[\r][\n]"
2024-02-02T10:10:53.543 DEBUG [or.ap.ht.wire] http-outgoing-0 >> "[\r][\n]"
2024-02-02T10:10:53.552 DEBUG [or.ap.ht.wire] http-outgoing-0 << "HTTP/1.1 200 [\r][\n]"
2024-02-02T10:10:53.552 DEBUG [or.ap.ht.wire] http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
2024-02-02T10:10:53.552 DEBUG [or.ap.ht.wire] http-outgoing-0 << "ETag: "dee801df668b7d852958708518f5a3e4"[\r][\n]"
2024-02-02T10:10:53.552 DEBUG [or.ap.ht.wire] http-outgoing-0 << "x-amz-server-side-encryption: AES256[\r][\n]"
2024-02-02T10:10:53.552 DEBUG [or.ap.ht.wire] http-outgoing-0 << "x-amz-request-id: f51347ba-5ae7-4457-a0cf-83af1ddd552b[\r][\n]"
2024-02-02T10:10:53.552 DEBUG [or.ap.ht.wire] http-outgoing-0 << "x-amz-id-2: s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=[\r][\n]"
2024-02-02T10:10:53.552 DEBUG [or.ap.ht.wire] http-outgoing-0 << "Content-Length: 0[\r][\n]"
2024-02-02T10:10:53.552 DEBUG [or.ap.ht.wire] http-outgoing-0 << "date: Fri, 02 Feb 2024 09:10:53 GMT[\r][\n]"
2024-02-02T10:10:53.552 DEBUG [or.ap.ht.wire] http-outgoing-0 << "server: hypercorn-h11[\r][\n]"
2024-02-02T10:10:53.552 DEBUG [or.ap.ht.wire] http-outgoing-0 << "[\r][\n]"
2024-02-02T10:10:53.557 DEBUG [so.am.aw.request] Received successful response: 200, Request ID: f51347ba-5ae7-4457-a0cf-83af1ddd552b, Extended Request ID: s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234=
from localstack.
bentsku
commented on June 23, 2024
Hi @pko7bc and sorry to hear the issue is still present. Looks like it might be some handling of 100 Continue which could be failing?
I can see:
http-outgoing-1 << "HTTP/1.1 100 Continue[\r][\n]"
http-outgoing-1 << "date: Wed, 28 Feb 2024 22:19:28 GMT[\r][\n]"
http-outgoing-1 << "server: hypercorn-h11[\r][\n]"
http-outgoing-1 << "[\r][\n]"
Being, like you said, "end of stream" for failing requests. This must be on our low level gateway level and not particularly in S3. Thanks a lot for the report, we'll keep looking.
from localstack.
Related Issues (20)
- enhancement request: Add support for RDS BatchExecuteStatement HOT 1
- enhancement request: Emulate more of AWS S3 errors (specifically "EntityTooLarge" and "Policy Condition failed") HOT 3
- bug: ACM-PCA doesn’t return the certificate chain
- bug: Localstack DNS resolution issues caused by resolv.conf HOT 2
- bug: ACM-PCA doesn’t implement `create_certificate_authority_audit_report`
- bug: AdminInitiateAuth with incorrect password produces incorrect response that sdks cannot deserialize
- bug: DynamoDB Streams crashing during PutRecord HOT 7
- bug: SNS publishing from step functions seems to not construct the Message attribute correctly HOT 3
- enhancement request: Addnew common string functions to vtl renderer HOT 1
- bug: Authorization Header missing from Cloudfront Requests
- bug: Error when running localstack in Kubernetes HOT 1
- DynamoDB DeleteTable does not delete global tables HOT 1
- bug: localstack lambda container can't connect to another container HOT 2
- enhancement request: Support XML templating in API Gateway HOT 1
- bug: Step Functions: get-execution-history, list-executions and describe-state-machine command returns InvalidName error in localstack v3.2.0
- StartMessageMoveTask fails when optional destination ARN is not supplied HOT 5
- bug: Unable to get-object from S3 Glacier Instant Retrieval HOT 2
- bug: Multi-Region KMS Key does not return expected data causing AWS provider in terraform to crash HOT 1
- enhancement request: publish cloudwatch metrics via Embedded Metric Format (EMF) HOT 1
- enhancement request: Native support for Kubernetes livenessProbe, readinessProbe HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from localstack.