okigan / awscurl Goto Github PK
View Code? Open in Web Editor NEWcurl-like access to AWS resources with AWS Signature Version 4 request signing.
License: MIT License
curl-like access to AWS resources with AWS Signature Version 4 request signing.
License: MIT License
In AWS EKS environment with IRSA. In version 0.22.
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
awscurl -X PUT https://example.com
Traceback (most recent call last):
File "/usr/local/bin/awscurl", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
inner_main(sys.argv[1:])
File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 393, in load_aws_config
cred = session.get_credentials()
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 441, in get_credentials
self._credentials = self._components.get_component(
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 937, in get_component
self._components[name] = factory()
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 151, in _create_credential_resolver
return botocore.credentials.create_credential_resolver(
File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 64, in create_credential_resolver
metadata_timeout = session.get_config_variable('metadata_service_timeout')
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 251, in get_config_variable
return self.get_component('config_store').get_config_variable(
File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 313, in get_config_variable
return provider.provide()
File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 410, in provide
value = provider.provide()
File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 471, in provide
scoped_config = self._session.get_scoped_config()
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 351, in get_scoped_config
raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found
Works fine in 0.21
is it possible to put it in python script: run command and get output to JSON file (without subprocess)
import requests
headers = {
'Content-Type': 'application/json',
}
params = (
('Action', 'ListUsers'),
('Version', '2010-05-08'),
)
response = requests.get('https://iam.amazonaws.com/', headers=headers, params=params)
with open('1.json', 'w') as outfile:
outfile.write(response.content)
and getting
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
<Error>
<Type>Sender</Type>
<Code>MissingAuthenticationToken</Code>
<Message>Request is missing Authentication Token</Message>
</Error>
<RequestId>bb3695a3-399f-11e8-b6b6-17d7b8ab932d</RequestId>
</ErrorResponse>
Hey ๐
That's a great tool! I am using this tool to perform requests against APIs which are protected by IAM-Authentication. Those apis contain query parameters which parameters may require url-encoding.
It looks like there is a problem with uris which are not url-encoded.
awscurl --service s3 https://awscurl-sample-bucket.s3.amazonaws.com/?query=para:meter
Results in an error:
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>.../AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20210608T045117Z
Even though it looks like the underlying implementation correctly escapes the url (like for example curl does) when performing the request, but the signature is calculated for the unescaped uri and this results in a signature mismatch.
Running with escaped uri parameters works perfectly fine.
awscurl --service s3 https://awscurl-sample-bucket.s3.amazonaws.com/?query%3Dpara%3Ameter --region us-east-1
First off, very neat project!
In regards to func load_aws_config
. Is there a specific reason you use a ConfigParser?
If you used boto3 you'd end up supporting most credential providers implicitly.
boto3.Session().get_credentials().get_frozen_credentials()
You could construct the session
object with a profile and/or keys if provided, or otherwise allow boto to default to its own chain of finding credentials. This would in turn allow it to work from an EC2 instance with an IAM instance profile while maintaining the existing functionality.
Hi,
please update awscurl in order to allow non URL encoded charecters.
E.g.
GET <host>/_task/taskId:123:abc
the character ":" is not RFC conform and must be URL encoded.
I would recommend to change line 182 as:
requests.utils.quote(canonical_uri)
If URI is not encoded properly AWS returns an error indicating that signature check has failed.
If you change request to:
GET <host>/_task/taskId%3A123%3Aabc
AWS tries to validate against
GET <host>/_task/taskId%253A123%253Aabc
So AWS encodes the char % separately to %25.
BR
MB
Created a private API with API Gateway and enabled AWS_IAM auth on a GET method for a resource. When invoking using the default Stage URL (using private DNS name), the request works fine as below -
awscurl -v https://API-ID.execute-api.REGION.amazonaws.com/STAGE/RESOURCE
However, making a request using the public DNS names leads to the signature calculation error. We are passing the Host header here for the VPC endpoint to be able to identify which API to hit.
awscurl -v https://vpce-ID.execute-api.REGION.vpce.amazonaws.com/STAGE/RESOURCE -H "Host: API-ID.execute-api.REGION.amazonaws.com"
If we replace Host header with x-api-gw-id as suggested here https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-api-test-invoke-url.html#w20aac13c16c28c11
The request works fine -
awscurl -v https://vpce-ID.execute-api.REGION.vpce.amazonaws.com/STAGE/RESOURCE -H "x-apigw-api-id: API-ID"
I suspect the issue is with how the host header value is used in the signature calculation. Can you please confirm if format is right?
After installing awscurl git-remote-codecommit stopped working and I cannot get it to work again. When I try to do git pull
for example I get the following error message: fatal: unable to find remote helper for 'codecommit'
.
Running setup.py install for awscurl
SyntaxError: ('invalid syntax', ('/usr/lib/python2.6/site-packages/awscurl/awscurl.py', 322, 23, ' headers = {k: v for (k, v) in map(lambda s: s.split(": "), args.header)}\n'))
Called with URL that passes "Mike's 1043" encoded as Mike%27s%201043
Example URL
https://xxxxxxxxx.execute-api.us-west-2.amazonaws.com/prod/add?user=XX&data=Mike%27s%201043
'Response code: 400\n'
{"message": "Could not parse request body into json: Unrecognized character escape ''' (code 39)\n at [Source: [B@742f0d86; line: 9, column: 34]"}
Traceback (most recent call last):
File "/usr/local/bin/awscurl", line 11, in
sys.exit(main())
File "/Users/Znn/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 276, in main
args.security_token
File "/Users/Znn/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 188, in make_request
return send_request(uri, data, headers, method)
File "/Users/Znn/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 218, in send_request
r.raise_for_status()
File "/Library/Python/2.7/site-packages/requests/models.py", line 862, in raise_for_status
raise HTTPError(http_error_msg, response=self)
awscurl currently restricts urllib to <1.23>=1.21.1
. Can this restriction be relaxed or awscurl updated appropriately to support the new version of urllib?
We have an API Gateway Resource path that redirects to a pre-signed s3 URL. When I make that request with awscurl
, it errors out with:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
and the pre-signed s3 URL in the python Traceback
...
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://our-bucket-name.s3.amazonaws.com/...
This makes sense because the signed request is not for the final URL. I'm guessing this will happen with any service that responds with a redirect.
I think it would make sense to add support for not following redirects as a command line flag.
It might even make sense to follow the pattern from curl
and only follow redirects if the caller explicitly request it, i.e. curl -L https://my.domain/redirecting
.
Happy to take crack at the implementation I already have a rough working implementation, but want your input in the right approach before I make a PR.
Our CI failed today due to an update of awscurl.
Looking at https://pypi.org/project/awscurl/0.14/#history I see you've released 0.13 and 0.14 (not committed to repo yet, I think?), but when I try to install awscurl now (either locally or on CI server), I get this:
โก pip install awscurl
Collecting awscurl
Downloading https://files.pythonhosted.org/packages/0f/c3/82ffdac5671b1f89f369c0c33aebd4122df3f4ab3c11c802bab41c2edb37/awscurl-0.14.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/zl/dqtyzbh92jl4v53l2gpxfzyh0000gp/T/pip-build-BjlhKY/awscurl/setup.py", line 5, in <module>
with open("README.md", "r") as fh:
IOError: [Errno 2] No such file or directory: 'README.md'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/zl/dqtyzbh92jl4v53l2gpxfzyh0000gp/T/pip-build-BjlhKY/awscurl/
I got error when trying to execute:
awscurl --service execute-api -X POST -d "hello" https://spjhtm1qcd.execute-api.us-east-1.amazonaws.com/test/@connections/atntqfjloAMCECg=
Also tried:
awscurl --service execute-api -X POST -d "hello" https://spjhtm1qcd.execute-api.us-east-1.amazonaws.com/test/\@connections/atntqfjloAMCECg\=
and
awscurl --service execute-api -X POST -d "hello" "https://spjhtm1qcd.execute-api.us-east-1.amazonaws.com/test/@connections/atntqfjloAMCECg="
The error:
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/test/%40connections/atntqfjloAMCECg%3D\n\nhost:spjhtm1qcd.execute-api.us-east-1.amazonaws.com\nx-amz-date:20190603T171519Z\n\nhost;x-amz-date\n2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20190603T171519Z\n20190603/us-east-1/execute-api/aws4_request\nb013b0bc1c19a4ebc9da8d82a6122379899465e0cd4808187a4ca276e999c56a'\n"}
Traceback (most recent call last):
File "/Users/anton.titkov/Library/Python/2.7/bin/awscurl", line 11, in <module>
sys.exit(main())
File "/Users/anton.titkov/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 373, in main
r.raise_for_status()
File "/Users/anton.titkov/Library/Python/2.7/lib/python/site-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://spjhtm1qcd.execute-api.us-east-1.amazonaws.com/test/@connections/atntqfjloAMCECg=
The load_aws_config function is responsible for getting credentials that are used to sign the request.
It's quite complex: a mixture of custom credential parsing and calls to botocore.
The PR #116 (add support for AWS SSO profiles) caused a rergression in #122 (instance credentials are not being loaded).
The tests for the load_aws_config function don't cover all the use cases, which makes it hard to add new functionality without breaking what already works.
Looks like the --profile command line argument does not work, awscurl uses default profile from ~/.aws/credentials despite the agument.
Running with AWS_PROFILE environment variable works fine.
Hi,
I installed the latest version of awscurl, 0.20. I attempt to invoke my Lambda function and it returned some error messages that I can't make out what went wrong.
The same request worked fine in Postman.
$ awscurl --service lambda --region eu-west-1 --profile default 'https://lambda.<region>.amazonaws.com/2015-03-31/functions/<function-name>/invocations?Qualifier=$LATEST' -H 'X-Amz-Invocation-Type:RequestResponse' -H 'X-Amz-Log-Type:Tail' -H 'Content-Type:application/json' -d '{"message":"Whats up."}'
The error message is:
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/bin/awscurl", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/awscurl/awscurl.py", line 500, in main
inner_main(sys.argv[1:])
File "/usr/local/lib/python3.7/site-packages/awscurl/awscurl.py", line 463, in inner_main
headers = {k: v for (k, v) in map(lambda s: s.split(": "), args.header)}
File "/usr/local/lib/python3.7/site-packages/awscurl/awscurl.py", line 463, in <dictcomp>
headers = {k: v for (k, v) in map(lambda s: s.split(": "), args.header)}
ValueError: not enough values to unpack (expected 2, got 1)
The doc of the API can be found here: https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html
I'm using a custom hostname for the URL and thus the server certificate can't be verified. I'm passing the --insecure
argument but it still prints the following warning:
/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'host.docker.internal'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
I think this should be suppressed.
It looks like the latest version doesn't work anymore when it's installed using pip.
pip install awscurl
The output of the install command is:
Collecting awscurl 15:40:56 Requirement already satisfied: configargparse in /usr/local/lib/python2.7/dist-packages (from awscurl) (1.5.3) 15:40:56 Requirement already satisfied: configparser in /usr/lib/python2.7/dist-packages (from awscurl) (3.5.0b2) 15:40:56 Requirement already satisfied: requests in /usr/local/lib/python2.7/dist-packages (from awscurl) (2.26.0) 15:40:56 Requirement already satisfied: urllib3[secure] in /usr/local/lib/python2.7/dist-packages (from awscurl) (1.26.7) 15:40:56 Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests->awscurl) (2021.10.8) 15:40:56 Requirement already satisfied: idna<3,>=2.5; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from requests->awscurl) (2.10) 15:40:56 Requirement already satisfied: chardet<5,>=3.0.2; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from requests->awscurl) (4.0.0) 15:40:56 Requirement already satisfied: pyOpenSSL>=0.14; extra == "secure" in /usr/local/lib/python2.7/dist-packages (from urllib3[secure]->awscurl) (21.0.0) 15:40:56 Requirement already satisfied: cryptography>=1.3.4; extra == "secure" in /usr/lib/python2.7/dist-packages (from urllib3[secure]->awscurl) (2.6.1) 15:40:56 Requirement already satisfied: ipaddress; python_version == "2.7" and extra == "secure" in /usr/lib/python2.7/dist-packages (from urllib3[secure]->awscurl) (1.0.17) 15:40:56 Requirement already satisfied: six>=1.5.2 in /usr/lib/python2.7/dist-packages (from pyOpenSSL>=0.14; extra == "secure"->urllib3[secure]->awscurl) (1.12.0) 15:40:56 Installing collected packages: awscurl 15:40:56 Successfully installed awscurl-0.26
All the commands fail with:
Traceback (most recent call last): 15:40:21 File "/usr/local/bin/awscurl", line 6, in <module> 15:40:21 from awscurl.__main__ import main 15:40:21 File "/usr/local/lib/python2.7/dist-packages/awscurl/__main__.py", line 6, in <module> 15:40:21 from .awscurl import main 15:40:21 File "/usr/local/lib/python2.7/dist-packages/awscurl/awscurl.py", line 19, in <module> 15:40:21 from urllib.parse import quote 15:40:21 ImportError: No module named parse
Note that I have installed the package on Mac and I didn't run in the previous errors so everything works just fine if the package is installed with Homebrew.
Thanks
Hi,
it seems that awscurl is not taking in account AWS_SESSION_TOKEN environment variable (when an user is assuming a role, in order to have some rights).
Is it possible to calculate the aws signature version 4 with this AWS_SESSION_TOKEN instead of only with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY?
Without taking in account this environment variable (created dynamically when an user is assuming a role), we obtain a 403 http error code.
Thanks
The default encoding in Python 2.x is ascii
so we are seeing the following backtrace when supplying UTF-8 arguments to the --data
flag:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/awscurl/awscurl.py", line 339, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/awscurl/awscurl.py", line 334, in main
args.security_token or args.session_token
File "/usr/local/lib/python2.7/site-packages/awscurl/awscurl.py", line 178, in make_request
payload_hash = sha256_hash(data)
File "/usr/local/lib/python2.7/site-packages/awscurl/awscurl.py", line 104, in sha256_hash
return hashlib.sha256(val.encode('utf-8')).hexdigest()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 93: ordinal not in range(128)
The fix seems to be to avoid the implicit call to decode(<default encoding>)
when dealing with val
being of type str
, by explicitly decoding val
as utf-8
first so we end of with a unicode
object to feed to encode('utf-8')
, like it expects.
Hence the fix looks like this: change
return hashlib.sha256(val.encode('utf-8')).hexdigest()
to
return hashlib.sha256(val.decode('utf-8').encode('utf-8')).hexdigest()
After install awscurl
on macOS Sierra, I get:
Traceback (most recent call last):
File "/usr/local/bin/awscurl", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 311, in main
args.security_token
File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 140, in make_request
canonical_querystring = normalize_query_string(query)
File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 227, in normalize_query_string
for p in sorted(kv))
File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 227, in <genexpr>
for p in sorted(kv))
TypeError: 'map' object is not subscriptable
Is there a way to fix it? Thanks.
What do you think about adding multipart/form-data
requests that are made with -F
option in curl?
It would be awesome for testing endpoints for file uploads that are located on API Gateways with AWS Auth enabled.
Example [aws]curl command:
awscurl \
-X POST \
-H "Content-Type: multipart/form-data" \
-F "field1=value1" \
-F 'field2={"some":"json"};type=application/json' \
-F "file=@/path/to/some/picture.png" \
https://redacted.execute-api.us-west-2.amazonaws.com/path
Expected request body:
--------------------------d9afcbc8f3fe869a
Content-Disposition: form-data; name="field1"
value1
--------------------------d9afcbc8f3fe869a
Content-Disposition: form-data; name="field2"
Content-Type: application/json
{"some":"json"}
--------------------------d9afcbc8f3fe869a
Content-Disposition: form-data; name="file"; filename="picture.png"
Content-Type: image/png
<binary data>
--------------------------d9afcbc8f3fe869a--
The project should have documentation to bootstrap new contributors, covering PR/build/etc steps.
I've been using awscurl to call API Gateway endpoints, and it has been working great for us up until now (thanks very much for making this!).
I recently started working on an endpoint that takes comma-delimited arrays, and found that the signature is not working properly in this case.
Example failure:
$ awscurl -v https://my-gateway-id.execute-api.us-east-1.amazonaws.com/live/my-path?arg=a,b,c
...
('\n'
'CANONICAL REQUEST = GET\n'
'/live/my-path\n'
'arg=a,b,c\n'
'host:my-gateway-id.execute-api.us-east-1.amazonaws.com\n'
'x-amz-date:20211202T033321Z\n'
'x-amz-security-token:IQoJb3JpZ2luX2VjEEsaCXVzLWVhc3QtMSJHMEUCIQC4m8wJYpB8Xch/XdJzdKHVbgT9refFSvzJ3CeRD9HYfQIgbWPPWfYZ6bYDupK/2SIbvkxruT8ssDn4Kl0qWcCxcF0qnAMIIxAAGgw3OTkyNDI4NjU0OTEiDALfa7r3/U7W1vFxMCr5AuxtcoYbB0iq58HexqEgxxe95yduKHkab03fDBNzwWl+ZG2ATBekfJLvi7Rhzr3U2uNckMXG32x3Nj/nirtwbqfecjqLj/7XfPhBNKgXzEuJ5bQ+oIUP3heXj5G8tJ/Yvwe0eRgghiLjarwzBzvMrA7u9o0jvIGTGHZqTMM0uUPkJrDWoGpWic73fvbe7GHG0j6Hcyy/agWmgLtG7NHh5GcAwdnZih8g9nzMOO5GL0wiu805NSo6xWc2BqnP/uyHQNRAxOc3Uxd+dYtqh6NosNvX3H5x8Gzj383hB3XDbsgZiRvvk+EMwlcZlsFMg+6Mg9abx516vpBgGaKsf5yoWhK4tnsvUnNsgF+2gwBjb93z6eKP7GdHyB3iirtetQcscwkFA6mt4cl1gz8FTNcfCN6BGPQhtUO+HXfpnOY8iapateIJnhjoibELSbcNYf9ql55Xm7bvOknGAJ6HCZvp3mpM/x/X7N6e+U0sdWtzVVFNt82cY/c3BknZML7UoI0GOqYB48W1RlChDe3KjQsHrKtTdgXBGdxU1IYbHdVSwzg9t3ZewwHEpVjfF77LQ0oSRD4ROuIC9cbAXXmr/QCPqpJpFaLtqFNlVCyLpAEygCLsXN+mxGQFj6laGmZR8nEVZrsnzZy1zhOEMOfhPZWQrULbaZeJ/ZtVWto3gnQ/Rl60H52jwf8ccU2EUijMLjlRgOH8MtAnS9WCbTKur07cxK4PXLVG21DTBw==\n'
'\n'
'host;x-amz-date;x-amz-security-token\n'
'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855')
...
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'GET\n/live/my-path\narg=a%2Cb%2Cc\nhost:my-gateway-id.execute-api.us-east-1.amazonaws.com\nx-amz-date:20211202T033321Z\nx-amz-security-token:IQoJb3JpZ2luX2VjEEsaCXVzLWVhc3QtMSJHMEUCIQC4m8wJYpB8Xch/XdJzdKHVbgT9refFSvzJ3CeRD9HYfQIgbWPPWfYZ6bYDupK/2SIbvkxruT8ssDn4Kl0qWcCxcF0qnAMIIxAAGgw3OTkyNDI4NjU0OTEiDALfa7r3/U7W1vFxMCr5AuxtcoYbB0iq58HexqEgxxe95yduKHkab03fDBNzwWl+ZG2ATBekfJLvi7Rhzr3U2uNckMXG32x3Nj/nirtwbqfecjqLj/7XfPhBNKgXzEuJ5bQ+oIUP3heXj5G8tJ/Yvwe0eRgghiLjarwzBzvMrA7u9o0jvIGTGHZqTMM0uUPkJrDWoGpWic73fvbe7GHG0j6Hcyy/agWmgLtG7NHh5GcAwdnZih8g9nzMOO5GL0wiu805NSo6xWc2BqnP/uyHQNRAxOc3Uxd+dYtqh6NosNvX3H5x8Gzj383hB3XDbsgZiRvvk+EMwlcZlsFMg+6Mg9abx516vpBgGaKsf5yoWhK4tnsvUnNsgF+2gwBjb93z6eKP7GdHyB3iirtetQcscwkFA6mt4cl1gz8FTNcfCN6BGPQhtUO+HXfpnOY8iapateIJnhjoibELSbcNYf9ql55Xm7bvOknGAJ6HCZvp3mpM/x/X7N6e+U0sdWtzVVFNt82cY/c3BknZML7UoI0GOqYB48W1RlChDe3KjQsHrKtTdgXBGdxU1IYbHdVSwzg9t3ZewwHEpVjfF77LQ0oSRD4ROuIC9cbAXXmr/QCPqpJpFaLtqFNlVCyLpAEygCLsXN+mxGQFj6laGmZR8nEVZrsnzZy1zhOEMOfhPZWQrULbaZeJ/ZtVWto3gnQ/Rl60H52jwf8ccU2EUijMLjlRgOH8MtAnS9WCbTKur07cxK4PXLVG21DTBw==\n\nhost;x-amz-date;x-amz-security-token\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20211202T033321Z\n20211202/us-east-1/execute-api/aws4_request\n38b3266af08b8f9f119ffed742e0063da244001dbdf2331c42f4f1af695e3e64'\n"}
Reviewing sigv4-create-canonical-request.html, it says:
Percent-encode all other characters with %XY, where X and Y are hexadecimal characters (0-9 and uppercase A-F). For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do) and extended UTF-8 characters must be in the form %XY%ZA%BC.
In my case, I believe that means it should be sending arg=a%2Cb%2Cc
instead of arg=a,b,c
Testing locally, the following patch appears to fix things for me, but I'm not sure how much fallout this might cause for other use cases:
diff --git a/awscurl/awscurl.py b/awscurl/awscurl.py
index cf09318..66a14c4 100755
--- a/awscurl/awscurl.py
+++ b/awscurl/awscurl.py
@@ -16,6 +16,7 @@ import configparser
import configargparse
import requests
from requests.structures import CaseInsensitiveDict
+from urllib.parse import urlencode
from .utils import sha256_hash, sha256_hash_for_binary_data, sign
@@ -315,9 +316,9 @@ def __normalize_query_string(query):
for s in query.split('&')
if len(s) > 0)
- normalized = '&'.join('%s=%s' % (p[0], p[1] if len(p) > 1 else '')
- for p in sorted(parameter_pairs))
- return normalized
+ normalized_pairs = [(p[0], p[1] if len(p) > 1 else '')
+ for p in [p for p in sorted(parameter_pairs)]]
+ return urlencode(normalized_pairs)
def __now():
Just encountered an issue using awscli against Amazon ElasticSearch service. Turns out some services require the x-amz-security-token header to be part of the signature, others don't.
I believe it's safe to just always include it in the signed payload when it's set.
From: Amazon V4 Signatures:
You can use temporary security credentials provided by the AWS Security Token Service (AWS STS) to sign a request. The process is the same as using long-term credentials, but requires an additional HTTP header or query string parameter for the security token. The name of the header or query string parameter is X-Amz-Security-Token, and the value is the session token (the string you received from AWS STS when you obtained temporary security credentials).
When you add the X-Amz-Security-Token parameter to the query string, some services require that you include this parameter in the canonical (signed) request. For other services, you add this parameter at the end, after you calculate the signature. For details, see the API reference documentation for that service.
Hi All,
I am trying to make a request to my API hosted on Amazon API Gateway. I put the request object in a file called request.json and am attempting to make a POST request as follows:
awscurl -v --service execute-api -X POST -d @"bin/request.json" "${endpoint}adduser"
Since I have enabled the verbose flag, I am seeing data field of my request as 'data': '@bin/request.json'
. Any help is appreciated.
Title says it all.
I think awscurl required a different version of python than I had on my system.
After running brew install awscurl
, I saw:
==> Installing dependencies for awscurl: [email protected], readline, xz and [email protected]
...
Python has been installed as
/usr/local/opt/[email protected]/bin/python3
You can install Python packages with
/usr/local/opt/[email protected]/bin/pip3 install <package>
They will install into the site-package directory
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages
See: https://docs.brew.sh/Homebrew-and-Python
[email protected] is keg-only, which means it was not symlinked into /usr/local,
because this is an alternate version of another formula.
If you need to have [email protected] first in your PATH run:
echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.bash_profile
For compilers to find [email protected] you may need to set:
export LDFLAGS="-L/usr/local/opt/[email protected]/lib"
For pkg-config to find [email protected] you may need to set:
export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig"
Now when I run simple tests with the AWS CLI, I see:
$ aws iam get-user
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512
Traceback (most recent call last):
File "/usr/local/bin/aws", line 19, in <module>
import awscli.clidriver
File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 17, in <module>
import botocore.session
File "/usr/local/lib/python2.7/site-packages/botocore/session.py", line 28, in <module>
import botocore.configloader
File "/usr/local/lib/python2.7/site-packages/botocore/configloader.py", line 19, in <module>
from botocore.compat import six
File "/usr/local/lib/python2.7/site-packages/botocore/compat.py", line 27, in <module>
from urllib3 import exceptions
File "/usr/local/lib/python2.7/site-packages/urllib3/__init__.py", line 7, in <module>
from .connectionpool import (
File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 30, in <module>
from .connection import (
File "/usr/local/lib/python2.7/site-packages/urllib3/connection.py", line 39, in <module>
from .util.ssl_ import (
File "/usr/local/lib/python2.7/site-packages/urllib3/util/__init__.py", line 6, in <module>
from .ssl_ import (
File "/usr/local/lib/python2.7/site-packages/urllib3/util/ssl_.py", line 8, in <module>
from hashlib import md5, sha1, sha256
ImportError: cannot import name md5
UPDATE: apparently its not just the aws
CLI. It's other things...
$ which pip
/usr/local/bin/pip
$ pip -v
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512
Traceback (most recent call last):
File "/usr/local/bin/pip", line 11, in <module>
load_entry_point('pip==19.0.3', 'console_scripts', 'pip')()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
return ep.load()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2411, in load
return self.resolve()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/local/lib/python2.7/site-packages/pip/_internal/__init__.py", line 19, in <module>
from pip._vendor.urllib3.exceptions import DependencyWarning
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/__init__.py", line 8, in <module>
from .connectionpool import (
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/connectionpool.py", line 29, in <module>
from .connection import (
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/connection.py", line 38, in <module>
from .util.ssl_ import (
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/__init__.py", line 6, in <module>
from .ssl_ import (
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 8, in <module>
from hashlib import md5, sha1, sha256
ImportError: cannot import name md5
https://github.com/allthings/awscurl has nice[er] docker instructions -- add similar level of details to https://github.com/okigan/awscurl/
Does awscurl already support AWS SSO?
I tried just now and I seem to be missing something.
I'm using the version installed today directly from Github.
I have a config file with a profile that uses AWS SSO to get temporary credentials.
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
[profile example]
sso_start_url = https://d-1111111111.awsapps.com/start
sso_region = eu-west-1
sso_account_id = 111111111111
sso_role_name = AdministratorAccess
region = eu-west-1
output = json
The profile works with AWS CLI v2.
$ aws sts get-caller-identity --profile example
{
"UserId": "AROAAAAAAAAAAAEXAMPLE:example",
"Account": "111111111111",
"Arn": "arn:aws:sts::111111111111:assumed-role/AWSReservedSSO_AdministratorAccess_1111111111111111/example"
}
But when I try to use it with awscurl I get a message about the token being expired.
$ export AWS_PROFILE=example
$ awscurl --region=eu-west-1 --service=s3 http://example.s3.eu-west-1.amazonaws.com/
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Token-0>IQoJ[...]96F0=</Token-0><RequestId>D[...]W</RequestId><HostId>GeFN[...]kfUTs=</HostId></Error>
Traceback (most recent call last):
File "/home/isme/.local/bin/awscurl", line 8, in <module>
sys.exit(main())
File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
inner_main(sys.argv[1:])
File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 494, in inner_main
response.raise_for_status()
File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://example.s3.eu-west-1.amazonaws.com/
Hi,
would it be possible to return the response object instead of returning a number (0, 1, etc.) for the success status of the request?
The reason behind that is, if one e.g. performs a GET with awscurl, it would be nice to have the response body available for further usage. Currently it is only piped to print() whcih is useless for automation...
BR & thanks
MB
When using 0.19 awscurl with python 3, encoding response.text results in a byte object being returned. Printing that directly results in the output being prefixed with b'
(see screenshots below), and thus json output cannot be parsed directly from the response. In Python 2 this is not an issue. I could also see this being an issue for (say) an xml based response as well (though I have not tested that).
My use case: I'm currently invoking awscurl in a unix pipe, where the result of awscurl
is piped to jq
for further processing (in CI); updating to the latest version broke the pipe. For the moment I've pinned the version in CI at 0.17
I'm facing an issue with executing a API Gateway URL.
./awscurl.py -X GET --region us-west-2 --access_key XXXXXXXXXXXX --secret_key XXXXXXXXXXXXXXXXX --service execute-api """https://xxxxx.execute-api.us-west-2.amazonaws.com/XX/YY/ZZZ?ids=[\{\"id\":\"aaaaa-bbb-cccc-dddd-1234567\"\}]&pid=aaaaa-bbb-cccc-dddd-1234567"""
I'm getting following response,
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}
Traceback (most recent call last):
File "awscurl.py", line 334, in
sys.exit(main())
File "awscurl.py", line 329, in main
args.security_token or args.session_token
File "awscurl.py", line 232, in make_request
return send_request(uri, data, headers, method)
File "awscurl.py", line 262, in send_request
r.raise_for_status()
File "/opt/VRTSitrpgw/venv27/lib/python2.7/site-packages/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url:
Any help is appreciated.
AWSCurl does not support profiles that assume role (for cross-account access) or MFA.
Hi,
we just tried to use v0.18 and got this exception:
Traceback (most recent call last):
File "/usr/local/bin/awscurl", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/awscurl/awscurl.py", line 491, in main
print(response.headers.encode('utf-8'), end='\n\n')
AttributeError: 'CaseInsensitiveDict' object has no attribute 'encode'
If you are uploading data to S3, how would I do that? This right here is my guess for just making a txt file that has Hello There, but it is not working.
awscurl -X 'POST' -d "Hello There" --service $SERVICE --region $REGION --access_key $ACCESS_KEY --secret_key $SECRET_KEY $URL
I'm confident that my query works fine if Elasticsearch domain allows open-access and I use curl
command to issue query.
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-2021> curl -XGET "https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty" -H "Content-Type: application/json" -d "@search-search-ngram-and-kuromoji-2.json"
{
"took" : 163,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 144.35374,
"hits" : [
{
"_index" : "search-ngram-and-kuromoji",
"_type" : "_doc",
"_id" : "ucNKC3gBNKbjtYmSAtQp",
"_score" : 144.35374,
"_source" : {
"section_url" : "001.html#topic_i5x_lkz_bgb"
},
"highlight" : {
"section_text" : [
"็ชใ่ก็ชใซ่ฟใ็ถๆ
๏ผ<em>SRSใจใขใใใฐใฎไฝๅใใใณ่ทฏไธ้ๅฎณ็ฉใจใฎๆฅ่งฆ</em>ใชใฉ๏ผใ็บ็ใใๆใซ่ป"
],
"section" : [
"ใคใใณใใใผใฟใฌใณใผใใผ"
]
}
}
]
}
}
I attached the search query: search-search-ngram-and-kuromoji-2.json zipped.
search-search-ngram-and-kuromoji-2.zip
awscurl
without specifying --data-binary
, no search result returned.I have changed domain access policy to exhibit open access and allow one IAM role named ESFullAccess. Also ESFullAccess has trusted IAM user called ESProgram.
The command-line Windows Power shell awsescurl.ps1
zipped.
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-2021> ./awsescurl.ps1 -X GET -d "@search-search-ngram-and-kuromoji-2.json" "https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty"
{'access_key': 'ASIAUPPPMJZZ2PAQ6B74',
'data': '@search-search-ngram-and-kuromoji-2.json',
'data_binary': False,
'header': ['Content-Type: application/json'],
'include': False,
'insecure': True,
'profile': 'default',
'region': 'yyyyyyyyy',
'request': 'GET',
'secret_key': 'IpYWCqt3OScsBxA0/dOmVrSFWN7NfEbX1VyQwye9',
'security_token': 'FwoGZXIvYXdzEMr//////////wEaDKOZeknTfG/fc5Ng+iKwAXTv+jazLkF0NMNGPiSYtytG3WqA1U1cUCU4ElfcHNixm+LFTOphsYQh9iY7xFO9cBh+iRrvF6qB10IeG7Ta+PJtcLZnzOUfOGE8w6a94YqpWciIRQ5CEAL3UDeYNru0IGeulJxVSzHaTRs8crJ7d3DOqSRDVGSKXfNpCQjzOKXwr/nam3JAkPGyyd4u2B8iWhOmPl9lhxORchF5fBb84Npw8YlSGDFPoLSsBM+NjX8jKJLTuoIGMi0eGpiRN7Wo1OKjCQy2C1V5UNVr6u66Q/cY7r0RveYwNeZhaz4DI/ThXpMPD4A=',
'service': 'es',
'session_token': None,
'uri': 'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty',
'verbose': True}
'pretty='
('\n'
'CANONICAL REQUEST = GET\n'
'/search-ngram-and-kuromoji/_search\n'
'pretty=\n'
'host:search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com\n'
'x-amz-date:20210315T002554Z\n'
'x-amz-security-token:FwoGZXIvYXdzEMr//////////wEaDKOZeknTfG/fc5Ng+iKwAXTv+jazLkF0NMNGPiSYtytG3WqA1U1cUCU4ElfcHNixm+LFTOphsYQh9iY7xFO9cBh+iRrvF6qB10IeG7Ta+PJtcLZnzOUfOGE8w6a94YqpWciIRQ5CEAL3UDeYNru0IGeulJxVSzHaTRs8crJ7d3DOqSRDVGSKXfNpCQjzOKXwr/nam3JAkPGyyd4u2B8iWhOmPl9lhxORchF5fBb84Npw8YlSGDFPoLSsBM+NjX8jKJLTuoIGMi0eGpiRN7Wo1OKjCQy2C1V5UNVr6u66Q/cY7r0RveYwNeZhaz4DI/ThXpMPD4A=\n'
'\n'
'host;x-amz-date;x-amz-security-token\n'
'41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211')
('\n'
'STRING_TO_SIGN = AWS4-HMAC-SHA256\n'
'20210315T002554Z\n'
'20210315/yyyyyyyyy/es/aws4_request\n'
'09176940c4429ffeab8230044aa84447ff1f6d91a76b84e0b0069495e2538a75')
'\nHEADERS++++++++++++++++++++++++++++++++++++'
{'Authorization': 'AWS4-HMAC-SHA256 '
'Credential=ASIAUPPPMJZZ2PAQ6B74/20210315/yyyyyyyyy/es/aws4_request, '
'SignedHeaders=host;x-amz-date;x-amz-security-token, '
'Signature=66a1ec2e970b6d5d76628f4e5493d3c6d8b6edc3a6cabd1f54b868290ae81418',
'Content-Type': 'application/json',
'x-amz-content-sha256': '41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211',
'x-amz-date': '20210315T002554Z',
'x-amz-security-token': 'FwoGZXIvYXdzEMr//////////wEaDKOZeknTfG/fc5Ng+iKwAXTv+jazLkF0NMNGPiSYtytG3WqA1U1cUCU4ElfcHNixm+LFTOphsYQh9iY7xFO9cBh+iRrvF6qB10IeG7Ta+PJtcLZnzOUfOGE8w6a94YqpWciIRQ5CEAL3UDeYNru0IGeulJxVSzHaTRs8crJ7d3DOqSRDVGSKXfNpCQjzOKXwr/nam3JAkPGyyd4u2B8iWhOmPl9lhxORchF5fBb84Npw8YlSGDFPoLSsBM+NjX8jKJLTuoIGMi0eGpiRN7Wo1OKjCQy2C1V5UNVr6u66Q/cY7r0RveYwNeZhaz4DI/ThXpMPD4A='}
'\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++'
('Request URL = '
'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty')
'\nRESPONSE++++++++++++++++++++++++++++++++++++'
'Response code: 200\n'
{'Date': 'Mon, 15 Mar 2021 00:25:55 GMT', 'Content-Type': 'application/json; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'close', 'Access-Control-Allow-Origin': '*', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding, User-Agent'}
{
"took" : 47,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
awscurl
with specifying --data-binary
, following error occurs.The command-line Windows Power shell awsescurl-db.ps1
zipped.
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-2021> ./awsescurl-db.ps1 -X GET -d "@search-search-ngram-and-kuromoji-2.json" "https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty"
{'access_key': 'ASIAUPPPMJZZUXODBZUO',
'data': '@search-search-ngram-and-kuromoji-2.json',
'data_binary': True,
'header': ['Content-Type: application/json'],
'include': False,
'insecure': True,
'profile': 'default',
'region': 'yyyyyyyyy',
'request': 'GET',
'secret_key': 'kfGOQpb/p4Ckm4vYMocotYu56K15BI+NIQq3VZei',
'security_token': 'FwoGZXIvYXdzEMr//////////wEaDJnWC1Gw7ePjDa6+fCKwASSe7Ro/gxR1BN7sts/Kn+RfFBJmYmjRZS+mvLClqjvg2OU/MdnZxJQeCsvxIP9Vzk5Ogdmq0tvp3uib3qVDQoEFHAfIf/AIPB/dRyp7KGLQpEXjjw5/v4/KRx3KDipAbH4aMFGjW0KrGymKsGiH1VMsuqfagWAl34tgclAtxoTQO+XGuxJTBafv1MEM9yAnMgN8S2zHtEEwNLOdJyb5CAQgTAAAJEgjSpjUgTpDsa4mKNLZuoIGMi0E3Ok7VjMeYV8WcpmwSwtMetlgSMIwvRRPArr8AYQVoEPINIzz6ufDE6IIquo=',
'service': 'es',
'session_token': None,
'uri': 'https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty',
'verbose': True}
'pretty='
Traceback (most recent call last):
File "c:\users\toshi\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\toshi\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\Scripts\awscurl.exe\__main__.py", line 7, in <module>
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 500, in main
inner_main(sys.argv[1:])
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 478, in inner_main
response = make_request(args.request,
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 100, in make_request
canonical_request, payload_hash, signed_headers = task_1_create_a_canonical_request(
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 200, in task_1_create_a_canonical_request
payload_hash = sha256_hash_for_binary_data(data) if data_binary else sha256_hash(data)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\utils.py", line 20, in sha256_hash_for_binary_data
return hashlib.sha256(val).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
I rarely know Python, but from error message "TypeError: Unicode-objects must be encoded before hashing", I've modified following code from:
https://github.com/okigan/awscurl/blob/master/awscurl/awscurl.py line 200
payload_hash = sha256_hash_for_binary_data(data) if data_binary else sha256_hash(data)
to
payload_hash = sha256_hash(data)
because hashing payload should be done with encoding UTF-8 if --data-binary
is specified or not.
However modified awscurl still reports following error:
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-2021> ./awsescurl-db.ps1 -X GET -d "@search-search-ngram-and-kuromoji-2.json" "https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty"
{'access_key': 'ASIAUPPPMJZZ2XEUBLF7',
'data': '@search-search-ngram-and-kuromoji-2.json',
'data_binary': True,
'header': ['Content-Type: application/json'],
'include': False,
'insecure': False,
'profile': 'default',
'region': 'yyyyyyyyy',
'request': 'GET',
'secret_key': 'd9Fd2UQbWWuQSo1YzJeGU74pmi+ERNvI4gN7RDre',
'security_token': 'FwoGZXIvYXdzEMr//////////wEaDIGnfgGrBvi2QJemgiKwAWOZo7D/VAIvgtVf5gL+z+yF610K45iNCG7q6HHf+vIpxFJfiji+uIEJZXQMWOHTkVONfMvm5dBz8g3Ss8aTVQxjEkXTP3tw1MUPiq15qLiYW6ZeRvv9+kw6gkM2r2TIZm1k3oGOknzz8GTwQQoHySjj+zaDqNdHxN1l/rXMcyCdsaghuH12FvNsAmZV0TelhGJ3ceo9X6omS8BqRHCO5YYhs4DV2ApRI80yCsCOAofxKPTguoIGMi1nDJqy/cmXak2pL0VHU2puG5pbnjSV5kgaVD3oerY2bcbCzDu+t8ml+TplHXI=',
'service': 'es',
'session_token': None,
'uri': 'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty',
'verbose': True}
'pretty='
('\n'
'CANONICAL REQUEST = GET\n'
'/search-ngram-and-kuromoji/_search\n'
'pretty=\n'
'host:search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com\n'
'x-amz-date:20210315T005516Z\n'
'x-amz-security-token:FwoGZXIvYXdzEMr//////////wEaDIGnfgGrBvi2QJemgiKwAWOZo7D/VAIvgtVf5gL+z+yF610K45iNCG7q6HHf+vIpxFJfiji+uIEJZXQMWOHTkVONfMvm5dBz8g3Ss8aTVQxjEkXTP3tw1MUPiq15qLiYW6ZeRvv9+kw6gkM2r2TIZm1k3oGOknzz8GTwQQoHySjj+zaDqNdHxN1l/rXMcyCdsaghuH12FvNsAmZV0TelhGJ3ceo9X6omS8BqRHCO5YYhs4DV2ApRI80yCsCOAofxKPTguoIGMi1nDJqy/cmXak2pL0VHU2puG5pbnjSV5kgaVD3oerY2bcbCzDu+t8ml+TplHXI=\n'
'\n'
'host;x-amz-date;x-amz-security-token\n'
'41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211')
('\n'
'STRING_TO_SIGN = AWS4-HMAC-SHA256\n'
'20210315T005516Z\n'
'20210315/yyyyyyyyy/es/aws4_request\n'
'21e16f8efede645709fa4ad0d9b0ca272cf17e4679ab9e75048a315d01ba1d45')
'\nHEADERS++++++++++++++++++++++++++++++++++++'
{'Authorization': 'AWS4-HMAC-SHA256 '
'Credential=ASIAUPPPMJZZ2XEUBLF7/20210315/yyyyyyyyy/es/aws4_request, '
'SignedHeaders=host;x-amz-date;x-amz-security-token, '
'Signature=30d36e27a4962fd248cd58052403e1da72d077c214acd5d67ab984d425927c63',
'Content-Type': 'application/json',
'x-amz-content-sha256': '41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211',
'x-amz-date': '20210315T005516Z',
'x-amz-security-token': 'FwoGZXIvYXdzEMr//////////wEaDIGnfgGrBvi2QJemgiKwAWOZo7D/VAIvgtVf5gL+z+yF610K45iNCG7q6HHf+vIpxFJfiji+uIEJZXQMWOHTkVONfMvm5dBz8g3Ss8aTVQxjEkXTP3tw1MUPiq15qLiYW6ZeRvv9+kw6gkM2r2TIZm1k3oGOknzz8GTwQQoHySjj+zaDqNdHxN1l/rXMcyCdsaghuH12FvNsAmZV0TelhGJ3ceo9X6omS8BqRHCO5YYhs4DV2ApRI80yCsCOAofxKPTguoIGMi1nDJqy/cmXak2pL0VHU2puG5pbnjSV5kgaVD3oerY2bcbCzDu+t8ml+TplHXI='}
'\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++'
('Request URL = '
'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty')
Traceback (most recent call last):
File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\Scripts\awscurl-script.py", line 33, in <module>
sys.exit(load_entry_point('awscurl==0.21', 'console_scripts', 'awscurl')())
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 499, in main
inner_main(sys.argv[1:])
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 477, in inner_main
response = make_request(args.request,
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 135, in make_request
return __send_request(uri, data, headers, method, verify)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 330, in __send_request
response = requests.request(method, uri, headers=headers, data=data, verify=verify)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\urllib3\connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\http\client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\http\client.py", line 1300, in _send_request
body = _encode(body, 'body')
File "c:\users\toshi\appdata\local\programs\python\python39\lib\http\client.py", line 164, in _encode
raise UnicodeEncodeError(
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 322-357: Body ('็นง๏ฝจ็นง๏ฝข็น่ใฃ็นง๏ฝฐ็ธบ๏ฝฎ่ดๆท่็ธบ็ฟซ๏ฝ็ธบ๏ฝณ) is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
According to the error message, I modified the following code:
https://github.com/okigan/awscurl/blob/master/awscurl/awscurl.py line 135
if data_binary:
return __send_request(uri, data, headers, method, verify)
else:
return __send_request(uri, data.encode('utf-8'), headers, method, verify)
to:
if data_binary:
return __send_request(uri, data.encode('utf-8'), headers, method, verify)
else:
return __send_request(uri, data.encode('utf-8'), headers, method, verify)
The error message has been vanished, but the query returns nothing.
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-2021> ./awsescurl-db.ps1 -X GET -d "@search-search-ngram-and-kuromoji-2.json" "https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty"
{'access_key': 'ASIAUPPPMJZZ5EN6WTOX',
'data': '@search-search-ngram-and-kuromoji-2.json',
'data_binary': True,
'header': ['Content-Type: application/json'],
'include': False,
'insecure': False,
'profile': 'default',
'region': 'yyyyyyyyy',
'request': 'GET',
'secret_key': 'F3kemHCI+/UNHAE19ms59sBi9XHcMaMN4vpb14as',
'security_token': 'FwoGZXIvYXdzEMv//////////wEaDPJUgKt3g2RxHCHWGiKwAaY60g/Lm1Hp48nED39tUP/34Ia2tqUT/Ljgqe2Rg2SBOGhAlTvQKpkypyNyAS8+vLFEmRGfw11UM6UOZvFmx3NeWi8g6zpV7QpeSCPFRKbwZLnSxTEn2r7n9p3QRXNNkNWlJSPCLDzTZecORr46FGYmdnX6ZkKL97p6dWpTiQ53O62FarMbUID90zReTEDKMEbM5n2oaS9hcfZCz8M7Zr7+zXC8C+5Pw01fi0TKGnYUKL7muoIGMi3hmp5Jw1uw3fZK4azfs7WUg+/EV2N5qN81sQyrF/sxH+X59xJ2cBjfeAYJvjQ=',
'service': 'es',
'session_token': None,
'uri': 'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty',
'verbose': True}
'pretty='
('\n'
'CANONICAL REQUEST = GET\n'
'/search-ngram-and-kuromoji/_search\n'
'pretty=\n'
'host:search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com\n'
'x-amz-date:20210315T010710Z\n'
'x-amz-security-token:FwoGZXIvYXdzEMv//////////wEaDPJUgKt3g2RxHCHWGiKwAaY60g/Lm1Hp48nED39tUP/34Ia2tqUT/Ljgqe2Rg2SBOGhAlTvQKpkypyNyAS8+vLFEmRGfw11UM6UOZvFmx3NeWi8g6zpV7QpeSCPFRKbwZLnSxTEn2r7n9p3QRXNNkNWlJSPCLDzTZecORr46FGYmdnX6ZkKL97p6dWpTiQ53O62FarMbUID90zReTEDKMEbM5n2oaS9hcfZCz8M7Zr7+zXC8C+5Pw01fi0TKGnYUKL7muoIGMi3hmp5Jw1uw3fZK4azfs7WUg+/EV2N5qN81sQyrF/sxH+X59xJ2cBjfeAYJvjQ=\n'
'\n'
'host;x-amz-date;x-amz-security-token\n'
'41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211')
('\n'
'STRING_TO_SIGN = AWS4-HMAC-SHA256\n'
'20210315T010710Z\n'
'20210315/yyyyyyyyy/es/aws4_request\n'
'0a5cea16f924e59ed1d26254b5239ff26a8dee5ee5583c6b51d6922f0eefb46e')
'\nHEADERS++++++++++++++++++++++++++++++++++++'
{'Authorization': 'AWS4-HMAC-SHA256 '
'Credential=ASIAUPPPMJZZ5EN6WTOX/20210315/yyyyyyyyy/es/aws4_request, '
'SignedHeaders=host;x-amz-date;x-amz-security-token, '
'Signature=d4243cc6a1264c77bc90743bca2e03cebe5a408b25b1bf238d8a638bea7bda9b',
'Content-Type': 'application/json',
'x-amz-content-sha256': '41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211',
'x-amz-date': '20210315T010710Z',
'x-amz-security-token': 'FwoGZXIvYXdzEMv//////////wEaDPJUgKt3g2RxHCHWGiKwAaY60g/Lm1Hp48nED39tUP/34Ia2tqUT/Ljgqe2Rg2SBOGhAlTvQKpkypyNyAS8+vLFEmRGfw11UM6UOZvFmx3NeWi8g6zpV7QpeSCPFRKbwZLnSxTEn2r7n9p3QRXNNkNWlJSPCLDzTZecORr46FGYmdnX6ZkKL97p6dWpTiQ53O62FarMbUID90zReTEDKMEbM5n2oaS9hcfZCz8M7Zr7+zXC8C+5Pw01fi0TKGnYUKL7muoIGMi3hmp5Jw1uw3fZK4azfs7WUg+/EV2N5qN81sQyrF/sxH+X59xJ2cBjfeAYJvjQ='}
'\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++'
('Request URL = '
'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty')
'\nRESPONSE++++++++++++++++++++++++++++++++++++'
'Response code: 200\n'
{'Date': 'Mon, 15 Mar 2021 01:07:12 GMT', 'Content-Type': 'application/json; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'close', 'Access-Control-Allow-Origin': '*', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding, User-Agent'}
{
"took" : 68,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
Do you have any ideas to fix this problem?
Regards,
When the -i
flag to include headers in response is provided, a python dict representation (actually the requests custom case-indifferent dict) of the headers are printed. Looks deceptively like JSON at a quick glance.
If the headers were dumped as valid JSON, tools like jq
could be used for digging into them, which aids in readability and using that header data as part of a pipeline.
Is the upper bound on urllib3
necessary?
Line 24 in 01cb53c
I built it locally using nixpkgs which has 1.25.3 using a custom patch and it seems to work fine, but I am not a python developer at all :)
When I run command from the Readme under the python:3.9.7-alpine3.14
docker image, I'm getting a lot of the following errors:
Downloading cryptography-3.1.tar.gz (534 kB)
|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 534 kB 10.0 MB/s
Installing build dependencies ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /tmp/pip-standalone-pip-huuba34w/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-e1ixhxe4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"''
cwd: None
Complete output (118 lines):
Collecting setuptools>=40.6.0
Using cached setuptools-58.0.0-py3-none-any.whl (816 kB)
Collecting wheel
Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Collecting cffi!=1.11.3,>=1.8
Using cached cffi-1.14.6.tar.gz (475 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: cffi
Building wheel for cffi (setup.py): started
Building wheel for cffi (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-l4232b8e
cwd: /tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/
Complete output (45 lines):
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.libera.chat.)
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/c
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Failed building wheel for cffi
Running setup.py clean for cffi
Failed to build cffi
Installing collected packages: pycparser, wheel, setuptools, cffi
Running setup.py install for cffi: started
Running setup.py install for cffi: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8kbxbpwh/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-e1ixhxe4/overlay --compile --install-headers /tmp/pip-build-env-e1ixhxe4/overlay/include/python3.9/cffi
cwd: /tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/
Complete output (45 lines):
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.libera.chat.)
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/c
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8kbxbpwh/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-e1ixhxe4/overlay --compile --install-headers /tmp/pip-build-env-e1ixhxe4/overlay/include/python3.9/cffi Check the logs for full command output.
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/12/be/c9cc7d7ab71dbcc9e4e517ead0cdd48e8c9a48d7b8bdddb738e90d08279a/cryptography-3.1.tar.gz#sha256=26409a473cc6278e4c90f782cd5968ebad04d3911ed1c402fc86908c17633e08 (from https://pypi.org/simple/cryptography/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*). Command errored out with exit status 1: /usr/local/bin/python /tmp/pip-standalone-pip-huuba34w/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-e1ixhxe4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.
Downloading cryptography-3.0.tar.gz (534 kB)
|โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ| 534 kB 17.6 MB/s
Installing build dependencies ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /tmp/pip-standalone-pip-0z1qgh1k/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xf8ws2lf/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"''
cwd: None
Complete output (118 lines):
Collecting setuptools>=40.6.0
Using cached setuptools-58.0.0-py3-none-any.whl (816 kB)
Collecting wheel
Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Collecting cffi!=1.11.3,>=1.8
Using cached cffi-1.14.6.tar.gz (475 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: cffi
Building wheel for cffi (setup.py): started
Building wheel for cffi (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ws9rz4du
cwd: /tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/
Complete output (45 lines):
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.libera.chat.)
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/c
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Failed building wheel for cffi
Running setup.py clean for cffi
Failed to build cffi
Installing collected packages: pycparser, wheel, setuptools, cffi
Running setup.py install for cffi: started
Running setup.py install for cffi: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-h7ub2wex/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-xf8ws2lf/overlay --compile --install-headers /tmp/pip-build-env-xf8ws2lf/overlay/include/python3.9/cffi
cwd: /tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/
Complete output (45 lines):
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.libera.chat.)
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/c
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-h7ub2wex/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-xf8ws2lf/overlay --compile --install-headers /tmp/pip-build-env-xf8ws2lf/overlay/include/python3.9/cffi Check the logs for full command output.
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/bf/ac/552fc8729d90393845cc3a2062facf4a89dcbe206fa78771d60ddaae7554/cryptography-3.0.tar.gz#sha256=8e924dbc025206e97756e8903039662aa58aa9ba357d8e1d8fc29e3092322053 (from https://pypi.org/simple/cryptography/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*). Command errored out with exit status 1: /usr/local/bin/python /tmp/pip-standalone-pip-0z1qgh1k/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xf8ws2lf/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.
If I install packages from the dockerfile, I'm getting the following:
/ # pip install awscurl
Collecting awscurl
Using cached awscurl-0.24-py3-none-any.whl (8.4 kB)
Collecting urllib3[secure]
Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting requests
Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting configparser
Using cached configparser-5.0.2-py3-none-any.whl (19 kB)
Collecting configargparse
Using cached ConfigArgParse-1.5.2-py3-none-any.whl (20 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting cryptography>=1.3.4
Using cached cryptography-3.4.8.tar.gz (546 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Collecting pyOpenSSL>=0.14
Using cached pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
Collecting cffi>=1.12
Using cached cffi-1.14.6-cp39-cp39-linux_x86_64.whl
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting six>=1.5.2
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: cryptography
Building wheel for cryptography (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmprjoz7zfw
cwd: /tmp/pip-install-d5d5bd9r/cryptography_7d40124d94904d0da3fabbd0ab736a9a
Complete output (165 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/cryptography
copying src/cryptography/utils.py -> build/lib.linux-x86_64-3.9/cryptography
copying src/cryptography/exceptions.py -> build/lib.linux-x86_64-3.9/cryptography
copying src/cryptography/fernet.py -> build/lib.linux-x86_64-3.9/cryptography
copying src/cryptography/__about__.py -> build/lib.linux-x86_64-3.9/cryptography
copying src/cryptography/__init__.py -> build/lib.linux-x86_64-3.9/cryptography
creating build/lib.linux-x86_64-3.9/cryptography/hazmat
copying src/cryptography/hazmat/_types.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
copying src/cryptography/hazmat/_der.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
copying src/cryptography/hazmat/_oid.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
copying src/cryptography/hazmat/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
creating build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/certificate_transparency.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/ocsp.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/name.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/oid.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/base.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/general_name.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/extensions.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/x509
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
copying src/cryptography/hazmat/backends/interfaces.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
copying src/cryptography/hazmat/backends/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/padding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/keywrap.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/cmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/poly1305.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/constant_time.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/_cipheralgorithm.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/_asymmetric.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/hashes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/_serialization.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/hmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings
copying src/cryptography/hazmat/bindings/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/x509.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/ocsp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/cmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/decode_asn1.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/poly1305.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/rsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/aead.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/ciphers.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/x448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/ed25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/dh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/ec.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/encode_asn1.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/ed448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/dsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/hashes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/backend.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/x25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/hmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
copying src/cryptography/hazmat/primitives/ciphers/modes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
copying src/cryptography/hazmat/primitives/ciphers/aead.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
copying src/cryptography/hazmat/primitives/ciphers/base.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
copying src/cryptography/hazmat/primitives/ciphers/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
copying src/cryptography/hazmat/primitives/ciphers/algorithms.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/padding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/rsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/x448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/ed25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/dh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/ec.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/ed448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/dsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/x25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
copying src/cryptography/hazmat/primitives/serialization/pkcs12.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
copying src/cryptography/hazmat/primitives/serialization/ssh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
copying src/cryptography/hazmat/primitives/serialization/pkcs7.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
copying src/cryptography/hazmat/primitives/serialization/base.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
copying src/cryptography/hazmat/primitives/serialization/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
copying src/cryptography/hazmat/primitives/twofactor/totp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
copying src/cryptography/hazmat/primitives/twofactor/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
copying src/cryptography/hazmat/primitives/twofactor/hotp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
copying src/cryptography/hazmat/primitives/twofactor/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/pbkdf2.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/scrypt.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/x963kdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/kbkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/concatkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/hkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
copying src/cryptography/hazmat/bindings/openssl/binding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
copying src/cryptography/hazmat/bindings/openssl/_conditional.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
copying src/cryptography/hazmat/bindings/openssl/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
running egg_info
writing src/cryptography.egg-info/PKG-INFO
writing dependency_links to src/cryptography.egg-info/dependency_links.txt
writing requirements to src/cryptography.egg-info/requires.txt
writing top-level names to src/cryptography.egg-info/top_level.txt
reading manifest file 'src/cryptography.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'docs/_build'
warning: no previously-included files found matching 'vectors'
warning: no previously-included files matching '*' found under directory 'vectors'
warning: no previously-included files matching '*' found under directory '.github'
warning: no previously-included files found matching 'release.py'
warning: no previously-included files found matching '.coveragerc'
warning: no previously-included files found matching 'codecov.yml'
warning: no previously-included files found matching '.readthedocs.yml'
warning: no previously-included files found matching 'dev-requirements.txt'
warning: no previously-included files found matching 'tox.ini'
warning: no previously-included files found matching 'mypy.ini'
warning: no previously-included files matching '*' found under directory '.zuul.d'
warning: no previously-included files matching '*' found under directory '.zuul.playbooks'
adding license file 'LICENSE'
adding license file 'LICENSE.APACHE'
adding license file 'LICENSE.BSD'
adding license file 'LICENSE.PSF'
writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
copying src/cryptography/py.typed -> build/lib.linux-x86_64-3.9/cryptography
running build_ext
generating cffi module 'build/temp.linux-x86_64-3.9/_padding.c'
creating build/temp.linux-x86_64-3.9
generating cffi module 'build/temp.linux-x86_64-3.9/_openssl.c'
running build_rust
=============================DEBUG ASSISTANCE=============================
If you are seeing a compilation error please try the following steps to
successfully install cryptography:
1) Upgrade to the latest pip and try again. This will fix errors for most
users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
2) Read https://cryptography.io/en/latest/installation.html for specific
instructions for your platform.
3) Check our frequently asked questions for more information:
https://cryptography.io/en/latest/faq.html
4) Ensure you have a recent Rust toolchain installed:
https://cryptography.io/en/latest/installation.html#rust
5) If you are experiencing issues with Rust for *this release only* you may
set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
=============================DEBUG ASSISTANCE=============================
error: can't find Rust compiler
If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
To update pip, run:
pip install --upgrade pip
and then retry package installation.
If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
This package requires Rust >=1.41.0.
----------------------------------------
ERROR: Failed building wheel for cryptography
Failed to build cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly
What I'm doing wrong?
Better setup instructions required I guess.
The AWS documentation for the environment variables that specify credential components for their own aws
cli specify that the STS tokens may be exposed via a AWS_SESSION_TOKEN
environment variable.
The awscurl
though expects the value in a different environment variable named AWS_SECURITY_TOKEN
. I believe it should use the same environment variable as the Amazon CLI.
pip install does not included --profile parameters
I'm trying to run awscurl to perform requests to the ES service through an SSH tunnel. The tunnel opens a local port connected to the ES service 443 port on the API endpoint. The tunnel is required to connect to a service not available to the public. The issue is that awscurl creates a signature using the URL passed in CLI. But the service hostname and port are different from the local URL. I need to make a signature using the real API endpoint, not using the local one. It would be nice to add this feature.
Now that this has been working for quite some time, could you make a release 1.0 or tag the code? I'd like to reference a version that is stable.
Please update setup.py
to install urllib3<1.24,>=1.21.1
, as it currently pulls in 1.24 and throws the following error:
File "/usr/local/bin/awscurl", line 7, in <module>
from awscurl.__main__ import main
File "/usr/local/lib/python2.7/dist-packages/awscurl/__main__.py", line 6, in <module>
from .awscurl import main
File "/usr/local/lib/python2.7/dist-packages/awscurl/awscurl.py", line 14, in <module>
import requests
File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 112, in <module>
from . import utils
File "/usr/local/lib/python2.7/dist-packages/requests/utils.py", line 26, in <module>
from ._internal_utils import to_native_string
File "/usr/local/lib/python2.7/dist-packages/requests/_internal_utils.py", line 11, in <module>
from .compat import is_py2, builtin_str, str
File "/usr/local/lib/python2.7/dist-packages/requests/compat.py", line 48, in <module>
from urllib3.packages.ordered_dict import OrderedDict
ImportError: No module named ordered_dict
When running in EC2, it's the preferred mechanism to get temporary credentials,
and it's supported by boto
It looks like the current pypi version of this package (0.11) still has "print(r.content)". I was using this with Amazon SageMaker and the results were coming back as a byte array ("b'...'") instead of text. When I do a git clone and run the latest version, it works great.
Could we get a pypi release of the latest version? Thanks!
(P.S. thanks for this really useful tool)
Hi
I just tested but
I run a Centos 6.10, python 3.6
~/test/testV2/awscurl-0.21/awscurl
$ ./awscurl.py --service s3 https://awscurl-sample-bucket.s3.amazonaws.com
Traceback (most recent call last):
File "./awscurl.py", line 19, in
from .utils import sha256_hash, sha256_hash_for_binary_data, sign
ModuleNotFoundError: No module named 'main.utils'; 'main' is not a package
May you help me?
regards
It'd be nice to be able to grab awscurl
via homebrew.
For instructions, see: https://docs.brew.sh/Formula-Cookbook
It would be nice to have a possibility to compile awscurl to a binary for optimal disk space usage especially in Docker containers. It's hard to pull all the dependencies required for the program to work. Particularly, I'm building an image for CI/CD that will have awscurl installed.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.