okigan / awscurl Goto Github PK
View Code? Open in Web Editor NEWcurl-like access to AWS resources with AWS Signature Version 4 request signing.
License: MIT License
awscurl's People
Contributors
Stargazers
Watchers
Forkers
robingolden lc0 ajcrites slowikps dojomadness gozer tamast dominic-kua ajmath akshay3030 jmahowald limitlesshq hax0rg1rl raniemi jonathancartwright mr-brody scott-roepnack-zocdoc mbayrak78 bradleyjames simondiep mariosotil dineshkarthik eodabas overdrive3000 iulianmac jmhummel soujanyakumar wileyew handymandandbr iddv telegrapher lrodorigo sverch markatwood zakvdm muziyoshiz p5k6 salewski sumikawa redavis22 5l1v3r1 marklap nvichare avineshwar sp-davidpichler perplexes dineshrathee12 beyarkay texanraj toshihikomakita sharp-pixel iainelder chowdarydinesh bibliotechy latkes dlvenable ekmixon varju nanofaroque ndavies-om1 azurecloudmonk jk1ng erwinverdonk zipabout fyunusa dlroy samhays adedayoominiyi vumdao maximelearning shinroo nrhtr khatwaninikhil gslahari arpitjain799 ryanl1997 erpel karancode azizou gilcu2 jlosito 0xffea kwehmeyer bfrobin446 vijaypabothu agrocrypto chenliu0831 offerup matthiassbawscurl's Issues
Can't install (pip install awscurl fails)
When I run command from the Readme under the python:3.9.7-alpine3.14 docker image, I'm getting a lot of the following errors:
Downloading cryptography-3.1.tar.gz (534 kB)
|████████████████████████████████| 534 kB 10.0 MB/s
Installing build dependencies ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /tmp/pip-standalone-pip-huuba34w/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-e1ixhxe4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"''
cwd: None
Complete output (118 lines):
Collecting setuptools>=40.6.0
Using cached setuptools-58.0.0-py3-none-any.whl (816 kB)
Collecting wheel
Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Collecting cffi!=1.11.3,>=1.8
Using cached cffi-1.14.6.tar.gz (475 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: cffi
Building wheel for cffi (setup.py): started
Building wheel for cffi (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-l4232b8e
cwd: /tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/
Complete output (45 lines):
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.libera.chat.)
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/c
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Failed building wheel for cffi
Running setup.py clean for cffi
Failed to build cffi
Installing collected packages: pycparser, wheel, setuptools, cffi
Running setup.py install for cffi: started
Running setup.py install for cffi: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8kbxbpwh/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-e1ixhxe4/overlay --compile --install-headers /tmp/pip-build-env-e1ixhxe4/overlay/include/python3.9/cffi
cwd: /tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/
Complete output (45 lines):
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.libera.chat.)
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/c
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"'; __file__='"'"'/tmp/pip-install-y1em43o4/cffi_e880ce50004945e28d75ff9d844d19f7/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-8kbxbpwh/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-e1ixhxe4/overlay --compile --install-headers /tmp/pip-build-env-e1ixhxe4/overlay/include/python3.9/cffi Check the logs for full command output.
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/12/be/c9cc7d7ab71dbcc9e4e517ead0cdd48e8c9a48d7b8bdddb738e90d08279a/cryptography-3.1.tar.gz#sha256=26409a473cc6278e4c90f782cd5968ebad04d3911ed1c402fc86908c17633e08 (from https://pypi.org/simple/cryptography/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*). Command errored out with exit status 1: /usr/local/bin/python /tmp/pip-standalone-pip-huuba34w/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-e1ixhxe4/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.
Downloading cryptography-3.0.tar.gz (534 kB)
|████████████████████████████████| 534 kB 17.6 MB/s
Installing build dependencies ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /tmp/pip-standalone-pip-0z1qgh1k/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xf8ws2lf/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"''
cwd: None
Complete output (118 lines):
Collecting setuptools>=40.6.0
Using cached setuptools-58.0.0-py3-none-any.whl (816 kB)
Collecting wheel
Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Collecting cffi!=1.11.3,>=1.8
Using cached cffi-1.14.6.tar.gz (475 kB)
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: cffi
Building wheel for cffi (setup.py): started
Building wheel for cffi (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ws9rz4du
cwd: /tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/
Complete output (45 lines):
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.libera.chat.)
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/c
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Failed building wheel for cffi
Running setup.py clean for cffi
Failed to build cffi
Installing collected packages: pycparser, wheel, setuptools, cffi
Running setup.py install for cffi: started
Running setup.py install for cffi: finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-h7ub2wex/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-xf8ws2lf/overlay --compile --install-headers /tmp/pip-build-env-xf8ws2lf/overlay/include/python3.9/cffi
cwd: /tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/
Complete output (45 lines):
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.libera.chat.)
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/c
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
error: command 'gcc' failed: No such file or directory
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"'; __file__='"'"'/tmp/pip-install-xup4hebv/cffi_02c44a64f72547d180b7185d0b042307/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-h7ub2wex/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-xf8ws2lf/overlay --compile --install-headers /tmp/pip-build-env-xf8ws2lf/overlay/include/python3.9/cffi Check the logs for full command output.
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/bf/ac/552fc8729d90393845cc3a2062facf4a89dcbe206fa78771d60ddaae7554/cryptography-3.0.tar.gz#sha256=8e924dbc025206e97756e8903039662aa58aa9ba357d8e1d8fc29e3092322053 (from https://pypi.org/simple/cryptography/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*). Command errored out with exit status 1: /usr/local/bin/python /tmp/pip-standalone-pip-0z1qgh1k/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xf8ws2lf/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.8,!=1.11.3; platform_python_implementation != '"'"'PyPy'"'"'' Check the logs for full command output.
If I install packages from the dockerfile, I'm getting the following:
/ # pip install awscurl
Collecting awscurl
Using cached awscurl-0.24-py3-none-any.whl (8.4 kB)
Collecting urllib3[secure]
Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting requests
Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting configparser
Using cached configparser-5.0.2-py3-none-any.whl (19 kB)
Collecting configargparse
Using cached ConfigArgParse-1.5.2-py3-none-any.whl (20 kB)
Collecting charset-normalizer~=2.0.0
Using cached charset_normalizer-2.0.4-py3-none-any.whl (36 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting idna<4,>=2.5
Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting cryptography>=1.3.4
Using cached cryptography-3.4.8.tar.gz (546 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Collecting pyOpenSSL>=0.14
Using cached pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB)
Collecting cffi>=1.12
Using cached cffi-1.14.6-cp39-cp39-linux_x86_64.whl
Collecting pycparser
Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting six>=1.5.2
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Building wheels for collected packages: cryptography
Building wheel for cryptography (PEP 517) ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmprjoz7zfw
cwd: /tmp/pip-install-d5d5bd9r/cryptography_7d40124d94904d0da3fabbd0ab736a9a
Complete output (165 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/cryptography
copying src/cryptography/utils.py -> build/lib.linux-x86_64-3.9/cryptography
copying src/cryptography/exceptions.py -> build/lib.linux-x86_64-3.9/cryptography
copying src/cryptography/fernet.py -> build/lib.linux-x86_64-3.9/cryptography
copying src/cryptography/__about__.py -> build/lib.linux-x86_64-3.9/cryptography
copying src/cryptography/__init__.py -> build/lib.linux-x86_64-3.9/cryptography
creating build/lib.linux-x86_64-3.9/cryptography/hazmat
copying src/cryptography/hazmat/_types.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
copying src/cryptography/hazmat/_der.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
copying src/cryptography/hazmat/_oid.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
copying src/cryptography/hazmat/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat
creating build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/certificate_transparency.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/ocsp.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/name.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/oid.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/base.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/general_name.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/extensions.py -> build/lib.linux-x86_64-3.9/cryptography/x509
copying src/cryptography/x509/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/x509
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
copying src/cryptography/hazmat/backends/interfaces.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
copying src/cryptography/hazmat/backends/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/padding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/keywrap.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/cmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/poly1305.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/constant_time.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/_cipheralgorithm.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/_asymmetric.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/hashes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/_serialization.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
copying src/cryptography/hazmat/primitives/hmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings
copying src/cryptography/hazmat/bindings/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/x509.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/ocsp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/cmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/decode_asn1.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/poly1305.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/rsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/aead.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/ciphers.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/x448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/ed25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/dh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/ec.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/encode_asn1.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/ed448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/dsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/hashes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/backend.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/x25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
copying src/cryptography/hazmat/backends/openssl/hmac.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/backends/openssl
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
copying src/cryptography/hazmat/primitives/ciphers/modes.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
copying src/cryptography/hazmat/primitives/ciphers/aead.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
copying src/cryptography/hazmat/primitives/ciphers/base.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
copying src/cryptography/hazmat/primitives/ciphers/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
copying src/cryptography/hazmat/primitives/ciphers/algorithms.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/ciphers
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/padding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/rsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/x448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/ed25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/dh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/ec.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/ed448.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/dsa.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
copying src/cryptography/hazmat/primitives/asymmetric/x25519.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/asymmetric
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
copying src/cryptography/hazmat/primitives/serialization/pkcs12.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
copying src/cryptography/hazmat/primitives/serialization/ssh.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
copying src/cryptography/hazmat/primitives/serialization/pkcs7.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
copying src/cryptography/hazmat/primitives/serialization/base.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
copying src/cryptography/hazmat/primitives/serialization/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/serialization
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
copying src/cryptography/hazmat/primitives/twofactor/totp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
copying src/cryptography/hazmat/primitives/twofactor/utils.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
copying src/cryptography/hazmat/primitives/twofactor/hotp.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
copying src/cryptography/hazmat/primitives/twofactor/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/twofactor
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/pbkdf2.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/scrypt.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/x963kdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/kbkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/concatkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
copying src/cryptography/hazmat/primitives/kdf/hkdf.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/primitives/kdf
creating build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
copying src/cryptography/hazmat/bindings/openssl/binding.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
copying src/cryptography/hazmat/bindings/openssl/_conditional.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
copying src/cryptography/hazmat/bindings/openssl/__init__.py -> build/lib.linux-x86_64-3.9/cryptography/hazmat/bindings/openssl
running egg_info
writing src/cryptography.egg-info/PKG-INFO
writing dependency_links to src/cryptography.egg-info/dependency_links.txt
writing requirements to src/cryptography.egg-info/requires.txt
writing top-level names to src/cryptography.egg-info/top_level.txt
reading manifest file 'src/cryptography.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'docs/_build'
warning: no previously-included files found matching 'vectors'
warning: no previously-included files matching '*' found under directory 'vectors'
warning: no previously-included files matching '*' found under directory '.github'
warning: no previously-included files found matching 'release.py'
warning: no previously-included files found matching '.coveragerc'
warning: no previously-included files found matching 'codecov.yml'
warning: no previously-included files found matching '.readthedocs.yml'
warning: no previously-included files found matching 'dev-requirements.txt'
warning: no previously-included files found matching 'tox.ini'
warning: no previously-included files found matching 'mypy.ini'
warning: no previously-included files matching '*' found under directory '.zuul.d'
warning: no previously-included files matching '*' found under directory '.zuul.playbooks'
adding license file 'LICENSE'
adding license file 'LICENSE.APACHE'
adding license file 'LICENSE.BSD'
adding license file 'LICENSE.PSF'
writing manifest file 'src/cryptography.egg-info/SOURCES.txt'
copying src/cryptography/py.typed -> build/lib.linux-x86_64-3.9/cryptography
running build_ext
generating cffi module 'build/temp.linux-x86_64-3.9/_padding.c'
creating build/temp.linux-x86_64-3.9
generating cffi module 'build/temp.linux-x86_64-3.9/_openssl.c'
running build_rust
=============================DEBUG ASSISTANCE=============================
If you are seeing a compilation error please try the following steps to
successfully install cryptography:
1) Upgrade to the latest pip and try again. This will fix errors for most
users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
2) Read https://cryptography.io/en/latest/installation.html for specific
instructions for your platform.
3) Check our frequently asked questions for more information:
https://cryptography.io/en/latest/faq.html
4) Ensure you have a recent Rust toolchain installed:
https://cryptography.io/en/latest/installation.html#rust
5) If you are experiencing issues with Rust for *this release only* you may
set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
=============================DEBUG ASSISTANCE=============================
error: can't find Rust compiler
If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.
To update pip, run:
pip install --upgrade pip
and then retry package installation.
If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
This package requires Rust >=1.41.0.
----------------------------------------
ERROR: Failed building wheel for cryptography
Failed to build cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly
What I'm doing wrong?
Better setup instructions required I guess.
Allow requests to not follow redirects
We have an API Gateway Resource path that redirects to a pre-signed s3 URL. When I make that request with awscurl, it errors out with:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
and the pre-signed s3 URL in the python Traceback
...
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://our-bucket-name.s3.amazonaws.com/...
This makes sense because the signed request is not for the final URL. I'm guessing this will happen with any service that responds with a redirect.
I think it would make sense to add support for not following redirects as a command line flag.
It might even make sense to follow the pattern from curl and only follow redirects if the caller explicitly request it, i.e. curl -L https://my.domain/redirecting.
Happy to take crack at the implementation I already have a rough working implementation, but want your input in the right approach before I make a PR.
Does awscurl not support fetching POST data from a json file?
Hi All,
I am trying to make a request to my API hosted on Amazon API Gateway. I put the request object in a file called request.json and am attempting to make a POST request as follows:
awscurl -v --service execute-api -X POST -d @"bin/request.json" "${endpoint}adduser"
Since I have enabled the verbose flag, I am seeing data field of my request as 'data': '@bin/request.json' . Any help is appreciated.
awscurl in python script ?
is it possible to put it in python script: run command and get output to JSON file (without subprocess)
import requests
headers = {
'Content-Type': 'application/json',
}
params = (
('Action', 'ListUsers'),
('Version', '2010-05-08'),
)
response = requests.get('https://iam.amazonaws.com/', headers=headers, params=params)
with open('1.json', 'w') as outfile:
outfile.write(response.content)
and getting
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
<Error>
<Type>Sender</Type>
<Code>MissingAuthenticationToken</Code>
<Message>Request is missing Authentication Token</Message>
</Error>
<RequestId>bb3695a3-399f-11e8-b6b6-17d7b8ab932d</RequestId>
</ErrorResponse>
Create Release Version
Now that this has been working for quite some time, could you make a release 1.0 or tag the code? I'd like to reference a version that is stable.
PR #77 returns raw bytes as output, resulting output that is not parsable as json in current(0.19) release when using python 3
When using 0.19 awscurl with python 3, encoding response.text results in a byte object being returned. Printing that directly results in the output being prefixed with b' (see screenshots below), and thus json output cannot be parsed directly from the response. In Python 2 this is not an issue. I could also see this being an issue for (say) an xml based response as well (though I have not tested that).
My use case: I'm currently invoking awscurl in a unix pipe, where the result of awscurl is piped to jq for further processing (in CI); updating to the latest version broke the pipe. For the moment I've pinned the version in CI at 0.17
Python 3 screenshot
Python 2 screenshot
Does not support EC2 Instance Profiles
First off, very neat project!
In regards to func load_aws_config. Is there a specific reason you use a ConfigParser?
If you used boto3 you'd end up supporting most credential providers implicitly.
boto3.Session().get_credentials().get_frozen_credentials()
You could construct the session object with a profile and/or keys if provided, or otherwise allow boto to default to its own chain of finding credentials. This would in turn allow it to work from an EC2 instance with an IAM instance profile while maintaining the existing functionality.
Does not support AssumeRole or MFA profiles
AWSCurl does not support profiles that assume role (for cross-account access) or MFA.
Add/update contributing instructions
The project should have documentation to bootstrap new contributors, covering PR/build/etc steps.
Doesn't recognize AWS_WEB_IDENTITY_TOKEN_FILE, botocore.exceptions.ProfileNotFound: The config profile (default) could not be found
In AWS EKS environment with IRSA. In version 0.22.
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
awscurl -X PUT https://example.com
Traceback (most recent call last):
File "/usr/local/bin/awscurl", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 504, in main
inner_main(sys.argv[1:])
File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 469, in inner_main
args.access_key, args.secret_key, args.session_token = load_aws_config(args.access_key,
File "/usr/local/lib/python3.9/site-packages/awscurl/awscurl.py", line 393, in load_aws_config
cred = session.get_credentials()
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 441, in get_credentials
self._credentials = self._components.get_component(
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 937, in get_component
self._components[name] = factory()
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 151, in _create_credential_resolver
return botocore.credentials.create_credential_resolver(
File "/usr/local/lib/python3.9/site-packages/botocore/credentials.py", line 64, in create_credential_resolver
metadata_timeout = session.get_config_variable('metadata_service_timeout')
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 251, in get_config_variable
return self.get_component('config_store').get_config_variable(
File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 313, in get_config_variable
return provider.provide()
File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 410, in provide
value = provider.provide()
File "/usr/local/lib/python3.9/site-packages/botocore/configprovider.py", line 471, in provide
scoped_config = self._session.get_scoped_config()
File "/usr/local/lib/python3.9/site-packages/botocore/session.py", line 351, in get_scoped_config
raise ProfileNotFound(profile=profile_name)
botocore.exceptions.ProfileNotFound: The config profile (default) could not be found
Works fine in 0.21
Exception occurred with v0.18 in python 2.7 & python 3.7.1
Hi,
we just tried to use v0.18 and got this exception:
Traceback (most recent call last):
File "/usr/local/bin/awscurl", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/awscurl/awscurl.py", line 491, in main
print(response.headers.encode('utf-8'), end='\n\n')
AttributeError: 'CaseInsensitiveDict' object has no attribute 'encode'
awscurl did not work with Lambda
Hi,
I installed the latest version of awscurl, 0.20. I attempt to invoke my Lambda function and it returned some error messages that I can't make out what went wrong.
The same request worked fine in Postman.
$ awscurl --service lambda --region eu-west-1 --profile default 'https://lambda.<region>.amazonaws.com/2015-03-31/functions/<function-name>/invocations?Qualifier=$LATEST' -H 'X-Amz-Invocation-Type:RequestResponse' -H 'X-Amz-Log-Type:Tail' -H 'Content-Type:application/json' -d '{"message":"Whats up."}'
The error message is:
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/bin/awscurl", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/awscurl/awscurl.py", line 500, in main
inner_main(sys.argv[1:])
File "/usr/local/lib/python3.7/site-packages/awscurl/awscurl.py", line 463, in inner_main
headers = {k: v for (k, v) in map(lambda s: s.split(": "), args.header)}
File "/usr/local/lib/python3.7/site-packages/awscurl/awscurl.py", line 463, in <dictcomp>
headers = {k: v for (k, v) in map(lambda s: s.split(": "), args.header)}
ValueError: not enough values to unpack (expected 2, got 1)
The doc of the API can be found here: https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html
Some services require the AWS Security Token as part of the signature
Just encountered an issue using awscli against Amazon ElasticSearch service. Turns out some services require the x-amz-security-token header to be part of the signature, others don't.
I believe it's safe to just always include it in the signed payload when it's set.
From: Amazon V4 Signatures:
You can use temporary security credentials provided by the AWS Security Token Service (AWS STS) to sign a request. The process is the same as using long-term credentials, but requires an additional HTTP header or query string parameter for the security token. The name of the header or query string parameter is X-Amz-Security-Token, and the value is the session token (the string you received from AWS STS when you obtained temporary security credentials).
When you add the X-Amz-Security-Token parameter to the query string, some services require that you include this parameter in the canonical (signed) request. For other services, you add this parameter at the end, after you calculate the signature. For details, see the API reference documentation for that service.
Add docker instructions
https://github.com/allthings/awscurl has nice[er] docker instructions -- add similar level of details to https://github.com/okigan/awscurl/
TypeError: 'map' object is not subscriptable
After install awscurl on macOS Sierra, I get:
Traceback (most recent call last):
File "/usr/local/bin/awscurl", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 311, in main
args.security_token
File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 140, in make_request
canonical_querystring = normalize_query_string(query)
File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 227, in normalize_query_string
for p in sorted(kv))
File "/usr/local/lib/python3.6/site-packages/awscurl/awscurl.py", line 227, in <genexpr>
for p in sorted(kv))
TypeError: 'map' object is not subscriptable
Is there a way to fix it? Thanks.
A release with print-content-as-text?
It looks like the current pypi version of this package (0.11) still has "print(r.content)". I was using this with Amazon SageMaker and the results were coming back as a byte array ("b'...'") instead of text. When I do a git clone and run the latest version, it works great.
Could we get a pypi release of the latest version? Thanks!
(P.S. thanks for this really useful tool)
Needs support for urllib 1.23
awscurl currently restricts urllib to <1.23>=1.21.1. Can this restriction be relaxed or awscurl updated appropriately to support the new version of urllib?
Proposal: Allow modifying signature hostname
I'm trying to run awscurl to perform requests to the ES service through an SSH tunnel. The tunnel opens a local port connected to the ES service 443 port on the API endpoint. The tunnel is required to connect to a service not available to the public. The issue is that awscurl creates a signature using the URL passed in CLI. But the service hostname and port are different from the local URL. I need to make a signature using the real API endpoint, not using the local one. It would be nice to add this feature.
Do you have an example of a POST or a PUT command
If you are uploading data to S3, how would I do that? This right here is my guess for just making a txt file that has Hello There, but it is not working.
awscurl -X 'POST' -d "Hello There" --service $SERVICE --region $REGION --access_key $ACCESS_KEY --secret_key $SECRET_KEY $URL
Support AWS SSO
Does awscurl already support AWS SSO?
I tried just now and I seem to be missing something.
I'm using the version installed today directly from Github.
I have a config file with a profile that uses AWS SSO to get temporary credentials.
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
[profile example]
sso_start_url = https://d-1111111111.awsapps.com/start
sso_region = eu-west-1
sso_account_id = 111111111111
sso_role_name = AdministratorAccess
region = eu-west-1
output = jsonThe profile works with AWS CLI v2.
$ aws sts get-caller-identity --profile example
{
"UserId": "AROAAAAAAAAAAAEXAMPLE:example",
"Account": "111111111111",
"Arn": "arn:aws:sts::111111111111:assumed-role/AWSReservedSSO_AdministratorAccess_1111111111111111/example"
}
But when I try to use it with awscurl I get a message about the token being expired.
$ export AWS_PROFILE=example
$ awscurl --region=eu-west-1 --service=s3 http://example.s3.eu-west-1.amazonaws.com/
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Token-0>IQoJ[...]96F0=</Token-0><RequestId>D[...]W</RequestId><HostId>GeFN[...]kfUTs=</HostId></Error>
Traceback (most recent call last):
File "/home/isme/.local/bin/awscurl", line 8, in <module>
sys.exit(main())
File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 500, in main
inner_main(sys.argv[1:])
File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/awscurl/awscurl.py", line 494, in inner_main
response.raise_for_status()
File "/home/isme/.local/pipx/venvs/awscurl/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://example.s3.eu-west-1.amazonaws.com/
AWS_SESSION_TOKEN not taken in account
Hi,
it seems that awscurl is not taking in account AWS_SESSION_TOKEN environment variable (when an user is assuming a role, in order to have some rights).
Is it possible to calculate the aws signature version 4 with this AWS_SESSION_TOKEN instead of only with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY?
Without taking in account this environment variable (created dynamically when an user is assuming a role), we obtain a 403 http error code.
Thanks
python broken after `brew install awscurl`
Title says it all.
I think awscurl required a different version of python than I had on my system.
After running brew install awscurl, I saw:
==> Installing dependencies for awscurl: [email protected], readline, xz and [email protected]
...
Python has been installed as
/usr/local/opt/[email protected]/bin/python3
You can install Python packages with
/usr/local/opt/[email protected]/bin/pip3 install <package>
They will install into the site-package directory
/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages
See: https://docs.brew.sh/Homebrew-and-Python
[email protected] is keg-only, which means it was not symlinked into /usr/local,
because this is an alternate version of another formula.
If you need to have [email protected] first in your PATH run:
echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.bash_profile
For compilers to find [email protected] you may need to set:
export LDFLAGS="-L/usr/local/opt/[email protected]/lib"
For pkg-config to find [email protected] you may need to set:
export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig"Now when I run simple tests with the AWS CLI, I see:
$ aws iam get-user
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512
Traceback (most recent call last):
File "/usr/local/bin/aws", line 19, in <module>
import awscli.clidriver
File "/usr/local/lib/python2.7/site-packages/awscli/clidriver.py", line 17, in <module>
import botocore.session
File "/usr/local/lib/python2.7/site-packages/botocore/session.py", line 28, in <module>
import botocore.configloader
File "/usr/local/lib/python2.7/site-packages/botocore/configloader.py", line 19, in <module>
from botocore.compat import six
File "/usr/local/lib/python2.7/site-packages/botocore/compat.py", line 27, in <module>
from urllib3 import exceptions
File "/usr/local/lib/python2.7/site-packages/urllib3/__init__.py", line 7, in <module>
from .connectionpool import (
File "/usr/local/lib/python2.7/site-packages/urllib3/connectionpool.py", line 30, in <module>
from .connection import (
File "/usr/local/lib/python2.7/site-packages/urllib3/connection.py", line 39, in <module>
from .util.ssl_ import (
File "/usr/local/lib/python2.7/site-packages/urllib3/util/__init__.py", line 6, in <module>
from .ssl_ import (
File "/usr/local/lib/python2.7/site-packages/urllib3/util/ssl_.py", line 8, in <module>
from hashlib import md5, sha1, sha256
ImportError: cannot import name md5UPDATE: apparently its not just the aws CLI. It's other things...
$ which pip
/usr/local/bin/pip
$ pip -v
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 147, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512
Traceback (most recent call last):
File "/usr/local/bin/pip", line 11, in <module>
load_entry_point('pip==19.0.3', 'console_scripts', 'pip')()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
return ep.load()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2411, in load
return self.resolve()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/local/lib/python2.7/site-packages/pip/_internal/__init__.py", line 19, in <module>
from pip._vendor.urllib3.exceptions import DependencyWarning
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/__init__.py", line 8, in <module>
from .connectionpool import (
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/connectionpool.py", line 29, in <module>
from .connection import (
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/connection.py", line 38, in <module>
from .util.ssl_ import (
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/__init__.py", line 6, in <module>
from .ssl_ import (
File "/usr/local/lib/python2.7/site-packages/pip/_vendor/urllib3/util/ssl_.py", line 8, in <module>
from hashlib import md5, sha1, sha256
ImportError: cannot import name md5Plans to add multipart requests support?
What do you think about adding multipart/form-data requests that are made with -F option in curl?
It would be awesome for testing endpoints for file uploads that are located on API Gateways with AWS Auth enabled.
Example [aws]curl command:
awscurl \
-X POST \
-H "Content-Type: multipart/form-data" \
-F "field1=value1" \
-F 'field2={"some":"json"};type=application/json' \
-F "file=@/path/to/some/picture.png" \
https://redacted.execute-api.us-west-2.amazonaws.com/pathExpected request body:
--------------------------d9afcbc8f3fe869a
Content-Disposition: form-data; name="field1"
value1
--------------------------d9afcbc8f3fe869a
Content-Disposition: form-data; name="field2"
Content-Type: application/json
{"some":"json"}
--------------------------d9afcbc8f3fe869a
Content-Disposition: form-data; name="file"; filename="picture.png"
Content-Type: image/png
<binary data>
--------------------------d9afcbc8f3fe869a--
`--data` with UTF-8 data raises UnicodeDecodeError
The default encoding in Python 2.x is ascii so we are seeing the following backtrace when supplying UTF-8 arguments to the --data flag:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/awscurl/awscurl.py", line 339, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/awscurl/awscurl.py", line 334, in main
args.security_token or args.session_token
File "/usr/local/lib/python2.7/site-packages/awscurl/awscurl.py", line 178, in make_request
payload_hash = sha256_hash(data)
File "/usr/local/lib/python2.7/site-packages/awscurl/awscurl.py", line 104, in sha256_hash
return hashlib.sha256(val.encode('utf-8')).hexdigest()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 93: ordinal not in range(128)
The fix seems to be to avoid the implicit call to decode(<default encoding>) when dealing with val being of type str, by explicitly decoding val as utf-8 first so we end of with a unicode object to feed to encode('utf-8'), like it expects.
Hence the fix looks like this: change
return hashlib.sha256(val.encode('utf-8')).hexdigest()
to
return hashlib.sha256(val.decode('utf-8').encode('utf-8')).hexdigest()
Improve tests for load_aws_config
The load_aws_config function is responsible for getting credentials that are used to sign the request.
It's quite complex: a mixture of custom credential parsing and calls to botocore.
The PR #116 (add support for AWS SSO profiles) caused a rergression in #122 (instance credentials are not being loaded).
The tests for the load_aws_config function don't cover all the use cases, which makes it hard to add new functionality without breaking what already works.
Doesn't handle URLs encoded with apostrophe correctly %27
Called with URL that passes "Mike's 1043" encoded as Mike%27s%201043
Example URL
https://xxxxxxxxx.execute-api.us-west-2.amazonaws.com/prod/add?user=XX&data=Mike%27s%201043
'Response code: 400\n'
{"message": "Could not parse request body into json: Unrecognized character escape ''' (code 39)\n at [Source: [B@742f0d86; line: 9, column: 34]"}
Traceback (most recent call last):
File "/usr/local/bin/awscurl", line 11, in
sys.exit(main())
File "/Users/Znn/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 276, in main
args.security_token
File "/Users/Znn/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 188, in make_request
return send_request(uri, data, headers, method)
File "/Users/Znn/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 218, in send_request
r.raise_for_status()
File "/Library/Python/2.7/site-packages/requests/models.py", line 862, in raise_for_status
raise HTTPError(http_error_msg, response=self)
awscurl doesn't work for AWS Elasticsearch when query contains CJK multi-byte Unicode characters
It works fine if domain allows open-access
I'm confident that my query works fine if Elasticsearch domain allows open-access and I use curl command to issue query.
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-2021> curl -XGET "https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty" -H "Content-Type: application/json" -d "@search-search-ngram-and-kuromoji-2.json"
{
"took" : 163,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 144.35374,
"hits" : [
{
"_index" : "search-ngram-and-kuromoji",
"_type" : "_doc",
"_id" : "ucNKC3gBNKbjtYmSAtQp",
"_score" : 144.35374,
"_source" : {
"section_url" : "001.html#topic_i5x_lkz_bgb"
},
"highlight" : {
"section_text" : [
"突や衝突に近い状態（<em>SRSエアバッグの作動および路上障害物との接触</em>など）が発生した時に車"
],
"section" : [
"イベントデータレコーダー"
]
}
}
]
}
}I attached the search query: search-search-ngram-and-kuromoji-2.json zipped.
search-search-ngram-and-kuromoji-2.zip
If I launch awscurl without specifying --data-binary, no search result returned.
I have changed domain access policy to exhibit open access and allow one IAM role named ESFullAccess. Also ESFullAccess has trusted IAM user called ESProgram.
The command-line Windows Power shell awsescurl.ps1 zipped.
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-2021> ./awsescurl.ps1 -X GET -d "@search-search-ngram-and-kuromoji-2.json" "https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty"
{'access_key': 'ASIAUPPPMJZZ2PAQ6B74',
'data': '@search-search-ngram-and-kuromoji-2.json',
'data_binary': False,
'header': ['Content-Type: application/json'],
'include': False,
'insecure': True,
'profile': 'default',
'region': 'yyyyyyyyy',
'request': 'GET',
'secret_key': 'IpYWCqt3OScsBxA0/dOmVrSFWN7NfEbX1VyQwye9',
'security_token': 'FwoGZXIvYXdzEMr//////////wEaDKOZeknTfG/fc5Ng+iKwAXTv+jazLkF0NMNGPiSYtytG3WqA1U1cUCU4ElfcHNixm+LFTOphsYQh9iY7xFO9cBh+iRrvF6qB10IeG7Ta+PJtcLZnzOUfOGE8w6a94YqpWciIRQ5CEAL3UDeYNru0IGeulJxVSzHaTRs8crJ7d3DOqSRDVGSKXfNpCQjzOKXwr/nam3JAkPGyyd4u2B8iWhOmPl9lhxORchF5fBb84Npw8YlSGDFPoLSsBM+NjX8jKJLTuoIGMi0eGpiRN7Wo1OKjCQy2C1V5UNVr6u66Q/cY7r0RveYwNeZhaz4DI/ThXpMPD4A=',
'service': 'es',
'session_token': None,
'uri': 'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty',
'verbose': True}
'pretty='
('\n'
'CANONICAL REQUEST = GET\n'
'/search-ngram-and-kuromoji/_search\n'
'pretty=\n'
'host:search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com\n'
'x-amz-date:20210315T002554Z\n'
'x-amz-security-token:FwoGZXIvYXdzEMr//////////wEaDKOZeknTfG/fc5Ng+iKwAXTv+jazLkF0NMNGPiSYtytG3WqA1U1cUCU4ElfcHNixm+LFTOphsYQh9iY7xFO9cBh+iRrvF6qB10IeG7Ta+PJtcLZnzOUfOGE8w6a94YqpWciIRQ5CEAL3UDeYNru0IGeulJxVSzHaTRs8crJ7d3DOqSRDVGSKXfNpCQjzOKXwr/nam3JAkPGyyd4u2B8iWhOmPl9lhxORchF5fBb84Npw8YlSGDFPoLSsBM+NjX8jKJLTuoIGMi0eGpiRN7Wo1OKjCQy2C1V5UNVr6u66Q/cY7r0RveYwNeZhaz4DI/ThXpMPD4A=\n'
'\n'
'host;x-amz-date;x-amz-security-token\n'
'41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211')
('\n'
'STRING_TO_SIGN = AWS4-HMAC-SHA256\n'
'20210315T002554Z\n'
'20210315/yyyyyyyyy/es/aws4_request\n'
'09176940c4429ffeab8230044aa84447ff1f6d91a76b84e0b0069495e2538a75')
'\nHEADERS++++++++++++++++++++++++++++++++++++'
{'Authorization': 'AWS4-HMAC-SHA256 '
'Credential=ASIAUPPPMJZZ2PAQ6B74/20210315/yyyyyyyyy/es/aws4_request, '
'SignedHeaders=host;x-amz-date;x-amz-security-token, '
'Signature=66a1ec2e970b6d5d76628f4e5493d3c6d8b6edc3a6cabd1f54b868290ae81418',
'Content-Type': 'application/json',
'x-amz-content-sha256': '41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211',
'x-amz-date': '20210315T002554Z',
'x-amz-security-token': 'FwoGZXIvYXdzEMr//////////wEaDKOZeknTfG/fc5Ng+iKwAXTv+jazLkF0NMNGPiSYtytG3WqA1U1cUCU4ElfcHNixm+LFTOphsYQh9iY7xFO9cBh+iRrvF6qB10IeG7Ta+PJtcLZnzOUfOGE8w6a94YqpWciIRQ5CEAL3UDeYNru0IGeulJxVSzHaTRs8crJ7d3DOqSRDVGSKXfNpCQjzOKXwr/nam3JAkPGyyd4u2B8iWhOmPl9lhxORchF5fBb84Npw8YlSGDFPoLSsBM+NjX8jKJLTuoIGMi0eGpiRN7Wo1OKjCQy2C1V5UNVr6u66Q/cY7r0RveYwNeZhaz4DI/ThXpMPD4A='}
'\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++'
('Request URL = '
'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty')
'\nRESPONSE++++++++++++++++++++++++++++++++++++'
'Response code: 200\n'
{'Date': 'Mon, 15 Mar 2021 00:25:55 GMT', 'Content-Type': 'application/json; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'close', 'Access-Control-Allow-Origin': '*', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding, User-Agent'}
{
"took" : 47,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
If I launch awscurl with specifying --data-binary, following error occurs.
The command-line Windows Power shell awsescurl-db.ps1 zipped.
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-2021> ./awsescurl-db.ps1 -X GET -d "@search-search-ngram-and-kuromoji-2.json" "https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty"
{'access_key': 'ASIAUPPPMJZZUXODBZUO',
'data': '@search-search-ngram-and-kuromoji-2.json',
'data_binary': True,
'header': ['Content-Type: application/json'],
'include': False,
'insecure': True,
'profile': 'default',
'region': 'yyyyyyyyy',
'request': 'GET',
'secret_key': 'kfGOQpb/p4Ckm4vYMocotYu56K15BI+NIQq3VZei',
'security_token': 'FwoGZXIvYXdzEMr//////////wEaDJnWC1Gw7ePjDa6+fCKwASSe7Ro/gxR1BN7sts/Kn+RfFBJmYmjRZS+mvLClqjvg2OU/MdnZxJQeCsvxIP9Vzk5Ogdmq0tvp3uib3qVDQoEFHAfIf/AIPB/dRyp7KGLQpEXjjw5/v4/KRx3KDipAbH4aMFGjW0KrGymKsGiH1VMsuqfagWAl34tgclAtxoTQO+XGuxJTBafv1MEM9yAnMgN8S2zHtEEwNLOdJyb5CAQgTAAAJEgjSpjUgTpDsa4mKNLZuoIGMi0E3Ok7VjMeYV8WcpmwSwtMetlgSMIwvRRPArr8AYQVoEPINIzz6ufDE6IIquo=',
'service': 'es',
'session_token': None,
'uri': 'https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty',
'verbose': True}
'pretty='
Traceback (most recent call last):
File "c:\users\toshi\appdata\local\programs\python\python39\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\toshi\appdata\local\programs\python\python39\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\Scripts\awscurl.exe\__main__.py", line 7, in <module>
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 500, in main
inner_main(sys.argv[1:])
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 478, in inner_main
response = make_request(args.request,
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 100, in make_request
canonical_request, payload_hash, signed_headers = task_1_create_a_canonical_request(
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 200, in task_1_create_a_canonical_request
payload_hash = sha256_hash_for_binary_data(data) if data_binary else sha256_hash(data)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\utils.py", line 20, in sha256_hash_for_binary_data
return hashlib.sha256(val).hexdigest()
TypeError: Unicode-objects must be encoded before hashing
Fixing TypeError
I rarely know Python, but from error message "TypeError: Unicode-objects must be encoded before hashing", I've modified following code from:
https://github.com/okigan/awscurl/blob/master/awscurl/awscurl.py line 200
payload_hash = sha256_hash_for_binary_data(data) if data_binary else sha256_hash(data)to
payload_hash = sha256_hash(data)because hashing payload should be done with encoding UTF-8 if --data-binary is specified or not.
However modified awscurl still reports following error:
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-2021> ./awsescurl-db.ps1 -X GET -d "@search-search-ngram-and-kuromoji-2.json" "https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty"
{'access_key': 'ASIAUPPPMJZZ2XEUBLF7',
'data': '@search-search-ngram-and-kuromoji-2.json',
'data_binary': True,
'header': ['Content-Type: application/json'],
'include': False,
'insecure': False,
'profile': 'default',
'region': 'yyyyyyyyy',
'request': 'GET',
'secret_key': 'd9Fd2UQbWWuQSo1YzJeGU74pmi+ERNvI4gN7RDre',
'security_token': 'FwoGZXIvYXdzEMr//////////wEaDIGnfgGrBvi2QJemgiKwAWOZo7D/VAIvgtVf5gL+z+yF610K45iNCG7q6HHf+vIpxFJfiji+uIEJZXQMWOHTkVONfMvm5dBz8g3Ss8aTVQxjEkXTP3tw1MUPiq15qLiYW6ZeRvv9+kw6gkM2r2TIZm1k3oGOknzz8GTwQQoHySjj+zaDqNdHxN1l/rXMcyCdsaghuH12FvNsAmZV0TelhGJ3ceo9X6omS8BqRHCO5YYhs4DV2ApRI80yCsCOAofxKPTguoIGMi1nDJqy/cmXak2pL0VHU2puG5pbnjSV5kgaVD3oerY2bcbCzDu+t8ml+TplHXI=',
'service': 'es',
'session_token': None,
'uri': 'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty',
'verbose': True}
'pretty='
('\n'
'CANONICAL REQUEST = GET\n'
'/search-ngram-and-kuromoji/_search\n'
'pretty=\n'
'host:search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com\n'
'x-amz-date:20210315T005516Z\n'
'x-amz-security-token:FwoGZXIvYXdzEMr//////////wEaDIGnfgGrBvi2QJemgiKwAWOZo7D/VAIvgtVf5gL+z+yF610K45iNCG7q6HHf+vIpxFJfiji+uIEJZXQMWOHTkVONfMvm5dBz8g3Ss8aTVQxjEkXTP3tw1MUPiq15qLiYW6ZeRvv9+kw6gkM2r2TIZm1k3oGOknzz8GTwQQoHySjj+zaDqNdHxN1l/rXMcyCdsaghuH12FvNsAmZV0TelhGJ3ceo9X6omS8BqRHCO5YYhs4DV2ApRI80yCsCOAofxKPTguoIGMi1nDJqy/cmXak2pL0VHU2puG5pbnjSV5kgaVD3oerY2bcbCzDu+t8ml+TplHXI=\n'
'\n'
'host;x-amz-date;x-amz-security-token\n'
'41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211')
('\n'
'STRING_TO_SIGN = AWS4-HMAC-SHA256\n'
'20210315T005516Z\n'
'20210315/yyyyyyyyy/es/aws4_request\n'
'21e16f8efede645709fa4ad0d9b0ca272cf17e4679ab9e75048a315d01ba1d45')
'\nHEADERS++++++++++++++++++++++++++++++++++++'
{'Authorization': 'AWS4-HMAC-SHA256 '
'Credential=ASIAUPPPMJZZ2XEUBLF7/20210315/yyyyyyyyy/es/aws4_request, '
'SignedHeaders=host;x-amz-date;x-amz-security-token, '
'Signature=30d36e27a4962fd248cd58052403e1da72d077c214acd5d67ab984d425927c63',
'Content-Type': 'application/json',
'x-amz-content-sha256': '41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211',
'x-amz-date': '20210315T005516Z',
'x-amz-security-token': 'FwoGZXIvYXdzEMr//////////wEaDIGnfgGrBvi2QJemgiKwAWOZo7D/VAIvgtVf5gL+z+yF610K45iNCG7q6HHf+vIpxFJfiji+uIEJZXQMWOHTkVONfMvm5dBz8g3Ss8aTVQxjEkXTP3tw1MUPiq15qLiYW6ZeRvv9+kw6gkM2r2TIZm1k3oGOknzz8GTwQQoHySjj+zaDqNdHxN1l/rXMcyCdsaghuH12FvNsAmZV0TelhGJ3ceo9X6omS8BqRHCO5YYhs4DV2ApRI80yCsCOAofxKPTguoIGMi1nDJqy/cmXak2pL0VHU2puG5pbnjSV5kgaVD3oerY2bcbCzDu+t8ml+TplHXI='}
'\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++'
('Request URL = '
'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty')
Traceback (most recent call last):
File "C:\Users\toshi\AppData\Local\Programs\Python\Python39\Scripts\awscurl-script.py", line 33, in <module>
sys.exit(load_entry_point('awscurl==0.21', 'console_scripts', 'awscurl')())
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 499, in main
inner_main(sys.argv[1:])
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 477, in inner_main
response = make_request(args.request,
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 135, in make_request
return __send_request(uri, data, headers, method, verify)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\awscurl\awscurl.py", line 330, in __send_request
response = requests.request(method, uri, headers=headers, data=data, verify=verify)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\site-packages\urllib3\connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\http\client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "c:\users\toshi\appdata\local\programs\python\python39\lib\http\client.py", line 1300, in _send_request
body = _encode(body, 'body')
File "c:\users\toshi\appdata\local\programs\python\python39\lib\http\client.py", line 164, in _encode
raise UnicodeEncodeError(
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 322-357: Body ('繧ｨ繧｢繝舌ャ繧ｰ縺ｮ菴懷虚縺翫ｈ縺ｳ) is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.
Last challenge: fixing "UnicodeEncodeError"
According to the error message, I modified the following code:
https://github.com/okigan/awscurl/blob/master/awscurl/awscurl.py line 135
if data_binary:
return __send_request(uri, data, headers, method, verify)
else:
return __send_request(uri, data.encode('utf-8'), headers, method, verify)to:
if data_binary:
return __send_request(uri, data.encode('utf-8'), headers, method, verify)
else:
return __send_request(uri, data.encode('utf-8'), headers, method, verify)The error message has been vanished, but the query returns nothing.
PS C:\Users\toshi\OneDrive\Documents\ElasticSearch\command-2021> ./awsescurl-db.ps1 -X GET -d "@search-search-ngram-and-kuromoji-2.json" "https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty"
{'access_key': 'ASIAUPPPMJZZ5EN6WTOX',
'data': '@search-search-ngram-and-kuromoji-2.json',
'data_binary': True,
'header': ['Content-Type: application/json'],
'include': False,
'insecure': False,
'profile': 'default',
'region': 'yyyyyyyyy',
'request': 'GET',
'secret_key': 'F3kemHCI+/UNHAE19ms59sBi9XHcMaMN4vpb14as',
'security_token': 'FwoGZXIvYXdzEMv//////////wEaDPJUgKt3g2RxHCHWGiKwAaY60g/Lm1Hp48nED39tUP/34Ia2tqUT/Ljgqe2Rg2SBOGhAlTvQKpkypyNyAS8+vLFEmRGfw11UM6UOZvFmx3NeWi8g6zpV7QpeSCPFRKbwZLnSxTEn2r7n9p3QRXNNkNWlJSPCLDzTZecORr46FGYmdnX6ZkKL97p6dWpTiQ53O62FarMbUID90zReTEDKMEbM5n2oaS9hcfZCz8M7Zr7+zXC8C+5Pw01fi0TKGnYUKL7muoIGMi3hmp5Jw1uw3fZK4azfs7WUg+/EV2N5qN81sQyrF/sxH+X59xJ2cBjfeAYJvjQ=',
'service': 'es',
'session_token': None,
'uri': 'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty',
'verbose': True}
'pretty='
('\n'
'CANONICAL REQUEST = GET\n'
'/search-ngram-and-kuromoji/_search\n'
'pretty=\n'
'host:search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com\n'
'x-amz-date:20210315T010710Z\n'
'x-amz-security-token:FwoGZXIvYXdzEMv//////////wEaDPJUgKt3g2RxHCHWGiKwAaY60g/Lm1Hp48nED39tUP/34Ia2tqUT/Ljgqe2Rg2SBOGhAlTvQKpkypyNyAS8+vLFEmRGfw11UM6UOZvFmx3NeWi8g6zpV7QpeSCPFRKbwZLnSxTEn2r7n9p3QRXNNkNWlJSPCLDzTZecORr46FGYmdnX6ZkKL97p6dWpTiQ53O62FarMbUID90zReTEDKMEbM5n2oaS9hcfZCz8M7Zr7+zXC8C+5Pw01fi0TKGnYUKL7muoIGMi3hmp5Jw1uw3fZK4azfs7WUg+/EV2N5qN81sQyrF/sxH+X59xJ2cBjfeAYJvjQ=\n'
'\n'
'host;x-amz-date;x-amz-security-token\n'
'41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211')
('\n'
'STRING_TO_SIGN = AWS4-HMAC-SHA256\n'
'20210315T010710Z\n'
'20210315/yyyyyyyyy/es/aws4_request\n'
'0a5cea16f924e59ed1d26254b5239ff26a8dee5ee5583c6b51d6922f0eefb46e')
'\nHEADERS++++++++++++++++++++++++++++++++++++'
{'Authorization': 'AWS4-HMAC-SHA256 '
'Credential=ASIAUPPPMJZZ5EN6WTOX/20210315/yyyyyyyyy/es/aws4_request, '
'SignedHeaders=host;x-amz-date;x-amz-security-token, '
'Signature=d4243cc6a1264c77bc90743bca2e03cebe5a408b25b1bf238d8a638bea7bda9b',
'Content-Type': 'application/json',
'x-amz-content-sha256': '41bb10889ba70ce26a2cda05a6d33b4d057c9caef53d6986f252093450167211',
'x-amz-date': '20210315T010710Z',
'x-amz-security-token': 'FwoGZXIvYXdzEMv//////////wEaDPJUgKt3g2RxHCHWGiKwAaY60g/Lm1Hp48nED39tUP/34Ia2tqUT/Ljgqe2Rg2SBOGhAlTvQKpkypyNyAS8+vLFEmRGfw11UM6UOZvFmx3NeWi8g6zpV7QpeSCPFRKbwZLnSxTEn2r7n9p3QRXNNkNWlJSPCLDzTZecORr46FGYmdnX6ZkKL97p6dWpTiQ53O62FarMbUID90zReTEDKMEbM5n2oaS9hcfZCz8M7Zr7+zXC8C+5Pw01fi0TKGnYUKL7muoIGMi3hmp5Jw1uw3fZK4azfs7WUg+/EV2N5qN81sQyrF/sxH+X59xJ2cBjfeAYJvjQ='}
'\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++'
('Request URL = '
'https://search-tmtest-xxxxxxxxxxxxxxxxxxxxxxxxxx.yyyyyyyyy.es.amazonaws.com/search-ngram-and-kuromoji/_search?pretty')
'\nRESPONSE++++++++++++++++++++++++++++++++++++'
'Response code: 200\n'
{'Date': 'Mon, 15 Mar 2021 01:07:12 GMT', 'Content-Type': 'application/json; charset=UTF-8', 'Transfer-Encoding': 'chunked', 'Connection': 'close', 'Access-Control-Allow-Origin': '*', 'Content-Encoding': 'gzip', 'Vary': 'Accept-Encoding, User-Agent'}
{
"took" : 68,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
Do you have any ideas to fix this problem?
Regards,
Signature mismatch on non url-encoded uris
Hey 👋
That's a great tool! I am using this tool to perform requests against APIs which are protected by IAM-Authentication. Those apis contain query parameters which parameters may require url-encoding.
It looks like there is a problem with uris which are not url-encoded.
awscurl --service s3 https://awscurl-sample-bucket.s3.amazonaws.com/?query=para:meterResults in an error:
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>.../AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20210608T045117Z
Even though it looks like the underlying implementation correctly escapes the url (like for example curl does) when performing the request, but the signature is calculated for the unescaped uri and this results in a signature mismatch.
Running with escaped uri parameters works perfectly fine.
awscurl --service s3 https://awscurl-sample-bucket.s3.amazonaws.com/?query%3Dpara%3Ameter --region us-east-1Issue with executing URL with query string.
I'm facing an issue with executing a API Gateway URL.
./awscurl.py -X GET --region us-west-2 --access_key XXXXXXXXXXXX --secret_key XXXXXXXXXXXXXXXXX --service execute-api """https://xxxxx.execute-api.us-west-2.amazonaws.com/XX/YY/ZZZ?ids=[\{\"id\":\"aaaaa-bbb-cccc-dddd-1234567\"\}]&pid=aaaaa-bbb-cccc-dddd-1234567"""
I'm getting following response,
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}
Traceback (most recent call last):
File "awscurl.py", line 334, in
sys.exit(main())
File "awscurl.py", line 329, in main
args.security_token or args.session_token
File "awscurl.py", line 232, in make_request
return send_request(uri, data, headers, method)
File "awscurl.py", line 262, in send_request
r.raise_for_status()
File "/opt/VRTSitrpgw/venv27/lib/python2.7/site-packages/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url:
Any help is appreciated.
Publish awscurl to homebrew
It'd be nice to be able to grab awscurl via homebrew.
For instructions, see: https://docs.brew.sh/Formula-Cookbook
--profile does nothing
Looks like the --profile command line argument does not work, awscurl uses default profile from ~/.aws/credentials despite the agument.
Running with AWS_PROFILE environment variable works fine.
All awscurl command fails after latest release
It looks like the latest version doesn't work anymore when it's installed using pip.
pip install awscurl
The output of the install command is:
Collecting awscurl 15:40:56 Requirement already satisfied: configargparse in /usr/local/lib/python2.7/dist-packages (from awscurl) (1.5.3) 15:40:56 Requirement already satisfied: configparser in /usr/lib/python2.7/dist-packages (from awscurl) (3.5.0b2) 15:40:56 Requirement already satisfied: requests in /usr/local/lib/python2.7/dist-packages (from awscurl) (2.26.0) 15:40:56 Requirement already satisfied: urllib3[secure] in /usr/local/lib/python2.7/dist-packages (from awscurl) (1.26.7) 15:40:56 Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests->awscurl) (2021.10.8) 15:40:56 Requirement already satisfied: idna<3,>=2.5; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from requests->awscurl) (2.10) 15:40:56 Requirement already satisfied: chardet<5,>=3.0.2; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from requests->awscurl) (4.0.0) 15:40:56 Requirement already satisfied: pyOpenSSL>=0.14; extra == "secure" in /usr/local/lib/python2.7/dist-packages (from urllib3[secure]->awscurl) (21.0.0) 15:40:56 Requirement already satisfied: cryptography>=1.3.4; extra == "secure" in /usr/lib/python2.7/dist-packages (from urllib3[secure]->awscurl) (2.6.1) 15:40:56 Requirement already satisfied: ipaddress; python_version == "2.7" and extra == "secure" in /usr/lib/python2.7/dist-packages (from urllib3[secure]->awscurl) (1.0.17) 15:40:56 Requirement already satisfied: six>=1.5.2 in /usr/lib/python2.7/dist-packages (from pyOpenSSL>=0.14; extra == "secure"->urllib3[secure]->awscurl) (1.12.0) 15:40:56 Installing collected packages: awscurl 15:40:56 Successfully installed awscurl-0.26
All the commands fail with:
Traceback (most recent call last): 15:40:21 File "/usr/local/bin/awscurl", line 6, in <module> 15:40:21 from awscurl.__main__ import main 15:40:21 File "/usr/local/lib/python2.7/dist-packages/awscurl/__main__.py", line 6, in <module> 15:40:21 from .awscurl import main 15:40:21 File "/usr/local/lib/python2.7/dist-packages/awscurl/awscurl.py", line 19, in <module> 15:40:21 from urllib.parse import quote 15:40:21 ImportError: No module named parse
Note that I have installed the package on Mac and I didn't run in the previous errors so everything works just fine if the package is installed with Homebrew.
Thanks
syntax error in setup.py reported
Running setup.py install for awscurl
SyntaxError: ('invalid syntax', ('/usr/lib/python2.6/site-packages/awscurl/awscurl.py', 322, 23, ' headers = {k: v for (k, v) in map(lambda s: s.split(": "), args.header)}\n'))
Installing awscurl broke git-remote-codecommit
After installing awscurl git-remote-codecommit stopped working and I cannot get it to work again. When I try to do git pull for example I get the following error message: fatal: unable to find remote helper for 'codecommit'.
requests 2.19.1 has requirement urllib3<1.24,>=1.21.1, but you'll have urllib3 1.24 which is incompatible.
Please update setup.py to install urllib3<1.24,>=1.21.1, as it currently pulls in 1.24 and throws the following error:
File "/usr/local/bin/awscurl", line 7, in <module>
from awscurl.__main__ import main
File "/usr/local/lib/python2.7/dist-packages/awscurl/__main__.py", line 6, in <module>
from .awscurl import main
File "/usr/local/lib/python2.7/dist-packages/awscurl/awscurl.py", line 14, in <module>
import requests
File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 112, in <module>
from . import utils
File "/usr/local/lib/python2.7/dist-packages/requests/utils.py", line 26, in <module>
from ._internal_utils import to_native_string
File "/usr/local/lib/python2.7/dist-packages/requests/_internal_utils.py", line 11, in <module>
from .compat import is_py2, builtin_str, str
File "/usr/local/lib/python2.7/dist-packages/requests/compat.py", line 48, in <module>
from urllib3.packages.ordered_dict import OrderedDict
ImportError: No module named ordered_dict
Use AWS_SESSION_TOKEN instead of custom AWS_SECURITY_TOKEN env variable by default
The AWS documentation for the environment variables that specify credential components for their own aws cli specify that the STS tokens may be exposed via a AWS_SESSION_TOKEN environment variable.
The awscurl though expects the value in a different environment variable named AWS_SECURITY_TOKEN. I believe it should use the same environment variable as the Amazon CLI.
need to obfuscate AWS credentials used for testing
Return of make_request
Hi,
would it be possible to return the response object instead of returning a number (0, 1, etc.) for the success status of the request?
The reason behind that is, if one e.g. performs a GET with awscurl, it would be nice to have the response body available for further usage. Currently it is only piped to print() whcih is useless for automation...
BR & thanks
MB
Soften upper bound on urllib3?
Is the upper bound on urllib3 necessary?
Line 24 in 01cb53c
I built it locally using nixpkgs which has 1.25.3 using a custom patch and it seems to work fine, but I am not a python developer at all :)
ModuleNotFoundError
Hi
I just tested but
I run a Centos 6.10, python 3.6
~/test/testV2/awscurl-0.21/awscurl
$ ./awscurl.py --service s3 https://awscurl-sample-bucket.s3.amazonaws.com
Traceback (most recent call last):
File "./awscurl.py", line 19, in
from .utils import sha256_hash, sha256_hash_for_binary_data, sign
ModuleNotFoundError: No module named 'main.utils'; 'main' is not a package
May you help me?
regards
Commas in querystring are not URL encoded
I've been using awscurl to call API Gateway endpoints, and it has been working great for us up until now (thanks very much for making this!).
I recently started working on an endpoint that takes comma-delimited arrays, and found that the signature is not working properly in this case.
Example failure:
$ awscurl -v https://my-gateway-id.execute-api.us-east-1.amazonaws.com/live/my-path?arg=a,b,c
...
('\n'
'CANONICAL REQUEST = GET\n'
'/live/my-path\n'
'arg=a,b,c\n'
'host:my-gateway-id.execute-api.us-east-1.amazonaws.com\n'
'x-amz-date:20211202T033321Z\n'
'x-amz-security-token:IQoJb3JpZ2luX2VjEEsaCXVzLWVhc3QtMSJHMEUCIQC4m8wJYpB8Xch/XdJzdKHVbgT9refFSvzJ3CeRD9HYfQIgbWPPWfYZ6bYDupK/2SIbvkxruT8ssDn4Kl0qWcCxcF0qnAMIIxAAGgw3OTkyNDI4NjU0OTEiDALfa7r3/U7W1vFxMCr5AuxtcoYbB0iq58HexqEgxxe95yduKHkab03fDBNzwWl+ZG2ATBekfJLvi7Rhzr3U2uNckMXG32x3Nj/nirtwbqfecjqLj/7XfPhBNKgXzEuJ5bQ+oIUP3heXj5G8tJ/Yvwe0eRgghiLjarwzBzvMrA7u9o0jvIGTGHZqTMM0uUPkJrDWoGpWic73fvbe7GHG0j6Hcyy/agWmgLtG7NHh5GcAwdnZih8g9nzMOO5GL0wiu805NSo6xWc2BqnP/uyHQNRAxOc3Uxd+dYtqh6NosNvX3H5x8Gzj383hB3XDbsgZiRvvk+EMwlcZlsFMg+6Mg9abx516vpBgGaKsf5yoWhK4tnsvUnNsgF+2gwBjb93z6eKP7GdHyB3iirtetQcscwkFA6mt4cl1gz8FTNcfCN6BGPQhtUO+HXfpnOY8iapateIJnhjoibELSbcNYf9ql55Xm7bvOknGAJ6HCZvp3mpM/x/X7N6e+U0sdWtzVVFNt82cY/c3BknZML7UoI0GOqYB48W1RlChDe3KjQsHrKtTdgXBGdxU1IYbHdVSwzg9t3ZewwHEpVjfF77LQ0oSRD4ROuIC9cbAXXmr/QCPqpJpFaLtqFNlVCyLpAEygCLsXN+mxGQFj6laGmZR8nEVZrsnzZy1zhOEMOfhPZWQrULbaZeJ/ZtVWto3gnQ/Rl60H52jwf8ccU2EUijMLjlRgOH8MtAnS9WCbTKur07cxK4PXLVG21DTBw==\n'
'\n'
'host;x-amz-date;x-amz-security-token\n'
'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855')
...
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'GET\n/live/my-path\narg=a%2Cb%2Cc\nhost:my-gateway-id.execute-api.us-east-1.amazonaws.com\nx-amz-date:20211202T033321Z\nx-amz-security-token:IQoJb3JpZ2luX2VjEEsaCXVzLWVhc3QtMSJHMEUCIQC4m8wJYpB8Xch/XdJzdKHVbgT9refFSvzJ3CeRD9HYfQIgbWPPWfYZ6bYDupK/2SIbvkxruT8ssDn4Kl0qWcCxcF0qnAMIIxAAGgw3OTkyNDI4NjU0OTEiDALfa7r3/U7W1vFxMCr5AuxtcoYbB0iq58HexqEgxxe95yduKHkab03fDBNzwWl+ZG2ATBekfJLvi7Rhzr3U2uNckMXG32x3Nj/nirtwbqfecjqLj/7XfPhBNKgXzEuJ5bQ+oIUP3heXj5G8tJ/Yvwe0eRgghiLjarwzBzvMrA7u9o0jvIGTGHZqTMM0uUPkJrDWoGpWic73fvbe7GHG0j6Hcyy/agWmgLtG7NHh5GcAwdnZih8g9nzMOO5GL0wiu805NSo6xWc2BqnP/uyHQNRAxOc3Uxd+dYtqh6NosNvX3H5x8Gzj383hB3XDbsgZiRvvk+EMwlcZlsFMg+6Mg9abx516vpBgGaKsf5yoWhK4tnsvUnNsgF+2gwBjb93z6eKP7GdHyB3iirtetQcscwkFA6mt4cl1gz8FTNcfCN6BGPQhtUO+HXfpnOY8iapateIJnhjoibELSbcNYf9ql55Xm7bvOknGAJ6HCZvp3mpM/x/X7N6e+U0sdWtzVVFNt82cY/c3BknZML7UoI0GOqYB48W1RlChDe3KjQsHrKtTdgXBGdxU1IYbHdVSwzg9t3ZewwHEpVjfF77LQ0oSRD4ROuIC9cbAXXmr/QCPqpJpFaLtqFNlVCyLpAEygCLsXN+mxGQFj6laGmZR8nEVZrsnzZy1zhOEMOfhPZWQrULbaZeJ/ZtVWto3gnQ/Rl60H52jwf8ccU2EUijMLjlRgOH8MtAnS9WCbTKur07cxK4PXLVG21DTBw==\n\nhost;x-amz-date;x-amz-security-token\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20211202T033321Z\n20211202/us-east-1/execute-api/aws4_request\n38b3266af08b8f9f119ffed742e0063da244001dbdf2331c42f4f1af695e3e64'\n"}
Reviewing sigv4-create-canonical-request.html, it says:
Percent-encode all other characters with %XY, where X and Y are hexadecimal characters (0-9 and uppercase A-F). For example, the space character must be encoded as %20 (not using '+', as some encoding schemes do) and extended UTF-8 characters must be in the form %XY%ZA%BC.
In my case, I believe that means it should be sending arg=a%2Cb%2Cc instead of arg=a,b,c
Testing locally, the following patch appears to fix things for me, but I'm not sure how much fallout this might cause for other use cases:
diff --git a/awscurl/awscurl.py b/awscurl/awscurl.py
index cf09318..66a14c4 100755
--- a/awscurl/awscurl.py
+++ b/awscurl/awscurl.py
@@ -16,6 +16,7 @@ import configparser
import configargparse
import requests
from requests.structures import CaseInsensitiveDict
+from urllib.parse import urlencode
from .utils import sha256_hash, sha256_hash_for_binary_data, sign
@@ -315,9 +316,9 @@ def __normalize_query_string(query):
for s in query.split('&')
if len(s) > 0)
- normalized = '&'.join('%s=%s' % (p[0], p[1] if len(p) > 1 else '')
- for p in sorted(parameter_pairs))
- return normalized
+ normalized_pairs = [(p[0], p[1] if len(p) > 1 else '')
+ for p in [p for p in sorted(parameter_pairs)]]
+ return urlencode(normalized_pairs)
def __now():
Compile to binary
It would be nice to have a possibility to compile awscurl to a binary for optimal disk space usage especially in Docker containers. It's hard to pull all the dependencies required for the program to work. Particularly, I'm building an image for CI/CD that will have awscurl installed.
Check this if you run into issues with execute-api WebSockets
I got error when trying to execute:
awscurl --service execute-api -X POST -d "hello" https://spjhtm1qcd.execute-api.us-east-1.amazonaws.com/test/@connections/atntqfjloAMCECg=
Also tried:
awscurl --service execute-api -X POST -d "hello" https://spjhtm1qcd.execute-api.us-east-1.amazonaws.com/test/\@connections/atntqfjloAMCECg\=
and
awscurl --service execute-api -X POST -d "hello" "https://spjhtm1qcd.execute-api.us-east-1.amazonaws.com/test/@connections/atntqfjloAMCECg="
The error:
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/test/%40connections/atntqfjloAMCECg%3D\n\nhost:spjhtm1qcd.execute-api.us-east-1.amazonaws.com\nx-amz-date:20190603T171519Z\n\nhost;x-amz-date\n2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20190603T171519Z\n20190603/us-east-1/execute-api/aws4_request\nb013b0bc1c19a4ebc9da8d82a6122379899465e0cd4808187a4ca276e999c56a'\n"}
Traceback (most recent call last):
File "/Users/anton.titkov/Library/Python/2.7/bin/awscurl", line 11, in <module>
sys.exit(main())
File "/Users/anton.titkov/Library/Python/2.7/lib/python/site-packages/awscurl/awscurl.py", line 373, in main
r.raise_for_status()
File "/Users/anton.titkov/Library/Python/2.7/lib/python/site-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://spjhtm1qcd.execute-api.us-east-1.amazonaws.com/test/@connections/atntqfjloAMCECg=
URI encoding
Hi,
please update awscurl in order to allow non URL encoded charecters.
E.g.
GET <host>/_task/taskId:123:abc
the character ":" is not RFC conform and must be URL encoded.
I would recommend to change line 182 as:
requests.utils.quote(canonical_uri)
If URI is not encoded properly AWS returns an error indicating that signature check has failed.
If you change request to:
GET <host>/_task/taskId%3A123%3Aabc
AWS tries to validate against
GET <host>/_task/taskId%253A123%253Aabc
So AWS encodes the char % separately to %25.
BR
MB
Support retrieving credentials for an IAM Instance Profile via STS
When running in EC2, it's the preferred mechanism to get temporary credentials,
and it's supported by boto
Consider printing headers as json
When the -i flag to include headers in response is provided, a python dict representation (actually the requests custom case-indifferent dict) of the headers are printed. Looks deceptively like JSON at a quick glance.
If the headers were dumped as valid JSON, tools like jq could be used for digging into them, which aids in readability and using that header data as part of a pipeline.
Pip install fails on 0.14
Our CI failed today due to an update of awscurl.
Looking at https://pypi.org/project/awscurl/0.14/#history I see you've released 0.13 and 0.14 (not committed to repo yet, I think?), but when I try to install awscurl now (either locally or on CI server), I get this:
⚡ pip install awscurl
Collecting awscurl
Downloading https://files.pythonhosted.org/packages/0f/c3/82ffdac5671b1f89f369c0c33aebd4122df3f4ab3c11c802bab41c2edb37/awscurl-0.14.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/zl/dqtyzbh92jl4v53l2gpxfzyh0000gp/T/pip-build-BjlhKY/awscurl/setup.py", line 5, in <module>
with open("README.md", "r") as fh:
IOError: [Errno 2] No such file or directory: 'README.md'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/zl/dqtyzbh92jl4v53l2gpxfzyh0000gp/T/pip-build-BjlhKY/awscurl/
download/clone version is different from pip install
pip install does not included --profile parameters
urllib's InsecureRequestWarning printed to stdout when using the --insecure parameter
I'm using a custom hostname for the URL and thus the server certificate can't be verified. I'm passing the --insecure argument but it still prints the following warning:
/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'host.docker.internal'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
I think this should be suppressed.
"The request signature we calculated does not match the signature you provided" for a Private API with IAM_AUTH enabled using public DNS names
Created a private API with API Gateway and enabled AWS_IAM auth on a GET method for a resource. When invoking using the default Stage URL (using private DNS name), the request works fine as below -
awscurl -v https://API-ID.execute-api.REGION.amazonaws.com/STAGE/RESOURCE
However, making a request using the public DNS names leads to the signature calculation error. We are passing the Host header here for the VPC endpoint to be able to identify which API to hit.
awscurl -v https://vpce-ID.execute-api.REGION.vpce.amazonaws.com/STAGE/RESOURCE -H "Host: API-ID.execute-api.REGION.amazonaws.com"
If we replace Host header with x-api-gw-id as suggested here https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-private-api-test-invoke-url.html#w20aac13c16c28c11
The request works fine -
awscurl -v https://vpce-ID.execute-api.REGION.vpce.amazonaws.com/STAGE/RESOURCE -H "x-apigw-api-id: API-ID"
I suspect the issue is with how the host header value is used in the signature calculation. Can you please confirm if format is right?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.