aio-libs / aiohttp-remotes Goto Github PK

Some proxys include the port number in the client following the pattern ipaddress:port

Currently the library is passing the header directly to the ip_address function and, in this scenario is throwing this exception:

ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_protocol.py", line 418, in start
    resp = await task
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_app.py", line 458, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.7/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/local/lib/python3.7/site-packages/aiohttp_remotes/x_forwarded.py", line 59, in middleware
    forwarded_for = self.get_forwarded_for(headers)
  File "/usr/local/lib/python3.7/site-packages/aiohttp_remotes/x_forwarded.py", line 24, in get_forwarded_for
    (a.strip() for a in forwarded_for)
  File "/usr/local/lib/python3.7/site-packages/aiohttp_remotes/x_forwarded.py", line 25, in <listcomp>
    if addr
  File "/usr/local/lib/python3.7/ipaddress.py", line 54, in ip_address
    address)
ValueError: '10.200.80.7:60604' does not appear to be an IPv4 or IPv6 address

I'll be sending a PR solving this issue.
Thanks

https://github.com/aio-libs/aiohttp-remotes/blob/master/aiohttp_remotes/cloudflare.py#L33

Can use asyncio.gather to get faster setup time

Not sure if this is a aiohttp-remotes issue, but it seems like default access logger uses old request without updated remote address.
I deploy my aiohttp server with XForwardedRelaxed helper behind nginx and assign X-Forwarded-For header: proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; I can confirm that request was updated properly by printing it from route, but access logs always point to localhost.
What can be done to fix access logger output?

XForwardedStrict always fails when not running behind a proxy.
I believe the XForwardedStrict middleware should immediately return await handler(request) if the result of self.get_forwarded_for(headers) is an empty list (although I might be missing some additional intent).

A test case without an X-Forwarded-For header is absent, I provide one below:

async def test_x_forwarded_strict_no_forwarding(aiohttp_client):
    async def handler(request):
        assert request.remote == '127.0.0.1'
        return web.Response()

    app = web.Application()
    app.router.add_get('/', handler)
    await _setup(app, XForwardedStrict([['20.20.20.20']]))
    cl = await aiohttp_client(app)
    resp = await cl.get('/')
    assert resp.status == 200

Deploy environments may have more than one proxy-like device in the HTTP(s) delivery path.
For example, Load Balancer forwards to nginx forwards to an aiohttp service.

It is also possible the client is connecting via a proxy, which adds it's own X-Forwarded-For header.

IFF we assume deployment is either always without a proxy (development) or behind a proxy (production), then it should be safe to check the last proxy on the list against the trusted list.

If the deployed environment is running more than one proxy, it would be possible to check proxies to some depth. Checking all the proxies is difficult since the first proxy on the list might be from the client's environment.

I'm using aiohttp with aiohttp_remotes, and I want to execute the following line:

client_ip = aiohttp_remotes.XForwardedStrict(trusted=trusted_proxies).get_forwarded_for(request.headers)

This raises a TooManyHeaders(hdrs.X_FORWARDED_FOR) error. Upon inspection, I found that the line

request.headers.getall(hdrs.X_FORWARDED_FOR, [])

returns, in my case:

['192.168.0.1', '192.168.0.1']

I am still new to aiohttp_remotes, and I might be misunderstaning the error. However, it seems to me that duplicate IPs should not raise a TooManyHeaders error. If so, an easy fix would be to change the following line in aiohttp_remotes/x_forwarded.py

    def get_forwarded_for(self, headers):
        forwarded_for = headers.getall(hdrs.X_FORWARDED_FOR, [])
...

to

    def get_forwarded_for(self, headers):
        forwarded_for = list(set(headers.getall(hdrs.X_FORWARDED_FOR, [])))
...

System Information

• aiohttp: 3.5.4
• aiohttp_remotes: 0.1.2
• Python: 3.6.7

I'm maintaining an authentication handler that is meant to be deployed behind the Kubernetes NGINX ingress as an auth_request handler (https://github.com/lsst-sqre/gafaelfawr). This deployment has the interesting property that the number of trusted proxies in front of the service is not fixed. When called as an auth_request handler, an extra proxy hop is added due to how this is implemented internally by NGINX and the ingress controller.

This means that neither XForwardedRelaxed nor XForwardedStrict quite works, since both need to know the count of trusted proxies in front of the service.

I therefore developed an alternate middleware that starts from the right end of the X-Forwarded-For header and filters out IPs that are on the trusted list until it finds the first non-trusted IP, and then uses that to set the request attributes and would like to contribute it to this project.

Code coming momentarily in a PR.

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Hello,

The X-Forwarded and Forwarded middleware is not compatible with sockets created by AF_INET6. As per the sockets documentation, for AF_INET6 a 4-tuple is used. The current middleware only handles 2-tuple case when sockets are created by AF_INET.

Since we only require the peer_ip, we would just take the first item of the tuple returned.

Do like this:

REPO_NAME={{ YOUR_REPO_NAME }}
travis encrypt -r "aio-libs/${REPO_NAME}" --api-endpoint 'https://api.travis-ci.com/'

Ref: https://github.com/orgs/aio-libs/teams/admins/discussions/9