Unclosed response/connection errors about aiobotocore HOT 11 CLOSED

we made it work like this before:

        async with self._boto_session.create_client('s3', region_name=_default_region) as s3_client:
            response = await s3_client.get_object(Bucket=self._s3_bucket, Key=self._s3_key)
            async with response['Body'] as stream:
                body = await stream.read()

reason being is that the stream needs to decide if it's going to close the stream at the end of the context (if not all bytes were read, since the connector can't be re-used) or release it back to the pool (if all bytes were read). Please do not revert this behavior.

from aiobotocore.

@scribu I am sorry for late reply (got distracted to aiohttp_admin and absolutely forgot about maintaining duties), you have unclosed resource

response['Body'].read()
response['Body'].close()

should help. We should think about better API, current one is ugly...

from aiobotocore.

I get the same errors even if I use session.create_client() as an async context manager, like so:

import os
import asyncio
import aiobotocore

session = aiobotocore.get_session()

async def fetch(summary):
    async with session.create_client('s3') as client:
        response = await client.get_object(
            Bucket=os.environ['S3_BUCKET_NAME'],
            Key=summary['Key'],
        )

    return await response['Body'].read()

async def main():
    async with session.create_client('s3') as client:
        response = await client.list_objects_v2(
            Bucket=os.environ['S3_BUCKET_NAME'],
        )

    for summary in response.get('Contents', []):
        body = await fetch(summary)

        print(summary['Key'], len(body))

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

from aiobotocore.

Thanks for report! I will take a look.

from aiobotocore.

I'm not sure whether this issue is related to what I'm experiencing, but when I run tests that use aiobotocore, I get "Unclosed client session" errors from aiohttp:

py35 installed: -e git://github.com/aio-libs/aiobotocore.git@86c11c2e4713a74507e2fb1b8616667517828239#egg=aiobotocore,aiohttp==1.1.5,async-timeout==1.1.0,botocore==1.4.28,chardet==2.3.0,docutils==0.12,flake8==3.2.0,jmespath==0.9.0,mccabe==0.5.2,multidict==2.1.2,otherxyz-awslib==0.2.25,otherxyz-utils==0.2.9,prometheus-client==0.0.17,py==1.4.31,pycodestyle==2.2.0,pyflakes==1.3.0,pytest==3.0.4,pytest-asyncio==0.5.0,python-dateutil==2.6.0,six==1.10.0,yarl==0.7.0
py35 runtests: PYTHONHASHSEED='4257097605'
py35 runtests: commands[0] | flake8 otherxyz tests
py35 runtests: commands[1] | py.test -rw tests
============================= test session starts ==============================
platform darwin -- Python 3.5.2, pytest-3.0.4, py-1.4.31, pluggy-0.4.0
rootdir: /Users/adam/Other/src/otherxyz-awslib, inifile: 
plugins: asyncio-0.5.0
collected 58 items 

tests/arn_test.py ..
tests/dynamodb/attributes_test.py ..
tests/dynamodb/conditions_test.py .................
tests/dynamodb/expressions_test.py ....................
tests/dynamodb/table_test.py ......Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10df72b00>
...
tests/sns/platform_application_test.py ....
tests/sns/platform_endpoint_test.py ....

========================== 58 passed in 1.27 seconds ===========================
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10e0be390>
___________________________________ summary ____________________________________
  py35: commands succeeded
  congratulations :)

My tests are succeeding, and this code is intended for use in a server environment where the session is expected to live as long as the process, but is there something I should be doing differently to when using aiobotocore.get_session() and session.create_client() to close the underlying aiohttp client session?

from aiobotocore.

client has close() coroutine, that should close aiohttp session. Probably bug sitting somewhere

from aiobotocore.

@jettify Yup sorry I just found that in the source. Calling that when I'm finished seems to be fine. Is it necessary to await it? This comment https://github.com/aio-libs/aiobotocore/blob/master/aiobotocore/client.py#L203 seems to indicate that calling it as a coroutine is not required.

from aiobotocore.

That comment is outdated (will fix that very soon), in new aiohttp sesson.close is coroutine http://aiohttp.readthedocs.io/en/stable/client_reference.html#aiohttp.ClientSession.close so await is required.

from aiobotocore.

Ok, thanks for the pointer.

from aiobotocore.

I think with the above we can close this?

from aiobotocore.

Closing for now, feel free to open new issue

from aiobotocore.