It seems something is not working properly.

https://travis-ci.org/aio-libs/aiodocker/jobs/291980136

Let's assume we have a public API function with the following signature: def f(arg, flag1=None, flag2=True, flag3=False).
User is supposed to call it as f('arg', flag2=False).
But now it might do it as f('arg', None, False) which makes a mess.

I suggest changing all such signatures to def f(arg, *, flag1=None, flag2=True, flag3=False).
It forces user to specify optional args by name only.

@achimnol ?

Let's provide more "pythonic" API using magic methods.

For instance, if we use __getitem__, we could do:

Current:

img_id = '...'
image = docker.images.image(img_id)
await image.do_something()

cid = '...'
container = docker.containers.container(cid)
await container.do_something()

Suggested:

img_id = '...'
image = docker.images[img_id]
await image.do_something()

cid = '...'
container = docker.containers[cid]
await container.do_something()

Note: there is no "DockerImage" (not "DockerImages") class yet. The above code is just an illustration.

Sprint participants often suffer from setting up the minimal testing environment with Docker registries. We need to provide more friendly guides and/or automated script to do it.

Several aio-libs projects use docker for testing, for example aiopg/aioodbc/aiohttp_admin etc. I think it is good idea to switch from dockerpy to aiodocker. This way aiodocker will be tested on each PR there, and bugs can be found faster.

I thinks aioodbc project is good candidated to start with, since it is not widely used, and we can experiment. https://github.com/aio-libs/aioodbc/blob/dc26608e4155b69c111f44a317a623ba52c79c6c/tests/conftest.py#L73-L113

There haven't been a new release on PyPI since 2015-01-24, is this repo still maintained?

Initial version borrow texts from aiohttp, feel free to update them if needed

Commit 2fd465e#diff-836389a575fe727c5e91b897394f2650R14 disable the test for Docker version 17.03 because it doesn't work.

Build logs: https://travis-ci.org/aio-libs/aiodocker/jobs/261764205

_________________________ test_build_from_remote_file __________________________
docker = <aiodocker.docker.Docker object at 0x7f605bc22fd0>
random_name = <function _random_name at 0x7f605d383598>
    @pytest.mark.asyncio
    async def test_build_from_remote_file(docker, random_name):
        remote = ("https://raw.githubusercontent.com/aio-libs/"
                  "aiodocker/master/tests/docker/Dockerfile")
    
        tag = "{}:1.0".format(random_name())
        params = {'tag': tag, 'remote': remote}
>       await docker.images.build(**params)
tests/test_images.py:15: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
aiodocker/images.py:218: in build
    data=local_context
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <aiodocker.docker.Docker object at 0x7f605bc22fd0>, path = 'build'
method = 'POST'
params = {'forcerm': False, 'nocache': False, 'pull': False, 'q': False, ...}
timeout = None, data = None, headers = {}, kwargs = {}
url = URL('tcp://localhost:27015/v1.27/build')
response = <ClientResponse(tcp://localhost:27015/v1.27/build?t=aiodocker-795cf8a:1.0&rm=1&remote=https://raw.githubusercontent.co...l': 'false', 'Server': 'Docker/17.03.2-ce (linux)', 'Date': 'Mon, 07 Aug 2017 10:08:08 GMT', 'Content-Length': '180')>
what = b'{"message":"Unknown instruction: \\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000"}\n'
content_type = 'application/json'

    async def _query(self, path, method='GET', params=None, timeout=None,
                     data=None, headers=None, **kwargs):
        '''
            Get the response object by performing the HTTP request.
            The caller is responsible to finalize the response object.
            '''
        url = self.canonicalize_url(path)
        if timeout is not None:
            kwargs['timeout'] = timeout
        try:
            response = await self.session.request(
                method, url,
                params=httpize(params), headers=headers,
                data=data, **kwargs)
        except asyncio.TimeoutError:
            raise
    
        if (response.status // 100) in [4, 5]:
            what = await response.read()
            content_type = response.headers.get('content-type', '')
            response.close()
            if content_type == 'application/json':
                raise DockerError(response.status,
>                                 json.loads(what.decode('utf8')))
E               aiodocker.exceptions.DockerError: DockerError(500, 'Unknown instruction: \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
aiodocker/docker.py:142: DockerError
=============== 1 failed, 25 passed, 5 skipped in 24.73 seconds ================

Cannot really install it through pypi.

Need pass event loop to docker to use one main event loop for the daemon.

0.8.5a0 is not correct, it should be changed to 0.8.6a0, my error.

Sorry if I'm just being "PyPi ignorant", but would it be worthwhile to sign releases?

(Of course, my motives are extremely selfish and I just want to play with uscan's ability to verify signatures, but I know you're a big fan of GPG so I figured it might be something you'd consider.)

♥

I have added https://pyup.io/ bot to track dependencies updates.

I find this tool is very valuable to detect backward incompatible changes, and we are using it almost in any other aio-libs project.
See first auto PR: #62

There is a collection of headers={"content-type": "application/json",} all around docker.py

Won't be better to move this to the client, specifically _query or _query_json?

Currently you cannot build images

Considering that also aiohttp (aio-libs/aiohttp#2343) plans to drop support for lower versions.

What do you think?

Long story short

Docker service update can return some random errors.

https://travis-ci.org/aio-libs/aiodocker/jobs/314907338

moby/moby#30794

• Expected behaviour:
• Actual behaviour:

How to reproduce

Your environment

@achimnol what do you think about typing, I see annotation in code and new PRs (#26), but no mypy checks. So my question is should aiodocker require type annotation for the new code in PRs?

Deploy should be done on travis.
BTW we've uploaded aiodocker 0.7 on PyPI without pinning a git tag :(

@achimnol please add me owner rights on PyPI, I'll setup everything.
My login on PyPI is Andrew.Svetlov

Just like for all other aio-libs.
Python's licence is also Apache 2 derived BTW.

She already got commit rights to aiohtp.

I do nominate Cecile Tonglet to aiodocker committers too.

As I said for aiohttp team Cecile is very active and responsive, she follow aio-libs rules:

• don't do wild commits
• ask for review
• discuss every significant change first

Anybody object?

In PyCon KR 2017 Sprint, we have several participants using Windows laptops.
Let's test aiodocker on Windows and check what should be done.

This library imports aiohttp, but doesn't specify it as a requirement in setup.py. This results in errors such as the following.

(aiodocker) $ pip install aiodocker
Collecting aiodocker
  Downloading aiodocker-0.6.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-wwzirc4b/aiodocker/setup.py", line 4, in <module>
        from aiodocker import __version__
      File "/tmp/pip-build-wwzirc4b/aiodocker/aiodocker/__init__.py", line 3, in <module>
        from .docker import Docker
      File "/tmp/pip-build-wwzirc4b/aiodocker/aiodocker/docker.py", line 5, in <module>
        import aiohttp
    ImportError: No module named 'aiohttp'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-wwzirc4b/aiodocker/

To give some background:
#125

when you call run to create and start a container in case container.start() fails you won't get the id of the container back.

@barrachri made significant improvement for the library during last weeks.
He is responsible and understand development process well.
Thus I suggest to give him commit rights.

It motivates very well. But please keep in mind, all development except very trivial changes should go through pull requests anyway, PR review is very appreciated.

@jettify @achimnol @paultag @zbyte64 please share your opinions

Traceback (most recent call last):
  File "docker_test.py", line 43, in <module>
    loop.run_until_complete(docker.close())
  File "/Users/christianbarra/miniconda3/envs/mcc.cassiny.io/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "/Users/christianbarra/miniconda3/envs/mcc.cassiny.io/lib/python3.6/site-packages/aiodocker/docker.py", line 101, in close
    await self.events.stop()
  File "/Users/christianbarra/miniconda3/envs/mcc.cassiny.io/lib/python3.6/site-packages/aiodocker/events.py", line 69, in stop
    await self.task
  File "/Users/christianbarra/miniconda3/envs/mcc.cassiny.io/lib/python3.6/site-packages/aiodocker/events.py", line 54, in run
    async for data in self.json_stream:
  File "/Users/christianbarra/miniconda3/envs/mcc.cassiny.io/lib/python3.6/site-packages/aiodocker/jsonstream.py", line 26, in __anext__
    data = yield from self._response.content.readline()
  File "/Users/christianbarra/miniconda3/envs/mcc.cassiny.io/lib/python3.6/site-packages/aiohttp/streams.py", line 570, in readline
    return (yield from super().readline())
  File "/Users/christianbarra/miniconda3/envs/mcc.cassiny.io/lib/python3.6/site-packages/aiohttp/streams.py", line 268, in readline
    yield from self._wait('readline')
  File "/Users/christianbarra/miniconda3/envs/mcc.cassiny.io/lib/python3.6/site-packages/aiohttp/streams.py", line 235, in _wait
    yield from waiter
  File "/Users/christianbarra/miniconda3/envs/mcc.cassiny.io/lib/python3.6/site-packages/aiohttp/helpers.py", line 733, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError
2017-08-31 07:53:05,869 Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x108063da0>

and this is the code

#!/usr/bin/env python3

import logging
import asyncio
from aiodocker.docker import Docker
from aiodocker.exceptions import DockerError

FORMAT = '%(asctime)-15s %(message)s'
logging.basicConfig(format=FORMAT, level=logging.DEBUG)
log = logging.getLogger(__name__)

async def docker_listener(docker):

    subscriber = docker.events.subscribe()
    log.debug("Docker logger started")

    while True:
        event = await subscriber.get()
        if event is None:
            log.debug("Break")
            break

        actions = ('start', 'remove', 'die')
        log.debug("-------------")
        if event['Type'] == 'container':
            log.debug(f"{event['Type']}")
            log.debug(f"{event['Action']}")
            log.debug(f"{event['Actor']['Attributes']['com.docker.swarm.service.name']}")
            log.debug(f"{event['Actor']['Attributes']['user_id']}")
            log.debug(f"{event['time']}")


if __name__ == '__main__':
    log.debug("Docker logger started")
    loop = asyncio.get_event_loop()
    docker = Docker()
    try:
        # do our stuffs.
        loop.run_until_complete(docker_listener(docker))
    finally:
        loop.run_until_complete(docker.close())
        loop.close()

I don't see strong reasons to don't do it -- 3.5 has await syntax.

The motivation for 3.5 is: I want to drop 3.4 for most aio-libs projects in a few months.
Debian 9 Stretch has released with Python 3.5 stable support.
Thus people have native 3.5 on their linux boxes. It's crucial for projects like chat bots and https://home-assistant.io/ -- their users are very often not professional programmers and python 3.6 requirement with manual installation or using docker images is too hard for them.

On other hand I want to switch test suite for projects like aiopg to aiodocker, so test suite should be executed by every supported python version.

Is there strong objection against Python 3.5?

In the next PRs would be nice to clarify the use of close() like here

Long story short

Currently the aiodocker uses:

I think the current stable release is 1.30

The project has pretty big API for fitting it into README page.
We need a documentation.
I suggest using sphinx tool as for many other aio-libs projects.

Add a specific exception for docker.containers.run.

More info: #136

Since project under active development lets agree on code style and check it on each PR. Should be relatively easy to do.

Description

I wanted to check something in particular: what happens if you only monitor events and the DOCKER_HOST is set to something unreachable.

Current Behavior

I receive an invalid event, plus the script waits for nothing and if I interrupt it I get an unclean exit.

Excepted Behavior

The script should not even start, it should notice that the connection cannot be made. Also it should exit cleanly.

How To Reproduce

To do the test, I picked the example events.py and removed about everything except the loop:

#!/usr/bin/env python3

import asyncio
from aiodocker.docker import Docker
from aiodocker.exceptions import DockerError


async def demo(docker):

    subscriber = docker.events.subscribe()

    while True:
        event = await subscriber.get()
        print(f"event: {event!r}")


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    docker = Docker()
    try:
        # start a monitoring task.
        event_task = loop.create_task(docker.events.run())
        # do our stuffs.
        loop.run_until_complete(demo(docker))
        # explicitly stop monitoring.
        event_task.cancel()
    finally:
        loop.run_until_complete(docker.close())
        loop.close()

Then I run the script with:

DOCKER_HOST=tcp://127.0.0.1:666 ./events.py

And I first get an invalid event:

event: None

Then the script enters the while True loop and wait forever without output unless I interrupt the process.

^CTraceback (most recent call last):
  File "/tmp/events.py", line 24, in <module>
    loop.run_until_complete(demo(docker))
  File "/usr/lib/python3.6/asyncio/base_events.py", line 454, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 421, in run_forever
    self._run_once()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 1389, in _run_once
    event_list = self._selector.select(timeout)
  File "/usr/lib/python3.6/selectors.py", line 445, in select
    fd_event_list = self._epoll.poll(timeout, max_ev)
KeyboardInterrupt
Task exception was never retrieved
future: <Task finished coro=<DockerEvents.run() done, defined at /home/cecile/.local/lib/python3.6/site-packages/aiodocker-0.7-py3.6.egg/aiodocker/docker.py:512> exception=ClientConnectorError(111, "Cannot connect to host 127.0.0.1:666 ssl:False [Can not connect to 127.0.0.1:666 [Connect call failed ('127.0.0.1', 666)]]")>
Traceback (most recent call last):
  File "/home/cecile/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 651, in _create_direct_connection
    local_addr=self._local_addr)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 776, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.6/asyncio/base_events.py", line 763, in create_connection
    yield from self.sock_connect(sock, address)
  File "/usr/lib/python3.6/asyncio/selector_events.py", line 451, in sock_connect
    return (yield from fut)
  File "/usr/lib/python3.6/asyncio/selector_events.py", line 481, in _sock_connect_cb
    raise OSError(err, 'Connect call failed %s' % (address,))
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 666)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cecile/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 374, in connect
    proto = yield from self._create_connection(req)
  File "/home/cecile/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 628, in _create_connection
    _, proto = yield from self._create_direct_connection(req)
  File "/home/cecile/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 678, in _create_direct_connection
    (req.host, req.port, exc.strerror)) from exc
aiohttp.client_exceptions.ClientConnectorError: [Errno 111] Can not connect to 127.0.0.1:666 [Connect call failed ('127.0.0.1', 666)]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cecile/.local/lib/python3.6/site-packages/aiodocker-0.7-py3.6.egg/aiodocker/docker.py", line 525, in run
    params=params,
  File "/home/cecile/.local/lib/python3.6/site-packages/aiodocker-0.7-py3.6.egg/aiodocker/docker.py", line 121, in _query
    data=data, **kwargs)
  File "/home/cecile/.local/lib/python3.6/site-packages/aiohttp/client.py", line 621, in __await__
    resp = yield from self._coro
  File "/home/cecile/.local/lib/python3.6/site-packages/aiohttp/client.py", line 225, in _request
    conn = yield from self._connector.connect(req)
  File "/home/cecile/.local/lib/python3.6/site-packages/aiohttp/connector.py", line 379, in connect
    .format(key, exc.strerror)) from exc
aiohttp.client_exceptions.ClientConnectorError: [Errno 111] Cannot connect to host 127.0.0.1:666 ssl:False [Can not connect to 127.0.0.1:666 [Connect call failed ('127.0.0.1', 666)]]
Exception ignored in: <coroutine object demo at 0x7f9912428f68>
Traceback (most recent call last):
  File "/tmp/events.py", line 13, in demo
  File "/home/cecile/.local/lib/python3.6/site-packages/aiodocker-0.7-py3.6.egg/aiodocker/channel.py", line 14, in get
  File "/usr/lib/python3.6/asyncio/queues.py", line 169, in get
  File "/usr/lib/python3.6/asyncio/base_events.py", line 573, in call_soon
  File "/usr/lib/python3.6/asyncio/base_events.py", line 357, in _check_closed
RuntimeError: Event loop is closed

For tracking release changelog

aiodocker/logs.py:55:13: E722 do not use bare except'

After the last update flake8 returns an error.

https://travis-ci.org/aio-libs/aiodocker/jobs/291980136

w/ progress callbacks somehow

Before I started working on this project, I had my own Docker client library for aiohttp;

https://github.com/tenforce/docker-py-aiohttp

But I took a very different approach. My idea was to make docker-py return futures instead of result directly. It took me a very long time before I finally figured out a solution but recently, I succeeded. I made some kind of wrapper library around docker-py that will re-use most of the code of docker-py but redirect the HTTP calls to aiohttp instead and return futures instead of directly the result. I only had to re-write the json stream parser and underlying helpers like that. At this point, the entire low-level API of docker-py is working and the high-level one will need a little bit more tweak.

Now, to be honest, I'm struggling between aiodocker and aiodockerpy. Both have pros and cons:

• For example, the code of aiodocker is clean and brand new while aiodockerpy is more hacking around the requests handlers.
• On the other hand, aiodockerpy doesn't need the same kind of maintenance: the API changes is actually handled by docker-py itself, not by aiodockerpy. I get almost everything "for free"... even the documentation is the same.

I would really like to heard your thoughts about all of this.

e.g., aiohttp.ClientSession.close() is no longer a coroutine since v2.0.

        if connector is None:
>           if _rx_tcp_schemes.search(docker_host):
E           TypeError: expected string or bytes-like object

Probably better to raise an error if docker_host is None.

Long story short

https://github.com/aio-libs/aiodocker/blob/master/aiodocker/events.py#L24

• Expected behaviour: only one task is created the first time, the other just subscribe normally
• Actual behaviour: a new task is created every time

How to reproduce

/

Your environment

/

I've disabled 'Merge' and 'Rebase' behavior for UI Merge button.
Everybody should use 'Squash' for merging PR into master.
I believe it hurts nobody.

According to Docker API, the container listing returns somewhat summarized version of container details while its inspection API returns full details.
The problem is that those two versions are NOT compatible. We need to serve them under a consistent view, regardless of whether the user has invoked container.show() or not.

Let's add support for swarm, network, service, and secret APIs.

We don't use Circle CI anyway

https://travis-ci.org/aio-libs/aiodocker/jobs/248039891

#52

I'll send a PR to handle it, but it is more like a temporary fix.

Right now many APIs look like

for image in (await docker.images.list()):
        print(f" {image['Id']} {image['RepoTags'][0] if image['RepoTags'] else ''}")

That's work and might be useful sometimes but I consider it ugly.
Better notation should be:

async for image in docker.images.list():
        print(f" {image['Id']} {image['RepoTags'][0] if image['RepoTags'] else ''}")

It's more native and doesn't require extra parenthesis.

Implementation should support both ways, sure.
It could be done by returning Context class from docker.images.list() non-coroutine call.
The context should support __await__ and __aiter__ for returning an async iterator with __anext__.

It's not very hard task but I have no time for it now unfortunately.
@achimnol would you do it? I'll be happy to review if needed.

Now we use pypandoc tool for converting markdown into rst.
RST format is required for long_description in setup.py.
It works but I pretty sure keeping files as-is without conversion makes everything simpler.

Any objections?

See https://travis-ci.org/aio-libs/aiodocker/jobs/248039891
Next run will be successful, sure.

Could we fix it?
Any ideas?

In order to maintain code quality we need coverage report for each PR, codecov good tool for this. Integration could be tricky since tests executed inside docker container.