Failed to import the required Python library about ansible-docker HOT 27 CLOSED

It just fails to connect with connect to host localhost port 2222: Connection refused and also trying to connect to the VM's IP over port 22 fails too with the same message.

Normally with my Vagrantfiles, I wouldn't bother with their insecure key and I would copy in my SSH key, so it's more similar to how it would be on a real server.

But in any case, when I spin up a new DO server I'm unable to import docker in that virtualenv too, and it's due to the docker pip package not being installed, which is interesting considering the package's state is set to present and the task showed it was changed.

I wonder if something changed since September 2019 and today with Ansible. I'm going to investigate this further today. I only did a quick test on it yesterday.

By the way, are you using Python 2 or 3 as your Ansible interpreter?

from ansible-docker.

Hint for everyone else who ran into this, upgrade your locally cached version 😉
ansible-galaxy install -f nickjj.docker

from ansible-docker.

Within that python prompt can you run import docker and import docker_compose without getting an error?

It says "module not found" when I try import docker.

Also you should be running your custom Docker Compose task as the Docker user.

I'm not sure how I would go about doing that.

from ansible-docker.

What happens if you set up a docker.yml playbook (check the README) and run that before you run your custom tasks in another playbook?

from ansible-docker.

@nickjj So a playbook docker.yml with the first two tasks of this one (Update and upgrade apt packages and Install Docker & Docker compose) right?

from ansible-docker.

No, just the update / upgrade apt task as a pre-task and then the rest of the file should be exactly what's in the readme example. The other task is something this role already does for you. You don't need to define that.

from ansible-docker.

I've run the following docker.yml:

- name: Deploy
  hosts: vagrant
  become: yes
  pre_tasks:
    - name: Update and upgrade apt packages
      apt:
        upgrade: 'yes'
        update_cache: yes
        cache_valid_time: 3600
        # Reference: https://github.com/ansible/ansible/issues/56832
        force_apt_get: yes 
  roles:
    - role: "nickjj.docker"
      tags: ["docker"]

then I tried to run /usr/bin/env python-docker and inside the interpreter import docker but I get:

>>> import docker
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'docker'

from ansible-docker.

If I spin up a Vagrant box using https://app.vagrantup.com/bento/boxes/ubuntu-18.04 and run the role along with running sudo apt-get update && sudo apt-get install -y python beforehand (to bootstrap Ansible), I'm able to SSH in as the user I configured in the Docker role, run /usr/bin/env python-docker and successfully run import docker without errors.

Those apt commands can also be rolled up into a raw task:

- name: Install Python without using Ansible modules
  raw: >
    bash -c "test -e /usr/bin/python \
    || (apt-get -y update && apt-get install -y python)"
  changed_when: false

You could add that to your list of pre-tasks, to run as the first thing. Or abstract it into a "bootstrap" role (this is what I do).

from ansible-docker.

@nickjj This is p confusing.
Can you please try with this vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu1804"
  config.vm.define "vagrant"

  config.vm.provision "deploy", type: 'ansible' do |ansible|
    ansible.compatibility_mode = "2.0"
    ansible.playbook = "ansible/docker.yml"
    ansible.groups = {
      "staging" => ["vagrant"]
    }
  end
end

and with the above playbook named as ansible/docker.yml and then run vagrant up?

And from there vagrant ssh and get into the /usr/bin/env python-docker interpreter to run import docker.

from ansible-docker.

Can you try it without using the built in ansible features of vagrant? That doesn't really resemble a production environment and I don't know what crazy things vagrant does behind the scenes.

You should spin up a brand new server without any extra vagrant provision steps and then:

1. Install python on the server and run an apt-get update

2. Create and configure some type of non-root user to SSH in as

3. Configure this Docker role to use that user you just created

4. Run the docker playbook

That is what I'm doing here. Running Ansible against a vagrant driven VM should be no different than how you would against a production server and the above steps work in both environments.

from ansible-docker.

Ok, I've removed the provision configuration in the vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu1804"
  config.vm.define "vagrant"
end

I've added the host explicitly in my vagrant file:

all:
  children:
    staging:
      hosts:
        vagrant:
          ansible_host: localhost
          ansible_user: vagrant
          ansible_port: 2222
          ansible_ssh_common_args: -i .vagrant/machines/vagrant/virtualbox/private_key

then I've run the playbook:

- name: Deploy
  hosts: vagrant
  become: yes
  pre_tasks:
    - name: Update and upgrade apt packages
      apt:
        upgrade: 'yes'
        update_cache: yes
        cache_valid_time: 3600
        # Reference: https://github.com/ansible/ansible/issues/56832
        force_apt_get: yes 
  roles:
    - role: "nickjj.docker"
      tags: ["docker"]

nothing changed. Everything happened exactly as if I was setting up provisioning with Vagrant.

With regards to the non-root to SSH to, is this user already created by the role?
How do I configure this ansible-docker role to use that user instead?

from ansible-docker.

This role doesn't create the user. That is a system level decision unrelated to using Docker.

Check the readme at: https://github.com/nickjj/ansible-docker#configuring-users-to-run-docker-without-root

You might be able to get away with using the vagrant user without having to make a new user. In that case, setting docker__users: ["vagrant"] might do it.

from ansible-docker.

I've modified my playbook with what you suggested:

- name: Deploy
  hosts: vagrant
  become: yes
  vars:                             # NEW
    docker__users: ["vagrant"]      # NEW
  pre_tasks:
    ...

but I'm still unable to import docker inside /usr/bin/env python-docker.

from ansible-docker.

What is the exact playbook and vagrantfile you're using now? I will try to set it up tonight and test it.

from ansible-docker.

Playbook:

- name: Deploy
  hosts: vagrant
  become: yes
  vars:
    docker__users: ["vagrant"]
  pre_tasks:
    - name: Update and upgrade apt packages
      apt:
        upgrade: 'yes'
        update_cache: yes
        cache_valid_time: 3600
        # Reference: https://github.com/ansible/ansible/issues/56832
        force_apt_get: yes

    - name: Install python
      apt:
        name:
          - python
        state: present
        
  roles:
    - role: "nickjj.docker"
      tags: ["docker"]

Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu1804"
  config.vm.define "vagrant"
end

Inventory file:

all:
  children:
    staging:
      hosts:
        vagrant:
          ansible_host: localhost
          ansible_user: vagrant
          ansible_port: 2222
          ansible_ssh_common_args: -i .vagrant/machines/vagrant/virtualbox/private_key

Thank you by the way. 😊

from ansible-docker.

In the mean time, if you wanted to try things out on a real host that would be a worthy test. It doesn't matter where. DigitalOcean, AWS, GCP -- anywhere really.

from ansible-docker.

@nickjj Sorry nick but I've just spent 8 hours only this issue and I feel like I'm going crazy. First thing tomorrow I'll give it a try either on digital ocean or on a local server we have.

from ansible-docker.

Can you set up a working Vagrantfile example that doesn't involve using vagrant's private key or any of vagrant's weird port forwarding?

Because I can't spin up your example using hyper-v. Every time I try to connect to localhost:2222 it fails.

from ansible-docker.

I'm not that skilled with Vagrant. I don't know how I would avoid those. I was able to ssh into it with:

ssh -i .vagrant/machines/vagrant/virtualbox/private_key -o StrictHostKeyChecking=no -p 2222 vagrant@localhost

If that doesn't work (nor does the inventory above) what error does it show up?

from ansible-docker.

By the way, are you using Python 2 or 3 as your Ansible interpreter?

Python 3

from ansible-docker.

I was using Python 2, so it seems to affect both, which is good at least.

So, here's a fun adventure. I re-installed Ansible on my machine using Python 3.x, which means the interpreter is also 3.x now.

Then I modified your playbook to install python3 instead of python, and also removed docker__users: ["vagrant"] since I'm not using vagrant.

Then I modified the hosts file to be:

[vagrant]
INSERT_DO_IP_HERE

(I know it's labelled vagrant but that doesn't matter)

Then I ran ansible-playbook -i /e/dockertest/hosts /e/dockertest/playbook.yml -u root

(DO's default user is root and /e/dockertest is where I made these files to test with vagrant before)

It reported 19 ok tasks and 15 changes with 0 failures.

I SSH'd into the server and ran python-docker and import docker fails. I ran ls -la /usr/local/bin and I see python-docker is symlinked to /usr/local/lib/docker/virtualenv/bin/python which is expected since this is what the role does.

Then I ran /usr/local/lib/docker/virtualenv/bin/python and when I do import docker it works.

I'll need to dig into why module imports fail through symlinks. This wasn't a problem before, I'm not sure what changed off the top of my head but if you have any suggestions I'm all ears.

from ansible-docker.

I figured it out. The difference between then and now is that virtualenv changed drastically and I didn't version lock it in the task that pip installs it in this role.

If you change that task to use virtualenv==16.3.0 then everything works (symlinks included).

Can you verify that? Alternatively you can remove the task that installs virtualenv and add virtualenv to the list of packages at docker__package_dependencies. I haven't tested the alternative approach but I like that better since it sticks to the distro's stable version of virtualenv, and will likely make that change to this role after testing it more extensively.

I also opened pypa/virtualenv#1599 to see if this could get fixed in the future, so we can use more modern versions of virtualenv.

from ansible-docker.

I can confirm that pinning that version for virtualenv solves the problem.

from ansible-docker.

This has been fixed in v1.9.2 which is available on the Ansible Galaxy.

Thanks a lot for the bug report!

from ansible-docker.

Thank you for dealing with this so quickly. 😃

from ansible-docker.

Hi
Newest version still doesn't work for me. My Debian distribution went with 15.x of virtualenv and it gets installed. However in /usr/local/lib/python3.7/dist-packages I see virtualenv-20.0.15.dist-info, and also pyvenv.cfg contains:

home = /usr
implementation = CPython
version_info = 3.7.3.final.0
virtualenv = 20.0.15
include-system-site-packages = false
base-prefix = /usr
base-exec-prefix = /usr
base-executable = /usr/bin/python3

What's desired state? Should I have no 20.x virtualenv installed at all? How can I uninstall it then? Simple removing /usr/local/lib/python3.7/dist-packages/viirtalenv* doesn't help - virtualenv iis not visible anymore.

There were suggested package pinning in earlier version. Was it supposed to pin apt package in Debian? I have no versions later than 15.x so it will not help me?

Task log:

TASK [nickjj.docker : Manage Docker registry login credentials] ******************************************************************************************************************************************************************************
failed: [myservice] (item={'registry_url': 'myregistry', 'username': 'admin', 'password': 'xxx', 'email': '[email protected]'}) => {"ansible_loop_var": "item", "changed": false, "item": {"email": "[email protected]", "password": "xxx", "registry_url": "myregistry", "username": "admin"}, "msg": "Failed to import the required Python library (Docker SDK for Python: docker (Python >= 2.7) or docker-py (Python 2.6)) on logging-service's Python /usr/local/bin/python-docker. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter, for example via pip install docker or pip install docker-py (Python 2.6). The error was: No module named 'docker'"}

from ansible-docker.

Hi,

The desired state should be having virtualenv installed with apt, and depending on your distro, that could vary but 15.x sounds about right.

20.x is the version with issues and that should be uninstalled. You can do that with pip uninstall virtualenv, but in your case since you're using Python 3.x you might need to do pip3 uninstall virtualenv.

from ansible-docker.