Giter VIP home page Giter VIP logo

docker-formula's Introduction

docker-formula

Extensible formula to manage Docker on MacOS, Windows, and GNU/Linux. Currently supports:

The default docker.software and docker.compose.software states support:

The other states support container managmement.

Travis CI Build Status Semantic Release

A SaltStack formula for Docker on MacOS, GNU/Linux, Windows and Raspberry Pi (4b).

See the full SaltStack Formulas installation and usage instructions. If you are interested in writing or contributing to formulas, please pay attention to the Writing Formula Section. If you want to use this formula, please pay attention to the FORMULA file and/or git tag, which contains the currently released version. This formula is versioned according to Semantic Versioning. See Formula Versioning Section for more details.

Commit message formatting is significant!!

Please see How to contribute for more details.

Meta-state (This is a state that includes other states).

This state installs the Docker solution (see https://docs.docker.io) for Raspberry Pi support please see Notes

Meta-state (This is a state that includes other states).

Stop Docker daemon and remove docker packages ('docker', 'docker-engine', 'docker-ce', etc) on Linux. To protect OS integrity, this state won't remove packages listed as dependencies (i.e. python is kept).

Configures the upstream Docker's repo on RedHat/Debian OS.

This state removes upstream Docker package repository only, on RedHat/Debian OS.

This state installs Docker (see https://docs.docker.com/engine/install and https://docs.docker.com/desktop/)

This state installs Dockerd daemon on Linux (systemd support).

This state stops Dockerd daemon on Linux (systemd support).

This state overrides default Docker options (i.e. /etc/default/docker):

docker:
  pkg:
    docker:
      config:
        - DOCKER_OPTS="-s btrfs --dns 8.8.8.8"
        - export http_proxy="http://172.17.42.1:3128"

This state uninstalls Docker overrides (i.e. /etc/default/docker).

This state uninstalls Docker software.

Pulls and runs a number of docker containers. See docker container API for docker.containers options:

docker:
  containers:
    running:
      - prometheus_simple
      - prometheus_detail

    prometheus_simple:
      image: "prom/prometheus:v1.7.1"

    prometheus_detail:
      image: "prom/prometheus:v1.7.1"
      # see https://docker-py.readthedocs.io/en/stable/containers.html

Saltstack dockercompose module state support (See https://docs.saltstack.com/en/2018.3/ref/modules/all/salt.modules.dockercompose.html).

The intent is to provide an interface similar to the specification provided by docker-compose. The hope is that you may provide pillar data similar to that which you would use to define services with docker-compose. The assumption is that you are already using pillar data and salt formulae to represent the state of your existing infrastructure.

No real effort had been made to support every possible feature of docker-compose. Rather, we prefer the syntax provided by the docker-compose whenever it is reasonable for the sake of simplicity.

It is worth noting that we have added one attribute which is decidedly absent from the docker-compose specification. That attribute is dvc. This is a boolean attribute which allows us to define data only volume containers which can not be represented with the docker.software.service.running state since they are not intended to include a long living service inside the container.

See the included pillar.example for a representative pillar data block. To use this formula, you might target a host with the following pillar:

docker:
  compose:
    ng:
      registry-datastore:
        dvc: true
        # image: &registry_image 'docker.io/registry:latest' ## Fedora
        image: &registry_image 'registry:latest'
        container_name: &dvc 'registry-datastore'
        command: echo *dvc data volume container
        volumes:
          - &datapath '/registry'
      registry-service:
        image: *registry_image
        container_name: 'registry-service'
        volumes_from:
          - *dvc
        environment:
          SETTINGS_FLAVOR: 'local'
          STORAGE_PATH: *datapath
          SEARCH_BACKEND: 'sqlalchemy'
          REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: '/registry'
        ports:
          - 127.0.0.1:5000:5000
        # restart: 'always'    # compose v1.9
        deploy:                # compose v3
          restart_policy:
            condition: on-failure
            delay: 5s
            max_attempts: 3
            window: 120s
      nginx-latest:
        # image: 'docker.io/nginx:latest'  ##Fedora
        image: 'nginx:latest'
        container_name: 'nginx-latest'
        links:
          - 'registry-service:registry'
        ports:
          - '80:80'
          - '443:443'
        volumes:
          - /srv/docker-registry/nginx/:/etc/nginx/conf.d
          - /srv/docker-registry/auth/:/etc/nginx/conf.d/auth
          - /srv/docker-registry/certs/:/etc/nginx/conf.d/certs
        working_dir: '/var/www/html'
        volume_driver: 'local'
        userns_mode: 'host'

Then you would target a host with the following states:

include:
  - base: docker
  - base: docker.compose.ng

Saltstack swarm module state support (See https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.swarm.html).

Opposite of docker.swarm state (See https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.swarm.html).

Create docker networks

Remove docker networks

Sub-states are available inside sub-directories.

There are some caveats with regard to the support of this module on Rasberry Pi 4b's.

  • This module has only been tested with Raspberry Pi 4b using Rasbian Os Version Buster
  • This module supports raspbian only when used from Salt 3002.6. Salt 3003.x fails with template isses.
  • Docker service is known to fail starting when freshly installed via this module on Rasbian Buster with all apt-get updates and upgrades performed. The error found in logs for failing to start is dockerd: failed to create NAT chain DOCKER

The Reason for this is as documented here . The following Fix followed by a restart fixes this. The summary reason is that the docker installer uses iptables for nat. Unfortunately Debian uses nftables. You can convert the entries over to nftables or just setup Debian to use the legacy iptables. On the target Raspberry Pi issue the following to resolve or incorporate the SLS before in your custom SLS

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
sudo shutdown -r 0  # Do a restart, Docker.d should then function

or the following SLS

iptables:
  alternatives.set:
    - path:  /usr/sbin/iptables-legacy
ip6tables:
  alternatives.set:
    - path:  /usr/sbin/ip6tables-legacy

The provisioning of docker to raspbian uses functionality from https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script. It specifically mentions Using these scripts is not recommended for production environments, and you should understand the potential risks before you use them: The reasons are stated as :

  • The scripts require root or sudo privileges to run. Therefore, you should carefully examine and audit the scripts before running them.
  • The scripts attempt to detect your Linux distribution and version and configure your package management system for you. In addition, the scripts do not allow you to customize any installation parameters. This may lead to an unsupported configuration, either from Docker’s point of view or from your own organization’s guidelines and standards.
  • The scripts install all dependencies and recommendations of the package manager without asking for confirmation. This may install a large number of packages, depending on the current configuration of your host machine.
  • The script does not provide options to specify which version of Docker to install, and installs the latest version that is released in the “edge” channel.
  • Do not use the convenience script if Docker has already been installed on the host machine using another mechanism.

Linux testing is done with kitchen-salt.

Requirements

  • Ruby
  • Docker
$ gem install bundler
$ bundle install
$ bin/kitchen test [platform]

Where [platform] is the platform name defined in kitchen.yml, e.g. debian-9-2019-2-py3.

bin/kitchen converge

Creates the Docker instance and runs the docker main state, ready for testing.

bin/kitchen verify

Runs the inspec tests on the actual instance.

bin/kitchen destroy

Removes the Docker instance.

bin/kitchen test

Runs all of the stages above in one go: i.e. destroy + converge + verify + destroy.

bin/kitchen login

Gives you SSH access to the instance for manual testing.

docker-formula's People

Contributors

aboe76 avatar alexproca avatar blbradley avatar cdarwin avatar dafyddj avatar danny-smit avatar davidjfelix avatar gravyboat avatar hatifnatt avatar japtain-cack avatar javierbertoli avatar keslerm avatar mingfang avatar msciciel avatar myii avatar nmadhok avatar noelmcloughlin avatar poil avatar puneetk avatar scornelissen85 avatar semantic-release-bot avatar solarchad avatar stamak avatar tedski avatar ticosax avatar tyhunt99 avatar vbh-jenkins avatar vladvasiliu avatar whiteinge avatar wwentland avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

docker-formula's Issues

Repository is empty

Did someone forget to push? Is this totally speculative? Not happening? Got my hopes up and then it gave me a bad first impression of the salt community. Some sort of explanation in the README explaining the purpose is called for at the very least, otherwise the community would be better off if this repository didn't exist.

KeyError: 'pip.installed' -- after successful initial installation

Initial installation successful:

          ID: docker-py requirements
    Function: pip.installed
        Name: docker-py
      Result: True
     Comment: All packages were successfully installed
     Started: 22:33:56.745286
    Duration: 3161.768 ms
     Changes:
              ----------
              docker-py==1.4.0:
                  Installed

Encountered the following error when running highstate again:

    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1161, in _thread_return
        return_data = func(*args, **kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 523, in highstate
        whitelist=kwargs.get('whitelist')
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2968, in call_highstate
        return self.state.call_high(high)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 2067, in call_high
        ret = dict(list(disabled.items()) + list(self.call_chunks(chunks).items()))
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1623, in call_chunks
        running = self.call_chunk(low, running, chunks)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1769, in call_chunk
        self._mod_init(low)
      File "/usr/lib/python2.7/dist-packages/salt/state.py", line 612, in _mod_init
        self.states['{0}.{1}'.format(low['state'], low['fun'])]  # pylint: disable=W0106
      File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 90, in __getitem__
        raise KeyError(key)
    KeyError: 'pip.installed'

Searched around and found that there have been various reports and suggestions about this error but no clear solution:

Tried a few ideas and put together the following workaround using pip.install instead:

docker-py requirements:
  pkg.installed:
    - name: python-pip
  module.run:
    - name: pip.install
    {%- if "pip_version" in docker %}
    - pkgs: docker-py {{ docker.pip_version }}
    {%- else %}
    - pkgs: docker-py
    {%- endif %}
    - require:
      - pkg: docker package
    - reload_modules: True

That allowed the procedure to complete without errors:

          ID: docker-py requirements
    Function: module.run
        Name: pip.install
      Result: True
     Comment: Module function pip.install executed
     Started: 01:55:31.301498
    Duration: 441.122 ms
     Changes:
              ----------
              ret:
                  ----------
                  pid:
                      18652
                  retcode:
                      0
                  stderr:
                  stdout:
                      Requirement already satisfied (use --upgrade to upgrade): docker-py in /usr/local/lib/python2.7/dist-packages
                      Requirement already satisfied (use --upgrade to upgrade): requests>=2.5.2 in /usr/local/lib/python2.7/dist-packages (from docker-py)
                      Requirement already satisfied (use --upgrade to upgrade): six>=1.4.0 in /usr/lib/python2.7/dist-packages (from docker-py)
                      Requirement already satisfied (use --upgrade to upgrade): websocket-client>=0.32.0 in /usr/local/lib/python2.7/dist-packages (from docker-py)
                      Requirement already satisfied (use --upgrade to upgrade): backports.ssl-match-hostname in /usr/local/lib/python2.7/dist-packages (from websocket-client>=0.32.0->docker-py)
                      Cleaning up...

Is there a better solution to this problem?

Missing dependencies on Ubuntu 14.04

When docker is installed there are missing 2 packages:
linux-image-extra-$(uname -r)
linux-image-extra-virtual
(see https://docs.docker.com/engine/installation/linux/ubuntu/)

Without these packages docker falls back to "devicemapper" storage driver in "loop-lvm" mode which is strongly discouraged as it causes massive problems with disk space usage especially with write-heavy containers.

As we only have Ubuntu 14.04 machines I added the 2 packages in:
/srv/formulas/docker-formula/docker/init.sls

docker package dependencies:
  pkg.installed:
    - pkgs:
      - apt-transport-https
      - iptables
      - ca-certificates
      - python-apt
      - linux-image-extra-{{ salt.cmd.run('uname -r') }}
      - linux-image-extra-virtual

With these modifications docker uses the "aufs" storage driver.

docker-formula on centos 7

when i run docker-formula on centos7 ,i have below information,please tell me why,thx

      ID: docker package repository
Function: pkgrepo.managed
    Name: deb https://apt.dockerproject.org/repo centos-CentOS Linux 7 (Core) main
  Result: False
 Comment: Failed to configure repo 'deb https://apt.dockerproject.org/repo centos-CentOS Linux 7 (Core) main': The repo does not exist and needs to be created, but either a baseurl or a mirrorlist needs to be given
 Started: 19:45:46.771018
Duration: 12.63 ms
 Changes:   

python-pip and pip issues with salt/docker

The compose.sls state uses pkg.installed for python-pip, and pip.installed for pip.
Do both need to be handled?

compose-pip:
  pkg.installed:
    - name: python-pip
  pip.installed:
    - name: pip
    - upgrade: True

After OS installation the python-pip package installs python modules located at /usr/lib/python/.
The formula init state does not install any pip unless install_docker-py: True.
However the compose state installs both python-pip (/usr/lib/) and PyPi pip (/usr/local/lib)`.

$ pip list | grep docker
docker (2.7.0)
docker-compose (1.19.0)
docker-pycreds (0.2.2)
dockerpty (0.4.1)

It becomes unclear which docker module salt is using.

# sudo salt-call --local cmd.run 'python -c "import docker ; import docker.utils; print(docker.version); help(docker)"'
local:
    3.0.1

Mixing pip's can cause issues

[ERROR   ] State 'docker_image.present' was not found in SLS 'docker.compose-ng'
Reason: 'docker_image' __virtual__ returned False: 'docker.version' is not available.

Maybe ...

^[[33mWARNING:^[[0m Dependency conflict: an older version of the 'docker-py' package may be polluting the namespace. If you're experiencing crashes, run the following command to remedy the issue: pip uninstall docker-py; pip uninstall docker; pip install docker
[WARNING ] /usr/local/lib/python2.7/dist-packages/docker/api/client.py:163: UserWarning: The minimum API version supported is 1.21, but you are using version 1.18. It is recommended you either upgrade Docker Engine or use an older version of Docker SDK for Python.
  'Python.'.format(MINIMUM_DOCKER_API_VERSION, self._version)
 
$ python
>>> import docker
>>> from docker import pulled
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name pulled

Mixing pips can be a hazard.

systemd detection command throws "garbage option" error

Applying the docker.containers SLS on CentOS 7 results in the following errors at render time:

[INFO    ] Executing command '/usr/bin/ps -p1 | grep -q systemd && echo systemd || echo upstart' in directory '/root'
[ERROR   ] Command '/usr/bin/ps -p1 | grep -q systemd && echo systemd || echo upstart' failed with return code: 1
[ERROR   ] output: error: garbage option

Usage:
 ps [options]

 Try 'ps --help <simple|list|output|threads|misc|all>'
  or 'ps --help <s|l|o|t|m|a>'
 for additional help text.

For more details see ps(1).

This is due to changes in the behavior of cmd.run in Salt 2017.7.0:
https://docs.saltstack.com/en/latest/topics/releases/2017.7.0.html#general-deprecations

Problem with reliably pulling docker containers after setting up docker

pillar data

{%- from "foobar/shared_vars.jinja" import shared_vars with context %}
docker-containers:
  lookup:
    ecs_agent:
      image: "amazon/amazon-ecs-agent:v1.7.1"
      runoptions:
        - "--detach=true"
        - "--restart=on-failure:10"
        - "--volume=/var/run/docker.sock:/var/run/docker.sock"
        - "--volume=/var/log/ecs/:/log"
        - "--volume=/var/lib/ecs/data:/data"
        - "--volume=/sys/fs/cgroup:/sys/fs/cgroup:ro"
        - "--volume=/var/run/docker/execdriver/native:/var/lib/docker/execdriver/native:ro"
        - "--publish=127.0.0.1:51678:51678"
        - "--env=ECS_LOGFILE=/log/ecs-agent.log"
        - "--env=ECS_LOGLEVEL=info"
        - "--env=ECS_DATADIR=/data"
        - "--env=ECS_CLUSTER={{ grains.tags.cluster }}"
{% endif %}

docker-pkg:
  lookup:
    config:
      - DOCKER_OPTS="{{ shared_vars.docker_opts }}"
    version: {{ shared_vars.docker_engine_version }}
    install_docker_py: False

state data

include:
  - docker
  - docker.containers

what we sometimes see in the logs:

[INFO    ] Completed state [docker] at time 10:17:12.994250
[INFO    ] Running state [docker pull amazon/amazon-ecs-agent:v1.7.1] at time 10:17:12.995713
[INFO    ] Executing state cmd.run for docker pull amazon/amazon-ecs-agent:v1.7.1
[INFO    ] Executing command 'docker pull amazon/amazon-ecs-agent:v1.7.1' in directory '/root'
[ERROR   ] Command 'docker pull amazon/amazon-ecs-agent:v1.7.1' failed with return code: 1
[ERROR   ] stdout: Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon. Is the docker daemon running on this host?). Using system default: https://index.docker.io/v1/
[ERROR   ] stderr: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
[ERROR   ] retcode: 1
[ERROR   ] {'pid': 5775, 'retcode': 1, 'stderr': 'Cannot connect to the Docker daemon. Is the docker daemon running on this host?', 'stdout': 'Warning: failed to get default registry endpoint from daemon (Cannot connect to the Docker daemon. Is the docker daemon running on this host?). Using system default: https://index.docker.io/v1/'}
[INFO    ] Completed state [docker pull amazon/amazon-ecs-agent:v1.7.1] at time 10:17:13.046047

It works mostly when I run salt again. so there seems to be a bit of time after docker is set up where the daemon is not reachable which imho the salt module should wait for.

Fedora 25 Support

Hi,

could you please add in the init.sls
Add Support for fedora.
For Fedora you dont need the managed repo.
So just if os_family redhat and os fedora dont manage the repo.

A dnf install docker is enough.
Thanks

Docker compose requires separate install

Docker-compose is bundled with Docker on Mac and Windows, but requires a separate install on other OSs (like Ubuntu). I don't see anywhere in the README or pillar.example about installing docker-compose.

Use refresh_repo only for latest version?

The docker.refresh_repo setting (defaulting to True) gets used for pkg.installed.

This causes apt-get update to run always.

But when specifying a specific version this could be skipped?! (since the repo gets updated through pkgrepo already in case it gets added)

Overriding Systemd file on Debian Jessie breaks Docker

Debian jessie without specifying a version of docker gives us docker.io from backports which is 1.6.2 currently. This package provides a functioning systemd file.

$ cat /lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
EnvironmentFile=-/etc/default/docker
ExecStart=/usr/bin/docker -d -H fd:// $DOCKER_OPTS
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

This formula however provides it's own systemd file which uses the docker daemon startup option that was introduced in docker 1.8, and so doesn't work with Jessie's 1.6

$ cat /etc/systemd/system/docker.service.d/docker-defaults.conf 
[Service]
EnvironmentFile=-/etc/default/docker
ExecStart=
ExecStart=/usr/bin/docker daemon $DOCKER_OPTS -H fd://

Obviously we can override the docker version to get a newer version not from backports that will work, but given that we provide the option for the backports version, we shouldn't override the systemd file.

Surely we can rely on the packaged systemd files instead of installing our own with this formula?

PGP key hardcoded?

Not really sure if this was a bug, just a passing critique and thought it warranted mentioning.

I'm not convinced that hardcoding the Docker PGP key is the best way to handle package signing. The docker key is already in the Ubuntu keyserver, you just have to tell apt that you're okay with using it and it will download it for you, like so:

pkgrepo.managed:
  ...
  - keyid: d8576a8ba88d21e9
  - keyserver: keyserver.ubuntu.com
  ...

I think this may be a better way to handle the key. The first reason is that it cleans up the repository, removing 1 file. The second is that it reduces the points of failure and lowers your burden for maintaining this package. By instructing apt to retrieve the key from a keyserver, it's impossible to add a key that has been revoked, since it will also check for revocation. Adding the key manually does not necessitate it hasn't been revoked nor that your server can reach a keyserver to determine if the key has been revoked.

docker-config does not respect systemd

docker.docker-config only edits /etc/default/docker which is unused on systemd nodes. Upon detection of systemd, the states should use a systemd drop-in to override the package-supplied unit file.

Init system detection fails on Ubuntu 16.04

Hi,

I'm having the following problem on Ubuntu 16.04:

          ID: docker-container-startup-config-exporter
    Function: file.managed
      Result: False
     Comment: Specified file docker-container-startup-config-exporter is not an absolute path

Investigating the logs, it turns out the command to identify the init system fails for some reason.

Minion log :

2017-09-13 16:25:36,232 [salt.loaded.int.module.cmdmod][INFO    ][7720] Executing command 'ps -p1 | grep -q systemd && echo systemd || echo upstart' in directory '/root'
2017-09-13 16:25:36,240 [salt.loaded.int.module.cmdmod][ERROR   ][7720] Command 'ps -p1 | grep -q systemd && echo systemd || echo upstart' failed with return code: 1
2017-09-13 16:25:36,241 [salt.loaded.int.module.cmdmod][ERROR   ][7720] output: error: garbage option

Usage:
 ps [options]

 Try 'ps --help <simple|list|output|threads|misc|all>'
  or 'ps --help <s|l|o|t|m|a>'
 for additional help text.

For more details see ps(1).

However, on the salt master:

salt 'elasticsearch-a.logs' cmd.run "ps -p1 | grep -q systemd && echo systemd || echo upstart"
elasticsearch-a.logs:
    systemd

And directly on the server:

» ps -p1 | grep -q systemd && echo systemd || echo upstart                                                  130 ↵
systemd

Replacing line 14 of containers.sls to use grains instead of a command, it works:

{# {%- set init_system = salt["cmd.run"]("ps -p 1 | grep -q systemd && echo systemd || echo upstart") %} #}
{%- set init_system = grains['init'] %}

Versions:

Master:

» salt -V
Salt Version:
           Salt: 2017.7.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.5
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.6
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Jan 19 2017, 14:48:08)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.2.1

System Versions:
           dist: Ubuntu 17.04 zesty
         locale: UTF-8
        machine: x86_64
        release: 4.10.0-33-generic
         system: Linux
        version: Ubuntu 17.04 zesty

Minion:

» salt-minion -V
Salt Version:
           Salt: 2017.7.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-93-generic
         system: Linux
        version: Ubuntu 16.04 xenial

Pip can fail and it's blocking

Here is what I get with the pip-formula, it should be better not to handle it here and let the user choose between pip formula or the working way for him. I'm block with this issue and I will need to extend docker-py requirements state to not depend from the python-pip package of jessie.

Here is what I get now:

root@ci-slave:/# apt-cache policy python-pip
python-pip:
  Installed: 1.5.6-5
  Candidate: 1.5.6-5
  Version table:
 *** 1.5.6-5 0
        500 http://ftp.fr.debian.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
root@ci-slave:/# apt-cache policy python-requests
python-requests:
  Installed: 2.4.3-6
  Candidate: 2.4.3-6
  Version table:
 *** 2.4.3-6 0
        500 http://ftp.fr.debian.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
root@ci-slave:/# pip version
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.5.6', 'console_scripts', 'pip')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 356, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2476, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2190, in load
    ['__name__'])
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 74, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/usr/lib/python2.7/dist-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/lib/python2.7/dist-packages/pip/download.py", line 25, in <module>
    from requests.compat import IncompleteRead
ImportError: cannot import name IncompleteRead
root@ci-slave:/#

then:

----------
          ID: docker-py requirements
    Function: pip.installed
        Name: docker-py
      Result: None
     Comment: Error installing 'docker-py': Traceback (most recent call last):
                File "/usr/bin/pip2", line 9, in <module>
                  load_entry_point('pip==1.5.6', 'console_scripts', 'pip2')()
                File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 356, in load_entry_point
                  return get_distribution(dist).load_entry_point(group, name)
                File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2476, in load_entry_point
                  return ep.load()
                File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2190, in load
                  ['__name__'])
                File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 74, in <module>
                  from pip.vcs import git, mercurial, subversion, bazaar  # noqa
                File "/usr/lib/python2.7/dist-packages/pip/vcs/mercurial.py", line 9, in <module>
                  from pip.download import path_to_url
                File "/usr/lib/python2.7/dist-packages/pip/download.py", line 25, in <module>
                  from requests.compat import IncompleteRead
              ImportError: cannot import name IncompleteRead
     Started: 19:17:04.062652
    Duration: 117.275 ms
     Changes:   
----------

Support other OSes too

As an avid gentoo user I feel kinda left out by the initial formula. I know that supporting every flavor of OS is kinda hard, but a little flexibility to at least fail gracefully when not dealing with debian.

A small note in the README that currently only debian and ubuntu are supported might suffice for the time being.

I will look into finding a way to integrate gentoo into the existing code but I expect it to be a little more than just one pull request.

version pillar should not be required

Having to specify an exact version string is a bit clunky. This is related to #80 as it would be nice to not have to specify a version at all, and just get the latest in a repository. I'm guessing you want to specify a version when using the docker repo's as they release versions so often. It just makes it hard to get started with this formula when specifying no version prevents you from using it due to systemd, and when specifying a version, you have to specify the exact version string of the package.

Even being able to do something like "1.11.*" would be nicer so we can have minor updates automatically but not major.

Define multiple Docker projects in pillar

Apologies for the possibly dumb question, but how can I define individual projects using Salt pillar?

Currently the pillar is docker:compose, but can I define docker:compose:project1 etc?

Test suite optimization

Last test run took over 30 minutes. We need better Docker images to solve this.

@gravyboat @whiteinge Either of you know what the status of SaltStack's Docker Hub is? It definitely lacks images of more modern OSes needed for this. Nice to see some tooling in this regard! 😁

Add Debian Support

The current implementation of the formula assumes Ubuntu. Since I use Debian, support for that platform interests me :)

script does not seem to work on ubuntu xenial

hey there,

great work. I was using this script for quite a while on ubuntu trusty. Now, having some vm's on ubuntu 16.04 LTS (xenial) the formular doesn't seem to work out of the box.

The script hangs forever on the master. And wenn I look into the apt-get log files on the minion (with sudo tail -f -n 1000 /var/log/apt/term.log), I see it stucked at the following line:

Setting up docker-engine (1.12.2-0~xenial) ...

Configuration file '/etc/default/docker'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
 ==> Keeping old config file as default.

hope this helps you guys.

Not clear how to use this

I get in principle that I should be able to add it to my gitfs_remotes and then include it, but I can't get it working.

In my master config:

fileserver_backend:
  - roots
  - git

gitfs_remotes:
  - https://github.com/saltstack-formulas/docker-formula

In my devops-packages.sls:

include:
  - docker

(I also tried docker.docker)

Then I ran:

sudo salt-call -l debug state.sls devops-packages

But I got:

Data failed to compile:
----------
    Specified SLS docker.docker in environment base is not available on the salt
 master

State fails due to changes in config file provided by docker pkg

They apparently updated the /etc/default/docker provided from the docker pkg for ubuntu. This change causes the package to fail to install due to dpkg requiring interaction when it detect changes to config files on install. It does not occurr with every run due to Salt randomizing order of the state. The key is docker must be installed prior to the /etc/default/docker:file.managed.

The method I used to fix it is to add a required to the /etc/default/docker to ensure it is run after pkg.install of docker.

You could also allow the ability to not run the /etc/defualt/docker file.managed based on either a parameter or grain data if the system does not need it.

If you need anymore information let me know

Add a list of user to group docker

I we want to use docker with another user than root, we need to add it to the docker group outside of the formula. A conveniant way to do it could be to add to the docker group with the formula according to a list of users from pillars:

docker-users:
  lookup:
    names:
      - jenkins
      - ...

Option for disabling repository configuration

Formula always enable docker repository and it should be at least a option to disable this behavior. We are using red hat satellite/spacewalk for local mirroring of packages so we don't want to fetch packages from external mirrors. Is it possible to add such feature to be configured via pillar ?

dockerfile ADD and COPY command

Hi,

why is it not possible to run a add or copy command in the formula.
I try to generate a elasticsearch container here my sls:

'elasticsearch master container':
  docker_container.running:
    - name: elasticsearch-master-{{ grains['id'] }}
    - image: docker.elastic.co/elasticsearch/elasticsearch:5.5.1
    - restart_policy: always
    - network_mode: {{ pillar.get('ELASTICSEARCH_NETWORK', 'bridge') }}
    - user: elasticsearch
    - add: salt://containers/elasticsearch/master/files/elasticsearch.yml /usr/share/elasticsearch/config/
    - copy: salt://containers/elasticsearch/master/files/jvm.options /usr/share/elasticsearch/config/
    - volumes:
      - /srv/elasticsearch/data:/usr/share/elasticsearch/data:rw
    - __monitoring__:
      - service: container-running
      - title: elasticsearch master container
      - container_name: elasticsearch-master-{{ grains['id'] }}

I dont want to bind a whole folder, is there any possibility to run a add and copy command?

Salt dockerio states/modules require docker-py library

While having a docker daemon set up on a host is somewhat convenient, to really use this formula in a production-like environment a user still needs to add the docker-py library herself in order to make use of native salt states and modules to manage docker images and containers.

Limit scope of container install to specific minions

Hi,

still fairly new to salt (just started using it, last friday), so please excuse if the answer to that one is actually obvious and I just can't see, yet.

Basically what I want to do is use this to start a swarm cluster. For this, I need to start a swarm master and several swarm minions. Is it possible to do this, with the current containers.sls formula? The way I understand it is, that it will iterate the containers containers on every node docker.containers gets applied to.

Support DOCKER_OPTS in pillar data

As a systems administrator, I would like to adjust docker daemon startup arguments through a known interface.

In Debian this is done through the file, /etc/default/docker. A list of options could be concatenated into an argument for file.append or something similar.

Recent docker and CentOS 7 support

Hi,
I think that formula needs some corrections or rewrite to follow https://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html and support recent versions of docker. Few changes are also required to support CentOS. If You agree I can provide pull request with newer version but it will break compatibility with old docker installations. I can also provide docker.v2 subformula pull request to support recents versions of docker on CentOS and Ubuntu. I could try to fix current code but it could hard or impossible to avoid compatibility issues.
I would be grateful if You could at least disagree with me. Formula in current state cannot be used with recent docker on CentOS 7. I'm open for discussion.

/etc/init.d/docker start" returns 1 and breaks docker-service state

I'm seeing this when requiring "service: docker-service" in another state:


      ID: docker-service
Function: service.running
    Name: docker
  Result: False
 Comment: Service docker is already enabled, and is dead
 Started: 00:19:18.436978
Duration: 100.658 ms
 Changes:   
          ----------
          docker:
              False

Not sure how salt verifies if a service is already running, but it looks like it's just trying to start docker and failing because docker's init script exits with 1 when the service is already running.

Using the signature seems work. Here's a pull request that should work in pretty much all OS'es:
#29

registry question

Hi guys,

I'm trying to understand the need for sqlalchemy in docker.registry.
Since the registry itself runs inside a container, I believe the sqlalchemy is already installed in the docker image. I'm not sure why sqlalchemy needs to be installed in the host level as well.
I'm bringing this up because the sqlalchemy requirement triggers to install a version of python2 that raises a bunch of package conflicts on ubuntu 14.04. Plus, if it's not needed, it's better if it's not there :)

Anything that I'm missing here?

Cheers,
g.

How to tell if it is using dockerio (deprecated) or dockerng?

I've run into an issue while using docker-formula's compose-ng. With the following pillar data:

docker:
  compose:
    registry-data:
      dvc: True
      image: registry:2
      container_name: &dvc 'registry-data'
      command: echo *dvc data volume container
      volumes:
        - /opt/docker-registry/data:/var/lib/registry
    registry:
      image: registry:2
      container_name: 'registry'
      restart: 'always'
      volumes_from:
        - 'registry-data'
      environment:
        REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: '/var/lib/registry'
      ports:
        - 127.0.0.1:5000:5000
    nginx:
      image: nginx:1.9
      restart: 'always'
      ports:
        - 5043:443
      links:
        - registry:registry
      volumes:
        - /etc/docker-registry/nginx/:/etc/nginx/conf.d
        - /etc/docker-registry/auth/:/etc/nginx/conf.d/auth
        - /etc/docker-registry/certs/:/etc/nginx/conf.d/certs

It ends up calling dockerio which seems to have a but parsing the image tag throwing an exception. I managed to overcome it by modifying the source code of /usr/lib/python2.7/dist-packages/salt/states/dockerio.py

FROM

def _get_image_name(image, tag):
    if ':' not in image:
        # backward compatibility: name could be already tagged
        return ':'.join((image, tag))
    return image

TO:

def _get_image_name(image, tag):
    if ':' not in image:
        # backward compatibility: name could be already tagged
        return ':'.join((image, str(tag)))
    return image

Which I consider ugly and unmaintainable.

Versions:
docker-py (1.6.0)
salt-common 2015.8.3+ds-1
salt-master 2015.8.3+ds-1
salt-minion 2015.8.3+ds-1

So, not sure if there's a mistake in my pillar or if it is something with dockerio or with this docker formula and how it passes the values to dockerio or if it should call something else. Sorry, still quite new on salt-docker. Let me know if I can provide further info.

[Request for information] Why use this formula over saltstack docker modules?

I'm looking to use docker to run a Jenkins server. I found that saltstack has It what looks to be native support for docker and docker-compose, handling both state and execution:

It looks like docker-compose does not handle the installation, while this formula does.

Are there other advantages of using this over salt's support? Are there disadvantages? When would someone use this formula?

I'll happily add any answers to docs if that's helpful.

Unreachable code

In init.sls the first 'jessie' condition (name and version block) is unreachable.
The "version in docker" checks contradict.

docker package:
  {%- if "version" in docker %}
  pkg.installed:
    {%- if grains["oscodename"]|lower == 'jessie' and "version" not in docker %}
    - name: docker.io
    - version: {{ docker.version }}
    {%- elif use_old_repo %}

ENH: Utilize the docker HTTPS repository (apt-transport-https)

Source: https://github.com/saltstack-formulas/docker-formula/blob/master/docker/init.sls#L14 :

docker_repo:
    pkgrepo.managed:
      - repo: 'deb http://get.docker.io/ubuntu docker main'
      - file: '/etc/apt/sources.list.d/docker.list'
      - key_url: salt://docker/docker.pgp

Docs: https://get.docker.io/ubuntu/ :

# Check that HTTPS transport is available to APT
if [ ! -e /usr/lib/apt/methods/https ]; then
    apt-get update
    apt-get install -y apt-transport-https
fi
# Add the repository to your APT sources
echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list
# Then import the repository key
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
# Install docker
apt-get update ; apt-get install -y lxc-docker

#
# Alternatively, just use the curl-able install.sh script provided at https://get.docker.io
#
  • Change line 14 of docker/init.sls
      - repo: 'deb https://get.docker.io/ubuntu docker main'

docker.containers fails to correctly start and manage containers (Start/Stop loop)

I believe we are hitting the issue with systemd managing the client, not containers. http://container-solutions.com/running-docker-containers-with-systemd/
The main docker issue is moby/moby#6791
A possible solution is https://github.com/ibuildthecloud/systemd-docker

OS: Debian Jessie
Docker version 1.11.1, build 5604cbe

Basically, https://github.com/saltstack-formulas/docker-formula/blob/master/docker/files/systemd.conf#L22 starts with ExecStart=/usr/bin/docker run -d which tells docker to start the container in the background and then exit. So the docker client does that. Systemd thinks that docker "crashed" so tries to restart it (after running the ExecStop commands that pull down the running container). And the cycle starts again until the retry limit.

Removing the -d works, as then the client doesn't exit once the container has started, but as others have pointed out in the above links, this means systemd is managing the client, not the container, and in some situations this will cause a correctly running container to be restarted.

Pinning pip to specific version isn't working when using docker.compose

The following state:

include:
  - docker
  - docker.compose

Using this pillar:

docker-pkg:
  lookup:
    pip:
      version: '== 8.1.1'

results in issues when the highstate ( salt-call state.highstate ----output-diff) is being executed:

local:
----------
          ID: docker-py requirements
    Function: pip.installed
        Name: pip == 8.1.1
      Result: True
     Comment: All packages were successfully installed
     Started: 13:17:40.906196
    Duration: 3594.701 ms
     Changes:   
              ----------
              pip==8.1.1:
                  Installed
----------
          ID: compose-pip
    Function: pip.installed
        Name: pip
      Result: True
     Comment: All packages were successfully installed
     Started: 13:17:46.380448
    Duration: 3871.755 ms
     Changes:   
              ----------
              pip==9.0.1:
                  Installed

Summary for local
-------------
Succeeded: 64 (changed=2)
Failed:     0
-------------
Total states run:     64
Total run time:   13.644 s

Initially it installs the requested version (== 8.1.1) however due to the upgrade: True in the compose-pip state-id this upgrades pip to the latest version thus overwriting the specific version pin set in docker-py requirements

Optional package update

In my opinion pkg.latest should not be the default state for package installation. Maybe it could be better to use pkg.installed and have a option to enable auto-update ? I feel more comfortable when packages are not updated with formulas by default during formula execution.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.