iamdork / compose Goto Github PK

Command line replacement for docker-compose, adding dork magics.

License: MIT License

Python 81.38% Shell 17.43% HTML 0.28% PHP 0.06% Nginx 0.86%

compose's Introduction

Dork Compose

What is this?

dork-compose is a drop in replacement for the docker-compose command line tool, adding some convenience features for running multiple compose projects on the same host.

Volume snapshots with automatic versioning.
Separation of Docker setup (Dockerfile and docker-compose.yml) from application source code.
Automatic launch of and connection to auxiliary services, like nginx or a DNS server.

Example use cases:

Your local development workstation, hosting multiple different projects built on the same framework, and therefore requiring similar infrastructure and setup steps.
A staging server running multiple versions of the same project, doing fast upgrade testing by using volume snapshots.
A continuous integration server, running automatic tests on pull requests.

Installation

dork-compose uses the same installation procedures as docker-compose. Either using pip:

pip install dork-compose

Plugins

Everything dork-compose does additionally to docker-compose is implemented using plugins. The DORK_PLUGINS environment variable controls which plugins are loaded and in which order they are executed. Plugins are able to alter environment variables, modify the configuration from docker-compose.yml and act on certain events throughout the docker-compose command processes.

By default the DORK_PLUGINS variable looks like this:

export DORK_PLUGINS="env:multi:lib:autobuild:hotcode:filesystem:proxy:dns:vault"

That's the default workstation setup. Plugins are executed from left to right. If two plugins to the same, the right one wins.
Let's run through this example:

env: Scans parent directories for .env or .dork.env files and populates the environment with their contents.
multi: Run multiple different projects on the same host. The project name will be the name ofthe containing directory. It is used to prefix snapshots and build the domain.
lib: If there is a DORK_LIBRARY environment variable that contains a valid directory, dork-compose will assume the docker-compose.yml is there. dork-compose will take care of handling the container build process accordingly.
autbuild: Automatically build docker images of application sources if using onbuild images.
hotcode: Mount code directories you work on into your local codebase and provide vendor code to your IDE.
filesystem: Implements snapshots as plain rsync. Not particularly fast or disk space economic, but works out of the box everywhere.
proxy: Spins up a proxy service that serves your project at http://project.dork.io.
dns: Runs a dns server and configures your system to use it. Enables you to use the domain dork.io.
vault: Exposes secrets (e.g. the github private token) to the build process without leaving any traces in the image.

There are no configuration files. Plugins can be configured using environment variables, which you define in your shell environment for by using the env plugin. For a complete list of plugins and their options please refer to Appendix: Plugins. For an in-action example of these plugins, please refer to the drupal-simple in the recipes repository.

Custom plugins

It's possible to create and load custom plugins. Simply create a Python file with one class called Plugin that extends dork_compose.plugin.Plugin and attach it to the DORK_PLUGINS variable:

export DORK_SOURCE="env:multi:lib:autobuild:hotcode:dependencies:filesystem:proxy:dns:vault:my_plugin=~/path/to/myplugin.py"

For example plugins have a look at the plugins directory inside the dork-compose source.

Snapshots

dork-compose is able to create snapshots of all data volumes used in a compose project. This is done by using the additional dork-compose snapshot command. For an example of how to work with snapshots, please refer to the drupal-simple example in the recipes repository.

Projects & Instances

Snapshots are organized in projects and instances. dork-compose assumes that instances of the same project are compatible. Aside from building the proxy domain, the major purpose is to restrict snapshots to be used by instances of the same project only.

The current project and instance is determined by plugins (like multi in the default setup) or by the DORK_PROJECT and DORK_INSTANCE environment variables.

Automatic snapshots

If the snapshot identifier is omitted from the snapshot save and snapshot load command, dork-compose will rely on plugins to provide one. The git plugin in the default setup for example will store snapshots by the current HEAD hash and will try to load the closest available ancestor to the current checkout. This avoids breaking your development database by switching between feature branches.

Appendix: Plugins

TODO: explain all builtin plugins.

compose's People

Contributors

Stargazers

Watchers

Forkers

sepal ohthehugemanatee

compose's Issues

Multiple proxy domains

If multiple projects define different base domains, it should still work.

nameserver added to the end of /etc/resolv.conf in Linux

Right now on my system, dork adds 127.0.0.1 to the end of my /etc/resolv.conf . That's a problem, because nslookup only tries subsequent servers only when the first server is not reachable. (see the resolver and resolv.conf manpages).

The local nameserver should be added to the TOP of the file.

autobuild plugin should provide .dockerignore file

Excluding the .dorkerfile and any folders listed in the dork.dependencies

DNS proxy is very slow

Ubuntu 17.04, but I also had the problem on 16.04 .

The DNS proxy is very slow resolving external hostnames.

 time nslookup google.com 127.0.0.1
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
Name:	google.com
Address: 172.217.18.110

nslookup google.com 127.0.0.1  0,01s user 0,01s system 0% cpu 4,033 total

4 seconds is a long time to wait for your primary nameserver. Cached requests and .dork.io requests are served quickly, it's only external requests that have this problem. I enabled logging on the container, and got

 dnsmasq[1]: started, version 2.75 cachesize 150
 dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
 dnsmasq[1]: reading /etc/resolv.conf
 dnsmasq[1]: using nameserver 127.0.0.11#53
 dnsmasq[1]: read /etc/hosts - 8 addresses

No additional logs are printed during resolution.

use rsync for dependencies plugin

sync dependencies using a smart rsync container, instead of docker fetch.

Stop proxy service when not needed any more

When all attached projects are "downed" and all networks are removed, the proxy service should be removed too.

no way to stop/restart dork-compose containers

After my last dork container has exited, I still get

docker ps
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                      NAMES
30cf52409bb5        vault                    "docker-entrypoint.sh"   10 days ago         Up 50 minutes       0.0.0.0:8200->8200/tcp                     dork_aux_vault_vault_1
89db5ec20dcb        andyshinn/dnsmasq:2.75   "dnsmasq -k -A /dork."   10 days ago         Up 50 minutes       0.0.0.0:53->53/tcp, 0.0.0.0:53->53/udp     dork_aux_dns_dns_1
ca077b14e38d        iamdork/dockergen        "/usr/local/bin/docke"   10 days ago         Up 50 minutes                                                  dork_aux_proxy_dockergen_1
76b28260097d        iamdork/nginx-proxy      "nginx -g 'daemon off"   10 days ago         Up 50 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   dork_aux_proxy_nginx_1

There's no way to end these containers without a manual docker stop.

Ensure dork-compose is not executed in wrong directories

Since the current directory does not necessarily contain a docker-compose.yml, it might be run from the wrong directory.

CORS support in the proxy

Doesn't exist right now. It should.

.no_auth.[service] should respect folder hirachy

While the .no_auth files respect the folder hirachy,the .no_auth.[service] file currently don't do that. It currently will remove the authentication for all sub directories, but it should actually behave like all the other auth files.

Make use of ssh auth socket

During container runtime, if SSH_AUTH_SOCKET is available, use it instead of mounting the ssh private key.

Proxy only supports HTTP(S)

Sometimes you want to access a non-HTTP service from outside the container, such as dbgp or mysql. Since our only proxy is NGINX, those requests aren't possible.

Per-project plugins

Get plugins per project from a cascading .dork.plugins file.

dork-compose info protocoll

dork-compose info should show the right protocoll for the proxy address. Currently it always displays http.

Auxiliary ELK service

collecting logs from containers

Upgrade to 1.10 and set userns

If docker is configured to work with user namespacing, containers that mount the docker socket (docker-gem for the proxy for example) are not able to do so. After the upgrade to docker compose 1.10 it should be possible to set userns: host on these containers.

Set maximum number of concurrent projects

If the threshold is met and an additional project is started, the oldest one should be turned downed.

Increase proxy buffer size.

As reported by @ohthehugemanatee:

2017/01/12 09:36:30 [error] 19#19: *1079 upstream sent too big header while reading response header from upstream, client: 172.26.0.1, server: ras-backend.dork.io, request: "GET /node/392/edit HTTP/1.1", upstream: "http://172.18.0.9:80/node/392/edit", host: "ras-backend.dork.io"

Probably solved by:

http {
  proxy_buffer_size   128k;
  proxy_buffers   4 256k;
  proxy_busy_buffers_size   256k;
}

GraphQL API

A graphql api based on https://github.com/graphql-python/graphene to list, status, start, stop ... projects.

dork-compose run does not invoke plugins on build

When the run command needs to build an image first, the source directory is not available because no plugins are invoked.

Remove unused .auth and .no_auth files

Currently they just remain where they are and confuse if access restrictions change.

Split proxy service into proxy and generator

For security reasons:
https://github.com/jwilder/nginx-proxy#separate-containers

Show dork-compose version on version flag

Currently dork-compose passes the version flag -v --version to docker-compose, but eventually you will also want to know the version of dork-compose itself.

use rsync for filesystem plugin

create snapshots using a smart rsync container (using volumes_from) instead of plain filesystem copy.

Compatibility policy

dork-compose doesn't work with the current docker-ce Stable release. At the least, it uses the "cached" mount mode which is only supported since 17.04... and stable is on 17.03.1 .

What's the official compatibility policy? It should be noted in Readme.md . Either we're good with stable, or we require edge. :)

Re-enable tests for vault plugin

The vault plugin tests have been disabled for now, because of strange errors with the hashicorps vault container and AUFS.

hashicorp/docker-vault#22

Force fresh images for auto builds

Autobuild should enforce image creation/deletion on up/down to ensure it's always based on the latest sources.

Ubuntu 17.04 compatibility

Steps to reproduce

(on a clean install of Ubuntu 17.04)

install docker-ce per the official instructions. Note that apparently no one bothered to rename the docker.io package to docker-ce on their Zesty repo, so you may have to install docker.io instead.
install dork-compose and docker-compose with sudo apt install -y python-pip && pip install dork-compose
dork-compose up -d --build on your favorite dork project.

Expected behavior

containers start normally.

Actual behavior

Traceback (most recent call last):
  File "/usr/local/bin/dork-compose", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python2.7/dist-packages/dork_compose/main.py", line 58, in run
    compose.cli.main.main()
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 88, in main
    command()
  File "/usr/local/lib/python2.7/dist-packages/dork_compose/injections.py", line 50, in dork_perform_command
    return perform_command(options, handler, command_options)
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 140, in perform_command
    handler(command, command_options)
  File "/usr/local/lib/python2.7/dist-packages/compose/cli/main.py", line 872, in up
    remove_orphans=remove_orphans)
  File "/usr/local/lib/python2.7/dist-packages/dork_compose/injections.py", line 249, in up
    containers = super(DorkProject, self).up(service_names, start_deps, strategy, do_build, timeout, detached, remove_orphans)
  File "/usr/local/lib/python2.7/dist-packages/compose/project.py", line 381, in up
    self.initialize()
  File "/usr/local/lib/python2.7/dist-packages/compose/project.py", line 425, in initialize
    self.networks.initialize()
  File "/usr/local/lib/python2.7/dist-packages/dork_compose/injections.py", line 211, in initialize
    super(DorkNetworks, self).initialize()
  File "/usr/local/lib/python2.7/dist-packages/compose/network.py", line 203, in initialize
    network.ensure()
  File "/usr/local/lib/python2.7/dist-packages/compose/network.py", line 59, in ensure
    data = self.inspect()
  File "/usr/local/lib/python2.7/dist-packages/compose/network.py", line 91, in inspect
    return self.client.inspect_network(self.full_name)
  File "/home/ohthehugemanatee/.local/lib/python2.7/site-packages/docker/utils/decorators.py", line 35, in wrapper
    return f(self, *args, **kwargs)
  File "/home/ohthehugemanatee/.local/lib/python2.7/site-packages/docker/api/network.py", line 180, in inspect_network
    return self._result(res, json=True)
  File "/home/ohthehugemanatee/.local/lib/python2.7/site-packages/docker/api/client.py", line 220, in _result
    self._raise_for_status(response)
  File "/home/ohthehugemanatee/.local/lib/python2.7/site-packages/docker/api/client.py", line 214, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localunixsocket/v1.22/networks/drupal_default

Other notes

other docker-compose projects have no problems
tried in bash and zsh to rule out shell versions
If I manually create the named network with docker network create drupal_default, the error message changes to the next network: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localunixsocket/v1.24/networks/dork_aux_proxy_default.
Docker service debug log:

May 11 11:20:26 Tesla dockerd[1293]: time="2017-05-11T11:20:26.263168557+02:00" level=debug msg="Calling GET /v1.22/info"
May 11 11:20:26 Tesla dockerd[1293]: time="2017-05-11T11:20:26.301624591+02:00" level=debug msg="Calling GET /v1.22/networks/drupal_default"
May 11 11:20:26 Tesla dockerd[1293]: time="2017-05-11T11:20:26.302114928+02:00" level=error msg="Handler for GET /v1.22/networks/ras-backend_default returned error: network drupal_default not found"

Environment substitution in .env files

Environment variables used in .env files should be substituted, for smarter environmental settings.
.env in /var/source

DORK_PLUGINS=a

.env in /var/source/project

DORK_PLUGINS=$DORK_PLUGINS;b

$DORK_PLUGINS should container 'a;b'

Support multiple libraries and local overrides.

It would be useful to aggregate multiple libraries (e.g. drupal + elk stack) within one project and rewire provided services in a local docker-compose.yml file. This behavior should not conflict with the $COMPOSE_FILE environment variable, but rather extend it.