network_mode and routing traffic through vpn about docker-traefik HOT 14 CLOSED

An update on this. To recap the issue;

Say you have a single docker compose with 2 containers, containerA and containerB. In the containerB compose, you set containerB to use the network of containerA with network_mode: service:containerA. If you run docker-compose up -d containerB, docker will up or reup containerA then containerB. Everything will work. If however containerA restarts outside of docker-compose or fails or is restarted by something like Watchtower then containerB has no way of knowing, loses network connectivity and stops working.

There's loads of chat about this on the Docker github and lots of the githubs for containers providing VPN because that is a large use case for network_mode: service:xxxxx. Until Docker provides a solution such as proposed here there are a few workarounds.

The one I'm using requires an autoheal container, and the addition of a Healthcheck to the dependent container (containerB) to check a known service running on containerA. The autoheal container will detect when the dependent (containerB) loses network access as a result of containerA restarting or failing or being updated by Watchtower and will restart the dependent giving it network access back.

More info: docker/compose#6329 (comment)

Also, I'm currently using Gluetun as my VPN container and route Jackett, Deemix, Lidarr-AMD and xTeVe through it. I've done some tests with this workaround by restarting Gluetun outside docker-compose and it works. I am waiting for Glutun to be updated to see if it also works when Watchtower updates the image.

It's worth noting that if you don't have your containerA image set to the :latest tag or don't use Watchtower or never have issues with your VPN provider you probably never experience this issue.

Here's an example of the workaround;

  gluetun:
    container_name: gluetun
    image: qmcgaw/private-internet-access:latest
    restart: always
    cap_add:
      - NET_ADMIN
    networks:
      t2_proxy:
        ipv4_address: 192.168.90.238
    ports:
      - "$GLUETUN_PROXY_PORT:8888/tcp" # HTTP proxy
      - "$GLUETUN_SHADOWPROXY_TCP_PORT:8388/tcp" # Shadowsocks TCP
      - "$GLUETUN_SHADOWPROXY_UDP_PORT:8388/udp" # Shadowsocks UDP
      - "$GLUETUN_CONTROL_PORT:8000/tcp" # Built-in HTTP control server
    volumes:
      - $CONFIG/gluetun:/gluetun
    environment:
      VPNSP: "private internet access"
      UID: $PUID
      GID: $PGID
      TZ: $TZ
      USER: $PIA_USERNAME
      PASSWORD: $PIA_PASSWORD
      REGION: $PIA_REGION
      PORT_FORWARDING: "on"
      PORT_FORWARDING_STATUS_FILE: "/tmp/gluetun/forwarded_port"
      FIREWALL_OUTBOUND_SUBNETS: "192.168.0.0/24"
      HTTPPROXY: "on"
      HTTPPROXY_LOG: "off"
      SHADOWSOCKS: "on"
      SHADOWSOCKS_LOG: "off"
      SHADOWSOCKS_PASSWORD: $GLUETUN_SHADOWSOCKS_PASSWORD
      SHADOWSOCKS_METHOD: "chacha20-ietf-poly1305"

  autoheal:
    container_name: autoheal
    image: willfarrell/autoheal
    restart: always
    networks:
      - t2_proxy
      - socket_proxy
    security_opt:
      - no-new-privileges:true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      AUTOHEAL_CONTAINER_LABEL: "all"
      # DOCKER_SOCK: "tcp://socket-proxy:2375" # not working with proxy

  jackett:
    image: ghcr.io/linuxserver/jackett:latest
    container_name: jackett
    restart: unless-stopped
    network_mode: service:gluetun
    security_opt:
      - no-new-privileges:true
    volumes:
      - $CONFIG/jackett:/config
      - $DATA/torrent/watch:/data/torrent/watch
      - /etc/localtime:/etc/localtime:ro
    environment:
      PUID: $PUID
      PGID: $PGID
      UMASK: 002
      TZ: $TZ
      AUTO_UPDATE: "false"
    healthcheck:
      test: ["CMD", "curl", "http://localhost:8000"] # HTTP Control Server running on Gluetun
      interval: 1m30s
      timeout: 2s
      retries: 1
    labels:
      - "traefik.enable=true"
      ## HTTP Routers
      - "traefik.http.routers.jackett-rtr.entrypoints=https"
      - "traefik.http.routers.jackett-rtr.rule=Host(`jackett.$DOMAINNAME`)"
      - "traefik.http.routers.jackett-rtr.priority=99"
      ## Middlewares
      - "traefik.http.routers.jackett-rtr.middlewares=chain-oauth@file"
      ## HTTP Services
      - "traefik.http.routers.jackett-rtr.service=jackett-svc"
      - "traefik.http.services.jackett-svc.loadbalancer.server.port=9117"
      ## HTTP Routers Auth Bypass
      - "traefik.http.routers.jackett-rtr-bypass.entrypoints=https"
      - "traefik.http.routers.jackett-rtr-bypass.rule=Host(`jackett.$DOMAINNAME`) && Headers(`X-Api-Key`, `$JACKETT_API_KEY`) || Query(`apikey`, `$JACKETT_API_KEY`)"
      - "traefik.http.routers.jackett-rtr-bypass.priority=100"
      - "traefik.http.routers.jackett-rtr-bypass.service=jackett-svc"
      - "traefik.http.routers.jackett-rtr-bypass.middlewares=chain-no-auth@file"

Closing because this issue is related to Docker Networking and not something specific to this stack.

from docker-traefik.

Same issue on the gluetun project at qdm12/gluetun#641

I'll address hopefully next week through https://github.com/qdm12/deunhealth

from docker-traefik.

Check out #34, which is related to haugene/docker-transmission-openvpn#771. Might make a difference to errors at startup.

from docker-traefik.

Thanks. I added that code already when I was trying to fix the issue but other than making transmission quicker to start, it didn’t help with my issue.

from docker-traefik.

I think we both mentioned this on another projects issue tracker. I still haven't been able to reproduce the inaccessible issue. I get the error about once or twice per service using the VPN during first five mins. I then don't see it for hours if at all. Maybe until Traefik check clashes with vpn health check or the VPN connection dropping or changing server. No idea.

I have everything in one compose file, so I have to change network_mode to service instead of container. I may have depends_on: listed for services using the VPN, but in terms of startup and health check it makes no difference. I haven't added anything to modify the health check, I just know it will take 5 mins to become available.

I am still using VPN for jackett, it's needed for geoblocking/censorship same privacy

from docker-traefik.

I can't get this to work in a reliable way. I even tried a different VPN container (arch-qbittorrentvpn).

I'd like to be able to run a number of containers through the VPN, but the containers either 404 or fail after a certain amount of time. I've tried network_mode: container:transmission-vpn and network_mode: service:transmission-vpn and #34 but no joy.

Does anyone have their docker compose file where they are successfully passing multiple containers through a container running a vpn (e.g. transmission-openvpn)? It might help me pin down what's wrong with my setup. Thanks.

from docker-traefik.

Does anyone have some guidance on this? My compose file is allowing the traffic through, but I get the same errors for all containers that run behind the VPN.

from docker-traefik.

My current configuration is stable and working as expected, since I'm on Unraid system i don't have a compose file (it works with docker run via GUI) but I can share my run output or my unraid xml template.

The only errors I keep seeing are in traefik for the container behind VPN:

time="2020-10-24T05:40:48+02:00" level=warning msg="Could not find network named 't2_proxy' for container '/qbittorrent'! Maybe you're missing the project's prefix in the label? Defaulting to first available network." serviceName=qbit-svc providerName=docker container=qbittorrent-3a12808bc04ea4d7f08eef3a32d93c8de70e23e7b5b96d1c5088d498f7876124

from docker-traefik.

My setup does work. However, if the VPN container restarts or reconnects or watchtower updates it, all the containers routed through it fail, returning a bad gateway through the reverse proxy. They are still running they just don’t have network access. Restarting them fixes the issue.

@rubbo898 If you restart your VPN container and wait until it’s back up then try and visit one of your containers routed through the VPN, do they work?

from docker-traefik.

@robflate my setup does the same thing when I remove the VPN container, watchtower seems to work ok for me however.

from docker-traefik.

@cwilford What about if you restart the VPN container. Do the routed containers work once the VPN container is back running?

sudo docker-compose up -d --force-recreate <vpn-container-name>

from docker-traefik.

@robflate same behaviour for me too, if you restart the VPN container the tunneled ones must be restarted too (with some latency)
For updates and generic rebuilding scenario Unraid OS automatically rebuild all the containers attached to the VPN one, so It's not a big deal.

from docker-traefik.

I moved this subject to discussions on VPN

from docker-traefik.

Amazing! Can’t wait to try it out.

from docker-traefik.