VPN Workflow Problem about kontena HOT 51 CLOSED

I'm playing with this now, might have a little cli command you could add to the docs to maybe get setup easier/faster.

from kontena.

Still having some issues:

~$ sudo openvpn --config ./kontena-vpn/konetna-vpn.ovpn 
Thu Sep  3 14:12:04 2015 OpenVPN 2.3.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Dec  1 2014
Thu Sep  3 14:12:04 2015 Control Channel Authentication: tls-auth using INLINE static key file
Thu Sep  3 14:12:04 2015 UDPv4 link local: [undef]
Thu Sep  3 14:12:04 2015 UDPv4 link remote: [AF_INET]{remote_ip}:1194
Thu Sep  3 14:12:05 2015 [54.210.193.104] Peer Connection Initiated with [AF_INET]{remote_ip}:1194
Thu Sep  3 14:12:08 2015 TUN/TAP device tun0 opened
Thu Sep  3 14:12:08 2015 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Thu Sep  3 14:12:08 2015 /sbin/ip link set dev tun0 up mtu 1500
Thu Sep  3 14:12:08 2015 /sbin/ip addr add dev tun0 local 192.168.255.6 peer 192.168.255.5
RTNETLINK answers: Invalid argument
Thu Sep  3 14:12:08 2015 ERROR: Linux route add command failed: external program exited with error status: 2
Thu Sep  3 14:12:08 2015 Initialization Sequence Completed

{Hangs, ^C pressed}

Thu Sep  3 14:21:59 2015 event_wait : Interrupted system call (code=4)
Thu Sep  3 14:21:59 2015 /sbin/ip addr del dev tun0 local 192.168.255.10 peer 192.168.255.9
Thu Sep  3 14:21:59 2015 SIGINT[hard,] received, process exiting

I thought it was strange it was looking to tunnel 192.168.x.x because the network I am on is a 10.* network. However that could just be my lack of VPN knowledge. Any thoughts?

from kontena.

IP Forwarding seems to be enabled....

~$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

from kontena.

What ubuntu version are you using (workstation)?

from kontena.

14.04LTS on all machines I am working with. Thanks!

On Thu, Sep 3, 2015, 2:53 PM Jari Kolehmainen [email protected]
wrote:

What ubuntu version are you using (workstation)?

—
Reply to this email directly or view it on GitHub.

from kontena.

192.168.255.0/24 network is used only with vpn client/server. Could you run openvpn command with more verbose output, like this:

~$ sudo openvpn --verb 6 --config ./kontena-vpn/konetna-vpn.ovpn 

And report what it says just before RTNETLINK answers: Invalid argument .

from kontena.

Ok, just tested this with default Ubuntu 14.04 virtual machine and it seems that docker bridge reserves the same ip address on workstation as inside the nodes. Easiest workaround for this is to change your workstation docker bridge address to something else than 172.17.0.0/16.

Edit /etc/default/docker and add --bip 10.200.42.1/16 (or some subnet that does not create conflict). Then reboot your workstation.

In addition to this you need to tweak ovpn file a bit:

• change route 172.17.42.1 255.255.255.254 to route 172.17.42.1 255.255.255.255
• (this is a bug, I'll add separate issue about it)
• add resolvconf scripts to end of file:

up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

After these changes you must start openvpn with --script-security 2 flag:

~$ sudo openvpn --script-security 2 --config ./kontena-vpn/konetna-vpn.ovpn

from kontena.

Perfect, I'll give that a shot, then we can maybe tweak the docs a bit for
the Ubuntu install to make this go a little cleaner for other folks. Thanks
for the ideas. I'll report back.

-Seth

On Fri, Sep 4, 2015 at 12:56 AM, Jari Kolehmainen [email protected]
wrote:

Ok, just tested this with default Ubuntu 14.04 virtual machine and it
seems that docker bridge reserves the same ip address on workstation as
inside the nodes. Easiest workaround for this is to change your workstation
docker bridge address to something else than 172.17.0.0/16.

Edit /etc/default/docker and add --bip 10.200.42.1/16 (or some subnet
that does not create conflict). Then reboot your workstation.

In addition to this you need to tweak ovpn file a bit:

• change route 172.17.42.1 255.255.255.254 to route 172.17.42.1
  255.255.255.255
  • (this is a bug, I'll add separate issue about it)
• add resolvconf scripts to end of file:

up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

After these changes you must start openvpn with --script-security 2 flag:

~$ sudo openvpn --script-security 2 --config ./kontena-vpn/konetna-vpn.ovpn

—
Reply to this email directly or view it on GitHub
#150 (comment).

from kontena.

@four43 did those instructions solve your problem?

from kontena.

Sorry @jakolehm, we were moving offices and networks, I wanted to make sure this was still an issue for me and that it wasn't caused by anything else. Now that we are all settled in, I can look at this again.

1. I am still able to duplicate the issue I was having earlier on our new network setup.
2. Output of OpenVPN, using the --verb 6 option, as you probably saw:

...
Wed Sep  9 10:52:56 2015 us=542991 PUSH: Received control message: 'PUSH_REPLY,dhcp-option DNS 8.8.4.4,dhcp-option DNS 8.8.8.8,route 192.168.255.1,topology net30,ping 10,ping-restart 60,ifconfig 192.168.255.6 192.168.255.5'
Wed Sep  9 10:52:56 2015 us=543073 OPTIONS IMPORT: timers and/or timeouts modified
Wed Sep  9 10:52:56 2015 us=543093 OPTIONS IMPORT: --ifconfig/up options modified
Wed Sep  9 10:52:56 2015 us=543108 OPTIONS IMPORT: route options modified
Wed Sep  9 10:52:56 2015 us=543122 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Wed Sep  9 10:52:56 2015 us=543360 ROUTE_GATEWAY 192.168.11.1/255.255.255.0 IFACE=wlan0 HWADDR=c4:e9:84:1d:9f:d0
Wed Sep  9 10:52:56 2015 us=543937 TUN/TAP device tun0 opened
Wed Sep  9 10:52:56 2015 us=543972 TUN/TAP TX queue length set to 100
Wed Sep  9 10:52:56 2015 us=544001 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Wed Sep  9 10:52:56 2015 us=544036 /sbin/ip link set dev tun0 up mtu 1500
Wed Sep  9 10:52:56 2015 us=545779 /sbin/ip addr add dev tun0 local 192.168.255.6 peer 192.168.255.5
Wed Sep  9 10:52:56 2015 us=548897 /sbin/ip route add 10.81.0.0/16 via 192.168.255.5
Wed Sep  9 10:52:56 2015 us=552937 /sbin/ip route add 172.17.42.1/31 via 192.168.255.5
RTNETLINK answers: Invalid argument

1. Added --bip 10.200.42.1/16 to the Docker conf, I am on a 192.168.x.x network now.
2. Restart Docker sudo service docker restart
3. Updated the VPN config route map from 255.255.255.254 to 255.255.255.255, added resolvers at the bottom.

That worked! Excellent fix! Now to make that a little less scary for others. Sounds like you have a patch up. Need help updating the docs perhaps?

Thanks for the help and looking into this promptly.

-Seth

from kontena.

OpenVPN image is already patched so the route problem should be fixed. Just shoot a pull request if you have improvements to docs, any help is appreciated :)

from kontena.

Still having some issues here. VPN is connected, it says "Initialization Sequence Completed" so I left a terminal window open with that running yet. However I'm still having issues resolving "registry.kontena.local" from within a script. I then tried it in my browser and got a ERR_NAME_NOT_RESOLVED error. If I run dig I get this:

~$ dig registry.kontena.local

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> registry.kontena.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59257
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;registry.kontena.local.        IN  A

;; ANSWER SECTION:
registry.kontena.local. 1   IN  A   10.81.21.50

;; Query time: 38 msec
;; SERVER: 172.17.42.1#53(172.17.42.1)
;; WHEN: Wed Sep 09 15:23:29 CDT 2015
;; MSG SIZE  rcvd: 78

docker push is still giving me fits though: dial tcp: lookup registry.kontena.local: no such host

Thoughts?

from kontena.

For reference:

~$ curl http://registry.kontena.local/v1/
curl: (6) Could not resolve host: registry.kontena.local

from kontena.

Did you add following lines to .ovpn file?

up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

(without those ubuntu does not seem to update it's dns settings)

from kontena.

I did, it added the entries to resolv.conf:

# /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 172.17.42.1
nameserver 8.8.4.4
nameserver 8.8.8.8
search kontena.local mydomain.com

Like I said, dig works, but others seem to fail, which is super odd.

from kontena.

As expected running host against that resolv.conf:

~$ host registry.kontena.local
registry.kontena.local has address 10.81.21.50
Host registry.kontena.local not found: 3(NXDOMAIN)
Host registry.kontena.local not found: 3(NXDOMAIN)

from kontena.

Talking with a friend of mine, he recommended checking out /etc/nsswitch.conf if things are resolving weird:

# /etc/nsswitch.conf
...
hosts:          files mdns4_minimal [NOTFOUND=return] dns mdns4
networks:       files
...

By default, glibc programs that do DNS queries will just use an mdns probe on the network and let whoever respond it seems.

I found this Avahi-Daemon that is part of Ubuntu by default. It is a zeroconf networking solution. It tries to listen for *.local address and resolve them via mDNS from what I can tell: http://ubuntuforums.org/showthread.php?t=1339516

Something something about .local unicast dns servers....

from kontena.

Interesting, what happens if you just try to resolve registry name (without kontena.local)?

from kontena.

Ah! That is working! Assuming because it is one of the search domains?

from kontena.

Yes, search domain causes that.. I think registry.<grid_name> should also work (might be handy if you have multiple grids)

from kontena.

ehhhh kind of:

~$ host registry.mygrid.kontena.local
registry.mygrid.kontena.local has address 10.81.21.50
Host registry.mygrid.kontena.local not found: 3(NXDOMAIN)
Host registry.mygrid.kontena.local not found: 3(NXDOMAIN)
~$ host registry.mygrid
Host registry.mygrid not found: 3(NXDOMAIN)
~$ host registry.mygrid.
Host registry.mygrid. not found: 3(NXDOMAIN)
$ host registry.mygrid.kontena.local
registry.mygrid.kontena.local has address 10.81.21.50
Host registry.mygrid.kontena.local not found: 3(NXDOMAIN)
Host registry.mygrid.kontena.local not found: 3(NXDOMAIN)
~$ host registry.mygrid
Host registry.mygrid not found: 3(NXDOMAIN)
~$ host registry
registry.kontena.local has address 10.81.21.50
Host registry.kontena.local not found: 3(NXDOMAIN)
Host registry.kontena.local not found: 3(NXDOMAIN)

from kontena.

So, now the new problem. If I do: docker push registry/namespace/container-name 1.0.0 it tries to push to the docker.io (their hub) lol. SO CLOSE

from kontena.

I have to admit that I didn't expect so many problems when using Ubuntu desktop (longtime user and usually linux has fewer problems vs win/osx when it comes to network related stuff). As a temporary workaround you can always push images using ip address (ip will change only when you update/deploy registry). Just remember to use dns name on service definition when pointing to internal images (so that they work even if ip changes).

from kontena.

@nevalla can you confirm if these problems exist on newer Ubuntu releases?

from kontena.

I can push to registry. (note the dot) to make sure docker knows it's a domain name, not just a namespace. This seems to work. It's ugly, but it works.

from kontena.

Things that seem to work well:

1. Changes mentioned above to VPN config, sounds like those are on un-released code already, sweet.
2. Changes made to binding IP of docker, I used --bip 192.168.254.1/24
3. Turning off Ubuntu's dumb avahi-daemon, not needed if your network is setup right.

sudo service avahi-daemon stop;
sudo sh -c "echo manual > /etc/init/avahi-daemon.override";

(via http://askubuntu.com/questions/205937/how-can-i-disable-avahi-daemon)

Now I can access the registry at registry.kontena.local

from kontena.

I think we could get around this if we didn't use .local. Thoughts on switching to just .kontena? registry.kontena for example. Possibly look into registry.mygrid.kontena if the registry is per grid?

from kontena.

Now I'm having issues connecting to the VPN again after the weekend.

kontena vpn config outputs: {"error":"Not found"}

Yet kontena vpn create outputs Vpn already exists

:(

from kontena.

The VPN client is trying to connect to one of the grid members, not that master. Is that normal? I forget if it did that before it broke...

from kontena.

Don't know the details but this sounds like a docker upgrade regression (1.6 -> 1.7 may loose reference to volume). Vpn should recover if you re-deploy it ( kontena service deploy vpn ).

from kontena.

Btw you can see vpn service details with following commands:

$ kontena service show vpn
$ kontena service logs vpn

Maybe it helps with debugging.

from kontena.

Thanks for the tips, I think I found the issue.

We are running our Kontena cluster on AWS using their Security Groups to power our firewall. I verified the IP the OpenVPN client is trying to connect to is the correct IP.

~$ kontena service show vpn
mygrid/vpn:
  status: running
  stateful: yes
  scaling: 1
  image: kontena/openvpn:ethwe
  dns: vpn.mygrid.kontena.local
  affinity: 
    - node==ip-10-0-0-99
  cmd: 
  env: 
    - OVPN_SERVER_URL=udp://{{OUR-VPN-IP}}:1194
  ports:
    - 1194:1194/udp
  volumes:
  volumes_from:
  links: 
  cap_add:
    - NET_ADMIN
  cap_drop:
  containers:

Up and running, looks good, right?

I ran openvpn --verb 6 --script-security 2 --config ./kontena-vpn/konetna-vpn.ovpn and found that the UDP packets don't seem to be making it to the server:

UDPv4 WRITE [42] to [AF_INET]{{OUR-VPN-IP}}:1194: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 pid=[ #1 ] [ ] pid=0 DATA len=0

It does that for a while, I tried telnet (TCP) too:

~$ telnet {{OUR-VPN-IP}} 1194
Trying {{OUR-VPN-IP}}...
telnet: Unable to connect to remote host: Connection refused

So what the heck! I logged into the grid server, the one hosting the VPN service and found this:

~$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1155/sshd       
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      5133/docker-proxy
tcp6       0      0 :::22                   :::*                    LISTEN      1155/sshd       
udp        0      0 0.0.0.0:43769           0.0.0.0:*                           646/dhclient    
udp        0      0 0.0.0.0:68              0.0.0.0:*                           646/dhclient    
udp6       0      0 :::1194                 :::*                                1395/docker-proxy
udp6       0      0 :::42286                :::*                                646/dhclient  

Port 1194 is only bound on an IPv6 address. I'm using IPv4 style addresses to connect. Could this be causing my issue?

Thanks for all the help guys, we'll get it!

EDIT: It might be binding to IPv4 too? moby/moby#2174 (comment) I admittedly am just like he described, I know enough about netstat to make assumptions.

iptables output:

~$ sudo iptables -t nat -nxvL
Chain PREROUTING (policy ACCEPT 4 packets, 306 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
    2181   166680 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 2 packets, 153 bytes)
    pkts      bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 4 packets, 262 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
   10801   888763 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 3 packets, 191 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
   12965  1054309 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
       0        0 MASQUERADE  udp  --  *      *       172.17.0.3           172.17.0.3           udp dpt:1194
       0        0 MASQUERADE  tcp  --  *      *       172.17.4.60          172.17.4.60          tcp dpt:8080

Chain DOCKER (2 references)
    pkts      bytes target     prot opt in     out     source               destination         
      18     1260 DNAT       udp  --  !docker0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:1194 to:172.17.0.3:1194
       0        0 DNAT       tcp  --  !docker0 *       0.0.0.0/0            127.0.0.1            tcp dpt:8080 to:172.17.4.60:8080

from kontena.

Have you opened port 1194 on Security Group's inbound rules?

from kontena.

I have for TCP and UDP. From what I can tell AWS Security groups only deal with IPv4 addresses at this time.

from kontena.

Perhaps more pressing, I'm getting: kontena vpn config outputting {"error":"Not found"} still, even after the re-deploy. showing the VPN service is same as above, looks to be running.

from kontena.

How about $ kontena service logs vpn? Are there any hints what's going on?

from kontena.

There are no entries since last Friday, the 11th, when everything was working. It's like the VPN service became totally dissociated from the rest of Kontena.

from kontena.

Is it the same result if you drop vpn service ($kontena vpn delete) and then re-create it?

from kontena.

Haven't tried that:

~$ kontena vpn delete
{"code":503,"message":"Node unavailable","backtrace":null}

Interesting.

~$ kontena grid list
Name                           Nodes    Services     Users     
mygrid *                       1        0            1    

from kontena.

how about $ kontena node list?

from kontena.

I updated and upgraded the servers to make sure I am on the latest version, still having issues.

~$ kontena node list
Name                           OS                                       Driver          Labels                         Status    
ip-10-0-0-99                   Ubuntu 14.04.2 LTS (3.13.0-48-generic)   aufs            -                              online  

That looks to be the correct node, the VPN service's affinity seems to point to that one.

Thanks for the help @nevalla , appreciating the input.

from kontena.

So I can stop and start the vpn container but I can't delete it.

~$ kontena service delete vpn = nope
~$ kontena service stop vpn = service stopped when I check it
~$ kontena service start vpn = service started again when I check it

from kontena.

Also checking the logs through kontena cli only shows log entries from last Friday, however I can check logs on the kontena node directly through docker and there are entries from Sunday, slightly better, still not today however.

from kontena.

can you also check logs from kontena-agent container. Seems that there might be something wrong.

from kontena.

On the Kontena Node, the kontena/agent:0.8.2 container:

I, [2015-09-15T16:40:00.399835 #1]  INFO -- QueueWorker: initialized
I, [2015-09-15T16:40:00.400846 #1]  INFO -- WebsocketClient: initialized with token Ln5to3nH8M+HOS3kIE1yKNoWV4Us085iCK05HCzrirXxTRRHY6z4swueguyxowsqIe2F5/v8trLAW/n5RtgcnQ==
I, [2015-09-15T16:40:00.401523 #1]  INFO -- LogWorker: initialized
I, [2015-09-15T16:40:00.402175 #1]  INFO -- WeaveAttacher: initialized
I, [2015-09-15T16:40:00.403291 #1]  INFO -- CadvisorLauncher: initialized
I, [2015-09-15T16:40:00.403334 #1]  INFO -- StatsWorker: initialized
I, [2015-09-15T16:40:00.403376 #1]  INFO -- EtcdLauncher: initialized
I, [2015-09-15T16:40:00.414118 #1]  INFO -- WeaveAttacher: fetching containers information
I, [2015-09-15T16:40:00.532226 #1]  INFO -- NodeInfoWorker: publishing node information
I, [2015-09-15T16:40:00.532361 #1]  INFO -- ContainerInfoWorker: fetching containers information
I, [2015-09-15T16:40:00.532765 #1]  INFO -- EventWorker: started to stream docker events
I, [2015-09-15T16:40:00.536490 #1]  INFO -- StatsWorker: waiting for cadvisor
I, [2015-09-15T16:40:00.543213 #1]  INFO -- LogWorker: starting to stream logs for container: 6857ac291cbbe7ccd36b564e1b406f32a0847966f6b516b88b9a0b958bab90d1
I, [2015-09-15T16:40:00.552073 #1]  INFO -- StatsWorker: cadvisor is running, starting stats loop
E, [2015-09-15T16:40:00.892273 #1] ERROR -- ContainerInfoWorker: publish_info: no such id: 6d9ec81b2b414661aea36b03f0c36b44ba8fb3b66ff05b0078104307321a47b8

E, [2015-09-15T16:40:00.965903 #1] ERROR -- ContainerInfoWorker: publish_info: no such id: 6d9ec81b2b414661aea36b03f0c36b44ba8fb3b66ff05b0078104307321a47b8

I, [2015-09-15T16:40:02.919854 #1]  INFO -- LogWorker: starting to stream logs for container: 91b64349330787e9e95943af6f04709ca357503b1d43938ccbf7798692f38bf9
I, [2015-09-15T16:40:20.554579 #1]  INFO -- WebsocketClient: connection closed with code: 1006
I, [2015-09-15T16:40:41.660170 #1]  INFO -- WebsocketClient: connection closed with code: 1006




I, [2015-09-15T16:41:00.979588 #1]  INFO -- ContainerInfoWorker: fetching containers information
I, [2015-09-15T16:41:00.981648 #1]  INFO -- WebsocketClient: connection closed with code: 1006
/usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/socket.rb:139:in `rescue in block in connect': connect timeout reached (Excon::Errors::Timeout)
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/socket.rb:119:in `block in connect'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/socket.rb:110:in `each'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/socket.rb:110:in `connect'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/socket.rb:28:in `initialize'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/connection.rb:389:in `new'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/connection.rb:389:in `socket'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/connection.rb:106:in `request_call'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/middlewares/mock.rb:47:in `request_call'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/middlewares/instrumentor.rb:22:in `request_call'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/middlewares/base.rb:15:in `request_call'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/middlewares/base.rb:15:in `request_call'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/middlewares/base.rb:15:in `request_call'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/middlewares/base.rb:15:in `request_call'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon/connection.rb:233:in `request'
    from /usr/lib/ruby/gems/2.2.0/gems/excon-0.45.4/lib/excon.rb:229:in `get'
    from /app/lib/kontena/helpers/node_helper.rb:8:in `node_info'
    from /app/lib/kontena/weave_adapter.rb:111:in `start!'
    from /app/lib/kontena/agent.rb:31:in `start!'
    from /app/bin/kontena-agent:27:in `<main>'

It doesn't seem to be connecting to the master?

I think we're on to something! The master seems happy though...

Kontena Master, issues with kontena-server-api:

E, [2015-09-15T16:38:16.503710 #1] ERROR -- : RPC error: RpcClient::Error Node unavailable
184.105.35.130 - - [15/Sep/2015 16:38:17] "GET /v1/services/mygrid/vpn HTTP/1.1" 200 576 0.0059
184.105.35.130 - - [15/Sep/2015 16:38:17] "GET /v1/services/mygrid/vpn HTTP/1.1" 200 576 0.0054
184.105.35.130 - - [15/Sep/2015 16:38:17] "GET /v1/services/mygrid/vpn/containers HTTP/1.1" 200 17 0.0035
184.105.35.130 - - [15/Sep/2015 16:39:51] "GET /v1/services/mygrid/vpn HTTP/1.1" 200 576 0.0056
184.105.35.130 - - [15/Sep/2015 16:39:51] "GET /v1/services/mygrid/vpn/containers HTTP/1.1" 200 17 0.0037

from kontena.

I GOT IT! The two Kontena servers couldn't talk to each other. Somewhere they got blocked on TCP port 8080. I destroyed the VPN and re-created it, downloaded the new config and it's working! Totally my fault.

from kontena.

That said, I think there is maybe an underlying issue with stale data and not being able to connect to the Kontena Nodes... All indications from the kontena-cli pointed to the node being okay, kontena node list made it seem fine.

from kontena.

I'm gonna open a separate issue for that, as it probably doesn't just pertain to VPN, then we can get back on topic, make some docs for Ubuntu specific business then call it good. Whew. Thanks @nevalla

from kontena.

Great to hear you got it working! We definitely need to check why Kontena master thought node was online. We have already pushed some effort to this issue #95, but apparently more investigation is needed.

from kontena.

Thanks @nevalla, appreciate the help as always.

from kontena.

Closing this because vpn should work after these:
#265
#267

from kontena.