Giter VIP home page Giter VIP logo

Comments (7)

Dbzman avatar Dbzman commented on July 20, 2024

Hey again,
I got it to work by downgrading Docker to 1.12.
Seems like I ran into the issue which is also described in the K8S 1.8 release notes:

Docker 1.13.1 and 17.03.2

Shared PID namespace, live-restore, and overlay2 were validated.

Known issues

The default iptables FORWARD policy was changed from ACCEPT to DROP, which causes outbound container traffic to stop working by default. See #40182 for the workaround.

The support for the v1 registries was removed.

The issue linked there kubernetes/kubernetes#40182 looks like it should've been fixed already since it's from January and the pod networks should've had enough time to circumvent this.
Looking further at that issue I see that weave had addressed this (or at least something related to that) in weaveworks/weave#2758 which it has been merged several days ago. Sadly it's not released yet.
What do you think? Should we downgrade Docker to 1.12 for now, wait for weave or add iptable rules ourselves?

Also: Are you able to reproduce that? It can be simply reproduced by using one of the above mentioned images on a worker node and calling "nslookup" with an external domain. Then it should already give the wrong ip.

EDIT:
I noticed this strange DNS issue again after working with the cluster the whole day. Restarting the cluster solved it partly.

from ansible-kubernetes-openshift-pi3.

rhuss avatar rhuss commented on July 20, 2024

Hmm, for me DNS seems to work also with Docker 17.03.2 without changing iptables (and using weave).

I get e.g. this on rpi-alpine:3.6

k exec -it test ash
/ # nslookup www.heise.de
nslookup: can't resolve '(null)': Name does not resolve

Name:      www.heise.de
Address 1: 193.99.144.85 www.heise.de
Address 2: 2a02:2e0:3fe:1001:7777:772e:2:85 www.heise.de

/ # ping www.heise.de
PING www.heise.de (193.99.144.85): 56 data bytes
64 bytes from 193.99.144.85: seq=0 ttl=248 time=13.606 ms
64 bytes from 193.99.144.85: seq=1 ttl=248 time=34.437 ms

which looks good to me. I don't know where the cant resolve (null) comes from

I used this pod for testing:

apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  containers:
  - image: "hypriot/rpi-alpine:3.6"
    name: sleep
    args:
    - "sleep"
    - "3600"

from ansible-kubernetes-openshift-pi3.

rhuss avatar rhuss commented on July 20, 2024

Your PR for the flannel support is still pending, which includes updating the iptable rules.

I'm going to review the PR and try to move the iptables change up so that is should be globally applied.

from ansible-kubernetes-openshift-pi3.

rhuss avatar rhuss commented on July 20, 2024

I just updated the scripts and added the iptables ruless for accepting forwards to the top-level kubernetes role.

I tested it with flannel and it worked without issues:

  • I could access other services internally by name
  • I could resolve external names via ping

I suspect you might have a different setup and might not have a proper NAT routing setup on your desktop so that the cluster node can reach the outside (and return packages are routed properly back to the node). For Mac OS X I had to setup the proper forwarding rules with https://github.com/Project31/ansible-kubernetes-openshift-pi3/blob/master/tools/setup_nat_on_osx.sh

from ansible-kubernetes-openshift-pi3.

Dbzman avatar Dbzman commented on July 20, 2024

Thanks for checking!
I think I found the issue. A collegue was pointing me to that yesterday.

So, what I experienced so far:

  • DNS sometimes suddenly stops working for most of my containers (I only get "62.138.239.45" for many hosts)
  • This IP address is the DNS error page of Telekom
  • Restarting the Cluster helps solving the problem

I thought that DNS did not work because I always got that IP back. In reality it worked properly but my internet provider returned that specific IP on failed DNS requests.

What happens in Kubernetes here is:

  • A container queries kube-dns to resolve a DNS name
  • kube-dns calls the nameserver and gets the above mentioned IP back (instead of an real error)
  • It's a valid IP, so kube-dns chaches this IP in dnsmasq
  • This wrong IP is now registered for a very long time for the requested host name

https://www.heise.de/newsticker/meldung/Telekom-leitet-DNS-Fehlermeldungen-um-213726.html
Luckily this behaviour can be turned off.
I did that and it works pretty well so I'll upgrade docker again.

Not sure if we really need that global iptables rules. It looks like my issues were completely related to the behaviour above.

Maybe we should mention somewhere that some providers can cause DNS problems since people will probably use this RaspberryPi setup also at home.

Also: Was it intentional that you switchted the default to flannel, again?^^

One last thing: Thank you so much for these scripts. It's so convenient and easy to provision a RaspberryPi cluster with that. You did an awesome job here! :)

from ansible-kubernetes-openshift-pi3.

rhuss avatar rhuss commented on July 20, 2024

Thanks for the compliments and that you like the playbooks ;-) 'hope I can continue to work on it soon, so to add an ingress controller (traefik) and rook for persistent volumes.

wrt the CNI plugin, I have no strong bias, so occasionally I switch back and forth ;-). In this case, I want to try whether flannel runs stable, too. Let's keep the iptable forward rules top-level, I don't think they harm (and are required for flannel anyway).

BTW, some time ago I had the same very same issue with the Deutsche Telekom DNS server. IMO this behaviour is totally bogus and since DNS issues are always hard to debug, this makes it even harder. I will add a FAQ section to the README and add this information.

thanks ...

from ansible-kubernetes-openshift-pi3.

Dbzman avatar Dbzman commented on July 20, 2024

I got ingress working on my setup already. Maybe I'll create pull request in the next days since I want to reprovision anyways for the Docker update. :)
Persistent volumes would be really nice. Currently I mount the host path which works but is not ideal.

Okay, I don't have a preference for the CNI as well. ;) At least it's always good to have the option to switch it if one of them has certain bugs or so.

I'm really happy that I got this solved now. I also think that behaviour is totally wrong as they completely violate the DNS protocol with that. Interesting that you had that issue was well before. :D

Thank you again!

from ansible-kubernetes-openshift-pi3.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.