Comments (7)
Hey again,
I got it to work by downgrading Docker to 1.12.
Seems like I ran into the issue which is also described in the K8S 1.8 release notes:
Docker 1.13.1 and 17.03.2
Shared PID namespace, live-restore, and overlay2 were validated.
Known issues
The default iptables FORWARD policy was changed from ACCEPT to DROP, which causes outbound container traffic to stop working by default. See #40182 for the workaround.
The support for the v1 registries was removed.
The issue linked there kubernetes/kubernetes#40182 looks like it should've been fixed already since it's from January and the pod networks should've had enough time to circumvent this.
Looking further at that issue I see that weave had addressed this (or at least something related to that) in weaveworks/weave#2758 which it has been merged several days ago. Sadly it's not released yet.
What do you think? Should we downgrade Docker to 1.12 for now, wait for weave or add iptable rules ourselves?
Also: Are you able to reproduce that? It can be simply reproduced by using one of the above mentioned images on a worker node and calling "nslookup" with an external domain. Then it should already give the wrong ip.
EDIT:
I noticed this strange DNS issue again after working with the cluster the whole day. Restarting the cluster solved it partly.
from ansible-kubernetes-openshift-pi3.
Hmm, for me DNS seems to work also with Docker 17.03.2 without changing iptables (and using weave).
I get e.g. this on rpi-alpine:3.6
k exec -it test ash
/ # nslookup www.heise.de
nslookup: can't resolve '(null)': Name does not resolve
Name: www.heise.de
Address 1: 193.99.144.85 www.heise.de
Address 2: 2a02:2e0:3fe:1001:7777:772e:2:85 www.heise.de
/ # ping www.heise.de
PING www.heise.de (193.99.144.85): 56 data bytes
64 bytes from 193.99.144.85: seq=0 ttl=248 time=13.606 ms
64 bytes from 193.99.144.85: seq=1 ttl=248 time=34.437 ms
which looks good to me. I don't know where the cant resolve (null)
comes from
I used this pod for testing:
apiVersion: v1
kind: Pod
metadata:
name: test
spec:
containers:
- image: "hypriot/rpi-alpine:3.6"
name: sleep
args:
- "sleep"
- "3600"
from ansible-kubernetes-openshift-pi3.
Your PR for the flannel support is still pending, which includes updating the iptable rules.
I'm going to review the PR and try to move the iptables change up so that is should be globally applied.
from ansible-kubernetes-openshift-pi3.
I just updated the scripts and added the iptables ruless for accepting forwards to the top-level kubernetes role.
I tested it with flannel and it worked without issues:
- I could access other services internally by name
- I could resolve external names via ping
I suspect you might have a different setup and might not have a proper NAT routing setup on your desktop so that the cluster node can reach the outside (and return packages are routed properly back to the node). For Mac OS X I had to setup the proper forwarding rules with https://github.com/Project31/ansible-kubernetes-openshift-pi3/blob/master/tools/setup_nat_on_osx.sh
from ansible-kubernetes-openshift-pi3.
Thanks for checking!
I think I found the issue. A collegue was pointing me to that yesterday.
So, what I experienced so far:
- DNS sometimes suddenly stops working for most of my containers (I only get "62.138.239.45" for many hosts)
- This IP address is the DNS error page of Telekom
- Restarting the Cluster helps solving the problem
I thought that DNS did not work because I always got that IP back. In reality it worked properly but my internet provider returned that specific IP on failed DNS requests.
What happens in Kubernetes here is:
- A container queries kube-dns to resolve a DNS name
- kube-dns calls the nameserver and gets the above mentioned IP back (instead of an real error)
- It's a valid IP, so kube-dns chaches this IP in dnsmasq
- This wrong IP is now registered for a very long time for the requested host name
https://www.heise.de/newsticker/meldung/Telekom-leitet-DNS-Fehlermeldungen-um-213726.html
Luckily this behaviour can be turned off.
I did that and it works pretty well so I'll upgrade docker again.
Not sure if we really need that global iptables rules. It looks like my issues were completely related to the behaviour above.
Maybe we should mention somewhere that some providers can cause DNS problems since people will probably use this RaspberryPi setup also at home.
Also: Was it intentional that you switchted the default to flannel, again?^^
One last thing: Thank you so much for these scripts. It's so convenient and easy to provision a RaspberryPi cluster with that. You did an awesome job here! :)
from ansible-kubernetes-openshift-pi3.
Thanks for the compliments and that you like the playbooks ;-) 'hope I can continue to work on it soon, so to add an ingress controller (traefik) and rook for persistent volumes.
wrt the CNI plugin, I have no strong bias, so occasionally I switch back and forth ;-). In this case, I want to try whether flannel runs stable, too. Let's keep the iptable forward rules top-level, I don't think they harm (and are required for flannel anyway).
BTW, some time ago I had the same very same issue with the Deutsche Telekom DNS server. IMO this behaviour is totally bogus and since DNS issues are always hard to debug, this makes it even harder. I will add a FAQ section to the README and add this information.
thanks ...
from ansible-kubernetes-openshift-pi3.
I got ingress working on my setup already. Maybe I'll create pull request in the next days since I want to reprovision anyways for the Docker update. :)
Persistent volumes would be really nice. Currently I mount the host path which works but is not ideal.
Okay, I don't have a preference for the CNI as well. ;) At least it's always good to have the option to switch it if one of them has certain bugs or so.
I'm really happy that I got this solved now. I also think that behaviour is totally wrong as they completely violate the DNS protocol with that. Interesting that you had that issue was well before. :D
Thank you again!
from ansible-kubernetes-openshift-pi3.
Related Issues (20)
- typo in header HOT 1
- docker daemon doesn't launch after running 'base node setup' playbook HOT 3
- Task: "Check for an already generated token" requires local sudo without password HOT 1
- Bug with kube-proxy
- Storage driver HOT 8
- persistentvolume-binder disabled by default HOT 4
- kubeadm 1.7.1 cannot start due to bug HOT 1
- Management Role includes "includes/install_k8s_resource.yml"? HOT 8
- Docker downgrade fails HOT 30
- After clean Install: Port occupied HOT 3
- Error to exec playbook "kubernetes.yml" HOT 6
- Suggestion: Pin Kubernetes components (like Docker) HOT 1
- Horizontal Pod Scaler cannot get resources
- Fail when run Kubernetes playbook.
- Extraneous '\n' in the system.yml file HOT 1
- TASK [base : Copy SSH Key] Problem.
- Getting an error running the kubernetes playbook HOT 4
- Not sure what's going on... HOT 15
- Two times a Destination not writeable error in kubernetes.yml
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ansible-kubernetes-openshift-pi3.