dns diagnosis tool proposal: test about dns HOT 15 CLOSED

cc @thockin @bowei

from dns.

Some thoughts:

From what I've seen, a lot of the problems are related to DNS working in general but not for a specific service entry, meaning testing latency across different kube-dns managed entries would be less informational. Some other things to possible for the tool to look at:

• QPS that is seen at each of the kube-dns instances; {in,out} of dnsmasq, kube-dns.
• Contents of kube-system:kube-dns.
• /etc/resolv.conf, /etc/hosts for the pod that is having DNS connectivity issues
• Number of UDP conntrack entries present on the node with problems.

from dns.

thanks @bowei, sounds great, for the last two tests/queries, how to identify nodes with problems? Maybe the execution should be split into two rounds, one for checking if there are faulty nodes (eg. QPS < threshold) and a second pass to extract more info about them?

from dns.

We can take a look at failing DNS pods. Frequently issues are opened stating that pods can't be scheduled or are bouncing between healthy/unhealthy.

from dns.

cool @cmluciano thanks! Do you think we need an issue for discussing the process to be performed by the tool itself?

from dns.

I think we have a few ideas captured in this issue that can be placed in a PR.

from dns.

Indeed :) on it

from dns.

Ok, I have already this https://github.com/kubernetes/dns/compare/master...fgimenez:diagnoser?expand=1 with e2e tests passing, it would be great to get some feedback before it gets too hairy, mainly about the general implementation, if that's ok i'll continue with the actual tasks.

Not sure if I should propose it as is or wait until it does something moderately useful, maybe in that case the PR will be big(ish) WDYT?

It follows the suggestion by @thockin of having a job that exits with error so that it gets rescheduled, currently the job definition is under a manifests directory, probably this should go somewhere else.

In order to get info about a real running kubedns in the e2e tests i'll need to set it up, should i deploy the complete addon? If so, how to do that from the repo? haven't found any example in the existing tests, probably the most simlar to this situation is the sidecar-e2e test, should i follow that pattern?

Thanks!

cc/ @thockin @cmluciano @bowei

from dns.

@fgimenez -- you can open that as a PR and assign it to me, it will be much easier to comment on in that form

from dns.

thanks @bowei, done #129 not able to assign it though

from dns.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

from dns.

/remove-lifecycle stale

from dns.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

from dns.

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

from dns.

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

from dns.