Inconsistency between ENI allocated IPs and OS configuration about cni-ipvlan-vpc-k8s HOT 16 OPEN

We're shipping a rc later this week that I'm hopeful will address this issue that you've been hitting -- this is part of a refactor in conjunction with us moving to k8s 1.10. Will keep you updated.

from cni-ipvlan-vpc-k8s.

Initial testing looks good, we are going to deploy to a larger cluster

from cni-ipvlan-vpc-k8s.

Just to confirm, since you're not using skipDeallocation, I assume you are also not using the "GC" mechanism?

I think we hit a case of this in a mass scale up ourselves this week, and I agree more logging around the actual plugin invocations would be beneficial here.

from cni-ipvlan-vpc-k8s.

I've been seeing the same behavior. I haven't checked the cloudtrail logs (for some reason, that idea hadn't occurred to me). This happens quite frequently for us though. For every 30-40 pods we launch, 2-3 of them get stuck in this state. @theatrus - are you able to share the configuration (cni config, OS, anything about # subnets, etc) that you run this in successfully?

from cni-ipvlan-vpc-k8s.

We run with skipDeallocation true and the GC stack cleaning up unused IPs every 5 minutes on a 15 minute expiration.

@paulnivin identified some race conditions which there is a PR in the works for free-IP detection which could be exasperated by not using skipDeallocation. When IPAM allocates a new IP address, its possible for a second IPAM run to grab that same IP address, which means both runs of the ipvlan plugin attempt to use the same address. Only one will succeed, and that same IP address will then be released by the teardown.

Using skipDeallocation avoids the issue of the IP being freed immediately (it will be marked in the file "registry" as free), which should side-step the issue until the racing patch can be added (the racing patch also uses the registry for this reason).

from cni-ipvlan-vpc-k8s.

I don't think the GC feature is documented in the README. Would it be possible for you to add it?

from cni-ipvlan-vpc-k8s.

@theatrus yes we are not using the gc mechanism (I only have seen references to it in the code and in the cli tool). We'll try with SkipDeallocation set to true

Just to understand better the race condition: when the CNI plugin is run at the same time for two pods, and grab the same address, one will fail (makes sense). What will trigger the teardown? An explicit delete from the runtime on the failed sandbox?

from cni-ipvlan-vpc-k8s.

@lbernail That is correct, as the IPvlan plugin will fail due to "address already in use" errors, the sandbox is then torn down. As both sandboxes "have" the same IP, the IP is released from AWS even though its actually in use elsewhere.

@gdearment Docs will be added shortly, I forgot to followup when adding the initial support here. The quick summary is, in order to free IPs, you need to run the -tool in a systemd timer (or cron). We run it every 5 minutes with the following unit:

[Unit]
Description=Garbage collect IPs unused for 15 minutes

[Service]
Type=oneshot
ExecStart=/usr/local/bin/cni-ipvlan-vpc-k8s-tool registry-gc --free-after=15m

Note that there is still a race condition possible here, but it should not lead to a fatal condition and can be safely retried. We're tracking a third, much rarer variant, where GC also releases an in-use IP but have only had one occurrence of this happen with stress testing.

from cni-ipvlan-vpc-k8s.

@theatrus Any risk in running with SkipDeallocation and no gc?
Since IP addresses will be marked as free in the local registry it should not be an issue right, they will still be reused? (only limit is that we will allocate more private IPs on the ENI than needed)

from cni-ipvlan-vpc-k8s.

No issue, except for never returning IPs to Amazon until the node terminates.

from cni-ipvlan-vpc-k8s.

Thanks!

Should we have seen the "address already in use" error somewhere? I found nothing in the kubelet logs

from cni-ipvlan-vpc-k8s.

OK running with SkipDeallocation=True has really helped a lot (no issue related to this this week)
I wonder if we could default this flag to True until the race condition has been fixed. What do you think?

from cni-ipvlan-vpc-k8s.

Hey @theatrus or @paulnivin - do you have any details or a pr to solve the race? We are running with SkipDeallocation but we still see periodic errors. They resolve automatically but they create event noise that complicates monitoring. 😞

from cni-ipvlan-vpc-k8s.

That's great new thanks

from cni-ipvlan-vpc-k8s.

@lbernail Now that #47 has been merged, can you give master a spin and see if this resolves the race you've been seeing?

from cni-ipvlan-vpc-k8s.

@paulnivin Sure. I'll just need to rebase our nodeport fix and to update our CNI configuration to move to chained execution. I'll probably have this deployed on a test cluster Monday. I'll let you know how it goes

from cni-ipvlan-vpc-k8s.