Comments (16)
We're shipping a rc later this week that I'm hopeful will address this issue that you've been hitting -- this is part of a refactor in conjunction with us moving to k8s 1.10. Will keep you updated.
from cni-ipvlan-vpc-k8s.
Initial testing looks good, we are going to deploy to a larger cluster
from cni-ipvlan-vpc-k8s.
Just to confirm, since you're not using skipDeallocation, I assume you are also not using the "GC" mechanism?
I think we hit a case of this in a mass scale up ourselves this week, and I agree more logging around the actual plugin invocations would be beneficial here.
from cni-ipvlan-vpc-k8s.
I've been seeing the same behavior. I haven't checked the cloudtrail logs (for some reason, that idea hadn't occurred to me). This happens quite frequently for us though. For every 30-40 pods we launch, 2-3 of them get stuck in this state. @theatrus - are you able to share the configuration (cni config, OS, anything about # subnets, etc) that you run this in successfully?
from cni-ipvlan-vpc-k8s.
We run with skipDeallocation true and the GC stack cleaning up unused IPs every 5 minutes on a 15 minute expiration.
@paulnivin identified some race conditions which there is a PR in the works for free-IP detection which could be exasperated by not using skipDeallocation. When IPAM allocates a new IP address, its possible for a second IPAM run to grab that same IP address, which means both runs of the ipvlan plugin attempt to use the same address. Only one will succeed, and that same IP address will then be released by the teardown.
Using skipDeallocation avoids the issue of the IP being freed immediately (it will be marked in the file "registry" as free), which should side-step the issue until the racing patch can be added (the racing patch also uses the registry for this reason).
from cni-ipvlan-vpc-k8s.
I don't think the GC feature is documented in the README. Would it be possible for you to add it?
from cni-ipvlan-vpc-k8s.
@theatrus yes we are not using the gc mechanism (I only have seen references to it in the code and in the cli tool). We'll try with SkipDeallocation set to true
Just to understand better the race condition: when the CNI plugin is run at the same time for two pods, and grab the same address, one will fail (makes sense). What will trigger the teardown? An explicit delete from the runtime on the failed sandbox?
from cni-ipvlan-vpc-k8s.
@lbernail That is correct, as the IPvlan plugin will fail due to "address already in use" errors, the sandbox is then torn down. As both sandboxes "have" the same IP, the IP is released from AWS even though its actually in use elsewhere.
@gdearment Docs will be added shortly, I forgot to followup when adding the initial support here. The quick summary is, in order to free IPs, you need to run the -tool in a systemd timer (or cron). We run it every 5 minutes with the following unit:
[Unit]
Description=Garbage collect IPs unused for 15 minutes
[Service]
Type=oneshot
ExecStart=/usr/local/bin/cni-ipvlan-vpc-k8s-tool registry-gc --free-after=15m
Note that there is still a race condition possible here, but it should not lead to a fatal condition and can be safely retried. We're tracking a third, much rarer variant, where GC also releases an in-use IP but have only had one occurrence of this happen with stress testing.
from cni-ipvlan-vpc-k8s.
@theatrus Any risk in running with SkipDeallocation and no gc?
Since IP addresses will be marked as free in the local registry it should not be an issue right, they will still be reused? (only limit is that we will allocate more private IPs on the ENI than needed)
from cni-ipvlan-vpc-k8s.
No issue, except for never returning IPs to Amazon until the node terminates.
from cni-ipvlan-vpc-k8s.
Thanks!
Should we have seen the "address already in use" error somewhere? I found nothing in the kubelet logs
from cni-ipvlan-vpc-k8s.
OK running with SkipDeallocation=True has really helped a lot (no issue related to this this week)
I wonder if we could default this flag to True until the race condition has been fixed. What do you think?
from cni-ipvlan-vpc-k8s.
Hey @theatrus or @paulnivin - do you have any details or a pr to solve the race? We are running with SkipDeallocation but we still see periodic errors. They resolve automatically but they create event noise that complicates monitoring. 😞
from cni-ipvlan-vpc-k8s.
That's great new thanks
from cni-ipvlan-vpc-k8s.
@lbernail Now that #47 has been merged, can you give master a spin and see if this resolves the race you've been seeing?
from cni-ipvlan-vpc-k8s.
@paulnivin Sure. I'll just need to rebase our nodeport fix and to update our CNI configuration to move to chained execution. I'll probably have this deployed on a test cluster Monday. I'll let you know how it goes
from cni-ipvlan-vpc-k8s.
Related Issues (20)
- Clarify configuration and setup instructions
- The CNI plugin only works on OS where interfaces are called ethX
- Additional interfaces could be in the same subnet HOT 8
- Plugin does not work with containerd runtime HOT 6
- missing routes on first pod for an interface HOT 3
- Nodeports not working properly HOT 8
- Incorrect handling of errors in IPAM HOT 1
- CNI plugin: `portmap` when chained with cni-ipvlan-vpc-k8s does not work
- Listing VPC CIDRs to create routes in pods does not filter on association state
- New release HOT 1
- Disabling ipMasq
- Cannot connect to pod outside VPC HOT 4
- Pods loosing connection to each other on particular node from time to time HOT 13
- Build binaries for 0.5.0 release HOT 1
- aws-vpc-cni-k8s vs lyft cni HOT 3
- IP Batching changes the default behavior HOT 1
- Brother, I compile for a day. Compilation is unsuccessful. Please find a compiled package.
- Binary Release for 0.6.1 HOT 3
- support for containerd
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cni-ipvlan-vpc-k8s.