Priority class calico-priority isn't high enough for the calico-node daemonset about operator HOT 9 CLOSED

I'm sorry I just understood that there were 2 different system-*-critical priority classes here. Even though in my first message I mentioned them both. I see what you were saying now. I think your suggestion sounds good

I think the perfect solution would be to default the Calico components to system-node-critical and system-cluster-critical if the Kubernetes version is v1.17 or greater.

from operator.

I would happily review a PR for this change. I added a backlog item for this but am not sure when it will get prioritized.
I did verify we do not need to maintain K8s v1.16 compatibility so it won't be necessary to maintain the previous calico priority class, we can remove that and just use the new system-*-critical classes. I wanted to pass that information along if you were inclined to submit a PR.

from operator.

Thanks @tmjd that was literally my next question. I'll create a PR for this, hopefully this week but if not it'll be the week after next due to annual leave.

from operator.

FYI the comment about system-node-critical only being usable in kube-system is no longer correct as of Kubernetes v1.17.

from operator.

If you are saturating a node with pods all with system-cluster-critical, then won't the system still evict pods, but since everything is system-node-critical then the pods evicted will be randomly chosen? I think the only thing replacing the calico-priority with system-cluster-critical would do is ensure that calico-node isn't the 1st pods evicted when a node is under pressure.

I'm not saying we shouldn't do this but I don't know if it is solving this expected behavior.

The calico-node daemonset should always be scheduled onto a node.

from operator.

I'd be willing to review a PR with this change. I'm not sure if we can just make this switch directly or need to include checking the kubernetes version of the cluster to continue using calico-priority on K8s older than v1.17.

from operator.

@tmjd I think the specifics here are that calico-node should use system-node-critical which has a higher priority than system-cluster-critical (used for deployments and stateful sets). If a daemonset can't be scheduled due to a lower priority it stays in the pending state, if it's a deployment then it will cause a node scale out event (if CA or equivalent is supported). In our case we have a set of tainted nodes for our system components which either use no priority class, system-node-critical or system-cluster-critical; if calico-node doesn't use system-node-critical it can't guarantee that it will be scheduled on these nodes.

I think the perfect solution would be to default the Calico components to system-node-critical and system-cluster-critical if the Kubernetes version is v1.17 or greater.

from operator.

@tmjd is this something you'd need a PR for or is it something that will be actively looked into?

from operator.

@tmjd I've opened #1473 to fix this.

from operator.