Add a strategy for taints and tolerations about descheduler HOT 27 CLOSED

any more details, please expand, I would be interested in contributing

Lets say a pod was scheduled on a node where there is mismatch between pod's toleration and node's taints, and it could be due to that the pod was scheduled on a node whose taints were later updated. Since, pod's tolerations were only checked at the time admission, and later node's taints were updated, pod continued running on the node, so it only applies for NoSchedule taints.

In summary, it would work as follows:

1. get a list of nods (which already exists)
2. get a list of pods on each node (which already exists)
3. Verify that the node's taints (NoSchedule) are still satisfied by its pods' tolerations.
4. As per previous step, If not satisfied, evict the pod.
5. If yes (still satisfied), continue checking other pods until you have reached the end of list of pods for that node.
6. Repeat above for all nodes.

from descheduler.

any more details, please expand, I would be interested in contributing

from descheduler.

@seanmalloy @damemi looks like this issue hasn't been picked up in long time. Is it okay if I give it a try? will post my queries here if any.

/assign

from descheduler.

Thanks @damemi for clarifying, I guess this issue will be closed now 🙂

from descheduler.

@paktek123 Take a look at this issue for a use case currently implemented with Draino that could be replaced by descheduler: kubernetes/node-problem-detector#199 (comment)

from descheduler.

Thanks for the explanation, I will hopefully try to contribute next week

from descheduler.

Nice idea! It is also very useful for customizing the migration time of POD.

from descheduler.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

from descheduler.

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

from descheduler.

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

from descheduler.

@fejta-bot: Closing this issue.

from descheduler.

/reopen

from descheduler.

@damemi: Reopened this issue.

from descheduler.

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

from descheduler.

@fejta-bot: Closing this issue.

from descheduler.

/reopen

from descheduler.

@isindir: You can't reopen an issue/PR unless you authored it or you are a collaborator.

from descheduler.

/reopen

from descheduler.

@seanmalloy: Reopened this issue.

from descheduler.

/remove-lifecycle rotten
/kind feature

from descheduler.

Please do!

from descheduler.

@damemi @aveshagarwal one query around this description mentioned here, Are we looking to add a new check in api itself like somewhere here: https://github.com/kubernetes-sigs/descheduler/tree/master/pkg/descheduler/node or this feature is going to be a new change w.r.t command line options as well maybe here: https://github.com/kubernetes-sigs/descheduler/tree/master/cmd/descheduler/app ?

from descheduler.

I'm not actually sure why this issue is still open, I might have mistakenly reopened it when it went stale... we have a Taints/Tolerations strategy that was merged out of this issue (https://github.com/kubernetes-sigs/descheduler#removepodsviolatingnodetaints)

Is there something that strategy is missing from the discussion here? Or can we close this?

from descheduler.

Just checking, but is it now possible to:

1. Detect permanent node problems and set Node Conditions using the Node Problem Detector and the scheduler's TaintNodesByCondition functionality.
2. Configure Descheduler to deschedule pods based on taints to cordon and drain nodes when they exhibit the NPD's KernelDeadlock condition, or a variant of KernelDeadlock we call VolumeTaskHung.
3. Let the Cluster Autoscaler scale down underutilised nodes, including the nodes Descheduler has drained.

If so, is there an example?

from descheduler.

@StevenACoffman yes, the descheduler will evict any pods that are currently running on a node that has any NoSchedule taint that the pods do not tolerate. So, your use case should work with the RemovePodsViolatingNodeTaints strategy

from descheduler.

/close

from descheduler.

@damemi: Closing this issue.

from descheduler.