Pod stay pending even though node has become schedulable about platform-aware-scheduling HOT 5 CLOSED

There are a number of optimizations in the k8s scheduler which prevent Pods from being attempted to be re-scheduled. Basically there are different queues inside the scheduler, and the scheduler then isn't telemetry savvy enough to retry the Pod scheduling just based on the fact that telemetry state has changed. But there is a 5 minute long last-resort fallback which retries unschedulable Pods periodically. Back in the days that delay was much shorter.

The good news is, that you can adjust the delay. The scheduler (deprecated) flag --pod-max-in-unschedulable-pods-duration duration Default: 5m0s

Don't be afraid of the deprecation. That flag has been deprecated for long and it isn't going away in 1.29 and perhaps not even in 1.30.

For conventional scheduler-plugins there is the EnqueueExtension API, but I'm not sure if that exists at all for extenders. And even if it did, there wouldn't be a telemetry based part in that API. But the gist of that API is that plugins can say what sort of changes in the cluster would trigger a move from the unschedulable queue to them other queues. For example a change in a node object could be such a trigger. But in the case of TAS, I suppose the only change TAS could do would be to update some label in the node.

Meanwhile, the cmd line flag is your workaround.

from platform-aware-scheduling.

Hi,

Let's try something quick: the pods that are stuck in "Pending" are they stuck in stat state forever? What happens to them after 5-6 mins?
Could you also add the TAS & K8s default scheduler logs here?

Thank you

from platform-aware-scheduling.

Hi,
Is seams that we hit the same issue. When pod is started while dontshedule policy is active then the pod get stuck in pending state. Even if we switch policy to scheduleonmetric pod remain in pending state. New pods are created properly. After 5 minutes pending pod wake up, shedule again and move to running state. Is that behaviour expected? Is it 5 minutes timeout somehow configurable?
We use tas version 0.5.0

from platform-aware-scheduling.

@jiriproX As far as I know, that behaviour is expected and it's coming from this K8s default scheduler parameter: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/ --pod-max-in-unschedulable-pods-duration duration (Default: 5m0s).

As for the configurable part, it seems to be ... but with a caveat: "This flag is deprecated and will be removed in 1.26".

So, according to the docs is you use K8s < 1.26 you should be able to tune this value. Now, I looked at the release notes for 1.26 and this deprecation isn't mentioned https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md#changes-by-kind-7 so I'd say this would need testing.

from platform-aware-scheduling.

Hello, yes I confirm I'm having the same thing, after waiting something like 5 minutes, they get scheduled again. But I'm fairly sure with my previous cluster version (1.25, now using 1.27 at the time of the issue) they would be rescheduled immediately, which is the behavior I would like to have. Thanks for the info @madalazar I will need to look into this.

from platform-aware-scheduling.