CA isn't scaling up due to unfamiliar extended resources requests about autoscaler HOT 8 OPEN

Hi @MaciekPytel. I found out a similar issue in #3852 but then got impression that CA does ignore extended resources which are unfamiliar to it and move on with scaling up. Has that behavior changed recently?

from autoscaler.

@x13n any thoughts on this?

from autoscaler.

CA will observe other nodes in a node group to get an idea how new nodes will look like - if they contain extended resources that are later required by a pod, a scale up should get triggered. However, if there are no nodes or if there's a mix of node with and without the extended resource, CA will have no idea about such extended resource and won't trigger scale up because it will think it won't help the pod.

from autoscaler.

bserve other nodes in a node group

Is there way to tell CA to ignore the resources it is unaware of ?

from autoscaler.

I don't think so - why do you ask? What would be the use case for ignoring some resources?

from autoscaler.

I don't think so - why do you ask? What would be the use case for ignoring some resources?

In my situation, there's a controller in place that adds extended resources to new nodes. These extended resources are later requested by some pods, just like the native resources CPU and memory. Let's take an example: imagine I have two nodes in my cluster that are almost maxed out, leaving no free resources for new pods. Now, I want to create a workload Pod that requests extended resources. The CA realizes it can't fit these Pods onto existing nodes due to the resource shortage (in this case, the extended resources like example.com/myresource). Since the CA doesn't recognize these extended resources, it would be helpful if it could still proceed with creating new nodes. Meanwhile, my external controller would continue its job of adding the extended resources to the nodes. Currently, these pods remain pending because there's no node with enough extended resources, and the CA can't create new nodes.

from autoscaler.

If CA ignored extended resources on both pods and nodes, it could scale a node group that won't have the extended resource later on. Or it could also create nodes for pods requesting, say, example.com/myresource: 1000, even though your controller can only set the allocatable up to, say, example.com/myresource: 10. Or it could create nodes with example.com/myresource even though the pod requested example.com/myotherresource. Such pods would stay pending, only now there will also be an empty node around. I guess in a controlled environment you could ensure these edge conditions won't happen, so some kind of optional feature in CA might make sense. On the other hand, isn't this a use case for Dynamic Resource Allocation? Current version doesn't really work with autoscaling, but IIUC there is ongoing work to make it compatible.

CC @towca @MaciekPytel

from autoscaler.