Unable to upgrade standalone k0s cluster on AWS about k0smotron HOT 4 OPEN

@a13x5 thanks for detailed analysis. You are absolutely correct it's using wrong node names in this case.

The only thing I'm wondering, why do we even need to specify node names in the Plan, couldn't we just use empty selector? @makhov WDYT?

What if we're running controller-only nodes as the CP, in that case there will be no nodename and hence the same logic would not apply. AFAIK we could also cover this case using just empty selector.

from k0smotron.

Interesting. We use AUTOPILOT_HOSTNAME to prevent this kind of situations, but looks like, it doesn't always work as expected.
@a13x5 could you show us cluster configuration to run some tests?

The only thing I'm wondering, why do we even need to specify node names in the Plan, couldn't we just use empty selector?

We can't use empty selector, because sometimes there some "leftover" ControlNode objects in the workload cluster from the controllers that don't exist anymore. Autopilot will get stuck and the Plan will never succeed.

It can be addressed by another reconciler, that will maintain ControlNode list up to date, but for now I created #673 to use nodeRef if it exists to fix the issue.

from k0smotron.

@makhov Sure thing here is a full yaml (.txt because of GH rules): k0s-standalone-aws.txt

I also should add that the ControlNode has wrong hostname as well (IP is correct though)

apiVersion: autopilot.k0sproject.io/v1beta2
kind: ControlNode
metadata:
  annotations:
    autopilot.k0sproject.io/mode: controller+worker
  creationTimestamp: "2024-08-12T17:13:06Z"
  generation: 1
  labels:
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: aws-cl-1-cp-0
    kubernetes.io/os: linux
  name: aws-cl-1-cp-0
  resourceVersion: "470"
  uid: 1c3b5b23-2fd6-4a41-b935-e05bf013cba5
status:
  addresses:
  - address: 10.0.67.254
    type: InternalIP
  - address: aws-cl-1-cp-0
    type: Hostname

from k0smotron.

@a13x5 thanks, now I understand, what's going on here. You are using controller+worker node with --hostname-override, which triggers a bug in k0s autopilot.

I've created an issue in k0s repo: k0sproject/k0s#4857

Unfortunately, this also means, that it can't be fixed just by using .status.nodeRef.name. I'll try to figure out a proper solution for the issue.

from k0smotron.