[🐛 Bug]: Helm post-upgrade hook fails when recreating selenium-grid-edge-scaledobject about docker-selenium HOT 24 CLOSED

I am trying to add a Job to run together with hooks to execute kubectl command to patch those. Hope it can be implemented

from docker-selenium.

0.30.0 works like a charm. Thank you!

from docker-selenium.

@oranlom, in chart 0.30.1, I renamed the label in object metadata. Hence, the command also changed something. The upgrade from 0.30.0 to 0.30.1 would result as failed. However, moving forward, the upgrade is expected to be stable

from docker-selenium.

@oranlom, thank you for creating this issue. We will troubleshoot it as soon as we can.

from docker-selenium.

Looks like the issue is in backward chart versions. Can you try 2 possible ways?

1. In your deployments/selenium-grid/envs/DEV/deployment.yaml, add these configs for KEDA sub-chart

keda:
  additionalAnnotations:
    "helm.sh/hook": pre-install

2. Upgrade to use recent chart versions (as of now 0.29.0+)

from docker-selenium.

Ah ok, I see you are using enableWithExistingKEDA: true. Probably above suggestions might not work. Let me try to reproduce this one

from docker-selenium.

Yes, those keda.additionalAnnotations are already there by default. Additionally, our selenium helm deploy starts in the different step of the infrastructure pipeline, when keda is already deployed. I'm currently trying to deploy the 0.29.0 version, but there's a lot of changes introduced so I need to tweak my helm-values first to avoid issues in other pipelines with some dependencies on selenium.

from docker-selenium.

So I upgraded the chart to version 0.29.0 but I got the same error.

Updated helm-values.yaml file:

basicAuth:
  enabled: false
ingress:
  enabled: false
serviceAccount:
  create: true
  nameOverride: selenium-serviceaccount
autoscaling:
  enableWithExistingKEDA: true
  scalingType: deployment
  scaledOptions:
    minReplicaCount: 0
    maxReplicaCount: 8
    advanced:
      horizontalPodAutoscalerConfig:
        behavior: # Optional. Use to modify HPA's scaling behavior
          scaleDown:
            stabilizationWindowSeconds: 300
            policies:
              - type: Percent
                value: 100
                periodSeconds: 15
secrets:
  create: false
  nameOverride: selenium-kv-secret
components:
  subPath: /

hub:
  nameOverride: selenium-hub
  serviceType: ClusterIP
  nodeSelector: { agentpool: "seleniumpool" }
  tolerations:
    - effect: NoSchedule
      key: role
      operator: Equal
      value: seleniumNode
  extraVolumeMounts:
    - name: qa-file-pv
      mountPath: /mnt/testautomation-files
    - name: selenium-kv-secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  extraVolumes:
    - name: qa-file-pv
      persistentVolumeClaim:
        claimName: qa-file-pvc
    - name: selenium-kv-secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: selenium-kv-secrets-provider
  extraEnvFrom:
    - secretRef:
        name: selenium-kv-secret

# Configuration for chrome nodes
chromeNode:
  nameOverride: selenium-chrome-node
  nodeSelector: { agentpool: "seleniumpool" }
  tolerations:
    - effect: NoSchedule
      key: role
      operator: Equal
      value: seleniumNode
  extraVolumeMounts:
    - name: qa-file-pv
      mountPath: /mnt/testautomation-files
    - name: selenium-kv-secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  extraVolumes:
    - name: qa-file-pv
      persistentVolumeClaim:
        claimName: qa-file-pvc
    - name: selenium-kv-secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: selenium-kv-secrets-provider
  extraEnvFrom:
    - secretRef:
        name: selenium-kv-secret
  scaledObjectOptions:
    scaleTargetRef:
      name: selenium-chrome-node

# Configuration for firefox nodes
firefoxNode:
  # Enable firefox nodes
  enabled: false

# Configuration for edge nodes
edgeNode:
  nameOverride: selenium-edge-node
  nodeSelector: { agentpool: "seleniumpool" }
  tolerations:
    - effect: NoSchedule
      key: role
      operator: Equal
      value: seleniumNode
  extraVolumeMounts:
    - name: qa-file-pv
      mountPath: /mnt/testautomation-files
    - name: selenium-kv-secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  extraVolumes:
    - name: qa-file-pv
      persistentVolumeClaim:
        claimName: qa-file-pvc
    - name: selenium-kv-secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: selenium-kv-secrets-provider
  extraEnvFrom:
    - secretRef:
        name: selenium-kv-secret
  scaledObjectOptions:
    scaleTargetRef:
      name: selenium-edge-node

Error message: Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade selenium-grid/templates/edge-node-hpa.yaml failed: scaledobjects.keda.sh "selenium-edge-node" already exists

Any ideas, or suggestions are more than welcome

from docker-selenium.

Looks like I also could reproduce the issue, but with the object scaledjobs. With using an existing KEDA and upgrade scenario, so I guess all KEDA crds already there, we can update the hook for post-install only. Can you try this config?

autoscaling:
  annotations:
    "helm.sh/hook": post-install

from docker-selenium.

Correct me if I'm wrong but this would actually mean that the helm-upgrade --install command in our pipeline will not trigger the redeployment of scaledobjects even if their configuration in values.yaml is changed. This is not really a desired solution :/

from docker-selenium.

As the usage of the chart hooks, without post-upgrade specification, it doesn't mean the redeployment does not happen even their configs changed. In a short description, the hooks allow us to configure an object that will be deployed (before or after) a release's life cycle (install, upgrade, or rollback).
For example: during the upgrade, all objects (configmap, svc, secrets, deployment, etc.) are deployed in parallel, once upgrade completed, the object scaledobject is deployed

from docker-selenium.

I understand the concept of chart hooks, but when the hook is set to post-install only, the deployment of object scaledobject will happen only at first time installation or clean install, not in the case of an upgrade, right? At least that's what my observation and testing showed.
Also, I don't seem to understand why the helm validation (helm-upgrade --install dry-run) outputs the manifests of an updated scaledobjects with the

  annotations:
    "helm.sh/hook": post-install

but in fact the scaledobject stay untouched with the original
annotations: helm.sh/hook: post-install, post-upgrade

So again, it seems like the redeployment of scaledobjects when upgrading will not happen when having post-install hook set. With post-upgrade hook on scaledobjects I'm getting the error shown in the description of this bug.

from docker-selenium.

How about if you validate by changing a configuration and rendering YAML output using the helm template? The new config is present in the output YAML?

from docker-selenium.

I used the following command helm template selenium-grid repo_selenium-grid/selenium-grid --create-namespace --namespace platform-test --version=0.29.0 -f deployments/selenium-grid/envs/DEV/deployment.yaml
The new config is indeed present in the output YAML.

Unfortunately after that with the following command:
helm upgrade --install selenium-grid repo_selenium-grid/selenium-grid --create-namespace --namespace platform-test --version 0.29.0 --values deployments/selenium-grid/envs/DEV/deployment.yaml
the change was not deployed.

from docker-selenium.

Ok, thanks for your info, I had misunderstood that the output in template will be deployed accordingly.
Let me relook on this right the way, first would be selenium-grid/templates/edge-node-hpa.yaml failed: object is being deleted. To see the object is being deleted in how long, or it was stuck until the hooks timed out.

from docker-selenium.

I just realized I'm not able to reproduce it by manually deploying via cli, I get this error in the pipeline exclusively, here's the raw log with timestamps from the last run, not sure if this is any helpful:

2024-04-15T12:40:02.7192162Z [command]/azp/_work/_tool/helm/3.9.0/x64/linux-amd64/helm upgrade --install --values /azp/_work/1/s/helm/deployments/selenium-grid/configurations/envs/DEV/deployment.yaml --wait --timeout 5m0s --create-namespace -n platform-test selenium-grid repo_selenium-grid/selenium-grid --version 0.29.0
2024-04-15T12:40:08.9650508Z Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade selenium-grid/templates/edge-node-hpa.yaml failed: scaledobjects.keda.sh "selenium-edge-node" already exists
2024-04-15T12:40:08.9807233Z ##[error]Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade selenium-grid/templates/edge-node-hpa.yaml failed: scaledobjects.keda.sh "selenium-edge-node" already exists

2024-04-15T12:40:08.9820775Z ##[section]Finishing: helm upgrade

from docker-selenium.

Sorry, this is the last one, it appears it's slightly different:

2024-04-18T11:22:25.5768644Z [command]/azp/_work/_tool/helm/3.9.0/x64/linux-amd64/helm upgrade --install --values /azp/_work/1/s/helm/deployments/selenium-grid/configurations/envs/DEV/deployment.yaml --wait --timeout 5m0s --create-namespace -n platform-test selenium-grid repo_selenium-grid/selenium-grid --version 0.29.0
2024-04-18T11:22:30.9688654Z Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade selenium-grid/templates/edge-node-hpa.yaml failed: object is being deleted: scaledobjects.keda.sh "selenium-edge-node" already exists
2024-04-18T11:22:30.9730983Z ##[error]Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade selenium-grid/templates/edge-node-hpa.yaml failed: object is being deleted: scaledobjects.keda.sh "selenium-edge-node" already exists

2024-04-18T11:22:30.9861278Z ##[section]Finishing: helm upgrade

from docker-selenium.

I read through the details in this ticket kedacore/keda#2507. The workaround mentioned is patch the finalizer

However, I also saw a log with 2024-01-03T01:28:17Z ERROR Failed to update ScaledObject after removing a finalizer {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"testingbug-scaled-object","namespace":"testingbug"}, "namespace": "testingbug", "name": "testingbug-scaled-object", "reconcileID": "927da8fb-3dcb-444d-b310-ac0af12153c7", "finalizer": "finalizer.keda.sh", "error": "admission webhook \"vscaledobject.kb.io\" denied the request: Deployment.apps \"testdeployment\" not found"}

In your KEDA deployment, is there pod keda-webhook running there?

from docker-selenium.

Yes, there is keda-admission-webhooks pod running in our cluster. I was able to delete the finalizer, but how would it fix the issue if after next pipeline run (helm upgrade) the finalizer is there again?

from docker-selenium.

Chart 0.30.0 is out with the job patch. You can try and share the experience

from docker-selenium.

Hi!

Could we reopen this issue? I have run into the same problem again. Sorry for reporting a success back then, but it seems like the issue does not occur in every pipeline run.
I do believe that only a small change to the patch-keda-objects-job.yaml is required. There seems to be a wrong label in the patch command itself, so the finalizer is eventually not removed. Instead of Release-name:

$ kubectl get ScaledObjects,ScaledJobs -n platform-test -l component.autoscaling=selenium-grid -o=json | jq '.metadata.finalizers = null' | kubectl apply -f - || true ;
error: no objects passed to apply

the component.autoscaling=true label should be used:

$ kubectl get ScaledObjects,ScaledJobs -n platform-test -l component.autoscaling=true -o=json | jq '.metadata.finali
zers = null' | kubectl apply -f - || true ;
scaledobject.keda.sh/selenium-chrome-node configured
scaledobject.keda.sh/selenium-edge-node configured

from docker-selenium.

Any estimation on how long could it take until the working version with this fix included is out?

from docker-selenium.

Any estimation on how long could it take until the working version with this fix included is out?

You mean a fix where any old chart versions is able to upgrade to new seamlessly (instead of uninstall the old and reinstall)?

from docker-selenium.

Oh, I'm sorry I didn't understood from your previous comment that manual uninstall before upgrading to 0.30.1 is the way I should proceed. It works fine now, thanks!

from docker-selenium.