Unexpected workspace restart due to scale down of a replicaset about che HOT 11 CLOSED

Adding devworkspace-operator label for initial investigation, however, it might completely unrelated to DWO

from che.

@amisevsk do you happen to have any ideas why deployment could be scaled down randomly?

from che.

I'm not sure the steps to reproduce is complete -- when I start a workspace from the provided devfile, the dashboard is not cloned and I don't see any commands (maybe a bug in the che-tasks-extension?). I cloned the che-dashboard repository and opened it in code but the devfile tasks failed to run at that point.

from che.

Hi @amisevsk ,

That's my fault, I missed some steps in the How to reproduce section. I've added missed steps, and hopefully, now it should work.

from che.

Hi @akurinnoy -- looks like your edits didn't get saved. The steps look the same as before to me :)

from che.

@amisevsk Finally I've updated the description, sorry for that :)

from che.

Thanks! I was able to reproduce the issue now (after running update dependencies as well, in case anyone runs into issues).

From the DWO logs, it is indeed updating the workspace deployment which is triggering a rollout (i.e. a new pod since the replicaset is updated). This seems to be due to something updating the DevWorkspace to have a new environment variable:

  		Name: \"CHE_DASHBOARD_URL\",
  		Value: strings.Join({
  			\"http\",
- 			\"://amisevsk-che-dashboard-local-server\",
+ 			\"s://che-dogfooding\",
  			\".<redacted>.com\",
  		}, \"\"),

which is coming from the che-code editor's DevWorkspaceTemplate.

@akurinnoy Is there a chance the dashboard, while running in dogfooding mode, is updating this DevWorkspaceTemplate to use its current dashboard URL?

from che.

@amisevsk let me check it.

from che.

Hi @amisevsk ,
I got two reloads in a row, and there were no requests that may update the DevWorkspaceTemplate...

from che.

I'll take another look.

from che.

I've tested it again on dogfooding and what I've found is that as soon as I load the dashboard in my browser, the URL for CHE_DASHBOARD_URL is updated with the current dashboard's URL (which is different in the running-within-a-workspace case).

Since DWO does not automatically trigger reconciles when DevWorkspaceTemplates are updated, this change is not initially noticed by the DevWorkspace Operator. However, when something does trigger a reconcile on the workspace, DWO will notice that the resolve workspace has a different CHE_DASHBOARD_URL and update the deployment, triggering a restart of the workspace. You can test this by manually triggering a reconcile by e.g. adding a nonsense annotation to the DevWorkspace:

oc annotate --overwrite dw $DEVWORKSPACE_NAME -n $DEVWORKSPACE_NAMESPACE "recon
cile=$(date +%s)"

(Run this in the workspace terminal)

To track changes to the CHE_DASHBOARD_URL env var, I used the following bash snippet:

while true; do
  date
  oc get dwt che-code-che-dashboard -o yaml | grep -A 1 CHE_DASHBOARD_URL
  sleep 1
done

while this is running, reload either the usual Che dashboard or the one running within your workspace to see the effect.

I'm still not sure where the env var is getting set, but it seems to be on first load of the dashboard page.

from che.