Allow pod-overrides of `volumes` and container-overrides of `volumeMounts` in Devfiles about che HOT 7 CLOSED

Pod/container overrides for volumes/volumemounts is now possible thanks to devfile/devworkspace-operator#1223

I'm able to add a CSI volume to a devworkspace pod/container by following these steps to create a SharedSecret: https://docs.openshift.com/container-platform/4.14/storage/container_storage_interface/ephemeral-storage-shared-resource-csi-driver-operator.html

Example overrides:

schemaVersion: 2.2.0
attributes:
  controller.devfile.io/storage-type: per-workspace
metadata:
  name: che-demo
  attributes:
    pod-overrides:
      spec:
        volumes:
          - csi:
              driver: csi.sharedresource.openshift.io
              readOnly: true
              volumeAttributes:
                sharedSecret: my-share
            name: my-csi-volume
components:
- name: workspace
  attributes:
    container-overrides:
      volumeMounts:
        - mountPath: /etc/my-csi-volume
           name: my-csi-volume
           readOnly: true
  container: 
    ...

I believe this covers your use case @cgruver ?

Yes! That covers the use case.

from che.

Mounting the SharedSecret like so: #22751 (comment) is working in the latest Che:

To get it working, I manually added csi in the list of volumes in the container-build scc (the default scc for workspaces).

from che.

There are currently a couple of options to mount a CSI volume:

1. Che administrator can specify a specific storage class using .spec.devEnvironments.storage.perWorkspaceStrategyPvcConfig.storageClass in the CheCluster CR
2. The developer can pre-create the CSI volume and a corresponding PVC with the label controller.devfile.io/mount-to-devworkspace: "true" and this will be automounted in workspaces.

@cgruver can you confirm that those 2 options don't work for your use case? And that what you are requesting is to add the capability in the Devfile API? For example:

schemaVersion: 2.2.0
metadata:
  name: mydevfile
components:
  - name: mydevfile
    container:
      (...)
      volumeMounts:
        - name: cache
          path: /.cache
  - name: cache
    volume:
      size: 2Gi
+      csi:
+        driver: some-csi-driver

My worry is this will make the Devfile less portable (i.e. it works on my cluster but doesn't on yours).

from che.

@l0rd The use case in question here is not for PVCs, but for things like AWS Secrets Manager or SPIFFE.

The CheCluster CR config will not work, because these CSI volumes will often be workspace specific, and it's possible that there will be a need for volumes from different drivers.

The easiest option is probably to allow them to be injected via pod-overrides and container-overrides

pod-overrides: {"spec": {"volumes": [{"name": "mysecret-volume", "csi": {"driver": "some-csi-driver", "readOnly" : true, "volumeAttributes": {"attrib1": "some-value"}}}]}}
container-overrides: {"volumeMounts": [{"name": "mysecret", "mountPath": "/var/my/path", "readOnly": true}]}

Portability is less of an issue, because in this scenario, the workspace is dependent on the volume anyway. If a given cluster does not have the needed CSI drivers, then the workspace won't work anyway.

from che.

@cgruver ok, I am still not sure if the developer can or not pre-create the volume. And adding this information in the devfile still looks like a hack. However I agree with you that the pod/container-overrides you are trying to apply should work and we need to figure out how to fix it (if that's a problem with the default SCC, then it's configurable and doesn't require any change in the code).

For a better solution to this use case: have you thought about leveraging the cloud-credential-operator? We introduced the support for SA token volume projection to add GCP credentials in workspaces (see GCP workload identity federation) when the cloud-credential-operator is installed and configured in the cluster. The cloud-credential-operator supports AWS out of the box (and I assume that SPIFFE could be supported too).

from che.

Pod/container overrides for volumes/volumemounts is now possible thanks to devfile/devworkspace-operator#1223

I'm able to add a CSI volume to a devworkspace pod/container by following these steps to create a SharedSecret: https://docs.openshift.com/container-platform/4.14/storage/container_storage_interface/ephemeral-storage-shared-resource-csi-driver-operator.html

Example overrides:

schemaVersion: 2.2.0
attributes:
  controller.devfile.io/storage-type: per-workspace
metadata:
  name: che-demo
  attributes:
    pod-overrides:
      spec:
        volumes:
          - csi:
              driver: csi.sharedresource.openshift.io
              readOnly: true
              volumeAttributes:
                sharedSecret: my-share
            name: my-csi-volume
components:
- name: workspace
  attributes:
    container-overrides:
      volumeMounts:
        - mountPath: /etc/my-csi-volume
           name: my-csi-volume
           readOnly: true
  container: 
    ...

I believe this covers your use case @cgruver ?

from che.

I have a permissions issue when trying with Che however (my cluster died):

status:
  conditions:
    - lastTransitionTime: '2024-01-31T01:54:49Z'
      message: 'Error creating DevWorkspace deployment: Detected unrecoverable deployment condition: FailedCreate pods "workspace07eac1a5666447ee-6d865d86cb-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: "SETGID": capability may not be added, provider restricted-v2: .initContainers[0].capabilities.add: Invalid value: "SETUID": capability may not be added, provider restricted-v2: .initContainers[0].allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "SETGID": capability may not be added, provider restricted-v2: .containers[0].capabilities.add: Invalid value: "SETUID": capability may not be added, provider restricted-v2: .containers[0].allowPrivilegeEscalation: Invalid value: true: Allowing privilege escalation for containers is not allowed, provider "restricted": Forbidden: not usable by user or serviceaccount, spec.volumes[0]: Invalid value: "csi": csi volumes are not allowed to be used, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]'

from che.