weka / csi-wekafs Goto Github PK

👋 Hi, are there any plans to add support for snapshots?
Thanks!

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency go to v1.22.5
• chore(deps): update alpine to v3.20
• fix(deps): update module github.com/container-storage-interface/spec to v1.10.0
• fix(deps): update module google.golang.org/grpc to v1.65.0
• fix(deps): update opentelemetry-go monorepo to v1.28.0 (go.opentelemetry.io/otel, go.opentelemetry.io/otel/sdk, go.opentelemetry.io/otel/trace)
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Hello,
I faced issue that PVC mount get timed out when one pod requests multiple PVC mount.
Pod trys to mount but any volume can't be mounted, these PVC mounts are retried several times and finally pod can't be startup.

Here's my condition.
I made two PV (hereby static provision I used), and these two PV are sharing same sub-directory path.

I ran an model and this model create its log in same path dir/v1/storageclass-test/jslouisyou/model/ in this case, and below two PV are attached.

spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: "1"
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: pv-output-model
    namespace: default
    resourceVersion: "27069727"
    uid: 3f985fdd-966b-452c-97c1-81d10cdc2e2b
  csi:
    driver: csi.weka.io
    volumeHandle: dir/v1/storageclass-test/jslouisyou/model/
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: "1"
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: pv-output-log
    namespace: default
    resourceVersion: "27069647"
    uid: 1b558ff7-b2c3-4052-a013-8340d2ae1d47
  csi:
    driver: csi.weka.io
    volumeHandle: dir/v1/storageclass-test/jslouisyou/model/
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

but when I tried to create a new pod, then Unable to attach or mount volumes, timed out waiting for the condition issue occurs and pod can't be started.

$ kubectl describe pod -n default test-0
...
Volumes:
  model:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pv-output-model
    ReadOnly:   false
  log:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pv-output-log
    ReadOnly:   false

...
Events:
  Type     Reason       Age                    From     Message
  ----     ------       ----                   ----     -------
  Warning  FailedMount  67s (x156 over 5h52m)  kubelet  (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[log model], unattached volumes=[${REDACTED} log model]: timed out waiting for the condition

I found that PVC mount can be done when I try to mount just one PV mount, but multiple mount with same subpath is failed.

This issue also happens in longhorn (NFS filesystem),
And people have similar issues with multiple mounts:
kubernetes/kubernetes#110158
https://stackoverflow.com/questions/53383994/error-marking-master-timed-out-waiting-for-the-condition-kubernetes

and this issue also happens in EFS CSI driver:
kubernetes-sigs/aws-efs-csi-driver#100 (comment)

I think that many CSI drivers share this issue but EFS CSI driver resolved it by itself.

So I'm wondering that is this expected behavior?
If any user try to mount multiple volume, then these multiple volume should have different sub-directory path?

Thanks.

Hi, using the driver requires me to mount WEKA file system on node as prerequisites ? the CSI driver can mount the volume for me ?

Hello,
Recently I faced some issues while using csi-wekafs plugin, which csi-wekafs-controller POD stucked by some reason and then all csi-wekafs-node PODs can't communicate with csi-wekafs-controller POD.

And It seems that csi-wekafs-controller POD which was spawned from statefulset is not configured as supporting HA, only number of replicas is set to 1.

I'm wondering that If I change replicas: 1 to replicas: 3 or something else, then all behaviors like attaching, resizing, mounting, unpublishing, ... are working well?

Thanks.

I don't seem to be able to get metrics for each PVC mounted by the system from the Weka CSI driver, such as:

kubelet_volume_stats_capacity_bytes - the per-PVC capacity in bytes.
kubelet_volume_stats_used_bytes - the per-PVC space usage in bytes.
kubelet_volume_stats_available_bytes - the per-PVC free space in bytes.

Is there a specific configuration option needed to enable these, or is it not currently supported by the driver at present?

The current driver doesn't appear to support fsGroupPolicy as described on the CSI Driver fsGroup Support page and the design doc.

With the default StorageClass permissions: "0750" value, I'm running into trouble starting pods as non-root users without getting initial permission problems on lots of files.

Suppose the following use-case: A Kubernetes-user creates a Weka FileSystem (either dynamically via PVC or statically). The user populates the volume with some data and later wants to share that volume with other users. But before sharing it, the volume should be marked read-only to prevent modification to the data. How to achieve this?

Options we have considered:

1. Changing the accessMode of the PV and the PVC to ReadOnlyMany does not seem to prevent a pod from writing data into the mounted weka filesystem. The wekafs is mounted rw anyhow, and files can be created, changed, deleted.
2. Changing the .spec.csi.readOnly to true in the PV is forbidden, because the spec is immutable.
3. Setting read-only mounts and volumes in Pod.spec.containers.volumeMounts[x].readOnly and Pod.spec.volumes.persistentVolumeClaim.readOnly resp. might work, but doesn't prevent a user from accidentally not setting it and therefore modifying data.
4. Using a Weka-user with "Read-Only" user role is not supported, because as per documentation at least an OrgaAdmin is required.
5. One could recursively go through the filesystem and mark every single file and directory read-only. Given that we're dealing with several 100000 files, this would take awful long and is no guarantee that a file isn't deleted accidentally by a root-user in a Kubernetes pod.
6. The Weka web UI doesn't seem to provide an option for marking a filesystem read-only.

Is there any other option to prevent modifications to an entire Weka filesystem?

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: renovate.json
Error type: Invalid JSON (parsing failed)
Message: Syntax error near // Renovat

Hello,

Are there any general limitations for mounting WEKA volume in k8s? I can't find any of these in kubernetes documentations or anywhere.

For example,

1. The maximum number of PersistentVolume or PersistentVolumeClaim can be created within same StorageClass across all nodes, is limited?
2. The maximum number of mounted filesystem limits within one node limited? (I'm using Ubuntu 20.04)

Thanks!

Does this mean I can only have 1024 PVCs?

Hi Sergey,

as discussed, we would like to be able to override the value for fsGroupPolicy of the CSIDriver:

Currently this is hardcoded to File. This causes a serious problem: Consider a statically provisioned volume with a few million files in it. If that volume is mounted by many users in many pods, the kubelet will chmod / chown all the files in that volume when starting pods. This can take a very long time, during which the kubelet is busy and not willing to e.g. terminate other pods.

This problem even occurs, when people do not specify fsGroup in their pod spec. In this case it just defaults to some value. And if that default owner / file mode doesn't match with the one in the volume (because e.g. someone as made a modification to a file or directory), the kubelet will chmod / chmown again.

Kubernetes suggests setting the fsGroupChangePolicy in each pod manifest to OnRootMismatch. But this is not sufficient either, because if a mismatch is detected in the root directory, the kubelet will still do chmod / chown on everything.

Hence, the only option we see is to set the fsGroupPolicy of the CSIDriver to None, to prevent the kubelet from messing with the permissions. So having a way to specify that in the Helm chart would be nice :)