piraeusdatastore / linstor-csi Goto Github PK

CSI plugin for LINSTOR

License: Apache License 2.0

Dockerfile 0.38% Makefile 0.19% Go 99.43%

csi kubernetes linstor sds persistent-storage

linstor-csi's Introduction

Linstor CSI Plugin

This CSI plugin allows for the use of LINSTOR volumes on Container Orchestrators that implement CSI, such as Kubernetes.

Building

If you wish to create a docker image for a local registry run make staticrelease.

Deployment

If you are looking to deploy a full LINSTOR setup with LINSTOR controller and satellites, take a look at our operator.

This project ONLY deploys the CSI components, a working LINSTOR cluster is required.

Kubernetes

The yaml file in examples/k8s/deploy shows an example configuration which will deploy the LINSTOR csi plugin along with the needed k8s sidecar containers. You will need to change all instances of LINSTOR_IP to point to the controller(s) of the LINSTOR cluster that you wish this plugin to interact with.

You will need to enable the following feature gates on both the kube-apiserver and all kubelets for this plugin to be operational: CSINodeInfo=true, CSIDriverRegistry=true,VolumeSnapshotDataSource=true. Please ensure that your version of Kubernetes is recent enough to enable these gates.

Usage

This project must be used in conjunction with a working LINSTOR cluster, version 0.9.11 or better. LINSTOR's documentation is the foremost guide on setting up and administering LINSTOR.

⚠️️ Known issues

Due to the way ZFS snapshots work, provisioning new Volumes from existing Volumes does not work using ZFS storage pools. The internal temporary snapshot cannot be deleted after the new volume is created.

As a workaround, first create a VolumeSnapshot of the existing volume and restore from that snapshot. Also read the issue below!
Deletion of VolumeSnapshots will fail for ZFS based volumes if a volume restored from the snapshot exists in the cluster.

Kubernetes

After the plugin has been deployed, you're free to create storage classes that point to the name of the external provisioner associateed with the CSI plugin and have your users start provisioning volumes from them. A basic storage class could look like this:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-basic-storage
provisioner: linstor.csi.linbit.com
allowVolumeExpansion: true
parameters:
  linstor.csi.linbit.com/placementCount: "2"
  linstor.csi.linbit.com/storagePool: "my-storage-pool"
  linstor.csi.linbit.com/resourceGroup: "linstor-basic-storage"
  csi.storage.k8s.io/fstype: xfs
  # You can override LINSTOR properties by adding the property.linstor.csi.linbit.com prefix:
  property.linstor.csi.linbit.com/DrbdOptions/auto-quorum: suspend-io

A full list of all parameters usable in a storage class is available here.

Ensure that all kubelets that are expected to use LINSTOR volumes have a running LINSTOR satellite that is configured to work with the LINSTOR controller configured in the plugin's deployment files and that the storage pool indicated in the storage class has been properly configured. This pool does not need to be present on the Kubelets themselves for volumes attached over the network.

Most of the documentation for using this project with Kubernetes is located here.

Kubevirt

An example of using the CSI driver in combination with kubevirt (block device mode, live migration) can be found in the examples/kubevirt/ directory.

linstor-csi's People

Contributors

Stargazers

Watchers

linstor-csi's Issues

Implement csi expand volume

What would you like to be added
For some reason we need for capacity expansion function in our business, but I found that linstor csi driver did not implement the capacity expansion function

Volumes with csi- prefix stoped working (volume not present in storage backend)

Hi, I've just updated csi-provisioner, which is probably implements this fix #3.

Now when I can create new PVCs, volumes are creating for them with the name like: pvc-bbd4682a-5024-11e9-82fd-c2b1dc5386a1 - this is working fine.

But old volumes with the name like csi-0964269a-b154-43a1-876c-d15989446479 are stopped working at all:

csi-attacher log:

I0327 00:31:03.233747       1 controller.go:173] Started VA processing "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233778       1 csi_handler.go:93] CSIHandler: processing VA "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233787       1 csi_handler.go:120] Attaching "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233793       1 csi_handler.go:259] Starting attach operation for "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.233846       1 csi_handler.go:214] PV finalizer is already set on "pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f"
I0327 00:31:03.236093       1 csi_handler.go:524] Can't get CSINodeInfo pve1: csinodeinfos.csi.storage.k8s.io "pve1" not found
I0327 00:31:03.236158       1 csi_handler.go:173] VA finalizer is already set on "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.236166       1 csi_handler.go:187] NodeID annotation is already set on "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.236180       1 connection.go:242] GRPC call: /csi.v1.Controller/ControllerPublishVolume
I0327 00:31:03.236187       1 connection.go:243] GRPC request: {"node_id":"pve1","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"storage.kubernetes.io/csiProvisionerIdentity":"1551278453900-8081-io.drbd.linstor-csi"},"volume_id":"deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23"}
I0327 00:31:03.330688       1 connection.go:245] GRPC response: {}
I0327 00:31:03.331017       1 connection.go:246] GRPC error: rpc error: code = NotFound desc = ControllerPublishVolume failed for deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23: volume not present in storage backend
I0327 00:31:03.331030       1 csi_handler.go:388] Saving attach error to "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.333783       1 controller.go:139] Ignoring VolumeAttachment "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec" change
I0327 00:31:03.333827       1 csi_handler.go:398] Saved attach error to "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec"
I0327 00:31:03.333844       1 csi_handler.go:103] Error processing "csi-c6daec176b656194f5c0e70768581463229899fa56b274fb062b27de39ee64ec": failed to attach: rpc error: code = NotFound desc = ControllerPublishVolume failed for deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23: volume not present in storage backend

Example PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: io.drbd.linstor-csi
  creationTimestamp: "2019-02-27T16:57:07Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/io-drbd-linstor-csi
  name: pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f
  resourceVersion: "40803092"
  selfLink: /api/v1/persistentvolumes/pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f
  uid: b7316819-3ab0-11e9-8fc7-f6cd6192cc92
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Mi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: eig
    namespace: default
    resourceVersion: "29068182"
    uid: 86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f
  csi:
    driver: io.drbd.linstor-csi
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1551278453900-8081-io.drbd.linstor-csi
    volumeHandle: deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23
  persistentVolumeReclaimPolicy: Delete
  storageClassName: linstor-basic-storage
  volumeMode: Filesystem
status:
  phase: Bound

Linstor does not provides any opportunity for rename already created resources, and Kubernetes does not provides to change spec.csi.volumeHandle for the already created PV.

Does it mean, that this situation have no other solution than manually edit linstor database for remove csi- prefix?

Invalid JSON in Aux/csi-volume-annotations property breaks other volumes provisioning

I'm not sure that it is expected behavior, but If any resource ddefinition in your cluster have Aux/csi-volume-annotations property with invalid JSON for some reason, then provisioning for any other volume will fail with the failed to unmarshal annotations for ResDef error.

replicas_on_same replicas_on_different stopped working

Hi, my storageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-a-b
parameters:
  ReplicasOnDifferent: side
  ReplicasOnSame: moonshot
  autoPlace: "2"
  storagePool: thindata
provisioner: linstor.csi.linbit.com

Provisioner runs:

curl -X 'POST' -d '{"select_filter":{"place_count":2,"storage_pool":"thindata","replicas_on_same":["moonshot"],"replicas_on_different":["side"]},"layer_list":["DRBD","STORAGE"]}' -H 'Accept: application/json' -H 'Content-Type: application/json' 'http://localhost:3370/v1/resource-definitions/pvc-01a17976-a143-4c31-b118-3bcccb7951e5/autoplace'

but correct commad is:

curl -X 'POST' -d '{"select_filter":{"place_count":2,"storage_pool":"thindata","replicas_on_same":["Aux/moonshot"],"replicas_on_different":["Aux/side"]},"layer_list":["DRBD","STORAGE"]}' -H 'Accept: application/json' -H 'Content-Type: application/json' 'http://localhost:3370/v1/resource-definitions/pvc-01a17976-a143-4c31-b118-3bcccb7951e5/autoplace'

version:

image: quay.io/linbit/linstor-csi:v0.7.2

Setting replication mode

According to https://www.linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-replication-protocols DRBD supports 3 replication modes.
Is it possible to define a storage class for each of these modes/protocols in order to decide per workload what the best choice for security and performance is?

Two pods on the same node cannot access a volume at the same time?

Hi!

I am trying to use Okteto with Linstor. Okteto is a tool for development in Kubernetes, and it creates a persistent volume to keep work files etc. There is a main deployment and optional additional services that can be activated, and these services are scheduled on the same node as the main one so they can share the volume. This works with Ceph, Longhorn and OpenEBS.

However with Linstor I get the following error when the second service/pod attempts to mount the volume:

MountVolume.SetUp failed for volume "pvc-d264ff30-cf3c-4494-a28e-38887156a454" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = NodePublishVolume failed for pvc-d264ff30-cf3c-4494-a28e-38887156a454: unable to get an exclusive open on /dev/drbd1009, check device health

Is there a workaround for this? Thanks!

SSL support

Hi, is there any option to use certificate based authentication with linstor-csi plugin?

mountOpts are not updating for existing volumes

Hi,
We trying to specify mountOpts in storageClass but old volumes still use old values.

I've also tried to restart csi pods, and update mount options inside Aux/csi-volume-annotations there is no changes.

Where exactly csi-driver takes options for mount filesystem?

volumeattachment can stuck on detachhing if volume already detached

Example, volumeattachment can't be removed due 404 error:

Name:         csi-b858cce844edca0ccc24195483a44f5c51cf041e042963054bcbf8524e117587
Namespace:    
Labels:       <none>
Annotations:  csi.alpha.kubernetes.io/node-id: m9c34
API Version:  storage.k8s.io/v1
Kind:         VolumeAttachment
Metadata:
  Creation Timestamp:             2020-01-22T07:16:29Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2020-01-22T09:56:27Z
  Finalizers:
    external-attacher/linstor-csi-linbit-com
  Resource Version:  2445479763
  Self Link:         /apis/storage.k8s.io/v1/volumeattachments/csi-b858cce844edca0ccc24195483a44f5c51cf041e042963054bcbf8524e117587
  UID:               912e3700-eae0-4160-9d3d-1d9f2cf93ea9
Spec:
  Attacher:   linstor.csi.linbit.com
  Node Name:  m9c34
  Source:
    Persistent Volume Name:  pvc-a5083d7f-5577-42ef-b535-f9a873da4d1a
Status:
  Attached:  true
  Detach Error:
    Message:  rpc error: code = Internal desc = ControllerpublishVolume failed for pvc-a5083d7f-5577-42ef-b535-f9a873da4d1a: 404 Not Found
    Time:     2020-01-22T12:59:20Z
Events:       <none>

but m9c34 node have no this resource attached:

# linstor r l -r pvc-a5083d7f-5577-42ef-b535-f9a873da4d1a
+----------------------------------------------------------------------------+
| ResourceName                             | Node | Port | Usage  |    State |
|============================================================================|
| pvc-a5083d7f-5577-42ef-b535-f9a873da4d1a | m9c4 | 8915 | Unused | UpToDate |
| pvc-a5083d7f-5577-42ef-b535-f9a873da4d1a | m9c5 | 8915 | Unused | UpToDate |
+----------------------------------------------------------------------------+

Workaround: create diskless resource on m9c34 and repeat the attempt

Proposed solution: consider 404 as successful response on detach operation

Error:Client API version '4' is incompatible with controller version '3' in k8s v1.13

I got an error when creating PVC for linstor-csi.
My environment was Kubernetes v1.13.

I succeeded applying cluster-csi.yaml and Storageclass.yaml.
Storageclass.yaml is shown below.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: two-replica
provisioner: io.drbd.linstor-csi
parameters:
  autoPlace: "2"
  storagePool: "pool0"
  filesystem: "xfs"

But after applying PVC.yaml , created PVC got "Pending" status.
PVC.yaml is shown below.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: two-replica
provisioner: io.drbd.linstor-csi
parameters:
  autoPlace: "2"
  storagePool: "pool0"
  filesystem: "xfs"
[ec2-user@ip-172-31-9-219 linstor]$ cat my-first-volume-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: my-first-volume
spec:
  storageClassName: two-replica
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

For understanding this situation, I put "kubectl describe pvc" command, and I got a result shown below.

Events:
  Type       Reason                Age                    From                                                                               Message
  ----       ------                ----                   ----                                                                               -------
  Normal     ExternalProvisioning  2m28s (x26 over 8m8s)  persistentvolume-controller                                                        waiting for a volume to be created, either by external provisioner "io.drbd.linstor-csi" or manually created by system administrator
  Normal     Provisioning          52s (x6 over 8m8s)     io.drbd.linstor-csi_linstor-csi-controller-0_0c2a3270-6288-11e9-9417-de9ddfef2681  External provisioner is provisioning volume for claim "default/my-first-volume"
  Warning    ProvisioningFailed    52s (x6 over 8m8s)     io.drbd.linstor-csi_linstor-csi-controller-0_0c2a3270-6288-11e9-9417-de9ddfef2681  failed to provision volume with StorageClass "two-replica": rpc error: code = Internal desc = CreateVolume failed for pvc-f45d1945-6288-11e9-b593-062a213f9ba0: exit status 1: Error: Client API version '4' is incompatible with controller version '3', please update your client.
Mounted By:  <none>

The error says "Client API version '4' is incompatible with controller version '3'" in io.drbd.linstor-csi_linstor-csi-controller.
I guess this error related with this issue: #6

When I use Kubernetes v1.13, how does linstor-csi work?

Missing documentation of many available StorageClass parameters

Hi Everyone,

In the example of https://raw.githubusercontent.com/piraeusdatastore/piraeus/master/deploy/all.yaml I've seen StorageClass parameters that are not documented neither at https://www.linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-kubernetes-advanced-configuration nor in this repo (also not in piraeus and piraeus-operator as I checked).

For example, the meaning of allowRemoteVolumeAccess or placementPolicy can only be found with a repo search in some Go source code and the changelog file:

I managed to get the information what I was searching for from the comments but since the user should create the SC on their own (piraeusdatastore/piraeus-operator#125 (comment)), and the operator repo doesn't mention that, there could be a place for improvement related to SC examples and documentation of the available options. A documentation page here with some examples of common combinations and a link back here from the operator readme's install guide could work nicely together.

Great project by the way, this is just an idea for enhancement :)

Various issues around snapshots

When working on ROX support, I noticed some issues in the current snapshot implementation:

Snapshot creation is not indempotent, i.e. the creation process may be stopped at a time where it is already present in LINSTOR but not in the metadata for CSI. This then leads to an unusable snapshot, as the driver tries to start the process from scratch. LINSTOR then complains that the snapshot is already present.
Snapshot-restore is not indempotent. Basically see above. A resource definition with restored volume definitions but without restored resources is currently not recovered.
Snapshots are marked as ReadyToUse immediatly, without waiting for the SUCCESS flag in LINSTOR.
For volume-from-volume scenarios, if restoring a volume fails, a snapshot is created for every retry instead of reusing the old one
For volume-from-volume scenarios, created snapshots are not deleted after restore
Snapshots are deleted if the PV is deleted. This violates CSI Spec: Either leave snapshots alone, or refuse to delete the PV.
See https://github.com/container-storage-interface/spec/blob/v1.2.0/spec.md#deletevolume
implementation issue: Why are we storing snapshot information as auxiliary property on resource definitions? We should rely on LINSTORs snapshot API for this.

Configuration files related to k8s deployment need to be updated

some configuration files, such as linstor-csi-[1.17|1.19|dev].yaml in examples/k8s/deploy folder, has linstor-csi-snapshotter-role define but lack of linstor-csi-snapshotter-binding:

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: linstor-csi-snapshotter-binding
subjects:
  - kind: ServiceAccount
    name: linstor-csi-controller-sa
    # You must change this to the current context
    namespace: default
roleRef:
  kind: ClusterRole
  name: linstor-csi-snapshotter-role
  apiGroup: rbac.authorization.k8s.io

ReadWriteMany accessMode

Hi,
I'm trying to create a StatefulSet with a ReadWriteMany accessMode.
It doesn't work. Is it possible to do ReadWriteMany access mode with Linstor and Lisntor CSI ?

Events on `test-0` :

Events:
  Type     Reason                  Age                    From                                  Message
  ----     ------                  ----                   ----                                  -------
  Warning  FailedScheduling        <unknown>              default-scheduler                     persistentvolumeclaim "data" not found
  Normal   Scheduled               <unknown>              default-scheduler                     Successfully assigned default/test-0 to node2
  Normal   SuccessfulAttachVolume  8m18s                  attachdetach-controller               AttachVolume.Attach succeeded for volume "pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196"
  Warning  FailedMount             7m31s (x3 over 8m3s)   kubelet, node2  MountVolume.SetUp failed for volume "pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = NodePublishVolume failed for pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196: mounting volume failed: invalid parameter: resourcegroup does not belong to paramKey values
  Warning  FailedMount             3m58s (x2 over 6m16s)  kubelet, node2  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[default-token-m9cnr data]: timed out waiting for the condition
  Warning  FailedMount             111s (x8 over 8m3s)    kubelet, node2  MountVolume.SetUp failed for volume "pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = NodePublishVolume failed for pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196: mounting volume failed: invalid parameter: drbdoptions/net/allow-two-primaries does not belong to paramKey values
  Warning  FailedMount             104s                   kubelet, node2  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data default-token-m9cnr]: timed out waiting for the condition

`kubectl logs linstor-csi-node-pvdjb -n linstor -c linstor-csi-plugin` :

time="2020-04-30T06:13:33Z" level=debug msg="method called" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=driver method=/csi.v1.Node/NodeGetCapabilities nodeID=node2 provisioner=linstor.csi.linbit.com req= resp="capabilities:<rpc:<type:GET_VOLUME_STATS > > " version=v0.7.4-9-g7383849
time="2020-04-30T06:13:33Z" level=debug msg="looking up resource by CSI volume id" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" csiVolumeID=pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 linstorCSIComponent=client
time="2020-04-30T06:13:33Z" level=info msg="curl -X 'GET' -H 'Accept: application/json' 'https://linstor-controller:3371/v1/resource-definitions/pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196'"
time="2020-04-30T06:13:33Z" level=debug msg="converted resource definition to volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=client resourceDefinition="{Name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 ExternalName:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 Props:map[Aux/csi-volume-annotations:{\"name\":\"pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196\",\"id\":\"pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2020-04-30T06:03:00.12667645Z\",\"sizeBytes\":1073741824,\"readonly\":false,\"parameters\":{\"DrbdOptions/Net/allow-two-primaries\":\"yes\",\"resourceGroup\":\"linstor_group\",\"storagePool\":\"pool_linstor\"},\"snapshots\":[]} DrbdOptions/Net/allow-two-primaries:yes DrbdOptions/Resource/quorum:off DrbdPrimarySetOn:NODE1 StorPoolName:pool_linstor] Flags:[] LayerData:[{Type:DRBD Data:0xc0003d2e10} {Type:STORAGE Data:<nil>}] Uuid:d4b77817-bf44-405a-93c0-c40035fca705}" volume="&{Name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 ID:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-30 06:03:00.12667645 +0000 UTC SizeBytes:1073741824 Readonly:false Parameters:map[DrbdOptions/Net/allow-two-primaries:yes resourceGroup:linstor_group storagePool:pool_linstor] Snapshots:[]}"
time="2020-04-30T06:13:33Z" level=debug msg="getting assignment info" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=client targetNode=node2 volume="&{Name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 ID:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-30 06:03:00.12667645 +0000 UTC SizeBytes:1073741824 Readonly:false Parameters:map[DrbdOptions/Net/allow-two-primaries:yes resourceGroup:linstor_group storagePool:pool_linstor] Snapshots:[]}"
time="2020-04-30T06:13:33Z" level=info msg="curl -X 'GET' -H 'Accept: application/json' 'https://linstor-controller:3371/v1/resource-definitions/pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196/resources/node2/volumes/0'"
time="2020-04-30T06:13:33Z" level=debug msg="found assignment info" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=client volumeAssignment="&{Vol:0xc000208800 Node:node2 Path:/dev/drbd1000}"
time="2020-04-30T06:13:33Z" level=error msg="method failed" func="github.com/sirupsen/logrus.(*Entry).Error" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:297" error="rpc error: code = Internal desc = NodePublishVolume failed for pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196: mounting volume failed: invalid parameter: drbdoptions/net/allow-two-primaries does not belong to paramKey values" linstorCSIComponent=driver method=/csi.v1.Node/NodePublishVolume nodeID=node2 provisioner=linstor.csi.linbit.com req="volume_id:\"pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196\" target_path:\"/var/lib/kubelet/pods/eecc1e3b-df42-4012-a2d9-24fadf59ef18/volumes/kubernetes.io~csi/pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196/mount\" volume_capability:<mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:\"csi.storage.k8s.io/ephemeral\" value:\"false\" > volume_context:<key:\"csi.storage.k8s.io/pod.name\" value:\"test-0\" > volume_context:<key:\"csi.storage.k8s.io/pod.namespace\" value:\"default\" > volume_context:<key:\"csi.storage.k8s.io/pod.uid\" value:\"eecc1e3b-df42-4012-a2d9-24fadf59ef18\" > volume_context:<key:\"csi.storage.k8s.io/serviceAccount.name\" value:\"default\" > volume_context:<key:\"storage.kubernetes.io/csiProvisionerIdentity\" value:\"1588184499057-8081-linstor.csi.linbit.com\" > " resp="<nil>" version=v0.7.4-9-g7383849

linstor controller

$ linstor storage-pool list
+----------------------------------------------------------------------------------------------------------------------------------+
| StoragePool          | Node  | Driver   | PoolName   | FreeCapacity | TotalCapacity | CanSnapshots | State |
|==================================================================================================================================|
| DfltDisklessStorPool | node1 | DISKLESS |            |              |               | False        | Ok    |
| DfltDisklessStorPool | node2 | DISKLESS |            |              |               | False        | Ok    |
| pool_linstor         | node1 | LVM      | vg_linstor |   334.34 GiB |    335.35 GiB | False        | Ok    |
| pool_linstor         | node2 | LVM      | vg_linstor |   335.35 GiB |    335.35 GiB | False        | Ok    |
+----------------------------------------------------------------------------------------------------------------------------------+

$ 
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Node  | Resource                                 | StoragePool          | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|=================================================================================================================================================================|
| node1 | pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 | pool_linstor         |     0 |    1000 | /dev/drbd1000 |  1.00 GiB | Unused | UpToDate |
| node2 | pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 | DfltDisklessStorPool |     0 |    1000 | /dev/drbd1000 |           | Unused | Diskless |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+

$ linstor resource-group list
+------------------------------------------------------------------------+
| ResourceGroup | SelectFilter                    | VlmNrs | Description |
|========================================================================|
| DfltRscGrp    | PlaceCount: 2                   |        |             |
|------------------------------------------------------------------------|
| linstor_group | PlaceCount: 1                   | 0      |             |
|               | StoragePool: pool_linstor       |        |             |
|               | LayerStack: ['DRBD', 'STORAGE'] |        |             |
+------------------------------------------------------------------------+

$ linstor resource-group list-properties linstor_group
+----------------------------------------------------+
| Key                                 | Value        |
|====================================================|
| DrbdOptions/Net/allow-two-primaries | yes          |
| StorPoolName                        | pool_linstor |
+----------------------------------------------------+

`drbdadm status` on node2

pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 role:Secondary
  disk:Diskless
  node1 role:Secondary
    peer-disk:UpToDate

`drbdsetup events` on node2

exists resource name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 role:Secondary suspended:no
exists connection name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 peer-node-id:0 conn-name:node1 connection:Connected role:Secondary
exists device name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 volume:0 minor:1000 disk:Diskless client:yes quorum:yes
exists peer-device name:pvc-e1d40fd7-0967-40c1-8c05-9c4a460b7196 peer-node-id:0 conn-name:node1 volume:0 replication:Established peer-disk:UpToDate peer-client:no resync-suspended:no
exists -

Definition

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  # The name used to identify this StorageClass.
  name: linstor-basic-storage-class
  # The name used to match this StorageClass with a provisioner.
  # linstor.csi.linbit.com is the name that the LINSTOR CSI plugin uses to identify itself
provisioner: linstor.csi.linbit.com
parameters:
  # LINSTOR will provision volumes from the drbdpool storage pool configured
  # On the satellite nodes in the LINSTOR cluster specified in the plugin's deployment
  storagePool: "pool_linstor"
  resourceGroup: "linstor_group"
  DrbdOptions/Net/allow-two-primaries: "yes"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test
  serviceName: test
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: test
    spec:
      securityContext:
        fsGroup: 1001
        runAsUser: 1001
      containers:
        - name: debug
          image: alpine:3.11
          command: ["/bin/sh"]
          stdin: true
          tty: true
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data
            readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: linstor-basic-storage-class
---

CreateVolume failed: unexpected end of JSON input

Today we faced with new problem when linstor-csi can not provision new volume:

Logs of linstor-csi-plugin

time="2019-08-07T09:19:54Z" level=debug msg="looking up resource by CSI volume name" func="github.com/LINBIT/linstor-csi/pkg/client.(*Linstor).GetByName" file="/home/hayley/projects/linstor-csi/pkg/client/linstor.go:478" annotationsKey=Aux/csi-volume-annotations csiVolumeName=pvc-0efd1fc6-b171-40f4-8405-05378533c265 linstorCSIComponent=client
time="2019-08-07T09:19:54Z" level=info msg="curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/resource-definitions'"
time="2019-08-07T09:19:54Z" level=error msg="method failed" func=github.com/LINBIT/linstor-csi/pkg/driver.Driver.Run.func1 file="/home/hayley/projects/linstor-csi/pkg/driver/driver.go:863" error="rpc error: code = Internal desc = CreateVolume failed for pvc-0efd1fc6-b171-40f4-8405-05378533c265: unexpected end of JSON input" linstorCSIComponent=driver method=/csi.v1.Controller/CreateVolume nodeID=pve2 provisioner=linstor.csi.linbit.com req="name:\"pvc-0efd1fc6-b171-40f4-8405-05378533c265\" capacity_range:<required_bytes:85899345920 > volume_capabilities:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:\"ReplicasOnDifferent\" value:\"side\" > parameters:<key:\"ReplicasOnSame\" value:\"moonshot\" > parameters:<key:\"autoPlace\" value:\"2\" > parameters:<key:\"mountOpts\" value:\"errors=remount-ro,discard\" > parameters:<key:\"storagePool\" value:\"thindata\" > " resp= version=v0.6.1

Logs of csi-provisioner

I0807 09:19:54.073540       1 controller.go:1196] provision "monitoring/vmstorage-volume-victoria-metrics-vmstorage-0" class "linstor-1": started
I0807 09:19:54.075472       1 controller.go:442] CreateVolumeRequest {Name:pvc-0efd1fc6-b171-40f4-8405-05378533c265 CapacityRange:required_bytes:85899345920  VolumeCapabilities:[mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[ReplicasOnDifferent:side ReplicasOnSame:moonshot autoPlace:2 mountOpts:errors=remount-ro,discard storagePool:thindata] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0807 09:19:54.075532       1 connection.go:180] GRPC call: /csi.v1.Controller/CreateVolume
I0807 09:19:54.075594       1 event.go:209] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"vmstorage-volume-victoria-metrics-vmstorage-0", UID:"0efd1fc6-b171-40f4-8405-05378533c265", APIVersion:"v1", ResourceVersion:"102791592", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/vmstorage-volume-victoria-metrics-vmstorage-0"
I0807 09:19:54.075539       1 connection.go:181] GRPC request: {"capacity_range":{"required_bytes":85899345920},"name":"pvc-0efd1fc6-b171-40f4-8405-05378533c265","parameters":{"ReplicasOnDifferent":"side","ReplicasOnSame":"moonshot","autoPlace":"2","mountOpts":"errors=remount-ro,discard","storagePool":"thindata"},"volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0807 09:19:54.090201       1 connection.go:183] GRPC response: {}
I0807 09:19:54.090697       1 connection.go:184] GRPC error: rpc error: code = Internal desc = CreateVolume failed for pvc-0efd1fc6-b171-40f4-8405-05378533c265: unexpected end of JSON input
I0807 09:19:54.090741       1 controller.go:979] Final error received, removing PVC 0efd1fc6-b171-40f4-8405-05378533c265 from claims in progress
W0807 09:19:54.090748       1 controller.go:886] Retrying syncing claim "0efd1fc6-b171-40f4-8405-05378533c265", failure 12
E0807 09:19:54.090764       1 controller.go:908] error syncing claim "0efd1fc6-b171-40f4-8405-05378533c265": failed to provision volume with StorageClass "linstor-1": rpc error: code = Internal desc = CreateVolume failed for pvc-0efd1fc6-b171-40f4-8405-05378533c265: unexpected end of JSON input
I0807 09:19:54.090803       1 event.go:209] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"vmstorage-volume-victoria-metrics-vmstorage-0", UID:"0efd1fc6-b171-40f4-8405-05378533c265", APIVersion:"v1", ResourceVersion:"102791592", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "linstor-1": rpc error: code = Internal desc = CreateVolume failed for pvc-0efd1fc6-b171-40f4-8405-05378533c265: unexpected end of JSON input

curl -X 'GET' -H 'Accept: application/json' 'http://localhost:3370/v1/resource-definitions'
[{"name":"aaa","uuid":"33202b7d-74fe-4336-b518-808c0a4442d3"},{"name":"csi-0964269a-b154-43a1-876c-d15989446479","props":{"Aux/csi-volume-annotations":"{\"name\":\"pvc-218e9550-3ab1-11e9-baf1-22b1a44f6c4f\",\"id\":\"0964269a-b154-43a1-876c-d15989446479\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-02-27T17:01:23.247750681Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"autoPlace\":\"2\",\"storagePool\":\"pve\"}}"},"layer_data":[{"type":"DRBD"},{"type":"STORAGE"}],"uuid":"4a12afc9-fa9d-4c3a-8e11-39514bf652f2"},{"name":"csi-0fb643e6-d818-4a88-accb-4bdacaacebbc","props":{"DrbdPrimarySetOn":"M1C7","Aux/csi-volume-annotations":"{\"name\":\"pvc-55b2cc6a-3ab0-11e9-baf1-22b1a44f6c4f\",\"id\":\"csi-0fb643e6-d818-4a88-accb-4bdacaacebbc\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-02-27T16:55:41.214480332Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferen\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"storagePool\":\"pve\"}}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7006,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"39eba9a3-38f1-43ee-bb91-fae4ced2bdbe"},{"name":"csi-39b1e7a8-7581-4a67-a15c-e2ec9016022e","props":{"DrbdPrimarySetOn":"M1C6","Aux/csi-volume-annotations":"{\"name\":\"pvc-5e2d3ab4-3ab1-11e9-baf1-22b1a44f6c4f\",\"id\":\"csi-39b1e7a8-7581-4a67-a15c-e2ec9016022e\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-02-27T17:03:04.942232824Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"storagePool\":\"data\"}}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7010,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"e4013414-3112-4469-85b0-7241c4cb9c68"},{"name":"csi-4bc19569-448d-497f-a7a4-9c3312603691","props":{"Aux/csi-volume-annotations":"{\"name\":\"pvc-aa8d7751-3aaf-11e9-baf1-22b1a44f6c4f\",\"id\":\"csi-4bc19569-448d-497f-a7a4-9c3312603691\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-02-27T16:50:54.081789038Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"ReplicasOnSame\":\"side\",\"autoPlace\":\"2\",\"storagePool\":\"pve\"}}"},"layer_data":[{"type":"DRBD"},{"type":"STORAGE"}],"uuid":"27fb6341-c1eb-44fd-9945-5eaea9f96852"},{"name":"csi-7499398a-5ccf-44a3-9c9c-bdfaf05374f1","props":{"DrbdPrimarySetOn":"M1C6","Aux/csi-volume-annotations":"{\"name\":\"pvc-503ab911-3ac1-11e9-baf1-22b1a44f6c4f\",\"id\":\"7499398a-5ccf-44a3-9c9c-bdfaf05374f1\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-02-27T18:57:21.290829372Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"storagePool\":\"data\"}}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7012,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"53020a85-c983-4315-bb18-cf6265f88c0b"},{"name":"csi-9b527570-bedb-438b-9745-3bada7bbc83e","props":{"DrbdPrimarySetOn":"PVE3","Aux/csi-volume-annotations":"{\"name\":\"pvc-a4303601-3ab0-11e9-baf1-22b1a44f6c4f\",\"id\":\"csi-9b527570-bedb-438b-9745-3bada7bbc83e\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-02-27T16:57:52.909364881Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferen\":\"side\",\"autoPlace\":\"2\",\"storagePool\":\"pve\"}}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7008,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"76c8e6ba-448b-4e9f-8765-b5d4fb8e7171"},{"name":"csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23","props":{"DrbdPrimarySetOn":"PVE2","Aux/csi-volume-annotations":"{\"name\":\"pvc-86e46c8e-3ab0-11e9-baf1-22b1a44f6c4f\",\"id\":\"csi-deb6b5da-3cd7-46ce-a5dd-61c1ef97ab23\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-02-27T16:57:06.125301711Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"autoPlace\":\"2\",\"storagePool\":\"pve\"}}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7007,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"6f530b5f-a63c-441c-b606-d9496764db62"},{"name":"default-task-pv-claim","props":{"DrbdPrimarySetOn":"M1C7","Aux/csi-volume-annotations":"{\"name\":\"opennebula-data-opennebula-db-0\",\"id\":\"opennebula-data-opennebula-db-0\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-02-26T16:05:16Z\",\"readonly\":false,\"parameters\":{}}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7013,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"f939b29a-ca86-44f9-871a-1b62f2a9597a"},{"name":"FFF","uuid":"edd9aac6-1a9f-4573-ae37-11bf166848aa"},{"name":"hosting-data-nfs1-stg1-0","props":{"DrbdPrimarySetOn":"M1C8"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7004,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"bc8a0b2d-f36c-4f6c-b3d1-18c561fd2586"},{"name":"hosting-vol-data-web-hs1-wd5-0","props":{"DrbdPrimarySetOn":"M1C6"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7001,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"9a2d2c07-bcc8-43a5-b919-acddc96ef875"},{"name":"nfs100","props":{"DrbdPrimarySetOn":"M1C4"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7002,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"9056f525-9ad9-46b8-9847-6ebcc0c6535b"},{"name":"one-image-10","props":{"DrbdPrimarySetOn":"M1C7","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7060,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"5ab00727-8042-4340-9c5d-c6e5034bc358"},{"name":"one-image-101","props":{"DrbdPrimarySetOn":"M1C7","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7016,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"c37e6754-55b1-43bb-a894-5247067dc403"},{"name":"one-image-102","props":{"DrbdPrimarySetOn":"M1C7","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7015,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"cbb34c2b-511a-45eb-9c40-a69a9d87bd6b"},{"name":"one-image-103","props":{"DrbdPrimarySetOn":"M1C8","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7017,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"33d2e8f7-6d8a-4236-bca4-0569f714575e"},{"name":"one-image-104","props":{"DrbdPrimarySetOn":"M1C7","Aux/one/SNAPSHOT_0/DISK_SIZE":"839270","Aux/one/SNAPSHOT_1/DISK_SIZE":"839270","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7018,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"3761563d-18d3-496e-87b3-519fec5c314b"},{"name":"one-image-105","props":{"DrbdPrimarySetOn":"M1C8","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7019,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"76fc8d46-fcb9-4faf-b1e6-8f3b046497f0"},{"name":"one-image-106","props":{"DrbdPrimarySetOn":"M1C7","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7022,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"0434d24a-a6c1-4611-9b9b-ef1705fa1af8"},{"name":"one-image-107","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/SNAPSHOT_0/DISK_SIZE":"839270","Aux/one/SNAPSHOT_1/DISK_SIZE":"839270","Aux/one/SNAPSHOT_3/DISK_SIZE":"839270","Aux/one/SNAPSHOT_4/DISK_SIZE":"839270","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7024,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"71f6e3ca-fafb-49ec-bd94-b553df26ddd3"},{"name":"one-image-108","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7132,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"79c2f910-2bb7-48d0-a2f4-fe41233dbaba"},{"name":"one-image-109","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7133,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"2dee2cd0-da6d-4ebc-90e1-079e364af94c"},{"name":"one-image-11","props":{"DrbdPrimarySetOn":"M1C7","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7061,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"534f681d-2527-4d0e-8c0f-28b4dd81b38f"},{"name":"one-image-111","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7135,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"1be7deaf-1ee5-4f6b-acb7-dcba07f6d8c1"},{"name":"one-image-112","props":{"DrbdPrimarySetOn":"M1C6","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7136,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"6adaad3d-41b0-4427-8e25-dfca1966b9d1"},{"name":"one-image-113","props":{"DrbdPrimarySetOn":"M1C6","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7137,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"a3ce268d-25a7-4d92-8d1b-96a4a0a19540"},{"name":"one-image-114","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7138,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"8c7df523-6b41-45af-a1b7-5cea03edfe9a"},{"name":"one-image-115","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7146,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"84e3ba1f-d765-4ee7-b240-caac09ef2ccd"},{"name":"one-image-116","props":{"DrbdPrimarySetOn":"M1C6","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7147,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"6a249b0b-29ae-44fb-8fc3-e66942e608dc"},{"name":"one-image-117","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7148,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"25967735-7045-4154-adff-c20d4938b924"},{"name":"one-image-119","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7134,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"fdb51f90-3979-4a8e-ae17-cdb86d030629"},{"name":"one-image-12","props":{"DrbdPrimarySetOn":"M1C8","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7062,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4378cebc-ba00-4ca7-a4f6-adf5de0985c2"},{"name":"one-image-120","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7152,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"d78a80d0-e684-4aea-9afc-187703be3ac3"},{"name":"one-image-121","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7154,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"8af68fe4-e64d-458a-8d66-069dee4670b4"},{"name":"one-image-122","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7155,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"438a63c9-ae5f-42c0-b274-6d0d4efd1eaf"},{"name":"one-image-123","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7156,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"1305df03-c210-4d6f-b9ce-8009133970f7"},{"name":"one-image-124","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7157,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"02bd8e85-096c-4d4b-8879-63ac5e5333b7"},{"name":"one-image-125","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7158,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"102d0dc0-5fb8-45e6-a683-85c7bc5df18f"},{"name":"one-image-126","props":{"DrbdPrimarySetOn":"M1C6","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7159,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"803c25f3-4490-4fe7-9f28-4c983322e350"},{"name":"one-image-127","props":{"DrbdPrimarySetOn":"M1C6","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7161,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"5b8a63ed-cd20-4c87-8f12-4f47fa2657e6"},{"name":"one-image-128","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7160,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"57ea74d1-4a19-4e1e-bf21-54adc7a4f62d"},{"name":"one-image-129","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7162,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"ad5c690a-2709-429a-b102-d7ceb9000850"},{"name":"one-image-13","props":{"DrbdPrimarySetOn":"M1C8","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7063,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"ad45cef7-ca7a-4811-b1de-3fe491b52000"},{"name":"one-image-130","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7172,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"ae7e26f1-f3a2-4347-8e9d-a7aad5682370"},{"name":"one-image-131","props":{"DrbdPrimarySetOn":"M1C6","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7199,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"11c5f632-23b3-4cea-9f99-c8b2166f0320"},{"name":"one-image-133","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7205,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"db4f2d9a-76aa-4030-bc19-5cf25f2047f0"},{"name":"one-image-134","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7206,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"cdc38d41-6c40-4a7f-801b-e636248fafa8"},{"name":"one-image-135","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7185,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"27f39147-27fb-4fbf-aa01-96320654ed0b"},{"name":"one-image-136","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7194,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"3a7b3f13-556e-4118-bff0-901112118da4"},{"name":"one-image-14","props":{"DrbdPrimarySetOn":"M1C8","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7064,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"e675d933-1dc7-4d47-ac3c-dba4ce60e528"},{"name":"one-image-141","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7195,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"d0a097f1-002e-4035-8d44-70defe85020e"},{"name":"one-image-15","props":{"DrbdPrimarySetOn":"M1C8","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7065,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"1f440c2c-74e3-462b-99f7-45d2981463c7"},{"name":"one-image-16","props":{"DrbdPrimarySetOn":"M1C8","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7066,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"c010f72d-fe54-4965-a909-e3cb62abd13c"},{"name":"one-image-17","props":{"DrbdPrimarySetOn":"M1C7","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7067,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4f3906ed-df64-4085-a6ca-616826ea7be9"},{"name":"one-image-18","props":{"DrbdPrimarySetOn":"M1C7","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7068,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"021c9b88-35e7-4f09-afc2-64478e85fb4b"},{"name":"one-image-19","props":{"DrbdPrimarySetOn":"M1C9","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7020,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"cb17fe90-36a1-42ae-9d14-7e79eac4fd72"},{"name":"one-image-20","props":{"DrbdPrimarySetOn":"M1C9","Aux/one/DISK_ID":"3","Aux/one/DS_ID":"107","Aux/one/VM_ID":"138"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7027,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"f8d23d5b-0b20-403a-887b-2282b4d8ae57"},{"name":"one-image-21","props":{"DrbdPrimarySetOn":"M1C9","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7035,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"150989c0-3919-4ec4-a753-a4d1fd6decf5"},{"name":"one-image-22","props":{"DrbdPrimarySetOn":"M1C9","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7037,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"d62e6ebc-5171-479d-a240-3b7d645df845"},{"name":"one-image-24","props":{"DrbdPrimarySetOn":"M1C9","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7038,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"70ca5e94-f565-4ce5-9f8e-fb73f256c651"},{"name":"one-image-28","props":{"DrbdPrimarySetOn":"M1C9","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7050,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"53b27cf7-21f0-40c5-9889-b833371d1415"},{"name":"one-image-29","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7080,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"891762e8-df21-4c1f-a892-23bc1bbc6396"},{"name":"one-image-30","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7095,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"59723c79-736a-4609-9846-45a7c0a45250"},{"name":"one-image-31","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7096,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"2a7d1f91-72fb-4678-aca5-cb64f1acf839"},{"name":"one-image-32","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/SNAPSHOT_0/DISK_SIZE":"314982","Aux/one/SNAPSHOT_1/DISK_SIZE":"314982","Aux/one/SNAPSHOT_2/DISK_SIZE":"314982","Aux/one/SNAPSHOT_3/DISK_SIZE":"314982","Aux/one/SNAPSHOT_4/DISK_SIZE":"311391","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7097,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"cfe7fc64-57a6-42e7-9486-043c3f6d5bbb"},{"name":"one-image-33","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7098,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"e95f2acc-8e31-4bcf-8c62-1706790ef4cb"},{"name":"one-image-41","props":{"DrbdPrimarySetOn":"M1C6","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7101,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"b7438e72-ac14-42c0-aefc-af93712bf958"},{"name":"one-image-42","props":{"DrbdPrimarySetOn":"M1C6","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7103,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"bd22862a-3f0c-402d-92fc-79dff4fc3756"},{"name":"one-image-43","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7104,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"84a35ffd-9ae7-46ba-a038-8f249d372aa8"},{"name":"one-image-45","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7107,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"f41b4887-959f-4a9d-8a0a-3eeddb13ce38"},{"name":"one-image-46","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/SNAPSHOT_0/DISK_SIZE":"52838","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7108,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"31281650-9eca-4f29-95f9-23f413f43aca"},{"name":"one-image-47","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7106,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"fec6ffc9-d4d4-4df5-8532-2d52aaba7fec"},{"name":"one-image-48","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7109,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"fd9fbdb2-de1f-4104-9c84-b7bce9f1c2d7"},{"name":"one-image-49","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7076,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"eb6dc47e-539c-43e9-8f42-1e10f109f680"},{"name":"one-image-50","props":{"DrbdPrimarySetOn":"M1C6","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"165","Aux/one/SNAPSHOT_0/DISK_SIZE":"53452","Aux/one/SNAPSHOT_1/DISK_SIZE":"53452"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7116,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"d9a746ff-f5a0-44a4-928c-cb75228e24f6"},{"name":"one-image-51","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7117,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"c2d59d94-a1d8-4592-a662-059ea13a59fd"},{"name":"one-image-53","props":{"DrbdPrimarySetOn":"M1C6","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7119,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"ec9b436a-177e-4b0e-965b-84636b4f9e8d"},{"name":"one-image-56","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7120,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"3781856f-1280-4bfe-9167-3a020114270e"},{"name":"one-image-59","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7123,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"e96dbdd4-1ac4-4d13-a295-548ab9315fdd"},{"name":"one-image-6","props":{"DrbdPrimarySetOn":"M1C8","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7056,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"fd88be01-ceaa-4c44-b3fb-d228abfe745a"},{"name":"one-image-60","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7125,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"ee2ba03f-03cc-4673-9bd2-0e9b8dc6750c"},{"name":"one-image-61","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7122,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"be4af19b-40a4-4819-b5d9-af58f3d5c690"},{"name":"one-image-62","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7126,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"1cfa3d37-cee7-474d-8c77-316656e0d207"},{"name":"one-image-63","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7127,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"263c1331-cb17-46ee-b21c-197e7bd410a1"},{"name":"one-image-64","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7129,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"e487ffd3-8640-4f09-b5eb-4c7383e5b4fa"},{"name":"one-image-65","props":{"DrbdPrimarySetOn":"M1C4","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7121,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"3ca3b357-1322-4eba-a6fc-37c6e85b75fb"},{"name":"one-image-66","props":{"DrbdPrimarySetOn":"M1C5","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7130,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"96dc86cd-1b63-40f4-8861-5128ff6024c4"},{"name":"one-image-7","props":{"DrbdPrimarySetOn":"M1C7","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7057,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"bccae1ac-2b2a-41f8-866c-e34ca4fd9ca2"},{"name":"one-image-8","props":{"DrbdPrimarySetOn":"M1C8","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7058,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"c1fdc541-d357-48c6-b56f-27c243cedcfb"},{"name":"one-image-9","props":{"DrbdPrimarySetOn":"M1C7","DrbdOptions/Net/allow-two-primaries":"yes"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7059,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"07ffffb1-e6a7-44e0-b391-10045ccec1c7"},{"name":"one-vm-101-disk-0","props":{"DrbdPrimarySetOn":"M1C9","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"101"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7021,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"6c9bfbfd-64d5-4671-8794-1a47a7c7c077"},{"name":"one-vm-103-disk-0","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7041,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"3d990840-99ff-4a0b-b8a0-62ac034acfe5"},{"name":"one-vm-104-disk-0","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7040,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"211b536b-6a42-46d3-9ccd-8dfd4ba44535"},{"name":"one-vm-105-disk-0","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7039,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"47abca81-c350-40d4-828a-57089f3dc9ad"},{"name":"one-vm-112-disk-0","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7043,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"2d508b63-0c22-4464-b4d3-78dbc636f93b"},{"name":"one-vm-113-disk-0","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7044,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"b7609401-0340-48b6-bae7-4a8fa999a7d0"},{"name":"one-vm-113-disk-2","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"2","Aux/one/DS_ID":"107","Aux/one/VM_ID":"113"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7099,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"3ec305f7-61df-47bb-b18d-456787181c70"},{"name":"one-vm-114-disk-0","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7045,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"f8dbab7c-7709-4f5d-9ee4-8a9f86b59c6f"},{"name":"one-vm-115-disk-0","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7046,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"769123e7-2769-4884-a9f7-1b94b3de8fa8"},{"name":"one-vm-115-disk-2","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7047,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"51f6ff4a-ec7b-4b68-a4c4-0e3063d2430f"},{"name":"one-vm-115-disk-3","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7048,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"a8a56f88-d1d9-4e03-ae46-1dad62d94630"},{"name":"one-vm-116-disk-0","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7049,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"0c1a3df1-3eb4-423e-a3d3-5e3efe7e697f"},{"name":"one-vm-120-disk-0","props":{"DrbdPrimarySetOn":"M1C9"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7051,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"7a397efa-30e0-4a2d-9ec9-0849159814da"},{"name":"one-vm-122-disk-0","props":{"DrbdPrimarySetOn":"M1C7"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7029,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"7badc38a-d216-44fd-ac69-a322844aa1d9"},{"name":"one-vm-122-disk-2","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"2","Aux/one/DS_ID":"107","Aux/one/VM_ID":"122"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7094,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"279656e7-a9ab-4dca-be99-ce543fbdeb5f"},{"name":"one-vm-123-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"123"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7069,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"54cd0cff-9e5a-4093-972d-85aa6dd97b76"},{"name":"one-vm-127-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"127"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7003,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"6a918351-09b6-4f3c-8b73-00fb36682f03"},{"name":"one-vm-128-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"128"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7085,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"87a6540c-aa07-4907-9761-c480a589cb57"},{"name":"one-vm-129-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"129"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7084,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"44c86585-504e-453c-bca7-6f5fdc42208f"},{"name":"one-vm-132-disk-0","props":{"DrbdPrimarySetOn":"M1C6","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"132"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7087,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"f38a6908-1a6d-49f6-8aec-624c577bc060"},{"name":"one-vm-133-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"133"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7086,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"902940bf-0a38-4d6d-996f-794a8d0bfe40"},{"name":"one-vm-134-disk-0","props":{"DrbdPrimarySetOn":"M1C6","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"134"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7088,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"e37a3a71-c220-4ee6-b735-1d7f9e7f62b3"},{"name":"one-vm-135-disk-0","props":{"DrbdPrimarySetOn":"M1C6","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"135"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7089,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"7c9826d6-66b1-4d91-b02c-f1fb41d1e858"},{"name":"one-vm-137-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"137"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7090,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"c1589b72-3a25-464c-9524-2b71b01e0f26"},{"name":"one-vm-145-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"145"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7110,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4f295483-6c01-409b-ba8b-8b2aca0aba70"},{"name":"one-vm-146-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"146"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7111,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"d55eb08e-b8f3-4648-b415-41f3a04df007"},{"name":"one-vm-147-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"147"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7112,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"9d89d462-b419-45c7-993c-fe1d9ac5adcf"},{"name":"one-vm-150-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"150"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7114,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"bdc1fb02-305f-46a2-8718-d40580e75581"},{"name":"one-vm-151-disk-2","uuid":"46b517ee-0d47-4d00-892d-72d93a1168fd"},{"name":"one-vm-157-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"157"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7032,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4fa46863-dcbd-4098-9d26-5cfd4e10a6a2"},{"name":"one-vm-163-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"163"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7100,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"d321bd16-9713-4c94-8fad-1daeb9558cd9"},{"name":"one-vm-164-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"164"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7105,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"c01d5da9-9cf8-4a2e-b1bb-cae87115e385"},{"name":"one-vm-165-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"165"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7202,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"b239b9ad-f35e-4222-920f-dba436e67c86"},{"name":"one-vm-166-disk-0","props":{"DrbdPrimarySetOn":"M1C7","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"166"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7203,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"92979da8-c0d1-47a1-8f04-e87d15036fdd"},{"name":"one-vm-166-disk-2","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"2","Aux/one/DS_ID":"107","Aux/one/VM_ID":"166"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7128,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"396c41f9-8926-46d8-8b84-4870429bc357"},{"name":"one-vm-167-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"167"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7115,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"bdfce639-9626-4d88-a9dd-8542024cd257"},{"name":"one-vm-168-disk-0","props":{"DrbdPrimarySetOn":"M1C7","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"168"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7118,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"781e5ee0-fa11-468f-89c8-6fba231dae7c"},{"name":"one-vm-169-disk-2","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"2","Aux/one/DS_ID":"107","Aux/one/VM_ID":"169"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7131,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"13115a9c-dee4-460f-ab94-93f3a5695587"},{"name":"one-vm-170-disk-0","props":{"DrbdPrimarySetOn":"M1C7","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"170"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7124,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"c1f0042c-e6c9-4277-8524-37592e49b613"},{"name":"one-vm-211-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"211"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7142,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"9462fa51-d75a-4124-9ecf-a3915365994e"},{"name":"one-vm-212-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"212"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7139,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"f106781b-fe62-41d0-9da3-47cd4710b10c"},{"name":"one-vm-213-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"213"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7141,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"5dc6999c-5ebd-47e4-9dd7-c32b755b5578"},{"name":"one-vm-214-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"214"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7143,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"813b9eee-0ee7-4f58-9072-4026bfc94093"},{"name":"one-vm-216-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"216"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7144,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"74ac1153-3174-4f7f-95ea-5f907ea72231"},{"name":"one-vm-217-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"217"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7150,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"ce9c77b2-500d-456c-a2cf-896bbc7f3309"},{"name":"one-vm-218-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"218"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7149,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"0913eb1b-98ba-43b4-8344-1848d4aac827"},{"name":"one-vm-219-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"219"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7151,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"8f8819da-ca65-4027-9f02-07801022a287"},{"name":"one-vm-221-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"221"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7145,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"45eafd96-6ed0-4c8d-93cd-ba85dffce75d"},{"name":"one-vm-222-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"222"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7153,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"ef464099-727d-43a4-8ca8-ef0e0cad8c30"},{"name":"one-vm-223-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"223"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7166,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"7caaa34b-b85a-4102-a36e-f6469e198444"},{"name":"one-vm-224-disk-2","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"2","Aux/one/DS_ID":"107","Aux/one/VM_ID":"224"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7163,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"84da4eed-1a31-4886-97b5-fdc5414fbbca"},{"name":"one-vm-225-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"225"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7164,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"d7e096a9-86cc-4c12-bfaa-2cf22474fd88"},{"name":"one-vm-226-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"226"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7165,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"62d289c6-9cca-48ce-8fba-f3be30ebbf3e"},{"name":"one-vm-227-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"227"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7167,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"f00450d5-8569-4e88-b278-253fe9f45a4e"},{"name":"one-vm-228-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"228"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7169,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4f8d3293-5bb9-4f94-b5e2-08934220044e"},{"name":"one-vm-229-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"229"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7170,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"cb0227d7-51a5-465a-a953-569a2662a913"},{"name":"one-vm-23-disk-0","props":{"DrbdPrimarySetOn":"M1C8"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7070,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4d3a0b50-cead-4820-aa2b-75459ffa42e6"},{"name":"one-vm-230-disk-0","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"230"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7168,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"fd3feff8-c2e7-4142-a795-5883b55a497d"},{"name":"one-vm-231-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"231"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7171,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"e4af9510-8048-45f2-9578-69a760fd1108"},{"name":"one-vm-232-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"232"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7173,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"618a8cb4-0813-42e2-82e4-d863990ad67a"},{"name":"one-vm-234-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"234"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7174,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4e21afc2-e25c-4911-9faf-87cc8e1356d6"},{"name":"one-vm-236-disk-0","props":{"DrbdPrimarySetOn":"M1C5","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"236"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7175,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"f491d111-6173-4ed6-b412-74fac41a6a33"},{"name":"one-vm-237-disk-0","props":{"DrbdPrimarySetOn":"M1C7","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"237"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7176,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"c95897d6-14e0-43ac-95a4-418756ecdfc9"},{"name":"one-vm-238-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"238"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7177,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"1567aabf-78d0-41a5-95f3-fa5919541e34"},{"name":"one-vm-239-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"239"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7178,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"33908436-0160-4e10-9a39-e91634c8a603"},{"name":"one-vm-24-disk-0","props":{"DrbdPrimarySetOn":"M1C7"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7071,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"c31c9288-5761-4853-b80d-482783a3af23"},{"name":"one-vm-240-disk-0","props":{"DrbdPrimarySetOn":"M1C7","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"240"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7179,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"1767b09a-78ed-44b5-b39b-e042bac39888"},{"name":"one-vm-241-disk-0","props":{"DrbdPrimarySetOn":"M1C7","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"241"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7180,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"aadc74e4-a143-418c-9e61-d4848801cff7"},{"name":"one-vm-242-disk-0","props":{"DrbdPrimarySetOn":"M1C7","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"107","Aux/one/VM_ID":"242"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7181,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"3f9d5f6c-b6e5-4121-9af9-f3b702adb1d0"},{"name":"one-vm-25-disk-0","props":{"DrbdPrimarySetOn":"M1C7"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7072,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"5157bf8f-f057-4dd3-bb42-86d2fdbddb91"},{"name":"one-vm-27-disk-0","props":{"DrbdPrimarySetOn":"M1C8"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7073,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"87cdce3d-2f19-4e90-ad0b-2dc30623dcc6"},{"name":"one-vm-28-disk-0","props":{"DrbdPrimarySetOn":"M1C7"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7074,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"b5cd8b3c-796c-4169-b623-b6493c730f5a"},{"name":"one-vm-29-disk-0","props":{"DrbdPrimarySetOn":"M1C8"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7077,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"484c5ef2-07fc-40ab-bf1a-a823c77f71cf"},{"name":"one-vm-30-disk-0","props":{"DrbdPrimarySetOn":"M1C8"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7075,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"c97a1b3d-4abd-480e-ac84-9b04e609db3c"},{"name":"one-vm-56-disk-0","props":{"DrbdPrimarySetOn":"M1C8"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7078,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"07e292e4-6be0-44e3-b7e0-334103ffd81f"},{"name":"one-vm-59-disk-0","props":{"DrbdPrimarySetOn":"M1C7"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7079,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"339f695a-8a67-42c7-b8ab-25d8b574e201"},{"name":"one-vm-77-disk-0","props":{"DrbdPrimarySetOn":"M1C8"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7081,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"0e463cbc-9559-47b1-9127-54d56c854cf6"},{"name":"one-vm-78-disk-0","props":{"DrbdPrimarySetOn":"M1C8"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7082,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"9bb5056c-2de4-4a6b-a7c2-1417d4f4c4cf"},{"name":"one-vm-78-disk-2","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"2","Aux/one/DS_ID":"107","Aux/one/VM_ID":"78"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7091,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"e7fdf6b1-3193-4ca2-af42-946a7f74673f"},{"name":"one-vm-78-disk-3","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"3","Aux/one/DS_ID":"107","Aux/one/VM_ID":"78"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7092,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"e23f673b-3258-456c-9176-a49b233c09b4"},{"name":"one-vm-78-disk-4","props":{"DrbdPrimarySetOn":"M1C4","Aux/one/DISK_ID":"4","Aux/one/DS_ID":"107","Aux/one/VM_ID":"78"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7093,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"2e81aa02-6d3b-47db-8ff8-577dc228b935"},{"name":"one-vm-90-checkpoint","props":{"DrbdPrimarySetOn":"M1C7"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7026,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"0d69ca02-85d2-469b-8d8c-52621b35d575"},{"name":"one-vm-90-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"0","Aux/one/VM_ID":"90"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7025,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"cd70f2a2-8c81-4957-ac4f-028123314f8e"},{"name":"one-vm-91-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"0","Aux/one/VM_ID":"91"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7028,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"e5461282-f996-4b6b-b017-82b683dd554e"},{"name":"one-vm-91-disk-2","props":{"DrbdPrimarySetOn":"M1C7","Aux/one/DISK_ID":"2","Aux/one/DS_ID":"0","Aux/one/VM_ID":"91"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7034,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"01a6d048-8358-4f83-a4fc-3f24876c8360"},{"name":"one-vm-93-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"0","Aux/one/VM_ID":"93"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7030,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"88b700ef-b3be-4205-a36d-54b3ea681b26"},{"name":"one-vm-94-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"0","Aux/one/VM_ID":"94"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7031,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"94fae592-4a5a-49c5-9ccb-a06310d419d5"},{"name":"one-vm-96-disk-0","props":{"DrbdPrimarySetOn":"M1C8","Aux/one/DISK_ID":"0","Aux/one/DS_ID":"0","Aux/one/VM_ID":"96"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7033,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"950af013-026c-46d2-b50f-1ea7f4ef1a91"},{"name":"one-vm-96-disk-2","props":{"DrbdPrimarySetOn":"M1C8"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7083,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"0c565795-163e-42a7-9b58-54ed2c59bca3"},{"name":"opennebula-data-opennebula-db-0","props":{"DrbdPrimarySetOn":"PVE3","Aux/csi-volume-annotations":"{\"name\":\"opennebula-data-opennebula-db-0\",\"id\":\"opennebula-data-opennebula-db-0\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-02-26T16:05:16Z\",\"readonly\":false,\"parameters\":{}}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7005,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"7c413df3-5786-42c0-8261-791706332360"},{"name":"pvc-0b292a0f-a259-11e9-b9e2-4677920cf3ed","props":{"Aux/csi-volume-annotations":"{\"name\":\"pvc-0b292a0f-a259-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-0b292a0f-a259-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-09T14:51:33.767486576Z\",\"sizeBytes\":10000000000,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"data\"},\"snapshots\":[]}"},"flags":["DELETE"],"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7184,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"231b7b8d-c95b-4027-b072-49370983bc32"},{"name":"pvc-13f59061-a2f9-11e9-b9e2-4677920cf3ed","props":{"DrbdPrimarySetOn":"M1C4","Aux/csi-volume-annotations":"{\"name\":\"pvc-13f59061-a2f9-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-13f59061-a2f9-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-10T09:57:16.047409037Z\",\"sizeBytes\":10000000000,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"thindata\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7186,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"8f7a98c5-fe30-499f-802b-27bca000171e"},{"name":"pvc-19b7244d-a2fa-11e9-b9e2-4677920cf3ed","props":{"DrbdPrimarySetOn":"M1C4","Aux/csi-volume-annotations":"{\"name\":\"pvc-19b7244d-a2fa-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-19b7244d-a2fa-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-10T10:04:31.057386071Z\",\"sizeBytes\":10000000000,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"thindata\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7187,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4de4e6ce-b439-4f73-aefc-b4c533d3986a"},{"name":"pvc-1e15efb1-a328-11e9-b9e2-4677920cf3ed","props":{"DrbdPrimarySetOn":"M1C5","Aux/csi-volume-annotations":"{\"name\":\"pvc-1e15efb1-a328-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-1e15efb1-a328-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-10T15:34:01.332773738Z\",\"sizeBytes\":10737418240,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"thindata\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7193,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4f090439-a674-4cad-902c-401ce96f106e"},{"name":"pvc-1e1616fb-a328-11e9-b9e2-4677920cf3ed","props":{"DrbdPrimarySetOn":"M1C4","Aux/csi-volume-annotations":"{\"name\":\"pvc-1e1616fb-a328-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-1e1616fb-a328-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-10T15:33:51.368775633Z\",\"sizeBytes\":10737418240,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"thindata\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7192,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"2b8837d6-d412-4b34-a20b-2742ecda123d"},{"name":"pvc-1e173607-a328-11e9-b9e2-4677920cf3ed","props":{"DrbdPrimarySetOn":"M1C5","Aux/csi-volume-annotations":"{\"name\":\"pvc-1e173607-a328-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-1e173607-a328-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-10T15:33:51.369455662Z\",\"sizeBytes\":10737418240,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"thindata\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7191,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4e37fd22-ecf9-49c9-9117-59209fa8455d"},{"name":"pvc-2ff9cd74-5090-11e9-8141-ea144b3eb49c","props":{"DrbdPrimarySetOn":"M1C7","Aux/csi-volume-annotations":"{\"name\":\"pvc-2ff9cd74-5090-11e9-8141-ea144b3eb49c\",\"id\":\"pvc-2ff9cd74-5090-11e9-8141-ea144b3eb49c\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-03-27T12:59:41.091215973Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"storagePool\":\"data\"}}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7009,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"67bc813b-dc3b-4896-ab53-9c4cff24c624"},{"name":"pvc-3e1b00c7-a322-11e9-b9e2-4677920cf3ed","props":{"DrbdPrimarySetOn":"M1C4","Aux/csi-volume-annotations":"{\"name\":\"pvc-3e1b00c7-a322-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-3e1b00c7-a322-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-10T14:51:48.069787862Z\",\"sizeBytes\":25000000512,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"thindata\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7190,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"8341db22-a389-4925-b879-cbb753036be2"},{"name":"pvc-7a80c798-a7a5-11e9-b9e2-4677920cf3ed","props":{"DrbdPrimarySetOn":"M1C4","Aux/csi-volume-annotations":"{\"name\":\"pvc-7a80c798-a7a5-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-7a80c798-a7a5-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-16T08:41:18.112900209Z\",\"sizeBytes\":3221225472,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"thindata\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7182,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"ed4a9ad7-6a09-4725-99ea-a44a931eb68d"},{"name":"pvc-7d7f3044-a313-11e9-b9e2-4677920cf3ed","props":{"DrbdPrimarySetOn":"M1C4","Aux/csi-volume-annotations":"{\"name\":\"pvc-7d7f3044-a313-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-7d7f3044-a313-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-10T13:06:11.980278965Z\",\"sizeBytes\":25000000512,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"thindata\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7189,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"51e4c1e4-cbd1-48c7-86d4-72795ae44948"},{"name":"pvc-804a73c1-7168-11e9-9e4c-9457a5d3b80e","props":{"DrbdPrimarySetOn":"PVE2","Aux/csi-volume-annotations":"{\"name\":\"pvc-804a73c1-7168-11e9-9e4c-9457a5d3b80e\",\"id\":\"pvc-804a73c1-7168-11e9-9e4c-9457a5d3b80e\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-05-08T08:09:00.161924832Z\",\"sizeBytes\":5000000512,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"pve\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7055,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"bd65f3bd-79a5-49f3-ae3b-f55754eb12d2"},{"name":"pvc-80874f70-7168-11e9-9e4c-9457a5d3b80e","props":{"DrbdPrimarySetOn":"PVE1","Aux/csi-volume-annotations":"{\"name\":\"pvc-80874f70-7168-11e9-9e4c-9457a5d3b80e\",\"id\":\"pvc-80874f70-7168-11e9-9e4c-9457a5d3b80e\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-05-08T08:09:00.11650875Z\",\"sizeBytes\":1000000512,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"pve\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7054,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"62bd7681-06db-4d1c-a12a-afe8de5d5446"},{"name":"pvc-80c443a9-7168-11e9-9e4c-9457a5d3b80e","props":{"DrbdPrimarySetOn":"PVE3","Aux/csi-volume-annotations":"{\"name\":\"pvc-80c443a9-7168-11e9-9e4c-9457a5d3b80e\",\"id\":\"pvc-80c443a9-7168-11e9-9e4c-9457a5d3b80e\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-05-08T08:08:46.480772462Z\",\"sizeBytes\":5000000512,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"pve\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7042,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"9a0b337a-f43d-449c-ad82-8652c04e4ca7"},{"name":"pvc-81f81b9c-7168-11e9-9e4c-9457a5d3b80e","props":{"DrbdPrimarySetOn":"PVE3","Aux/csi-volume-annotations":"{\"name\":\"pvc-81f81b9c-7168-11e9-9e4c-9457a5d3b80e\",\"id\":\"pvc-81f81b9c-7168-11e9-9e4c-9457a5d3b80e\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-05-08T08:08:48.509698442Z\",\"sizeBytes\":1000000512,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"pve\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7052,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"f8039105-7e5c-4ab9-b814-d457aa864aba"},{"name":"pvc-82365b9a-7168-11e9-9e4c-9457a5d3b80e","props":{"DrbdPrimarySetOn":"PVE3","Aux/csi-volume-annotations":"{\"name\":\"pvc-82365b9a-7168-11e9-9e4c-9457a5d3b80e\",\"id\":\"pvc-82365b9a-7168-11e9-9e4c-9457a5d3b80e\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-05-08T08:08:49.915628966Z\",\"sizeBytes\":1000000512,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"pve\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7053,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"bf6d6f13-9479-437e-9c3e-15d9c8860c0c"},{"name":"pvc-a1dada5b-50ab-11e9-b606-52b5c95dfe5c","props":{"DrbdPrimarySetOn":"M1C7","Aux/csi-volume-annotations":"{\"name\":\"pvc-a1dada5b-50ab-11e9-b606-52b5c95dfe5c\",\"id\":\"pvc-a1dada5b-50ab-11e9-b606-52b5c95dfe5c\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-03-27T16:16:23.062197723Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"storagePool\":\"data\"}}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7014,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"9f06fda6-f8e6-479a-bc2f-dcf13a8ddb58"},{"name":"pvc-a727272a-aa45-11e9-b9e2-4677920cf3ed","props":{"DrbdPrimarySetOn":"M1C5","Aux/csi-volume-annotations":"{\"name\":\"pvc-a727272a-aa45-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-a727272a-aa45-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-19T16:52:54.83861487Z\",\"sizeBytes\":32212254720,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"thindata\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7201,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"09b9d1b8-cf01-4bbc-98e9-c642d5673182"},{"name":"pvc-bbd4682a-5024-11e9-82fd-c2b1dc5386a1","props":{"DrbdPrimarySetOn":"M1C6","Aux/csi-volume-annotations":"{\"name\":\"pvc-bbd4682a-5024-11e9-82fd-c2b1dc5386a1\",\"id\":\"pvc-bbd4682a-5024-11e9-82fd-c2b1dc5386a1\",\"createdBy\":\"io.drbd.linstor-csi\",\"creationTime\":\"2019-03-27T00:10:37.398254293Z\",\"sizeBytes\":52428800,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"storagePool\":\"data\"}}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7000,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"bfc76cf3-1820-4180-9d0d-15ba07f1d0df"},{"name":"pvc-d0314157-a257-11e9-b9e2-4677920cf3ed","props":{"Aux/csi-volume-annotations":"{\"name\":\"pvc-d0314157-a257-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-d0314157-a257-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-09T14:42:45.315096126Z\",\"sizeBytes\":50000000000,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"data\"},\"snapshots\":[]}"},"flags":["DELETE"],"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7036,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"5583e1b8-eee8-4c04-af73-dad10e2c2427"},{"name":"pvc-eb768b86-a311-11e9-b9e2-4677920cf3ed","props":{"DrbdPrimarySetOn":"M1C4","Aux/csi-volume-annotations":"{\"name\":\"pvc-eb768b86-a311-11e9-b9e2-4677920cf3ed\",\"id\":\"pvc-eb768b86-a311-11e9-b9e2-4677920cf3ed\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2019-07-10T12:55:01.199114797Z\",\"sizeBytes\":10737418240,\"readonly\":false,\"parameters\":{\"ReplicasOnDifferent\":\"side\",\"ReplicasOnSame\":\"moonshot\",\"autoPlace\":\"2\",\"mountOpts\":\"errors=remount-ro\",\"storagePool\":\"thindata\"},\"snapshots\":[]}"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7188,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"a4c850e3-5cc0-4da9-a97f-fd1b4746b282"},{"name":"test","props":{"DrbdPrimarySetOn":"M1C7"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7011,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"15b12c2f-dfad-42de-8740-02761b220505"},{"name":"test1","props":{"DrbdPrimarySetOn":"M1C4"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7102,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"6aca0652-18d3-4412-84ff-6a873ff15477"},{"name":"test2","props":{"DrbdPrimarySetOn":"M1C6"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7023,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"4f2795f4-24c2-4771-a2b4-0bd9f9fcdff0"},{"name":"testb","uuid":"e61f96de-92f7-42ce-86e6-6c14a4fbad9e"},{"name":"testc","props":{"DrbdPrimarySetOn":"M1C6"},"layer_data":[{"type":"DRBD","data":{"peer_slots":7,"al_stripes":1,"al_stripe_size_kib":32,"port":7113,"transport_type":"IP","secret":"<redacted>","down":false}},{"type":"STORAGE"}],"uuid":"3d781703-3dd2-42ea-abda-060a9dd83809"}]

drbd sds based k8s persistent vol is not stable

Hi there,

Provisioning k8s pod with drbd persistent volume through linstor-csi works.
However, these sds volumes randomly turn into "outdated" "consistent" status, then cause pod crash.
I am using two replicas setting, sds storage is running on separate servers and connect to k8s cluster.
k8s nodes joined sds cluster as combinded diskless satellite. k8s version is 1.13.3

Btw, our staging environment (proxmox) is using the same sds cluster as well, without any problem.

Regards,
Tony

Incorrectly named driver

Per the CSI spec the name of a CSI driver must conform to the following requirements:

The name MUST follow domain name notation format (https://tools.ietf.org/html/rfc1035#section-2.3.1).

It SHOULD include the plugin's host company name and the plugin name, to minimize the possibility of collisions.

It MUST be 63 characters or less, beginning and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), dots (.), and alphanumerics between.

This field is REQUIRED.

While CSI 0.x required reverse domain notation, CSI 1.0 dropped the "reverse" part of the requirement. The expectation is that the name will be normal (forward) domain notation.

So for CSI 1.0, instead of io.drbd.linstor-csi the driver name should be linstor.csi.drbd.io (I would do .csi. instead of -csi., but that's just my personal opinion).

Cached information in aux-propertes might be non-actual

Hi, during the volume creation linstor-csi provisioner saves all PVC and StorageClass parameters into Aux/csi-volume-annotations property:

+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Key                               | Value                                                                                                                                                                                                                                                                                                                                                                                |
|==========================================================================================================================================================================================================================================================================================================================================================================================================================|
| Aux/csi-volume-annotations        | {"name":"pvc-ff7aa90e-ceed-48e1-af8c-1b0bd5baf875","id":"pvc-ff7aa90e-ceed-48e1-af8c-1b0bd5baf875","createdBy":"linstor.csi.linbit.com","creationTime":"2020-01-21T17:22:30.165608656Z","sizeBytes":1073741824000,"readonly":false,"parameters":{"autoPlace":"2","disklessStoragePool":"diskless","replicasOnSame":"Aux/hosting-2-storage","storagePool":"thindata"},"snapshots":[]} |
| DrbdOptions/Resource/on-no-quorum | io-error                                                                                                                                                                                                                                                                                                                                                                             |
| DrbdOptions/Resource/quorum       | majority                                                                                                                                                                                                                                                                                                                                                                             |
| DrbdPrimarySetOn                  | M9C7                                                                                                                                                                                                                                                                                                                                                                                 |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Problem is that csi-driver is also uses this information after the creation.

Example: after the linstor-server v1.4.2 release we've updated our storageclasses to use DfltDisklessStorPool instead of diskless to make auto-tiebreaker feature working, then added few new nodes which have no diskless pool configured. In result the old volumes can't be created on these nodes:

  Warning  FailedMount         6s                  kubelet, m5c38           MountVolume.WaitForAttach failed for volume "pvc-ff7aa90e-ceed-48e1-af8c-1b0bd5baf875" : rpc error: code = Internal desc = ControllerPublishVolume failed for pvc-ff7aa90e-ceed-48e1-af8c-1b0bd5baf875: Message: 'The storage pool 'diskless' for resource 'pvc-ff7aa90e-ceed-48e1-af8c-1b0bd5baf875' for volume number '0' is not deployed on node 'm5c38'.'; Cause: 'Dependency not found'; Details: 'The resource which should be deployed had at least one volume definition (volume number '0') which LinStor tried to automatically create. The storage pool name for this new volume was looked for in order in the properties of the resource, volume definition, resource definition and node, and finally in a system wide default storage pool name defined by the LinStor controller.
Node(s): 'm5c38', Resource: 'pvc-ff7aa90e-ceed-48e1-af8c-1b0bd5baf875'; Reports: '[5E455B1B-00000-000000]'

5E455B1B-00000-000000.log

Proposed solution:

Do not cache the PV and StorageClass parameters in resource-definition, use actual information from the Kubernetes instead.

Volume stays in "InUse" state even when no pods are using it

Hi! I am having an issue where if for example I scale a deployment to zero or delete the pod or upgrade the deployment (e.g. changing the image), the volume remains in "InUse" state according to the Linstor CLI, and then the new pod cannot use the volume:

Multi-Attach error for volume "pvc-9802166a-b921-11e9-bbca-9600002d2434" Volume is already exclusively attached to one node and can't be attached to another

This happens even if the deployment is configured to kill the old pod before starting the new one, so that only one pod is using the volume at any time.

Unless I am missing something I would expect that if for example if the deployment doesn't have any pods, the volume should become "Unused" so that it can be attached/mounted to the new pod when scaling the deployment to 1 or anyway when the new pod is created.

Here's the most recent logs: https://pastebin.com/kAXQQsTF

Thanks!

Recommended upgrade path for linstor-csi

I didn't find something about it in the documentation, so I guess I'll ask here.

Is there a recommended way of updating the CSI?
We've deployed the CSI a while back with the 1.14 version of the deployment file. Now the cluster got upgraded over time to now k8s 1.17 while the linstor CSI remained unchanged. Is it possible to remove the CSI and deploy the newest release without the existing volumes getting "lost" or "orphaned"?

Sorry if this is a stupid question, I'm still new to the Kubernetes world.
I appreciate your insights!

ZFS Clonning on the Linstor/DRBD9?

Hello!

We perform development with use of one Dev and many Feature environments. Dev environment files/databases are located on ZFS filesystem and exported by shareNFS to Kubernetes pods via dynamic NFS volumes. The task is to automatically create new Feature environments by cloning all files/databases in seconds, then running manual or autotests in that environment. Each Feature enviroment has it own Kubernetes namespace with pods identical to Dev environment pods. Performance of running tests doen't matter in this case, the gole is to make new enviroment avaliable for tests as fast as possible.

My idea is to join HA oportunities of Linstor/DRBD, with ZFS clonning features and Kubernetes balancing.

I'de like to create Kubernetes Pod similar to docker nfs server, but based on top of Linstor PVC and with ZFS cloning oportunities.

The question is how to get working ZFS filesystem on the Linstor persistent volume claim, to be availiable in Kuberntes Pod with purpose to make clone of existing data and export clone using ZFS shareNfs?

Regards, Sergey

ClusterRoleBinding for linstor-csi-snapshotter-role missing in example for 1.17

Since I've never tried the snapshot functionality so far I did not realize it.
Until now, after the logs of csi-snapshotter grew to a noticable size, filled with messages like this:

E0831 06:31:15.580904       1 reflector.go:156] github.com/kubernetes-csi/external-snapshotter/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1beta1.VolumeSnapshotClass: volumesnapshotclasses.snapshot.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:linstor-csi-controller-sa" cannot list resource "volumesnapshotclasses" in API group "snapshot.storage.k8s.io" at the cluster scope

After a bit of digging I noticed that in the example for 1.17, there was a ClusterRole for the Snapshotter (linstor-csi-snapshotter-role) but no binding that attaches the Role to the linstor-csi-controller-sa ServiceAccount.

I guess adding a ClusterRoleBinding like the one found in the examples for older versions should do it:

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: linstor-csi-snapshotter-binding
subjects:
  - kind: ServiceAccount
    name: linstor-csi-controller-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: linstor-csi-snapshotter-role
  apiGroup: rbac.authorization.k8s.io

Or is there a particular reason this role binding is removed from the 1.17 example?

The source volume and his snapshot are deleted when I try to create a new PVC from a VolumeSnapshot

Hi, when I try to create a new PVC from a Volume Snapshot the source linstor volume and his snapshot are deleted and a new one linstor volume is created.

I follow kubernetes-1-17-feature-cis-volume-snapshot-beta and the CSI Snapshotter repo too.

I added the new CRDs from CSI Snapshotter repo by:
kubectl create -f config/crd
and the Snapshot Controller by:
kubectl create -f deploy/kubernetes/snapshot-controller

My config:
kubernetes v1.17.4
Kube-linstor: v1.6.0

Storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor
provisioner: linstor.csi.linbit.com
parameters:
  autoPlace: "3"
  storagePool: "linstor-pool"

Volume Snapshot Class:

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
  name: linstor-snapshot
driver: linstor.csi.linbit.com
deletionPolicy: Delete

PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-volume
spec:
  storageClassName: linstor
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

Volume Snapshot:

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshot
metadata:
  name: test-snapshot
spec:
  volumeSnapshotClassName: linstor-snapshot
  source:
    persistentVolumeClaimName: test-volume

From Linstor CLI:

LINSTOR:~$ vd list
+---------------------------------------------------------------------------------------------+
| ResourceName                             | VolumeNr | VolumeMinor | Size    | Gross | State |
|=============================================================================================|
| pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 | 0        | 1000        | 500 MiB |       | ok    |
+---------------------------------------------------------------------------------------------+
LINSTOR:~$  s list
+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ResourceName                             | SnapshotName                                  | NodeNames                                | Volumes    | State      |
|===============================================================================================================================================================|
| pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 | snapshot-c3c1f0d7-4f82-4bfc-a497-346a14006966 | n7-worker-01, n7-worker-02, n7-worker-03 | 0: 500 MiB | Successful |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

When I try to create a new PVC derived from the current volume snapshot, the previous linstor volume and snapshot are deleted and a new one linstor volume is created:

PVC from current VolumeSnapshot:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: restore-pvc
spec:
  storageClassName: linstor
  dataSource:
    name: test-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

Result
From linstor CLI:

LINSTOR:~$ vd list
+---------------------------------------------------------------------------------------------+
| ResourceName                             | VolumeNr | VolumeMinor | Size    | Gross | State |
|=============================================================================================|
| pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98 | 0        | 1001        | 500 MiB |       | ok    |
+---------------------------------------------------------------------------------------------+
LINSTOR:~$  s list
+-----------------------------------------------------------+
| ResourceName | SnapshotName | NodeNames | Volumes | State |
|===========================================================|
+-----------------------------------------------------------+

The cluster status is not updated and shows the missing volume and snapshot.

foo@bar:~$ kubectl get volumesnapshot
NAME            READYTOUSE   SOURCEPVC     SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS      SNAPSHOTCONTENT                                    CREATIONTIME   AGE
test-snapshot   true         test-volume                           500Mi         linstor-snapshot   snapcontent-c3c1f0d7-4f82-4bfc-a497-346a14006966   32m            32m
foo@bar:~$ kubectl get pvc
NAME          STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
restore-pvc   Pending                                                                        linstor        32m
test-volume   Bound     pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3   500Mi      RWO            linstor        50m

The new linstor volume is created but the status of the new cluster PVC is "pending"

foo@bar:~$ kubectl describe restore-pvc

Name:          restore-pvc
Namespace:     default
StorageClass:  linstor
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
DataSource:
  APIGroup:  snapshot.storage.k8s.io
  Kind:      VolumeSnapshot
  Name:      test-snapshot
Mounted By:  <none>
Events:
  Type     Reason                Age                   From                                                                                          Message
  ----     ------                ----                  ----                                                                                          -------
  Warning  ProvisioningFailed    40m                   linstor.csi.linbit.com_linstor-linstor-csi-controller-0_1f54e77c-4fe9-4176-a529-aa2d76b219c8  failed to provision volume with StorageClass "linstor": volume content source missing
  Normal   Provisioning          2m22s (x16 over 40m)  linstor.csi.linbit.com_linstor-linstor-csi-controller-0_1f54e77c-4fe9-4176-a529-aa2d76b219c8  External provisioner is provisioning volume for claim "default/restore-pvc"
  Warning  ProvisioningFailed    2m22s (x15 over 40m)  linstor.csi.linbit.com_linstor-linstor-csi-controller-0_1f54e77c-4fe9-4176-a529-aa2d76b219c8  failed to provision volume with StorageClass "linstor": rpc error: code = NotFound desc = CreateVolume failed for pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98: snapshot not found in storage backend
  Normal   ExternalProvisioning  55s (x162 over 40m)   persistentvolume-controller                                                                   waiting for a volume to be created, either by external provisioner "linstor.csi.linbit.com" or manually created by system administrator

Some logs:

foo@bar:~$ kubectl logs -n linstor linstor-linstor-csi-controller-0 linstor-csi-plugin

CREATING FISRT VOLUME:
time="2020-04-20T11:23:44Z" level=debug msg="creating new volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" lin
storCSIComponent=driver newVolume="&{Name:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 ID: CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-20 11:23:44.481367665 +0000 UTC m=+236468.5
15719442 SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mountOpts:sync,noatime,nodiratime storagePool:linstor-pool] Snapshots:[]}" nodeID=n7-worker-01 provisioner=linsto
r.csi.linbit.com size="500.0 MiB" version=v0.7.4-9-g7383849

time="2020-04-20T11:23:44Z" level=debug msg="{0   { pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 map[Aux/csi-volume-annotations:{\"name\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"id\
":\"\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2020-04-20T11:23:44.481367665Z\",\"sizeBytes\":524288000,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mou
ntOpts\":\"sync,noatime,nodiratime\",\"storagePool\":\"linstor-pool\"},\"snapshots\":[]}] [] [{DRBD <nil>} {STORAGE <nil>}] }}"

time="2020-04-20T11:23:44Z" level=debug msg="{[] map[Aux/csi-volume-annotations:{\"name\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"id\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3
\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2020-04-20T11:23:44.481367665Z\",\"sizeBytes\":524288000,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOp
ts\":\"sync,noatime,nodiratime\",\"storagePool\":\"linstor-pool\"},\"snapshots\":[]}] []}"

time="2020-04-20T11:23:44Z" level=info msg="curl -X 'PUT' -d '{\"override_props\":{\"Aux/csi-volume-annotations\":\"{\\\"name\\\":\\\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\\\",\\\"i
d\\\":\\\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\\\",\\\"createdBy\\\":\\\"linstor.csi.linbit.com\\\",\\\"creationTime\\\":\\\"2020-04-20T11:23:44.481367665Z\\\",\\\"sizeBytes\\\":52
4288000,\\\"readonly\\\":false,\\\"parameters\\\":{\\\"autoPlace\\\":\\\"3\\\",\\\"mountOpts\\\":\\\"sync,noatime,nodiratime\\\",\\\"storagePool\\\":\\\"linstor-pool\\\"},\\\"snapshots
\\\":[]}\"}}\n' -H 'Accept: application/json' -H 'Content-Type: application/json' 'https://linstor-linstor-controller:3371/v1/resource-definitions/pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f86
1d3'"

time="2020-04-20T11:25:33Z" level=debug msg="converted resource definition to volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]
.2/entry.go:277" linstorCSIComponent=client resourceDefinition="{Name:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 ExternalName:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 Props:map[Aux/csi-v
olume-annotations:{\"name\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"id\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\
"2020-04-20T11:23:44.481367665Z\",\"sizeBytes\":524288000,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"sync,noatime,nodiratime\",\"storagePool\":\"linstor-poo
l\"},\"snapshots\":[]} DrbdOptions/Resource/on-no-quorum:io-error DrbdOptions/Resource/quorum:majority DrbdPrimarySetOn:N7-WORKER-02] Flags:[] LayerData:[{Type:DRBD Data:0xc00003a4b0}
{Type:STORAGE Data:<nil>}] Uuid:0aa5d2cb-7ca1-41ab-9b85-671218739ea9}" volume="&{Name:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 ID:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 CreatedBy:lin
stor.csi.linbit.com CreationTime:2020-04-20 11:23:44.481367665 +0000 UTC SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mountOpts:sync,noatime,nodiratime storagePool:lin
stor-pool] Snapshots:[]}"


CREATING SNAPSHOT:
time="2020-04-20T11:27:25Z" level=debug msg="created new snapshot" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" cs
iSnapshot="{Name:snapshot-c3c1f0d7-4f82-4bfc-a497-346a14006966 CsiSnap:size_bytes:524288000 snapshot_id:\"snapshot-c3c1f0d7-4f82-4bfc-a497-346a14006966\" source_volume_id:\"pvc-d9037b0
5-7f9e-4e10-b3fb-f6ddc7f861d3\" creation_time:<seconds:1587382045 nanos:837353143 > ready_to_use:true }" linstorCSIComponent=client linstorSnapshot="{Name:snapshot-c3c1f0d7-4f82-4bfc-a
497-346a14006966 ResourceName:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 Nodes:[n7-worker-01 n7-worker-02 n7-worker-03] Props:map[Aux/csi-volume-annotations:{\"name\":\"pvc-d9037b05-7f9e
-4e10-b3fb-f6ddc7f861d3\",\"id\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2020-04-20T11:23:44.481367665Z\",\"sizeBytes\"
:524288000,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"sync,noatime,nodiratime\",\"storagePool\":\"linstor-pool\"},\"snapshots\":[]} DrbdOptions/Resource/on-
no-quorum:io-error DrbdOptions/Resource/quorum:majority DrbdPrimarySetOn:N7-WORKER-02 SequenceNumber:1] Flags:[SUCCESSFUL] VolumeDefinitions:[{VolumeNumber:0 SizeKib:512000}] Uuid:0acf
3612-da8a-4091-bac7-ac35a30d03b8}"

time="2020-04-20T11:27:26Z" level=debug msg="method called" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCS
IComponent=driver method=/csi.v1.Controller/CreateSnapshot nodeID=n7-worker-01 provisioner=linstor.csi.linbit.com req="source_volume_id:\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\" nam
e:\"snapshot-c3c1f0d7-4f82-4bfc-a497-346a14006966\" " resp="snapshot:<size_bytes:524288000 snapshot_id:\"snapshot-c3c1f0d7-4f82-4bfc-a497-346a14006966\" source_volume_id:\"pvc-d9037b05
-7f9e-4e10-b3fb-f6ddc7f861d3\" creation_time:<seconds:1587382045 nanos:837353143 > ready_to_use:true > " version=v0.7.4-9-g7383849

time="2020-04-20T11:29:00Z" level=debug msg="converted resource definition to volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]
.2/entry.go:277" linstorCSIComponent=client resourceDefinition="{Name:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 ExternalName:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 Props:map[Aux/csi-v
olume-annotations:{\"name\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"id\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\
"2020-04-20T11:23:44.481367665Z\",\"sizeBytes\":524288000,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"sync,noatime,nodiratime\",\"storagePool\":\"linstor-poo
l\"},\"snapshots\":[{\"name\":\"snapshot-c3c1f0d7-4f82-4bfc-a497-346a14006966\",\"csiSnapshot\":{\"size_bytes\":524288000,\"snapshot_id\":\"snapshot-c3c1f0d7-4f82-4bfc-a497-346a1400696
6\",\"source_volume_id\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"creation_time\":{\"seconds\":1587382045,\"nanos\":837353143},\"ready_to_use\":true}}]} DrbdOptions/Resource/on-n
o-quorum:io-error DrbdOptions/Resource/quorum:majority DrbdPrimarySetOn:N7-WORKER-02] Flags:[] LayerData:[{Type:DRBD Data:0xc0005e3220} {Type:STORAGE Data:<nil>}] Uuid:0aa5d2cb-7ca1-41
ab-9b85-671218739ea9}" volume="&{Name:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 ID:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-20 11:2
3:44.481367665 +0000 UTC SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mountOpts:sync,noatime,nodiratime storagePool:linstor-pool] Snapshots:[0xc000117f60]}"


CREATING NEW VOLUME FROM SNAPSHOT
time="2020-04-20T11:29:00Z" level=debug msg="creating new volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" lin
storCSIComponent=driver newVolume="&{Name:pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98 ID: CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-20 11:29:00.401225087 +0000 UTC m=+236784.4
35576921 SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mountOpts:sync,noatime,nodiratime storagePool:linstor-pool] Snapshots:[]}" nodeID=n7-worker-01 provisioner=linsto
r.csi.linbit.com size="500.0 MiB" version=v0.7.4-9-g7383849

time="2020-04-20T11:29:00Z" level=info msg="creating volume from snapshot" func="github.com/sirupsen/logrus.(*Entry).Info" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:
285" linstorCSIComponent=client snapshot="&{Name:snapshot-c3c1f0d7-4f82-4bfc-a497-346a14006966 CsiSnap:size_bytes:524288000 snapshot_id:\"snapshot-c3c1f0d7-4f82-4bfc-a497-346a14006966\
" source_volume_id:\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\" creation_time:<seconds:1587382045 nanos:837353143 > ready_to_use:true }" volume="&{Name:pvc-5fb1b881-918c-4ad7-97c0-c5e0
ac0c6c98 ID: CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-20 11:29:00.401225087 +0000 UTC m=+236784.435576921 SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mou
ntOpts:sync,noatime,nodiratime storagePool:linstor-pool] Snapshots:[]}"

time="2020-04-20T11:29:00Z" level=debug msg="{0   { pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98 map[Aux/csi-volume-annotations:{\"name\":\"pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98\",\"id\
":\"\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2020-04-20T11:29:00.401225087Z\",\"sizeBytes\":524288000,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mou
ntOpts\":\"sync,noatime,nodiratime\",\"storagePool\":\"linstor-pool\"},\"snapshots\":[]}] [] [{DRBD <nil>} {STORAGE <nil>}] }}"

time="2020-04-20T11:29:00Z" level=info msg="curl -X 'POST' -d '{\"resource_definition\":{\"external_name\":\"pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98\",\"props\":{\"Aux/csi-volume-anno
tations\":\"{\\\"name\\\":\\\"pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98\\\",\\\"id\\\":\\\"\\\",\\\"createdBy\\\":\\\"linstor.csi.linbit.com\\\",\\\"creationTime\\\":\\\"2020-04-20T11:2
9:00.401225087Z\\\",\\\"sizeBytes\\\":524288000,\\\"readonly\\\":false,\\\"parameters\\\":{\\\"autoPlace\\\":\\\"3\\\",\\\"mountOpts\\\":\\\"sync,noatime,nodiratime\\\",\\\"storagePool
\\\":\\\"linstor-pool\\\"},\\\"snapshots\\\":[]}\"},\"layer_data\":[{\"type\":\"DRBD\"},{\"type\":\"STORAGE\"}]}}\n' -H 'Accept: application/json' -H 'Content-Type: application/json' '
https://linstor-linstor-controller:3371/v1/resource-definitions'"

time="2020-04-20T11:29:00Z" level=info msg="curl -X 'GET' -H 'Accept: application/json' 'https://linstor-linstor-controller:3371/v1/resource-definitions'"

time="2020-04-20T11:29:00Z" level=debug msg="{[] map[Aux/csi-volume-annotations:{\"name\":\"pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98\",\"id\":\"pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98
\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\"2020-04-20T11:29:00.401225087Z\",\"sizeBytes\":524288000,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOp
ts\":\"sync,noatime,nodiratime\",\"storagePool\":\"linstor-pool\"},\"snapshots\":[]}] []}"

time="2020-04-20T11:29:00Z" level=info msg="curl -X 'PUT' -d '{\"override_props\":{\"Aux/csi-volume-annotations\":\"{\\\"name\\\":\\\"pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98\\\",\\\"i
d\\\":\\\"pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98\\\",\\\"createdBy\\\":\\\"linstor.csi.linbit.com\\\",\\\"creationTime\\\":\\\"2020-04-20T11:29:00.401225087Z\\\",\\\"sizeBytes\\\":52
4288000,\\\"readonly\\\":false,\\\"parameters\\\":{\\\"autoPlace\\\":\\\"3\\\",\\\"mountOpts\\\":\\\"sync,noatime,nodiratime\\\",\\\"storagePool\\\":\\\"linstor-pool\\\"},\\\"snapshots
\\\":[]}\"}}\n' -H 'Accept: application/json' -H 'Content-Type: application/json' 'https://linstor-linstor-controller:3371/v1/resource-definitions/pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6
c98'"

time="2020-04-20T11:29:01Z" level=debug msg="method called" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCS
IComponent=driver method=/csi.v1.Controller/CreateVolume nodeID=n7-worker-01 provisioner=linstor.csi.linbit.com req="name:\"pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98\" capacity_range:<r
equired_bytes:524288000 > volume_capabilities:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:\"autoPlace\" value:\"3\" > parameters:<key:\"mountOpt
s\" value:\"sync,noatime,nodiratime\" > parameters:<key:\"storagePool\" value:\"linstor-pool\" > volume_content_source:<snapshot:<snapshot_id:\"snapshot-c3c1f0d7-4f82-4bfc-a497-346a140
06966\" > > " resp="volume:<capacity_bytes:524288000 volume_id:\"pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98\" accessible_topology:<segments:<key:\"linbit.com/hostname\" value:\"n7-worker
-01\" > > accessible_topology:<segments:<key:\"linbit.com/hostname\" value:\"n7-worker-02\" > > accessible_topology:<segments:<key:\"linbit.com/hostname\" value:\"n7-worker-03\" > > >
" version=v0.7.4-9-g7383849

time="2020-04-20T11:29:01Z" level=debug msg="converted resource definition to volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]
.2/entry.go:277" linstorCSIComponent=client resourceDefinition="{Name:pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98 ExternalName:pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98 Props:map[Aux/csi-v
olume-annotations:{\"name\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"id\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\
"2020-04-20T11:23:44.481367665Z\",\"sizeBytes\":524288000,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"sync,noatime,nodiratime\",\"storagePool\":\"linstor-poo
l\"},\"snapshots\":[]} DrbdOptions/Resource/on-no-quorum:io-error DrbdOptions/Resource/quorum:majority DrbdPrimarySetOn:N7-WORKER-02 SequenceNumber:1] Flags:[] LayerData:[{Type:DRBD Da
ta:0xc00003bf90} {Type:STORAGE Data:<nil>}] Uuid:d0d02f13-23fe-481e-974c-b52f2f55e402}" volume="&{Name:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 ID:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f86
1d3 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-20 11:23:44.481367665 +0000 UTC SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mountOpts:sync,noatime,nodiratim
e storagePool:linstor-pool] Snapshots:[]}"


DELETING FIRST VOLUME (SNAPSHOT DELETED AUTOMATICALLY?)
time="2020-04-20T11:29:01Z" level=debug msg="found existing volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" e
xistingVolume="&{Name:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 ID:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-20 11:23:44.481367665 +
0000 UTC SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mountOpts:sync,noatime,nodiratime storagePool:linstor-pool] Snapshots:[]}" linstorCSIComponent=driver nodeID=n7-w
orker-01 provisioner=linstor.csi.linbit.com version=v0.7.4-9-g7383849

time="2020-04-20T11:29:01Z" level=info msg="deleting volume" func="github.com/sirupsen/logrus.(*Entry).Info" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:285" linstorCS
IComponent=client volume="&{Name:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 ID:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-20 11:23:44.
481367665 +0000 UTC SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mountOpts:sync,noatime,nodiratime storagePool:linstor-pool] Snapshots:[]}"

time="2020-04-20T11:29:03Z" level=debug msg="converted resource definition to volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]
.2/entry.go:277" linstorCSIComponent=client resourceDefinition="{Name:pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98 ExternalName:pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98 Props:map[Aux/csi-v
olume-annotations:{\"name\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"id\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\
"2020-04-20T11:23:44.481367665Z\",\"sizeBytes\":524288000,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"sync,noatime,nodiratime\",\"storagePool\":\"linstor-poo
l\"},\"snapshots\":[]} DrbdOptions/Resource/on-no-quorum:io-error DrbdOptions/Resource/quorum:majority DrbdPrimarySetOn:N7-WORKER-02 SequenceNumber:1] Flags:[] LayerData:[{Type:DRBD Da
ta:0xc000720690} {Type:STORAGE Data:<nil>}] Uuid:d0d02f13-23fe-481e-974c-b52f2f55e402}" volume="&{Name:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 ID:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f86
1d3 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-20 11:23:44.481367665 +0000 UTC SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mountOpts:sync,noatime,nodiratim
e storagePool:linstor-pool] Snapshots:[]}"


FAILED RETRY CREATING VOLUME FROM SNAPSHOT
time="2020-04-20T11:29:03Z" level=debug msg="creating new volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" lin
storCSIComponent=driver newVolume="&{Name:pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98 ID: CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-20 11:29:03.852616028 +0000 UTC m=+236787.8
86967808 SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mountOpts:sync,noatime,nodiratime storagePool:linstor-pool] Snapshots:[]}" nodeID=n7-worker-01 provisioner=linsto
r.csi.linbit.com size="500.0 MiB" version=v0.7.4-9-g7383849

time="2020-04-20T11:29:03Z" level=info msg="curl -X 'GET' -H 'Accept: application/json' 'https://linstor-linstor-controller:3371/v1/resource-definitions'"

time="2020-04-20T11:29:03Z" level=debug msg="converted resource definition to volume" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]
.2/entry.go:277" linstorCSIComponent=client resourceDefinition="{Name:pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98 ExternalName:pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98 Props:map[Aux/csi-v
olume-annotations:{\"name\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"id\":\"pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3\",\"createdBy\":\"linstor.csi.linbit.com\",\"creationTime\":\
"2020-04-20T11:23:44.481367665Z\",\"sizeBytes\":524288000,\"readonly\":false,\"parameters\":{\"autoPlace\":\"3\",\"mountOpts\":\"sync,noatime,nodiratime\",\"storagePool\":\"linstor-poo
l\"},\"snapshots\":[]} DrbdOptions/Resource/on-no-quorum:io-error DrbdOptions/Resource/quorum:majority DrbdPrimarySetOn:N7-WORKER-02 SequenceNumber:1] Flags:[] LayerData:[{Type:DRBD Da
ta:0xc000720910} {Type:STORAGE Data:<nil>}] Uuid:d0d02f13-23fe-481e-974c-b52f2f55e402}" volume="&{Name:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f861d3 ID:pvc-d9037b05-7f9e-4e10-b3fb-f6ddc7f86
1d3 CreatedBy:linstor.csi.linbit.com CreationTime:2020-04-20 11:23:44.481367665 +0000 UTC SizeBytes:524288000 Readonly:false Parameters:map[autoPlace:3 mountOpts:sync,noatime,nodiratim
e storagePool:linstor-pool] Snapshots:[]}"

time="2020-04-20T11:29:03Z" level=error msg="method failed" func="github.com/sirupsen/logrus.(*Entry).Error" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:297" error="rp
c error: code = NotFound desc = CreateVolume failed for pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98: snapshot not found in storage backend" linstorCSIComponent=driver method=/csi.v1.Contr
oller/CreateVolume nodeID=n7-worker-01 provisioner=linstor.csi.linbit.com req="name:\"pvc-5fb1b881-918c-4ad7-97c0-c5e0ac0c6c98\" capacity_range:<required_bytes:524288000 > volume_capab
ilities:<mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > > parameters:<key:\"autoPlace\" value:\"3\" > parameters:<key:\"mountOpts\" value:\"sync,noatime,nodiratime\"
> parameters:<key:\"storagePool\" value:\"linstor-pool\" > volume_content_source:<snapshot:<snapshot_id:\"snapshot-c3c1f0d7-4f82-4bfc-a497-346a14006966\" > > " resp= version=v0.7.4-9-g
7383849

Failed to list *v1alpha1.CSIDriver: csidrivers.csi.storage.k8s.io is forbidden

Hi there,

Linstor-CSI works with the latest version k8s (1.13.3), we can use dynamic provision drbd sds volume in k8s cluster and mout/unmount them to specific pods. However, we still continually got error message from kubelet log as below, "master03" is my one of k8s node name, is it permission issue?

kubelet[1410374]: E0204 14:15:56.104528 1410374 reflector.go:134] k8s.io/csi-api/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1alpha1.CSIDriver: csidrivers.csi.storage.k8s.io is forbidden: User "system:node:master03" cannot list resource "csidrivers" in API group "csi.storage.k8s.io" at the cluster scope.

Regards,
Tony

Volume detaching is not working properly

There is also some problem with detaching volume from the node.
When you remove the pod, diskless replica continue existing on the node.

csi-attacher log:

I0327 15:31:22.085056       1 controller.go:173] Started VA processing "csi-6e426e2e00ebf7a92fd912d7dc1e1d71e9046e72b32f02c8cdb0dd716206f0bf"
I0327 15:31:22.085074       1 csi_handler.go:93] CSIHandler: processing VA "csi-6e426e2e00ebf7a92fd912d7dc1e1d71e9046e72b32f02c8cdb0dd716206f0bf"
I0327 15:31:22.085080       1 csi_handler.go:144] Starting detach operation for "csi-6e426e2e00ebf7a92fd912d7dc1e1d71e9046e72b32f02c8cdb0dd716206f0bf"
I0327 15:31:22.085113       1 csi_handler.go:151] Detaching "csi-6e426e2e00ebf7a92fd912d7dc1e1d71e9046e72b32f02c8cdb0dd716206f0bf"
I0327 15:31:22.086956       1 csi_handler.go:524] Can't get CSINodeInfo pve3: csinodeinfos.csi.storage.k8s.io "pve3" not found
I0327 15:31:22.086986       1 connection.go:242] GRPC call: /csi.v1.Controller/ControllerUnpublishVolume
I0327 15:31:22.086991       1 connection.go:243] GRPC request: {"node_id":"pve3","volume_id":"pvc-2ff9cd74-5090-11e9-8141-ea144b3eb49c"}
I0327 15:31:22.087689       1 connection.go:245] GRPC response: {}
I0327 15:31:22.088023       1 connection.go:246] GRPC error: <nil>
I0327 15:31:22.088029       1 csi_handler.go:377] Detached "csi-6e426e2e00ebf7a92fd912d7dc1e1d71e9046e72b32f02c8cdb0dd716206f0bf"
I0327 15:31:22.088055       1 util.go:71] Marking as detached "csi-6e426e2e00ebf7a92fd912d7dc1e1d71e9046e72b32f02c8cdb0dd716206f0bf"
I0327 15:31:22.091038       1 controller.go:203] Started PV processing "pvc-2ff9cd74-5090-11e9-8141-ea144b3eb49c"
I0327 15:31:22.091052       1 csi_handler.go:418] CSIHandler: processing PV "pvc-2ff9cd74-5090-11e9-8141-ea144b3eb49c"
I0327 15:31:22.091058       1 csi_handler.go:422] CSIHandler: processing PV "pvc-2ff9cd74-5090-11e9-8141-ea144b3eb49c": no deletion timestamp, ignoring
I0327 15:31:22.091157       1 util.go:82] Finalizer removed from "csi-6e426e2e00ebf7a92fd912d7dc1e1d71e9046e72b32f02c8cdb0dd716206f0bf"
I0327 15:31:22.091172       1 csi_handler.go:164] Fully detached "csi-6e426e2e00ebf7a92fd912d7dc1e1d71e9046e72b32f02c8cdb0dd716206f0bf"
I0327 15:31:22.091183       1 csi_handler.go:109] CSIHandler: finished processing "csi-6e426e2e00ebf7a92fd912d7dc1e1d71e9046e72b32f02c8cdb0dd716206f0bf"

linstor-csi-plugin log:

time="2019-03-27T15:31:22Z" level=debug msg="method called" func=github.com/LINBIT/linstor-csi/pkg/driver.Driver.Run.func1 file="/home/hayley/projects/linstor-csi/pkg/driver/driver.go:708" linstorCSIComponent=driver method=/csi.v1.Controller/ControllerUnpublishVolume nodeID=pve2 provisioner=io.drbd.linstor-csi req="volume_id:\"pvc-2ff9cd74-5090-11e9-8141-ea144b3eb49c\" node_id:\"pve3\" " resp= version=v0.3.2

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

CSI attacher version: v1.0.1-0-gb7dadac
Linstor csi driver container:

quay.io/linbit/linstor-csi@sha256:30e257a9a66dcdfe8dce1c63e32535722345871b58f2cb6ff6a784eec39481d2

armhf + k3s = attacher failed

Using a raspberry pi 4 armhf k3s cluster with 3 nodes everything works fine (installed using the manual, cross compiled the images for armhf) - but the attachment of the pvs to the pods is failing.

The attacher log shows:

I1021 21:31:02.646672       1 main.go:85] Version: v2.0.0-1-g4dd6201e-dirty
I1021 21:31:02.670671       1 connection.go:151] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
I1021 21:31:05.882641       1 common.go:111] Probing CSI driver for readiness
I1021 21:31:05.882745       1 connection.go:180] GRPC call: /csi.v1.Identity/Probe
I1021 21:31:05.882793       1 connection.go:181] GRPC request: {}
I1021 21:31:05.897815       1 connection.go:183] GRPC response: {}
I1021 21:31:05.899767       1 connection.go:184] GRPC error: <nil>
I1021 21:31:05.900046       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo
I1021 21:31:05.900345       1 connection.go:181] GRPC request: {}
I1021 21:31:05.918433       1 connection.go:183] GRPC response: {"name":"linstor.csi.linbit.com","vendor_version":"v0.7.2-9-g82bdc84-dirty"}
I1021 21:31:05.919945       1 connection.go:184] GRPC error: <nil>
I1021 21:31:05.919981       1 main.go:128] CSI driver name: "linstor.csi.linbit.com"
I1021 21:31:05.920005       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I1021 21:31:05.920022       1 connection.go:181] GRPC request: {}
I1021 21:31:05.922691       1 connection.go:183] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I1021 21:31:05.929761       1 connection.go:184] GRPC error: <nil>
I1021 21:31:05.929794       1 connection.go:180] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1021 21:31:05.929814       1 connection.go:181] GRPC request: {}
I1021 21:31:05.932448       1 connection.go:183] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":4}}},{"Type":{"Rpc":{"type":8}}}]}
I1021 21:31:05.947097       1 connection.go:184] GRPC error: <nil>
I1021 21:31:05.952587       1 main.go:152] CSI driver supports ControllerPublishUnpublish, using real CSI handler
I1021 21:31:05.956172       1 reflector.go:123] Starting reflector *v1.Node (10m0s) from k8s.io/client-go/informers/factory.go:133
I1021 21:31:05.956172       1 reflector.go:123] Starting reflector *v1beta1.VolumeAttachment (10m0s) from k8s.io/client-go/informers/factory.go:133
I1021 21:31:05.956494       1 controller.go:113] Starting CSI attacher
I1021 21:31:05.957211       1 reflector.go:123] Starting reflector *v1beta1.CSINode (10m0s) from k8s.io/client-go/informers/factory.go:133
I1021 21:31:05.957474       1 reflector.go:123] Starting reflector *v1.PersistentVolume (10m0s) from k8s.io/client-go/informers/factory.go:133
I1021 21:31:05.960396       1 reflector.go:161] Listing and watching *v1.PersistentVolume from k8s.io/client-go/informers/factory.go:133
I1021 21:31:05.960398       1 reflector.go:161] Listing and watching *v1beta1.VolumeAttachment from k8s.io/client-go/informers/factory.go:133
I1021 21:31:05.960397       1 reflector.go:161] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:133
I1021 21:31:05.960499       1 reflector.go:161] Listing and watching *v1beta1.CSINode from k8s.io/client-go/informers/factory.go:133
I1021 21:31:06.056800       1 shared_informer.go:123] caches populated
I1021 21:31:06.057039       1 controller.go:205] Started PV processing "pvc-51ab78fc-392b-4c43-b203-93b40d166019"
I1021 21:31:06.057212       1 csi_handler.go:444] CSIHandler: processing PV "pvc-51ab78fc-392b-4c43-b203-93b40d166019"
I1021 21:31:06.057366       1 csi_handler.go:448] CSIHandler: processing PV "pvc-51ab78fc-392b-4c43-b203-93b40d166019": no deletion timestamp, ignoring
I1021 21:31:06.057243       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:06.057658       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:06.057769       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:06.057911       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:06.058125       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:31:06.058305       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:06.057573       1 controller.go:205] Started PV processing "pvc-e1957a70-cecc-403f-aba7-e66a2faf7d7c"
I1021 21:31:06.058656       1 csi_handler.go:444] CSIHandler: processing PV "pvc-e1957a70-cecc-403f-aba7-e66a2faf7d7c"
I1021 21:31:06.058998       1 csi_handler.go:480] CSIHandler: processing PV "pvc-e1957a70-cecc-403f-aba7-e66a2faf7d7c": VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" found
I1021 21:31:06.103007       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:06.103113       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:31:06.104358       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:31:07.103423       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:07.103492       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:07.103526       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:07.103657       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:07.103701       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:31:07.103731       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:07.116285       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:07.116342       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:31:07.118022       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:31:09.116676       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:09.116779       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:09.116853       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:09.117084       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:09.117177       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:31:09.117241       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:09.132317       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:09.132424       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:31:09.136760       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:31:13.133045       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:13.133235       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:13.133343       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:13.133747       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:13.133923       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:31:13.134059       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:13.153233       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:13.153356       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:31:13.155010       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:31:21.153871       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:21.154094       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:21.154211       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:21.154742       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:21.154926       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:31:21.155037       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:21.174187       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:31:21.175257       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:21.175353       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:31:37.175745       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:37.175854       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:37.175924       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:37.176171       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:37.176284       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:31:37.176351       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:37.192899       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:31:37.193014       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:31:37.195716       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:32:09.193532       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:32:09.193656       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:32:09.194171       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:32:09.195109       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:32:09.195334       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:32:09.195500       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:32:09.211456       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:32:09.211572       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:32:09.215098       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:33:13.212001       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:33:13.212203       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:33:13.212264       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:33:13.212441       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:33:13.212542       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:33:13.212611       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:33:13.230631       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:33:13.230745       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:33:13.233589       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:35:21.231335       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:35:21.231540       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:35:21.231610       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:35:21.231766       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:35:21.231860       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:35:21.231924       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:35:21.249026       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:35:21.249137       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:35:21.252320       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:38:17.025112       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1beta1.VolumeAttachment total 9 items received
I1021 21:39:07.008475       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1beta1.CSINode total 0 items received
I1021 21:39:25.025166       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1.PersistentVolume total 0 items received
I1021 21:39:37.249616       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:39:37.249816       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:39:37.249884       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:39:37.250132       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:39:37.250238       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:39:37.250307       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:39:37.275578       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:39:37.275679       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:39:37.294831       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:40:48.020597       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1.Node total 27 items received
I1021 21:41:06.022835       1 reflector.go:235] k8s.io/client-go/informers/factory.go:133: forcing resync
I1021 21:41:06.022988       1 controller.go:205] Started PV processing "pvc-51ab78fc-392b-4c43-b203-93b40d166019"
I1021 21:41:06.023028       1 csi_handler.go:444] CSIHandler: processing PV "pvc-51ab78fc-392b-4c43-b203-93b40d166019"
I1021 21:41:06.023111       1 csi_handler.go:448] CSIHandler: processing PV "pvc-51ab78fc-392b-4c43-b203-93b40d166019": no deletion timestamp, ignoring
I1021 21:41:06.023171       1 controller.go:205] Started PV processing "pvc-e1957a70-cecc-403f-aba7-e66a2faf7d7c"
I1021 21:41:06.023201       1 csi_handler.go:444] CSIHandler: processing PV "pvc-e1957a70-cecc-403f-aba7-e66a2faf7d7c"
I1021 21:41:06.023265       1 reflector.go:235] k8s.io/client-go/informers/factory.go:133: forcing resync
I1021 21:41:06.023350       1 csi_handler.go:480] CSIHandler: processing PV "pvc-e1957a70-cecc-403f-aba7-e66a2faf7d7c": VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" found
I1021 21:41:06.023366       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:41:06.023400       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:41:06.023425       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:41:06.023478       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:41:06.023517       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:41:06.023544       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:41:06.030694       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:41:06.030746       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:41:06.032592       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:44:37.276091       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:44:37.276239       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:44:37.276302       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:44:37.276537       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:44:37.276631       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:44:37.276696       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:44:37.300483       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:44:37.300602       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:44:37.300767       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:44:44.029151       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1.PersistentVolume total 0 items received
I1021 21:45:24.027418       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1beta1.VolumeAttachment total 3 items received
I1021 21:46:34.024944       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1.Node total 18 items received
I1021 21:47:33.012910       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1beta1.CSINode total 0 items received
I1021 21:49:37.300979       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:49:37.301329       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:49:37.301554       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:49:37.302002       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:49:37.302262       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:49:37.302448       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:49:37.319575       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:49:37.319890       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:49:37.323078       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:50:13.033202       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1.PersistentVolume total 0 items received
I1021 21:51:06.023140       1 reflector.go:235] k8s.io/client-go/informers/factory.go:133: forcing resync
I1021 21:51:06.023424       1 controller.go:205] Started PV processing "pvc-51ab78fc-392b-4c43-b203-93b40d166019"
I1021 21:51:06.023524       1 csi_handler.go:444] CSIHandler: processing PV "pvc-51ab78fc-392b-4c43-b203-93b40d166019"
I1021 21:51:06.023677       1 csi_handler.go:448] CSIHandler: processing PV "pvc-51ab78fc-392b-4c43-b203-93b40d166019": no deletion timestamp, ignoring
I1021 21:51:06.023787       1 controller.go:205] Started PV processing "pvc-e1957a70-cecc-403f-aba7-e66a2faf7d7c"
I1021 21:51:06.023845       1 csi_handler.go:444] CSIHandler: processing PV "pvc-e1957a70-cecc-403f-aba7-e66a2faf7d7c"
I1021 21:51:06.024220       1 reflector.go:235] k8s.io/client-go/informers/factory.go:133: forcing resync
I1021 21:51:06.024334       1 csi_handler.go:480] CSIHandler: processing PV "pvc-e1957a70-cecc-403f-aba7-e66a2faf7d7c": VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" found
I1021 21:51:06.025001       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:51:06.025234       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:51:06.025446       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:51:06.025816       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:51:06.026058       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:51:06.026280       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:51:06.041269       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:51:06.041375       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:51:06.046043       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:51:54.030065       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1beta1.VolumeAttachment total 2 items received
I1021 21:54:08.032449       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1.Node total 24 items received
I1021 21:54:37.320513       1 controller.go:175] Started VA processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:54:37.320616       1 csi_handler.go:89] CSIHandler: processing VA "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:54:37.320691       1 csi_handler.go:140] Starting detach operation for "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:54:37.320914       1 csi_handler.go:147] Detaching "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:54:37.321007       1 csi_handler.go:550] Can't get CSINode host2: csinode.storage.k8s.io "host2" not found
I1021 21:54:37.321074       1 csi_handler.go:428] Saving detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:54:37.339042       1 csi_handler.go:439] Saved detach error to "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430"
I1021 21:54:37.339204       1 csi_handler.go:99] Error processing "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430": failed to detach: node "host2" has no NodeID annotation
I1021 21:54:37.339290       1 controller.go:141] Ignoring VolumeAttachment "csi-ab1e8c5f78cbcad5b31d7bc02b5a0b2f342395a46b4e60b1f1a232c4e6068430" change
I1021 21:56:17.040482       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1.PersistentVolume total 0 items received
I1021 21:56:37.019863       1 reflector.go:370] k8s.io/client-go/informers/factory.go:133: Watch close - *v1beta1.CSINode total 0 items received

I supposed the CSINodeInfo feature is no working correctly in k3s, is this correct or does the log point to another problem?

Link the the (possible) k3s bug

Unimplemented message during driver registration

I have strange messages during restart kubelet service:

kubelet[12486]: E0302 04:44:15.533230   12486 plugin_watcher.go:120] error failed to get plugin info using RPC GetInfo at socket /var/lib/kubelet/plugins/io.drbd.linstor-csi/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration when handling create event: "/var/lib/kubelet/plugins/io.drbd.linstor-csi/csi.sock": CREATE

# kubelet --version
Kubernetes v1.13.4

Auto recovery after server crash.

Hello!
As I know, if my server completely broke, linstor plugin won't setup automatically another copy in other server and I need some manual action to recovery.
Can you make it happen without my manual action?
Unfortunately, I didn't work with linstor, but I know If I use ceph for example it will be done without my intervention.

Version 0.7.2 does not work with examples/k8s/deploy/linstor-csi-1.16.yaml

Hi, I changed the version numbers in linstor-csi-1.16.yaml in order to use v0.7.2 instead of v0.7.0 and it did not work out. I could not attach a volume to a pod. Kubernetes gave me an error 404. When I switched back to v0.7.0 the problem disappeared.
I think the current linstor-csi-1.16.yaml are not compatible with v0.7.2

I use k8s 1.16.3

One more question, the master node in my cluster is a satellite as other two nodes (workers) are. Do I need to have linstor-csi-node running on the master node too?

More informative error if resource have connection problems

When drbd-resource have connection issues, eg. when diskless replica have no connection to data-replocas, csi-driver reports:

rpc error: code = Unknown desc = couldn't create ext4 filesystem on /dev/drbd1055: exit status 1: "mke2fs 1.42.13 (17-May-2015)\nCould not open /dev/drbd1055: Wrong medium type\n"

This is totally confusing, It would be nice to see more suitable error in this case, instead couldn't create ext4 filesystem

example linstor-csi.yaml is not compatible with Kubernetes 1.14

After upgrade to v1.14 csi driver is not working.
Any attempt to attach the volume ends up with the error err=no volume plugin matched.
I guess problem was caused by moving CSIDriver from csi.storage.k8s.io/v1alpha1 to storage.k8s.io/v1beta1

csi-cluster-driver-registrar log says:

I0326 16:18:08.628649       1 k8s_register.go:69] CSIDriver CRD already had been registered

but kubectl get csidrivers.csi.storage.k8s.io returns nothing.

csi-cluster-driver-registrar uses quay.io/k8scsi/csi-node-driver-registrar:v1.0.2 image

snapshot permissions need to be updated and example manifest snapshotter to reflect new naming convention

Perhaps on related to kubernetes 1.15.0+ but this one took me a while to get working because the driver name in the example manifests is wrong.

In order to get this working I had to first change the snapshotter to:

kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1alpha1
metadata:
  annotations:
    k10.kasten.io/is-snapshot-class: "true" # required for kasten.io to use the snapshot class
  name: linstor-csi-snapshot-class
  namespace: kube-system
snapshotter: linstor.csi.linbit.com

and once I got that working I finally got some action in the csi-snapshotter logs and received an error related to a permission for the volumesnapshots/status api. Additionally reference the official csi manifest I also added the "update" verb to persistentvolumeclaims resources. my linstor-csi-snapshotter-role looks like this:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: linstor-csi-snapshotter-role
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["create", "get", "list", "watch", "update", "delete"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["create", "list", "watch", "delete"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots/status"]
    verbs: ["update"]

after doing this snapshots are finally working.

Hope this helps :)

Unable to reuse mounted volume on same node

Kubernetes provides opportunity to use the same ReadWriteOnce volume multiple times inside single node.

PR #22 brakes this opportunity by implementing strict check for exclusive use only: fff3467#diff-98aaa3a1d99d08020a7d226e3b70232bR1196-R1198

error:

MountVolume.SetUp failed for volume "pvc-eb768b86-a311-11e9-b9e2-4677920cf3ed" : rpc error: code = Internal desc = NodePublishVolume failed for pvc-eb768b86-a311-11e9-b9e2-4677920cf3ed: unable to get an exclusive open on /dev/drbd1188, check device health

Bump csi images version

Hi @haySwim, I have successfully tested linstor with new csi images:

quay.io/k8scsi/csi-node-driver-registrar:v1.1.0
quay.io/k8scsi/csi-provisioner:v1.2.0
quay.io/k8scsi/csi-attacher:v1.1.1
quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1

Mount failed: /dev/drbd1000 already mounted on

Hey,

for backup purposes i want to mount a PVC in a second pod as read only.
This mount failes with the following error:

Events:
  Type     Reason       Age                 From                     Message
  ----     ------       ----                ----                     -------
  Normal   Scheduled    <unknown>           default-scheduler        Successfully assigned default/wordpress-mysql-backup-76b44f7777-jdnkq to bastel-cluster
  Warning  FailedMount  3m4s (x4 over 12m)  kubelet, bastel-cluster  Unable to attach or mount volumes: unmounted volumes=[mysql-persistent-storage], unattached volumes=[mysql-persistent-storage default-token-ddnbk]: timed out waiting for the condition
  Warning  FailedMount  54s (x8 over 14m)   kubelet, bastel-cluster  MountVolume.SetUp failed for volume "pvc-1e2c764e-5af5-48b7-9a91-9d028244f537" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = NodePublishVolume failed for pvc-1e2c764e-5af5-48b7-9a91-9d028244f537: mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t ext4 -o ro,,defaults /dev/drbd1000 /var/lib/kubelet/pods/05416e83-0c6d-418e-a5c1-79034eaaf06c/volumes/kubernetes.io~csi/pvc-1e2c764e-5af5-48b7-9a91-9d028244f537/mount
Output: mount: /var/lib/kubelet/pods/05416e83-0c6d-418e-a5c1-79034eaaf06c/volumes/kubernetes.io~csi/pvc-1e2c764e-5af5-48b7-9a91-9d028244f537/mount: /dev/drbd1000 already mounted on /var/lib/kubelet/pods/3bbdbd7c-08e7-4b36-90fe-9cd470f01e2b/volumes/kubernetes.io~csi/pvc-1e2c764e-5af5-48b7-9a91-9d028244f537/mount.
  Warning  FailedMount  47s (x2 over 5m18s)  kubelet, bastel-cluster  Unable to attach or mount volumes: unmounted volumes=[mysql-persistent-storage], unattached volumes=[default-token-ddnbk mysql-persistent-storage]: timed out waiting for the condition

The deployments for reference:

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  ports:
    - port: 3306
  selector:
    app: wordpress
    tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: MYSQLROOTPW12345678
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: wordpress-mysql-backup
  labels:
    app: wordpress-backup
spec:
  selector:
    matchLabels:
      app: wordpress-backup
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress-backup
        tier: mysql
    spec:
      containers:
      - image: busybox
        command:
        - sleep
        - "3600"
        name: busybox
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /data
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim
          readOnly: true

I am running a linstor-csi image that i build directly from master.

Can't resuse volume after delayed deletion

Hi, we're using latest STORK plugin from the upstream, by default it is coming with health-monitor enabled:

   --health-monitor                           Enable health monitoring of the storage driver (default: true)

And today we faced with the painful issue. We have many nodes, sometimes some of them might be overloaded, they are flapping between Online and OFFLINE state.

STORK detects these nodes and trying to reattach the volumes and restart the pods on place, example log message:

time="2020-10-10T19:46:16Z" level=info msg="Deleting Pod from Node m9c17 due to volume driver status: Offline ()" Namespace=hosting Owner=ReplicaSet/hc1-wd48-678d9888fb PodName=hc1-wd48-678d9888fb-p8gck

This causes really weird behavior from the linstor-csi driver:

Events:
  Type     Reason                  Age                   From                     Message
  ----     ------                  ----                  ----                     -------
  Normal   Scheduled               10m                   default-scheduler        Successfully assigned hosting/hc1-wd48-678d9888fb-fsmcq to m9c17
  Warning  FailedMount             9m39s (x11 over 10m)  kubelet, m9c17           MountVolume.WaitForAttach failed for volume "pvc-ddd150c5-94eb-48a2-9126-4d1339811752" : volume attachment is being deleted
  Warning  FailedMount             9m35s (x10 over 10m)  kubelet, m9c17           MountVolume.SetUp failed for volume "pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = NodePublishVolume failed for pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4: checking device path failed: path "" does not exist
  Warning  FailedMount             9m7s                  kubelet, m9c17           MountVolume.WaitForAttach failed for volume "pvc-ddd150c5-94eb-48a2-9126-4d1339811752" : volume pvc-ddd150c5-94eb-48a2-9126-4d1339811752 has GET error for volume attachment csi-9ff6fcc944f9e40da6106d5175b34c3e53f7449ee0a990f6c2c69ba07764d9e1: volumeattachments.storage.k8s.io "csi-9ff6fcc944f9e40da6106d5175b34c3e53f7449ee0a990f6c2c69ba07764d9e1" is forbidden: User "system:node:m9c17" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: no relationship found between node "m9c17" and this object
  Warning  FailedMount             8m20s                 kubelet, m9c17           Unable to attach or mount volumes: unmounted volumes=[vol-data-backup vol-data-web], unattached volumes=[wd48-vol-data-global run vol-data-backup wd48-vol-shared default-token-jt2jk cgroup fuse vol-data-web wd48-vol-data-proxy]: timed out waiting for the condition
  Normal   SuccessfulAttachVolume  8m14s                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-ddd150c5-94eb-48a2-9126-4d1339811752"
  Warning  FailedMount             3m55s (x4 over 9m3s)  kubelet, m9c17           MountVolume.SetUp failed for volume "pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Internal desc = NodePublishVolume failed for pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4: 404 Not Found

The volume might stuck on DELETING:

╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node  ┊ Port  ┊ Usage  ┊ Conns ┊    State ┊ CreatedOn ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4 ┊ m9c17 ┊ 55207 ┊        ┊ Ok    ┊ DELETING ┊           ┊
┊ pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4 ┊ m5c18 ┊ 55207 ┊ Unused ┊ Ok    ┊ UpToDate ┊           ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

The csi-attacher logs says:

I1010 16:40:34.311031       1 request.go:581] Throttling request took 1.950726244s, request: PATCH:https://10.96.0.1:443/apis/storage.k8s.io/v1/volumeattachments/csi-3df4c97cd2abde367eebc52e875e7af60add6daa8a4afe6dbbb87445ff222c8a/status
I1010 16:40:34.320360       1 csi_handler.go:612] Saved detach error to "csi-3df4c97cd2abde367eebc52e875e7af60add6daa8a4afe6dbbb87445ff222c8a"
I1010 16:40:34.320403       1 csi_handler.go:226] Error processing "csi-3df4c97cd2abde367eebc52e875e7af60add6daa8a4afe6dbbb87445ff222c8a": failed to detach: rpc error: code = Internal desc = ControllerpublishVolume failed for pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4: Message: 'Resource 'pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4' is still in use.'; Cause: 'Resource is mounted/in use.'; Details: 'Node: m9c17, Resource: pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4'; Correction: 'Un-mount resource 'pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4' on the node 'm9c17'.'; Reports: '[5F81D04D-00000-024056]'

After a while the diskless resource will be removed from the node:

╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                             ┊ Node  ┊ Port  ┊ Usage  ┊ Conns ┊    State ┊ CreatedOn ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4 ┊ m5c18 ┊ 55207 ┊ Unused ┊ Ok    ┊ UpToDate ┊           ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

But volumeattachment will continue existing on the node

# kubectl get volumeattachments.storage.k8s.io | grep pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4
csi-3df4c97cd2abde367eebc52e875e7af60add6daa8a4afe6dbbb87445ff222c8a   linstor.csi.linbit.com   pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4   m9c17    true       138m

However it will not allow pod to start, because drbd device is missing, the one of possible way to fix it, is to create resource manually, to satisfy existing volumeattachment

linstor r c m9c17 pvc-712ea0dc-5378-41fc-8c8a-5db8f50c8db4 --diskless

I guess this is exact case mentioned by @rck in #52 (comment)

Volume attachment doesn't work with examples/k8s/deploy/linstor-csi-1.16.yaml

Steps to reproduce

Apply fix from #39
kubectl apply -f linstor-csi-1.16.yaml
kubectl apply -f storage-class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-r1
parameters:
  autoPlace: "1"
  storagePool: linstor-pool
provisioner: linstor.csi.linbit.com

kubectl apply -f busybox.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: busybox
spec:
  selector:
    matchLabels:
      app: "busybox"
  serviceName: "default"
  replicas: 2
  template:
    metadata:
      labels:
        app: "busybox"
    spec:
      containers:
      - name: debian
        image: debian:9-slim
        command: ["/bin/sleep"]
        args: [ "3600" ]
        volumeMounts:
        - name: vol
          mountPath: /mnt
  volumeClaimTemplates:
  - metadata:
      name: vol
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "linstor-r1"
      resources:
        requests:
          storage: 100Gi

kubectl describe pod busybox-0

Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Warning  FailedMount             58m (x294 over 14h)  kubelet, es-101          Unable to attach or mount volumes: unmounted volumes=[vol], unattached volumes=[vol default-token-gfq6q]: timed out waiting for the condition
  Warning  FailedAttachVolume      58m (x210 over 14h)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-fea159a6-a8cd-4810-b15e-920bd421b180" : attachment timeout for volume pvc-fea159a6-a8cd-4810-b15e-920bd421b180
  Warning  FailedMount             54m (x77 over 14h)   kubelet, es-101          Unable to attach or mount volumes: unmounted volumes=[vol], unattached volumes=[default-token-gfq6q vol]: timed out waiting for the condition

kubectl logs linstor-csi-controller-0 csi-attacher -n kube-system

I1013 10:56:39.080950       1 csi_handler.go:412] Saving attach error to "csi-65c63822999778b93d5bf243cdb998d49d5c91dce9ad0b99c0e22e2f33c56769"
I1013 10:56:39.083240       1 csi_handler.go:123] Failed to save attach error to "csi-65c63822999778b93d5bf243cdb998d49d5c91dce9ad0b99c0e22e2f33c56769": volumeattachments.storage.k8s.io "csi-65c63822999778b93d5bf243cdb998d49d5c91dce9ad0b99c0e22e2f33c56769" is forbidden: User "system:serviceaccount:kube-system:linstor-csi-controller-sa" cannot patch resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope
I1013 10:56:39.083266       1 csi_handler.go:99] Error processing "csi-65c63822999778b93d5bf243cdb998d49d5c91dce9ad0b99c0e22e2f33c56769": failed to attach: could not add PersistentVolume finalizer: persistentvolumes "pvc-fea159a6-a8cd-4810-b15e-920bd421b180" is forbidden: User "system:serviceaccount:kube-system:linstor-csi-controller-sa" cannot patch resource "persistentvolumes" in API group "" at the cluster scope

How to fix

Add patch permission for linstor-csi-attacher-role on persistentvolumes and volumeattachments in linstor-csi-1.16.yaml like that:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: linstor-csi-attacher-role
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update", "patch"]

Unable to find the device path of volume zero of resource is spamming my events

This is quite usual problem if you're creating new PVCs and then immediately start using them, eg. when applying new helm chart.

This is events for the pod:

pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
Successfully assigned harbor/harbor-harbor-redis-0 to pve3
AttachVolume.Attach succeeded for volume "pvc-219355c9-70a5-11e9-86b2-001999d764e2"
MountVolume.MountDevice failed for volume "pvc-219355c9-70a5-11e9-86b2-001999d764e2" : rpc error: code = Unknown desc = unable to find the device path of volume zero of resource pvc-219355c9-70a5-11e9-86b2-001999d764e2 on node pve3 in [<very_big_map>]
MountVolume.MountDevice failed for volume "pvc-219355c9-70a5-11e9-86b2-001999d764e2" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
MountVolume.MountDevice failed for volume "pvc-219355c9-70a5-11e9-86b2-001999d764e2" : rpc error: code = Unknown desc = unable to mount device: exit status 32: mount: /dev/drbd1231 is already mounted or /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-219355c9-70a5-11e9-86b2-001999d764e2/globalmount busy

Where <very_big_map> is map which contains all my resources in linstor.
Right now it is containing 588231 symbols, and weight 576 kilobytes!. This is really huge for single event, after a while it might reach the limit of 1M in etcd.

https://github.com/LINBIT/golinstor/blob/3455ca3e0ab482722dd2952df048e57460a124f6/linstor.go#L1176

Volumes are creating with nodeAffinity set

When you having hundreds of PV it might create overload the api-server and provisioner, also there is no opportunity to using PVs on newly created nodes afterwards, example:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: linstor.csi.linbit.com
  creationTimestamp: "2020-01-29T16:21:58Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/linstor-csi-linbit-com
  name: pvc-ffc3e321-8f4a-4036-ae67-0ed27b27fb96
  resourceVersion: "2510909725"
  selfLink: /api/v1/persistentvolumes/pvc-ffc3e321-8f4a-4036-ae67-0ed27b27fb96
  uid: 98e518b9-0277-442d-bc75-e1973e407740
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 150Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: vol-data-mariadb-hc1-md14-1-0
    namespace: hosting
    resourceVersion: "2510907409"
    uid: ffc3e321-8f4a-4036-ae67-0ed27b27fb96
  csi:
    driver: linstor.csi.linbit.com
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1576375528840-8081-linstor.csi.linbit.com
    volumeHandle: pvc-ffc3e321-8f4a-4036-ae67-0ed27b27fb96
  mountOptions:
  - errors=remount-ro
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m10c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m11c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m12c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m13c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m14c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m15c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m16c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m5c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m6c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m7c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m8c9
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c10
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c11
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c12
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c13
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c14
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c15
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c16
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c17
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c18
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c19
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c20
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c21
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c22
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c23
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c24
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c25
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c26
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c27
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c28
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c29
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c3
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c30
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c31
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c32
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c33
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c34
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c35
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c36
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c37
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c38
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c39
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c4
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c40
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c41
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c42
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c43
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c44
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c45
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c5
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c6
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c7
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - m9c8
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - pve1
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - pve2
      - matchExpressions:
        - key: linbit.com/hostname
          operator: In
          values:
          - pve3
  persistentVolumeReclaimPolicy: Delete
  storageClassName: linstor-hosting-2
  volumeMode: Filesystem
status:
  phase: Bound

fsGroup does not apply

Linstor csi: 0.9.1
Kubernetes: v1.18.3.
Pool: ZFS.
DRBD: DRBDADM_API_VERSION=2 DRBD_KERNEL_VERSION=9.0.25 DRBDADM_VERSION=9.15.0

I use securityContext to run not as root user and i have no write access to the mounted volume.
My test Pod:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: linstor-hdd-r2
parameters:
  autoPlace: "2"
  storagePool: hdd
provisioner: linstor.csi.linbit.com

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: task-pv-claim
spec:
  storageClassName: linstor-hdd-r2
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi

---
kind: Pod
metadata:
  name: task-pv-pod-test2
spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 1000
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: task-pv-claim
  containers:
    - name: task-pv-container
      image: busybox
      command:
        - sh
        - -c
        - 'while true; do date > /mnt/index.html; sleep $(($RANDOM % 5 + 5)); done'
      securityContext:
        runAsUser: 1000
      volumeMounts:
        - mountPath: "/mnt"
          name: task-pv-storage

result:

k exec -it task-pv-pod-test2 -- ls -al | grep mnt
drwxr-xr-x    3 root     root          4096 Oct 28 08:55 mnt

---
stat  /mnt
  File: /mnt
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: 409305h/4231941d        Inode: 2           Links: 3
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)

---
k logs task-pv-pod-test2
sh: can't create /mnt/index.html: Permission denied
sh: can't create /mnt/index.html: Permission denied
sh: can't create /mnt/index.html: Permission denied

kubelet logs:

docker logs kubelet 2>&1 | grep pvc-182197ac-2d36-47ac-8214-8245db8fea2b
I1029 08:33:43.097319    2586 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71")
E1029 08:33:43.097419    2586 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b podName: nodeName:}" failed. No retries permitted until 2020-10-29 08:33:43.597381467 +0000 UTC m=+902268.100838580 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-182197ac-2d36-47ac-8214-8245db8fea2b\" (UniqueName: \"kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b\") pod \"task-pv-pod-test2\" (UID: \"c530cd9f-0f69-4ed4-9d46-14d4fb600c71\") "
I1029 08:33:43.599519    2586 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71")
E1029 08:33:43.599719    2586 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b podName: nodeName:}" failed. No retries permitted until 2020-10-29 08:33:44.599644374 +0000 UTC m=+902269.103101514 (durationBeforeRetry 1s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-182197ac-2d36-47ac-8214-8245db8fea2b\" (UniqueName: \"kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b\") pod \"task-pv-pod-test2\" (UID: \"c530cd9f-0f69-4ed4-9d46-14d4fb600c71\") "
I1029 08:33:44.604658    2586 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71")
E1029 08:33:44.604877    2586 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b podName: nodeName:}" failed. No retries permitted until 2020-10-29 08:33:46.604810337 +0000 UTC m=+902271.108267471 (durationBeforeRetry 2s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-182197ac-2d36-47ac-8214-8245db8fea2b\" (UniqueName: \"kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b\") pod \"task-pv-pod-test2\" (UID: \"c530cd9f-0f69-4ed4-9d46-14d4fb600c71\") "
I1029 08:33:46.614297    2586 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71")
E1029 08:33:46.614417    2586 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b podName: nodeName:}" failed. No retries permitted until 2020-10-29 08:33:50.614386514 +0000 UTC m=+902275.117843586 (durationBeforeRetry 4s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-182197ac-2d36-47ac-8214-8245db8fea2b\" (UniqueName: \"kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b\") pod \"task-pv-pod-test2\" (UID: \"c530cd9f-0f69-4ed4-9d46-14d4fb600c71\") "
I1029 08:33:50.632247    2586 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71")
E1029 08:33:50.632449    2586 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b podName: nodeName:}" failed. No retries permitted until 2020-10-29 08:33:58.632382821 +0000 UTC m=+902283.135839928 (durationBeforeRetry 8s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-182197ac-2d36-47ac-8214-8245db8fea2b\" (UniqueName: \"kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b\") pod \"task-pv-pod-test2\" (UID: \"c530cd9f-0f69-4ed4-9d46-14d4fb600c71\") "
I1029 08:33:58.664551    2586 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71")
I1029 08:33:58.667989    2586 operation_generator.go:1332] Controller attach succeeded for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71") device path: ""
I1029 08:33:58.764954    2586 reconciler.go:269] operationExecutor.MountVolume started for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71")
I1029 08:33:58.767841    2586 operation_generator.go:558] MountVolume.WaitForAttach entering for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71") DevicePath ""
I1029 08:33:58.772203    2586 operation_generator.go:567] MountVolume.WaitForAttach succeeded for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71") DevicePath "csi-0b30f019420a3a1ac2a9c35dfd41dc7c4340f9210584545ba276b5b7068ca0fc"
I1029 08:33:58.778347    2586 operation_generator.go:596] MountVolume.MountDevice succeeded for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71") device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-182197ac-2d36-47ac-8214-8245db8fea2b/globalmount"
I1029 08:34:01.145037    2586 operation_generator.go:657] MountVolume.SetUp succeeded for volume "pvc-182197ac-2d36-47ac-8214-8245db8fea2b" (UniqueName: "kubernetes.io/csi/linstor.csi.linbit.com^pvc-182197ac-2d36-47ac-8214-8245db8fea2b") pod "task-pv-pod-test2" (UID: "c530cd9f-0f69-4ed4-9d46-14d4fb600c71")
I1029 08:34:01.321747    2586 kuberuntime_manager.go:422] No sandbox for pod "task-pv-pod-test2_default(c530cd9f-0f69-4ed4-9d46-14d4fb600c71)" can be found. Need to start a new one
I1029 08:34:01.987395    2586 kubelet.go:1952] SyncLoop (PLEG): "task-pv-pod-test2_default(c530cd9f-0f69-4ed4-9d46-14d4fb600c71)", event: &pleg.PodLifecycleEvent{ID:"c530cd9f-0f69-4ed4-9d46-14d4fb600c71", Type:"ContainerStarted", Data:"269baf56ec4c93041a7f03732c3dcc4f9d85d83fc1353c6dfe079fa8a0eb00ca"}
I1029 08:34:05.229805    2586 kubelet.go:1952] SyncLoop (PLEG): "task-pv-pod-test2_default(c530cd9f-0f69-4ed4-9d46-14d4fb600c71)", event: &pleg.PodLifecycleEvent{ID:"c530cd9f-0f69-4ed4-9d46-14d4fb600c71", Type:"ContainerStarted", Data:"97cce30e3a25f897d92f9af24379f96470773d0e41aaf4ad5b79bbfcfdd0410d"}
W1029 08:34:05.397550    2586 volume_linux.go:49] Setting volume ownership for /var/lib/kubelet/pods/c530cd9f-0f69-4ed4-9d46-14d4fb600c71/volumes/kubernetes.io~secret/default-token-smxm5 and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow, see https://github.com/kubernetes/kubernetes/issues/69699

With "layerlist: storage", "placementPolicy: FollowTopology" creates a drbd device instead of an LV

Hi!
I'm using the latest piraeus-operator. With layerlist: storage, placementPolicy: FollowTopology creates a drbd device instead of an LV.

sc.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-hdd-lvm
provisioner: linstor.csi.linbit.com
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
parameters:
  layerlist: storage
  placementCount: "1"
  placementPolicy: FollowTopology
  allowRemoteVolumeAccess: "false"
  disklessOnRemaining: "false"
  csi.storage.k8s.io/fstype: xfs
  mountOpts: noatime,discard
  storagePool: hdd

oc -n piraeus-demo get pvc

NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
linstor-lvm   Bound    pvc-15523a8a-e0d2-48c3-9fa4-e79725def3cd   23Gi       RWO            linstor-hdd-lvm   9s

kubectl -n piraeus-demo describe pv pvc-15523a8a-e0d2-48c3-9fa4-e79725def3cd

Name:              pvc-15523a8a-e0d2-48c3-9fa4-e79725def3cd
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: linstor.csi.linbit.com
Finalizers:        [kubernetes.io/pv-protection external-attacher/linstor-csi-linbit-com]
StorageClass:      linstor-hdd-lvm
Status:            Bound
Claim:             piraeus-demo/linstor-lvm
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          23Gi
Node Affinity:
  Required Terms:
    Term 0:        linbit.com/hostname in [intel-1]
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            linstor.csi.linbit.com
    FSType:            xfs
    VolumeHandle:      pvc-15523a8a-e0d2-48c3-9fa4-e79725def3cd
    ReadOnly:          false
    VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity=1608004506593-8081-linstor.csi.linbit.com
Events:                <none>

┊ intel-1 ┊ pvc-15523a8a-e0d2-48c3-9fa4-e79725def3cd ┊ raid10 ┊ 0 ┊ 1013 ┊ /dev/drbd1013 ┊ 23.01 GiB ┊ InUse ┊ UpToDate ┊

Crash in NodePublishVolume

This happened in a recent run of the CSI E2E tests. Can't really say how to reproduce this, but anyways, this is the log:

time="2020-11-25T15:37:22Z" level=debug msg="method called" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=driver method=/csi.v1.Node/NodeGetCapabilities nodeID=demo1.linstor-days.at.linbit.com provisioner=linstor.csi.linbit.com req= resp="capabilities:<rpc:<type:GET_VOLUME_STATS > > capabilities:<rpc:<type:EXPAND_VOLUME > > " version=0.10.1
time="2020-11-25T15:37:22Z" level=debug msg="method called" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=driver method=/csi.v1.Node/NodeGetCapabilities nodeID=demo1.linstor-days.at.linbit.com provisioner=linstor.csi.linbit.com req= resp="capabilities:<rpc:<type:GET_VOLUME_STATS > > capabilities:<rpc:<type:EXPAND_VOLUME > > " version=0.10.1
time="2020-11-25T15:37:22Z" level=debug msg="looking up resource by CSI volume id" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" csiVolumeID=pvc-12bcfee8-1cfa-469f-b281-12b125ec9d44 linstorCSIComponent=client
time="2020-11-25T15:37:22Z" level=debug msg="curl -X 'GET' -H 'Accept: application/json' 'http://linstor-cs.linstor.svc:3370/v1/resource-definitions/pvc-12bcfee8-1cfa-469f-b281-12b125ec9d44'"
time="2020-11-25T15:37:22Z" level=debug msg="Status code not within 200 to 400, but 404 (Not Found)\n"
time="2020-11-25T15:37:22Z" level=debug msg="getting assignment info" func="github.com/sirupsen/logrus.(*Entry).Debug" file="/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:277" linstorCSIComponent=client targetNode=demo1.linstor-days.at.linbit.com volume="<nil>"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x12042f2]

goroutine 78 [running]:
github.com/piraeusdatastore/linstor-csi/pkg/client.(*Linstor).GetAssignmentOnNode(0xc00027ce10, 0x16380a0, 0xc000508ae0, 0x0, 0x7ffd1b038bd1, 0x20, 0x0, 0x0, 0x0)
	/buildroot/pkg/client/linstor.go:956 +0x292
github.com/piraeusdatastore/linstor-csi/pkg/driver.Driver.NodePublishVolume(0x1643860, 0xc00027ce10, 0x1646860, 0xc00027ce10, 0x162ae60, 0xc00027ce10, 0x16438c0, 0xc00027ce10, 0x1607e60, 0xc00027ce10, ...)
	/buildroot/pkg/driver/driver.go:309 +0x6d1
github.com/container-storage-interface/spec/lib/go/csi._Node_NodePublishVolume_Handler.func1(0x16380a0, 0xc000508ae0, 0x1405400, 0xc000382480, 0x135759e, 0x1ef, 0xc000589758, 0x75cf38)
	/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5594 +0x89
github.com/piraeusdatastore/linstor-csi/pkg/driver.Driver.Run.func1(0x16380a0, 0xc000508ae0, 0x1405400, 0xc000382480, 0xc00025c840, 0xc00025c860, 0xc000399a48, 0x592486, 0x13c7aa0, 0xc000508ae0)
	/buildroot/pkg/driver/driver.go:1042 +0x7e
github.com/container-storage-interface/spec/lib/go/csi._Node_NodePublishVolume_Handler(0x146d3a0, 0xc000219140, 0x16380a0, 0xc000508ae0, 0xc00003ad20, 0xc0002a8480, 0x16380a0, 0xc000508ae0, 0xc0005c2480, 0x21c)
	/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5596 +0x150
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000298480, 0x1646920, 0xc000092480, 0xc000334200, 0xc00027d410, 0x1e29630, 0x0, 0x0, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:971 +0x486
google.golang.org/grpc.(*Server).handleStream(0xc000298480, 0x1646920, 0xc000092480, 0xc000334200, 0x0)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1250 +0xcac
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003845f0, 0xc000298480, 0x1646920, 0xc000092480, 0xc000334200)
	/go/pkg/mod/google.golang.org/[email protected]/server.go:690 +0xa5
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/pkg/mod/google.golang.org/[email protected]/server.go:688 +0xa5

In any case, a 404 should not result in a nil dereference

ProvisioningFailed unsupported protocol scheme

Hi,

I could not find an up-to-date documentation how to deploy CSI plugin and make linstor working with k8s. Thus, I have changed LISTEN_IP and value linstor.example.com in linstor-1.16 yaml file. Pushed all yaml files into my k8s cluster. Now, I am trying to create a volume and get an error:

ProvisioningFailed	failed to provision volume with StorageClass "linstor": rpc error: code = Internal desc = CreateVolume failed for pvc-6a36d62a-143d-48a0-ab04-dad0f9c08892: Get /v1/resource-definitions: unsupported protocol scheme "

Any suggestions on that please? Thank you

With "layerlist: storage", "placementPolicy: FollowTopology" fails to create a volume

I'm using the latest piraeus-operator. with layerlist: storage, placementPolicy: FollowTopology fails to create a volume, where as placementPolicy: AutoPlace creates a drbd device instead of an LV.

Image versions:

  pluginImage: quay.io/piraeusdatastore/piraeus-csi:v0.9.0
  csiAttacherImage: quay.io/k8scsi/csi-attacher:v2.2.0
  csiNodeDriverRegistrarImage: quay.io/k8scsi/csi-node-driver-registrar:v1.3.0
  csiProvisionerImage: quay.io/k8scsi/csi-provisioner:v1.6.0
  csiSnapshotterImage: quay.io/k8scsi/csi-snapshotter:v2.1.0
  csiResizerImage: quay.io/k8scsi/csi-resizer:v0.5.0
  controllerImage: quay.io/piraeusdatastore/piraeus-server:v1.7.1
  kernelModImage: quay.io/piraeusdatastore/drbd9-centos7:v9.0.24

Storage pool:

linstor sp l
+----------------------------------------------------------------------------------------------------------------------------------------+
| StoragePool          | Node       | Driver   | PoolName                          | FreeCapacity | TotalCapacity | CanSnapshots | State |
|========================================================================================================================================|
| DfltDisklessStorPool | k8s-node-1 | DISKLESS |                                   |              |               | False        | Ok    |
| DfltDisklessStorPool | k8s-node-2 | DISKLESS |                                   |              |               | False        | Ok    |
| DfltDisklessStorPool | k8s-node-3 | DISKLESS |                                   |              |               | False        | Ok    |
| DfltStorPool         | k8s-node-1 | LVM_THIN | linstor_DfltStorPool/DfltStorPool |    99.79 GiB |     99.80 GiB | True         | Ok    |
| DfltStorPool         | k8s-node-2 | LVM_THIN | linstor_DfltStorPool/DfltStorPool |    99.80 GiB |     99.80 GiB | True         | Ok    |
| DfltStorPool         | k8s-node-3 | LVM_THIN | linstor_DfltStorPool/DfltStorPool |    99.80 GiB |     99.80 GiB | True         | Ok    |
+----------------------------------------------------------------------------------------------------------------------------------------+

placementPolicy: FollowTopology

sc.yaml:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: piraeus-dflt-raw
provisioner: linstor.csi.linbit.com
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  layerlist: storage
  placementCount: "1"
  placementPolicy: FollowTopology
  allowRemoteVolumeAccess: "false"
  disklessOnRemaining: "false"
  csi.storage.k8s.io/fstype: xfs
  mountOpts: noatime,discard
  storagePool: DfltStorPool

kubectl -n piraeus-demo get pvc
NAME           STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS       AGE
demo-rwo-raw   Pending                                      piraeus-dflt-raw   104s

kubectl describe pvc
Events:
  Type     Reason                Age                From                                                                                                    Message
  ----     ------                ----               ----                                                                                                    -------
  Normal   Provisioning          31s (x6 over 60s)  linstor.csi.linbit.com_piraeus-op-csi-controller-6db954d4b9-gkjrz_d4ef26d8-9182-42bc-84bd-4f31a7dc9676  External provisioner is provisioning volume for claim "piraeus-demo/demo-rwo-raw"
  Warning  ProvisioningFailed    30s (x6 over 60s)  linstor.csi.linbit.com_piraeus-op-csi-controller-6db954d4b9-gkjrz_d4ef26d8-9182-42bc-84bd-4f31a7dc9676  failed to provision volume with StorageClass "piraeus-dflt-raw": rpc error: code = Internal desc = CreateVolume failed for pvc-508bb7e2-8f1f-4aa4-9081-9b55a9300b5d: no volume topologies, unable to schedule volume pvc-508bb7e2-8f1f-4aa4-9081-9b55a9300b5d
  Normal   ExternalProvisioning  15s (x5 over 60s)  persistentvolume-controller                                                                             waiting for a volume to be created, either by external provisioner "linstor.csi.linbit.com" or manually created by system administrator

placementPolicy: AutoPlace

sc.yaml:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: piraeus-dflt-raw
provisioner: linstor.csi.linbit.com
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  layerlist: storage
  placementCount: "1"
  placementPolicy: AutoPlace
  allowRemoteVolumeAccess: "false"
  disklessOnRemaining: "false"
  csi.storage.k8s.io/fstype: xfs
  mountOpts: noatime,discard
  storagePool: DfltStorPool

kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
demo-rwo-raw   Bound    pvc-d2a914e7-c4ba-41b9-af50-e5430fbc7488   10Gi       RWO            piraeus-dflt-raw   9m40s

linstor v l
+----------------------------------------------------------------------------------------------------------------------------------------+
| Node       | Resource                                 | StoragePool  | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|========================================================================================================================================|
| k8s-node-1 | pvc-d2a914e7-c4ba-41b9-af50-e5430fbc7488 | DfltStorPool |     0 |    1000 | /dev/drbd1000 |  2.05 MiB | Unused | UpToDate |
+----------------------------------------------------------------------------------------------------------------------------------------+

Error in examples/k8s/deploy/linstor-csi-1.16.yaml

Hello!

Deploy of linstor-csi-1.16.yaml finishes with error:

root@kube1:~/linstor/demo# kubectl apply -f linstor-csi-1.16.yaml
error: error validating "linstor-csi-1.16.yaml": error validating data: ValidationError(StatefulSet.spec): missing required field "selector" in io.k8s.api.apps.v1.StatefulSetSpec; if you choose to ignore these errors, turn validation off with --validate=false

Swithing off validation raise error too :

root@kube1:~/linstor/demo# kubectl apply -f linstor-csi-1.16.yaml --validate=false
serviceaccount/linstor-csi-controller-sa created
clusterrole.rbac.authorization.k8s.io/linstor-csi-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/linstor-csi-provisioner-binding created
clusterrole.rbac.authorization.k8s.io/linstor-csi-attacher-role created
clusterrolebinding.rbac.authorization.k8s.io/linstor-csi-attacher-binding created
clusterrole.rbac.authorization.k8s.io/linstor-csi-cluster-driver-registrar-role created
clusterrolebinding.rbac.authorization.k8s.io/linstor-csi-cluster-driver-registrar-binding created
daemonset.apps/linstor-csi-node created
serviceaccount/linstor-csi-node-sa created
clusterrole.rbac.authorization.k8s.io/linstor-csi-driver-registrar-role created
clusterrolebinding.rbac.authorization.k8s.io/linstor-csi-driver-registrar-binding created
clusterrole.rbac.authorization.k8s.io/linstor-csi-snapshotter-role created
clusterrolebinding.rbac.authorization.k8s.io/linstor-csi-snapshotter-binding created
The StatefulSet "linstor-csi-controller" is invalid:
* spec.selector: Required value
* spec.template.metadata.labels: Invalid value: map[string]string{"app":"linstor-csi-controller", "role":"linstor-csi"}: `selector` does not match template `labels`

I'm using Kubernetes 16.1:

root@kube1:~/linstor/demo# kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

Thanks.

Update driver snapshotter to v1.2.0

Currently, Linstor CSI driver has csi-snapshotter version of v1.1.0.
Our use case (prebound snapshots) is not well-supported in csi-snapshotter v1.1.0.
Can you update csi-snapshotter from v1.1.0 to v1.2.x?

Not able to use satellite nodes outside of Kubernetes cluster

May be this is not an issue and that I'm not aware of how to configure but then I started with the setup of K8s cluster with its own nodes and the Linstor cluster completely outside of it, with its own controller and two satellite nodes (6 Linux servers totally). With that, when PV is provisioned, it has nodeaffinity set so that pods could be created only in linbit.com/hostname=<satellite>. This is a problem since K8s cluster nodes are completely different and no pods would get scheduled. Then, I modified the setup to run Linstor converged - that is, I have a separate controller server but then use the K8s minion nodes as Linstor controllers. So, four servers in all - K8s master, Linstor controller, and two K8s minion/satellites. Now, the problem is, when a minion/satellite goes down, K8s is not able to schedule the pod. ~~because of the classic "Multi-Attach" error, since I use AWS EBS volumes for the storage pool. Eventually, it worked with EBS volume getting detached.~~ The issue I'd like to raise is the lack of ability to run the linstor cluster completely outside so that regardless of the node where the K8s pod is scheduled, the storage can be accessed. Again, this might not be an issue. Just that, I don't know how to configure. If that's the case, please feel free to close this issue. If you can provide me some pointers, it'd be greatly appreciated.

UPDATE: I mistook the multi-attach for EBS issue. Looks like it is not. Sorry about that.

More informative error if linstor-csi-node pod can't reach linstor-controller

When linstor-csi-node can't reach linstor-controller druring mount operation, it will add an error to the pod's events:

MountVolume.MountDevice failed for volume "pvc-69bf68de-6b3a-11e9-86b2-001999d764e2" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

It would be nice if this error will be more informative.

Helm chart to deploy on GKE

Is there an Helm chart to deploy Linstron on Google Kubernetes Engine?

linstor-csi is not compatible with `--enable-controller-attach-detach=false`

Hi, my pods are stuck on Init because attaching volumes is not working, here is kubelet log:

kubelet[5246]: I0302 04:17:45.983773    5246 reconciler.go:227] operationExecutor.AttachVolume started for volume "pvc-c4fd2215-3c18-11e9-bffa-001999d764e2" (UniqueName: "kubernetes.io/csi/io.drbd.linstor-csi^bf753dd6-48df-4157-a421-74d802a59e44") pod "hc1-wd43-0" (UID: "13418de2-3c97-11e9-bffa-001999d764e2")
kubelet[5246]: E0302 04:17:45.986787    5246 csi_attacher.go:103] kubernetes.io/csi: attacher.Attach failed: volumeattachments.storage.k8s.io is forbidden: User "system:node:m12c16" cannot create resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: can only get individual resources of this type

It seems something wrong with RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:node
rules:
...
- apiGroups:
  - storage.k8s.io
  resources:
  - volumeattachments
  verbs:
  - get

kubectl version

Client Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.0-alpha.0.1172+29b3ec91089d0e-dirty", GitCommit:"29b3ec91089d0e213c15118e3c39087bcfbb6400", GitTreeState:"dirty", BuildDate:"2018-12-20T23:05:11Z", GoVersion:"go1.11.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

piraeusdatastore / linstor-csi Goto Github PK

linstor-csi's Introduction

Linstor CSI Plugin

Building

Deployment

Kubernetes

Usage

⚠️️ Known issues

Kubernetes

Kubevirt

linstor-csi's People

Contributors

Stargazers

Watchers

Forkers

linstor-csi's Issues

Events on test-0 :

kubectl logs linstor-csi-node-pvdjb -n linstor -c linstor-csi-plugin :

linstor controller

drbdadm status on node2

drbdsetup events on node2

Definition

Steps to reproduce

How to fix

Recommend Projects

Recommend Topics

Recommend Org

Events on `test-0` :

`kubectl logs linstor-csi-node-pvdjb -n linstor -c linstor-csi-plugin` :

`drbdadm status` on node2

`drbdsetup events` on node2