From kubernetes-announce:
"On the 3rd of April 2023, the old registry k8s.gcr.io will be frozen and no further images for Kubernetes and related subprojects will be pushed to the old registry.

This registry registry.k8s.io replaced the old one and has been generally available for several months. We have published a blog post about its benefits to the community and the Kubernetes project."

The leader election package specified 3 variables that are related to the time interval of acquiring and renewing leases. We should allow these options to be configurable in external-health-monitor-controller. See detailed description in the following issue:
kubernetes-csi/external-provisioner#556

See a PR that addressed this:
kubernetes-csi/external-provisioner#643

We need to make the changes here:
https://github.com/kubernetes-csi/external-health-monitor/blob/master/cmd/csi-external-health-monitor-controller/main.go

Reason

When I integrated the external-health-monitor with host-path-driver, I found its release script to get the rbac yaml files of csi components based on image tag. Because external-health-monitor has been released as v0.1.0, it will take https://raw.githubusercontent.com/kubernetes-csi/external-health-monitor/v0.1.0/deploy/kubernetes/external-health-monitor-controller/rbac.yaml

But, there is a bug in rbac file of external-health-monitor-controller. It was fixed in this commit: 78fb385#diff-4f92853217f76a2e31d2f601fcf8a532c199a96999ed40318cb2c6a93c7d9604L38

So, I want to do a new release for resolving the issue I encountered

I'd love to see the readonly attribute of the publish request passed into this method. A specific use-case I'm addressing is when an iscsi connection drops and then comes back online the mount becomes ro instead of rw. However without knowing specifically if it supposed to be ro I can for certain say if it's unhealthy or not.

In my case I support staging so on the same node it's theoretically possible (at least from a csi perspective...maybe not k8s) to have the same volume published in 1 case ro and another case rw. I support this by staging rw and then doing a bind mount with ro if appropriate. Not sure if the stats method gets called on the staged target or the published target but that could make the logic more complex.

Thanks!

We have an issue #75 to change the code to only watch Pods and Nodes when the Node Watcher component is enabled. We still need to address the scalability issue when Node Watcher is enabled:

kubernetes/kubernetes#102452 (comment)

For local storage (e.g. LVM disks) it makes sense to support a deployment model where the driver only runs locally on nodes together with sidecars. This is already support for provisioning (kubernetes-csi/external-provisioner#524, https://github.com/kubernetes-csi/external-provisioner#deployment-on-each-node).

Currently, when we enable health monitoring on the node, is there a way to regulate the interval for kubelet to make calls to NodeGetVolumeStats as part of collecting Volume health condition similar to how we can specify the interval when health monitor is enabled on controller ?

Images https://quay.io/k8scsi/csi-external-health-monitor-controller:canary and https://quay.io/k8scsi/csi-external-health-monitor-agent:canary are not available. After release is cut, new images will be accessible from a different repo.

For now, the deployment file should be modified to point to a placeholder, not an actual location.

My k8s version is 1.20.11, still need to deploy csi-external-health-monitor-agent service.

When I use block type PV, the staging_target_path address provided by kubelet is /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-9fb798fe-7e26-4488-85c4-4c8e660f7351

The address provided by csi-external-health-monitor-agent during monitoring is /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-9fb798fe-7e26-4488-85c4-4c8e660f7351

We need to update README with the following information:

Add a compatibility matrix as follows:
https://github.com/kubernetes-csi/external-snapshotter#compatibility

Document all the flags as follows:
https://github.com/kubernetes-csi/external-snapshotter#snapshot-controller-command-line-options
https://github.com/kubernetes-csi/external-snapshotter#csi-external-snapshotter-sidecar-command-line-options
We should have sections for csi-external-health-monitor-controller-sidecar-command-line-options and csi-external-health-monitor-agent-sidecar-command-line-options.

if the pv is not found, the response of the pv is nil, so the pv.Name will causes the panic.

CC @xing-yang

Reproduce:

1. deploy the external-provisioner in election-leader mode and with replicatins >=2.
2. disable "ControllerListVolume" capability in hostpath driver.
3. deploy the hostpath driver in daemonset.
4. use "ControllerGetVolume" to get volume'status.
5. deploy the healthe monitor sidecar with election-leader mode.
6. create pvc in the local.

Fix:

1. if the backend storage is local, we should filter the pv with node name. Only monitor the pv in the same node with the sidecar

Suggestion:

1. If backend storege is local disk, Not only "COntrollerGetVolume" or "ControllerListVolume" should monitor the local volume that with the monitor sidecar in the same node.

CC @xing-yang

Currently an event will be recorded on pvc/pod when the controller/agent has successfully retrieved an abnormal volume condition from the storage system. However when other errors occur in the controller/agent, the errors will be logged but not recorded as events. Before moving to beta, the controller/agent should be modified to record an event when other errors occur.

What happened?

when the state of pv goes from bound->available->bound, heath-monitor sidecar could not continued to monitoring pv health

What did you expect to happen?

pv health monitoring always work when pv status is bound

How can we reproduce it (as minimally and precisely as possible)?

1. external-health-monitor work with csi plugin which not support supportListVolumes and support controlVolumeStatus
2. create a pvc and then provisioing pv success, and reclaimPolicy=Retain in sc
3. in some case, need delete pvc and delete spec.claimRef in pv, and the pv status is change to available
4. create the deleted pvc, and the pvc status is change to bound
5. mock to the volume is not health, but the external-health-monitor is not event abnormal event

the case is exist, for example when resize the volume capacity failure, need to delete pvc and create again to resume the status of pvc and pv.

Anything else we need to know?

in pkg/controller/pv_monitor_controller.go:229, get pv from ctrl.pvQueue to check. when pv.Status.Phase is not bound, the ctrl.pvEnqueued[pv] is always true. so the pv not be re-enqueue ctrl.pvQueue

env version

kubernetes is v1.28.0
external-health-monitor is v0.9.0

/kind bug
/assign

If ListVolumes() CSI call does not find a volume that exists in K8s cluster, external health monitor controller should treat it as not found and report an event.

In the external-health-monitor-controller, we always watch all PVCs, Pods, Nodes:
https://github.com/kubernetes-csi/external-health-monitor/blob/v0.3.0/cmd/csi-external-health-monitor-controller/main.go#L210

We need to watch them to support the Node Watcher functionality which is disabled by default. This has caused scalability problems.
https://github.com/kubernetes-csi/external-health-monitor/blob/v0.3.0/cmd/csi-external-health-monitor-controller/main.go#L66

We should change the code to only watch Pods and Nodes when the Node Watcher component is enabled.

kubernetes/kubernetes#102452 (comment)

When volumes recover from unhealthy condition, check if abnormal events are sent, if so, send recovery events to indicate the volumes are healthy now.

We are planning to implement the volume health interface within our csi plugins. Since the volume abnormality signals are exposed as events on pvc (from controller) or pods (from node agent), they can’t be exposed as prometheus metrics since kube-state-metrics doesn’t support exporting k8s event resources due to high cardinality (no mention of events in documentation). Adding up pv failure metrics to prometheus are essential for setting up alerts in multi-tenant k8s cluster.

Is it possible to expose these volume health abnormalities as prometheus metrics from external-health-monitor controller and node agent itself?

Initial Thread:
https://kubernetes.slack.com/archives/C09QZFCE5/p1620665651243600

Open this issue to track the implementation of volume health monitoring agent.

We should implement health monitoring in CSI hostpath driver so that we can add e2e tests for health monitoring.

This can be done after the alpha release.

Hello,

At the line https://github.com/kubernetes-csi/external-health-monitor/blob/master/pkg/agent/pv_monitor_agent.go#L209 is incorrect. If the error is at line https://github.com/kubernetes-csi/external-health-monitor/blob/master/pkg/agent/pv_monitor_agent.go#L204 is not nil and it is not found, we should not re-queue checking.

Edit: Why?
When cronjob or job completed, the pod will be exists on the node but it's volumes were unpublished. The code still tries to get stats for the completed pods. however there is no volume which is mounted.

The external-health-monitor's worker thread mechanism (10 by default) is blind to the number of volumes. If there are fewer than 10 volumes then there is a burst of repeated probes of the same volume: if there is only one then there is a rapid fire succession of 10 probes of the same volume every minute.

This is not a high priority issue.

With this PR kubernetes/kubernetes#99284, the health monitoring logic is moved from the external agent to Kubelet. We should remove the agent in 1.21.

Open this issue to track the implementation of volume health monitoring controller.

See https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-csi

/assign @xing-yang

We need to change the image version to v0.1.0.

https://github.com/kubernetes-csi/external-health-monitor/blob/master/deploy/kubernetes/external-health-monitor-agent/daemonset.yaml#L21
https://github.com/kubernetes-csi/external-health-monitor/blob/master/deploy/kubernetes/external-health-monitor-controller/deployment.yaml#L22

when use Block PV, this error would occur：

E0901 10:33:25.153213       1 server.go:110] GRPC error: rpc error: code = NotFound 
desc = Could not get file information
 from /var/lib/kubelet/pods/a38a090d-deb6-457b-be20-dd3d5be90fad/volumes/kubernetes.io~csi/pvc-16d05a9d-d67a-45d5-8ac6-7c06778a9150/mount: 
stat /var/lib/kubelet/pods/a38a090d-deb6-457b-be20-dd3d5be90fad/volumes/kubernetes.io~csi/pvc-16d05a9d-d67a-45d5-8ac6-7c06778a9150/mount:
 no such file or directory

This is caused by CheckNodeVolumeStatus only consider FileSystem PV, but Block PV have different path.

We need to update Readme with information on how to use the health monitoring feature.
Example: https://github.com/kubernetes-csi/external-snapshotter/blob/master/README.md

Bug Report

I have observed same kind of issue in various kubernetes-csi project.
this happens because after the localization there are too much modifications done in the various directories.
I have observed same issue in this page also.

It has one broken link of the contributes cheat sheet which needs to fix.
I will try to look in further csi repo as well and try to fix it as soon as I can

/kind bug
/assign