Comments (12)
VA %s for volume %s has attached status %v but actual state %v
comes from periodic sync of VolumeAttachment objects ("attached status") with result of ListVolumes()
CSI call ("actual state"). It tries to attach volumes that were detached outside of the external-attacher, e.g. by a user. Or, on some clouds, a part of the cloud that's responsible for ListVolumes()
CSI call does not have the information that corresponding ControllerPublishVolume()
has succeeded (their "eventual consistency" pattern). Or there is a bug in the CSI driver or the attacher, of couse.
from external-attacher.
Hi @jsafrane,
thank you for your answer. 2 questions arise there with me.
- If I understand you correctly, the log entry should disappear if I remount the PersistentVolume once. The attachment should now come from the CSI driver and the necessary event/information should be available?
- So to know where the information from the ListVolumes comes from I would have to check where the concrete CSI implementation gets the values from?
from external-attacher.
If I understand you correctly, the log entry should disappear if I remount the PersistentVolume once. The attachment should now come from the CSI driver and the necessary event/information should be available?
Ideally, it should heal by itself - the attacher sees that something that should be attached (has VolumeAttachment) is not attached and attach it. Apparently, it does not happen in your case and I would be curious why.
So to know where the information from the ListVolumes comes from I would have to check where the concrete CSI implementation gets the values from?
Yes.
from external-attacher.
Hello @jsafrane,
The logs seem to occur when a PVC exists that has been mounted once in the past and then the pod is shut down (e.g. for autoscaling). As soon as you clean up the unused PVCs the log entry disappears.
So the log entry makes sense again, because an unused PVC has of course the status "attached: false". I would argue that this is actually unclean and an unused / unmounted PVC/PV should not show up in the logs.
How do you see it?
Thanks again for your support!
from external-attacher.
If you see volume XYZ has attached status true but actual state false
, then VolumeAttachment.status.attached is true
. Where do you see it as attached: false
?
from external-attacher.
Hi @jsafrane ,
that was my original question and I couldn't really answer it until now. All I know now is that this log entry is created for PVCs that are created, were consumed in the past but do not have a pod as a consumer at the moment of the log entry.
We have gone through various OpenStack commands and kubectl commands but where the attached: false
is read from we could not figure out.
from external-attacher.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity,
lifecycle/stale
is applied - After 30d of inactivity since
lifecycle/stale
was applied,lifecycle/rotten
is applied - After 30d of inactivity since
lifecycle/rotten
was applied, the issue is closed
You can:
- Mark this issue as fresh with
/remove-lifecycle stale
- Close this issue with
/close
- Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
from external-attacher.
Hello @Payback159,
we are facing the same problem (Kube+Cinder/OpenStack), and the CSI Cinder logs are full of these messages and attaching/detaching volumes are longer and longer.
Do you find any solution ?
from external-attacher.
Hello @vhurtevent,
apparently the log entries occur when you have PVC's in the cluster that were mounted once in the past and then left and never cleaned up. In our case it was PVC's coming from multiple statefulsets scaled via horizontal pod autoscaler.
We were able to "remove" the logs by cleaning up the leftover PVCs.
However, since in our case they were not the cause of our problem at the time, we stopped further analysis of these log entries.
from external-attacher.
New findings in this issue : kubernetes/cloud-provider-openstack#2295
from external-attacher.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity,
lifecycle/stale
is applied - After 30d of inactivity since
lifecycle/stale
was applied,lifecycle/rotten
is applied - After 30d of inactivity since
lifecycle/rotten
was applied, the issue is closed
You can:
- Mark this issue as fresh with
/remove-lifecycle rotten
- Close this issue with
/close
- Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
from external-attacher.
@vhurtevent has already pointed out in the previous comment that the problem is apparently solved in the cinder-csi-plugin. Only in the sense of a clean conclusion:
We were able to install the new version of the cinder-csi-plugin in our environment and can confirm that the log entries have also disappeared here.
In our larger clusters, this has also meant that pods with RWO volumes that have ended up on another node as a result of rescheduling can now mount their volume also more quickly.
Thank you very much @jsafrane and @vhurtevent !
from external-attacher.
Related Issues (20)
- csi-attacher:v3.3.0 image unavailable HOT 3
- csi-attacher image is having vulneraility HOT 5
- Attachment reconciler is incorrectly using nodeid annotation HOT 5
- Retry on attach error does not respect exponential backoff
- csi-attacher:v3.4.0 image unavailable HOT 1
- change default fstype from "ext4" to empty string HOT 4
- Single timeout for attachment/detachment and reconcile resync operations not always appropriate HOT 16
- Emit events on detach errors HOT 18
- Version 3.5.0 vulnerability with CVE-2022-1996 HOT 11
- Question about reconciling (reconcileVA) based on RPC_LIST_VOLUMES_PUBLISHED_NODES HOT 6
- Broken link of `contributor cheat sheet` needs to fix
- csi attacher report panic in log HOT 5
- Uncertain handling for attach HOT 14
- Readme has incorrect compatibility information for kubernetes version HOT 5
- 7 High Security vulnerability on latest CSI-attacher:v4.3.0 sidecar image HOT 7
- `fault.CnsNotRegisteredFault.summary` while consuming a volume by a pod HOT 2
- Attacher doesn't allow to set volumes limit in the ListVolume request
- VolumeAttachment takes too long to remove HOT 6
- ListVolumes : Panic detected on v4.4.1 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from external-attacher.