Comments (4)
I see in the delete log:
rbdplugin: I1008 08:05:16.470638 1 controllerserver.go:163] deleting volume pvc-59a6fe14cad011e8
...
provisioner: I1008 08:05:26.470676 1 controller.go:107] GRPC error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
provisioner: E1008 08:05:26.470695 1 controller.go:1138] Deletion of volume "pvc-59a6fe14cad011e8" failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
provisioner: E1008 08:05:26.470797 1 goroutinemap.go:150] Operation for "delete-pvc-59a6fe14cad011e8[6b1db6e5-cad0-11e8-a12b-deadbeef0100]" failed. No retries permitted until 2018-10-08 08:05:26.970758008 +0000 UTC m=+1036.417161060 (durationBeforeRetry 500ms). Error: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
rbdplugin: I1008 08:05:32.984535 1 rbd_util.go:341] rbd: Deleting file for Volume: csi-rbd-622a252c-cad0-11e8-9112-deadbeef0101 at: /var/lib/kubelet/plugins/csi-rbdplugin/controller resulting path: /var/lib/kubelet/plugins/csi-rbdplugin/controller/csi-rbd-622a252c-cad0-11e8-9112-deadbeef0101.json
it seems the provisioner timed out before rbd delete finished.
for rbd driver, deletion started without validating if the volume ID was still there. For sanity, we need to check local cache first around here https://github.com/ceph/ceph-csi/blob/master/pkg/rbd/controllerserver.go#L163
@pohly can you help with the sanity check? thanks
from ceph-csi.
@rootfs: I agree with your analysis. To make the call idempotent, it would have to return success when the volume info cannot be loaded.
But I am still worried that this local .json file is needed at all. Do you agree that this can only work across cluster and node restarts if the provisioner always runs on the same host? Any plans to address this?
from ceph-csi.
@pohly we already have an issue for this #66
from ceph-csi.
@pohly
agree, local json is persistent and won't be crash resistent.
as discussed with @gman0 , for kubernetes use case, we should consider a more robust cluster level storage for meta info. A configmap is a way to go for that.
@pohly if you are volunteering for #66, that'll be a great contribution, thanks.
from ceph-csi.
Related Issues (20)
- Can we share VolumeLocks for NodeServer/ControllerServer? HOT 3
- Not able to start CI job because of no nodes HOT 1
- RBD-Images are not shown in the Dashbord: Failed to execute RBD [errno 19] error generating diff from snapshot None HOT 2
- Concurrency issue when pvc is mounted to multiple pods HOT 23
- CentOS Stream 8 is EOL HOT 4
- Add support for ModifyVolume HOT 5
- allow podSecurityContexts to be set in `nodeplugin` and `provisioner` of chart `ceph-csi-cephfs` HOT 2
- Add http health endpoint for ceph-csi-cephfs and ceph-csi-rbd HOT 4
- cephfs-csi Pod has always been in containercreating HOT 3
- CephFS keyring requires nonsensicaly enormous and insecure privileges to work HOT 6
- Update modprobe in csi-rbdplugin to support zstd compressed rbd and nbd kernel module HOT 7
- Unable to create CephFS subvolume dynamically (`no available topology found`) HOT 2
- cephFS: Remove the 400 snapshot limitation from the doc HOT 1
- Allow to change mounter option from an existing PV HOT 8
- cephfs csi error,mds mds status Start request repeated too quickly. Failed with result 'signal'. HOT 3
- New csiplugin-configmap.yaml setting to override PVs's volumeAttributes.mounter HOT 2
- 'luksOpen'` encrypted volumes HOT 1
- rbd remap on network failure HOT 7
- Remove podSecurityPolicy from the helm documentation
- Ensure ceph-lock is always released when staging encrypted cephfs volume HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ceph-csi.