Comments (8)
from external-attacher.
Looking at the logs, exponential backoff is weird. Controller got synced at these intervals:
I1128 12:32:23.206454 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:23.233230 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:23.247519 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:23.257638 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:23.317422 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:23.420913 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:23.587099 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:23.913460 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:24.561902 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:24.572704 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:25.852872 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:25.860689 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:30.980237 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:31.009714 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:31.030087 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:51.490190 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:32:51.508139 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:35:35.348500 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:35:47.998133 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
I1128 12:37:28.546844 1 controller.go:167] Started VA processing "csi-1fe974024220d00085419a3757b0ccee5ebadeff7672fc3fe0ed469132a41aea"
Recalculated to 10 of milliseconds between calls:
2.7
1.4
1.0
6.0
10.3
16.6
32.6
64.8
1.1
128.0
0.8
512.0
2.9
2.0
2046.0
1.8
16384.0
1265.0
10054.9
It shows some exponential growth from 16 through 32, 64, 128, but it misses 256, 1024 and lot of numbers after 2048.
Unfortunately, informer does not log anything.
As side note, these short 1-2ms intervals in between are caused by timestamp in VolumeAttachment.Status.Error - when backoff timer fires up, the controller re-tries attach and writes new timestamp to Status. This triggers new "VolumeAttachment changed" event and the controller reconciles the object again. It tries to attach the volume again and writes new .Status. At this point, the timestamp is the same as before (it's rounded to seconds), so it does not generate new "VolumeAttachment changed" event. IMO we could do something about it and not to try attaching the volume so quickly, however, it most probably is not root cause of this bug.
from external-attacher.
This "VolumeAttachment changed" is related to the bug. New attach fails and VolumeAttachment is re-enqueued with new interval, overwriting the old one. So VolumeAttachment is enqueued with timeout 2*N instead of N, as we can see aboce. I thought that workqueue is smart enough not to enqueue an item if the item is still in the queue. I'll ignore "VolumeAttachment updated" events if the only difference is VolumeAttachment.Status.Error.
That does not explain why there is such a long gap between 2046.0 (=20s) and 16384 (=168s) though. And that's where these 2.5 minutes are, the previous syncs take ~30 seconds together.
from external-attacher.
I create #104, it does not retry to attach/detach on VolumeAttachment changed events caused by AttachError write.
from external-attacher.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale
from external-attacher.
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten
.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
from external-attacher.
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen
.
Mark the issue as fresh with /remove-lifecycle rotten
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
from external-attacher.
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity.
Reopen the issue with/reopen
.
Mark the issue as fresh with/remove-lifecycle rotten
.Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from external-attacher.
Related Issues (20)
- csi-attacher:v3.3.0 image unavailable HOT 3
- csi-attacher image is having vulneraility HOT 5
- Attachment reconciler is incorrectly using nodeid annotation HOT 5
- Retry on attach error does not respect exponential backoff
- csi-attacher:v3.4.0 image unavailable HOT 1
- change default fstype from "ext4" to empty string HOT 4
- Single timeout for attachment/detachment and reconcile resync operations not always appropriate HOT 14
- Emit events on detach errors HOT 18
- Version 3.5.0 vulnerability with CVE-2022-1996 HOT 11
- Question about reconciling (reconcileVA) based on RPC_LIST_VOLUMES_PUBLISHED_NODES HOT 6
- Broken link of `contributor cheat sheet` needs to fix
- csi attacher report panic in log HOT 5
- Uncertain handling for attach HOT 13
- VolumeAttachment has attached status true but actual state false HOT 12
- Readme has incorrect compatibility information for kubernetes version HOT 5
- 7 High Security vulnerability on latest CSI-attacher:v4.3.0 sidecar image HOT 7
- `fault.CnsNotRegisteredFault.summary` while consuming a volume by a pod HOT 2
- Attacher doesn't allow to set volumes limit in the ListVolume request
- VolumeAttachment takes too long to remove HOT 6
- ListVolumes : Panic detected on v4.4.1 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from external-attacher.