VolumeAttachment takes too long to remove about external-attacher HOT 6 CLOSED

This is a known issue between two threads running in the kubelet and explained in detail here kubernetes/kubernetes#115148 (comment). In summary node-shutdown-manager terminates all Pods (including the CSI Driver node plugin) which means that there's no component in the node to safely perform a unmount on workload Pods, therefore, external-attacher waits for 6m before forcefully detaching the volume. While a proper fix would be to add synchronization between the threads so that node-shutdown-manager waits for the CSI Node plugin to finish doing the unmount we found a simple workaround, we added a preStop hook on the EBS CSI Node plugin so that it's not terminated immediately when node-shutdown-manager receives the termination signal:

        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 30; exit 0"]

That way it has enough time to do unmount. Note that it's not guaranteed that termination of this Pod will happen in 30s because I believe node-shutdown-manager reacts to the Spot instance interruption action instead of the two-minute Spot Instance interruption notice.

from external-attacher.

Thanks @mauriciopoppe. Makes sense. Just to confirm is that something I need to put in as I can't see it in the current ebs csi node configuration.

from external-attacher.

Yes, this would be new config in the CSI plugin DaemonSet on the EBS CSI Driver container, for more info please check https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/

from external-attacher.

@mauriciopoppe Right. I might just hold off as I don't want to duplicate off a different configuration and maintain it. I can't see anything in the available chart that would allow me to inject as it stands. Unless you can point me to where I'm missing it.

https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/charts/aws-ebs-csi-driver/templates/node.yaml

Sounds like there might be some disconnect / disagreement with this project and requirements as well.

from external-attacher.

I think you could raise an issue there to see if they can add the lifecycle hook through an additional config property in their helm chart. Feel free to add a pointer in the issue to this issue or kubernetes/kubernetes#115148 explaining why they'd need to support it in their helm chart.

I'm closing this issue because we know it's a known problem tracked in kubernetes/kubernetes#115148

/close

from external-attacher.

@mauriciopoppe: Closing this issue.

from external-attacher.