[Bug] Cannot start container after adding SOCI index about soci-snapshotter HOT 5 CLOSED

To clarify a few details:

1. Does the container hang forever or is it just much slower to start?
2. The setup is that you're trying to run Jenkins with the $JENKINS_HOME directory on EFS, right? It works if you've pre-populated EFS, right? Is there any chance you can run it without EFS? Does that still cause the container to hang forever?

Do you happen to have a more minimal reproduction?

from soci-snapshotter.

Actually, I was able to reproduce this using your Dockerfile, entrypoint script, and the plugin list from your repo with a local snapshotter.

I modified the Dockerfile to install strace and ran it with:

sudo nerdctl run --rm -it --net host --snapshotter soci --entrypoint strace --cap-add SYS_PTRACE $IMAGE -f -o /tmp/trace /usr/bin/entrypoint

From the trace, I can see it getting stuck in an lseek loop:

29    lseek(3, 1774757, SEEK_SET)       = 1774757
29    lseek(3, 1774757, SEEK_DATA)      = 1774757
29    lseek(3, 1774757, SEEK_HOLE)      = 1774757
29    lseek(3, 1774757, SEEK_SET)       = 1774757
29    lseek(3, 1774757, SEEK_DATA)      = 1774757
29    lseek(3, 1774757, SEEK_HOLE)      = 1774757
29    lseek(3, 1774757, SEEK_SET)       = 1774757
29    lseek(3, 1774757, SEEK_DATA)      = 1774757
29    lseek(3, 1774757, SEEK_HOLE)      = 1774757
29    lseek(3, 1774757, SEEK_SET)       = 1774757

This is a known issue in one of our dependencies: hanwen/go-fuse#487. We have a PR out to fix that bug, but this is blocked until that gets merged.

from soci-snapshotter.

Looking a bit deeper at the trace, SOCI was treating the file as a sparse (hence the SEEK_DATA/SEEK_HOLE flags), but overlayFS wasn't. Something seems off about that, so I ran stat on the file both with an without SOCI:

OverlayFS:

$ stat /usr/share/jenkins/ref/plugins/apache-httpcomponents-client-4-api.jpi
  File: /usr/share/jenkins/ref/plugins/apache-httpcomponents-client-4-api.jpi
  Size: 1774757   	Blocks: 3472       IO Block: 4096   regular file

SOCI:

$ stat /usr/share/jenkins/ref/plugins/apache-httpcomponents-client-4-api.jpi
  File: /usr/share/jenkins/ref/plugins/apache-httpcomponents-client-4-api.jpi
  Size: 1774757     Blocks: 434        IO Block: 4096   regular file

OverlayFS is correctly reporting the number of 512-byte blocks, but SOCI is reporting the number of IO Block-byte blocks.

https://github.com/awslabs/soci-snapshotter/blob/main/fs/layer/node.go#L817

That's a bug in SOCI that we can fix. Even without that fix, there shouldn't be any functional difference between SOCI and OverlayFS (ignoring the current upstream bug), but this is probably causing a bunch of unnecessary lstats looking for data in sequential files.

from soci-snapshotter.

Hi Kern,

Thanks for considering this issue. In case it still helps, here is the answer to your question:

1. The container hangs forever and is killed by ECS.
2. The mount point in the container is JENKINS_HOME=/var/jenkins_home.

It works if you've pre-populated EFS, right?

Yes, it works. Basically, if I run the service with an image on DockerHub and switch to ECR with SOCI, it works. It doesn't work If I first run with ECR + SOCI and empty EFS.

Is there any chance you can run it without EFS? Does that still cause the container to hang forever?

I can try it if still needed.

from soci-snapshotter.

Hi @Kern--

Sorry to comment on a closed issue (I can open a new one if needed) but I'm still having the same issue with the updated version. I built this commit locally using this Dockerfile (the image is here on Dockerhub), pushed the index and started the service: the container is not able to start.

I also tested the service without the EFS same behavior.

from soci-snapshotter.