issue in COPY from one stage to next about imagebuilder HOT 8 CLOSED

Figuring out how to reproduce this was easier with the full context, i.e., knowing that the build was of the "v0.14-rh" branch of https://github.com/ViaQ/vector.

When copying from one location to another, imagebuilder generally asks the docker daemon to archive a parent directory that includes what it's attempting to copy. It then processes the stream it gets back from the docker daemon, skipping past entries which it determines shouldn't be copied to the destination, and renaming entries which it needs to pass along so that they'll be written to the right place when the archive stream is extracted at the destination. The resulting stream is passed back to the daemon for extraction in the destination location.

The error is that imagebuilder skips copying "/usr/bin/vector" because its entry is a hard link entry. A hard link entry itself contains no file contents, but rather contains the name of the file (which occurred earlier in the source archive stream) to which it should be hard linked in its header. At the point where imagebuilder's copying logic first sees the hard link entry that will become "/usr/bin/vector", the entry for the file to which it should be linked (name in the source container: "release/deps/vector-397a93b5c6350cf5") has been skipped, so hard links to it are also getting skipped, which is a bug.

For reference, prior to #158, the reproducer would have failed with a "link /release/deps/vector-397a93b5c6350cf5 /usr/bin/vector: no such file or directory" error when the daemon, as part of attempting to extract the archive stream, attempted to create the target file as a hard link to a file that didn't exist. That PR fixed the cases where the link target was copied and renamed, but left for another time the cases where the link target had been skipped entirely.

This is why copying the file to any other location in the earlier stage, and then copying it from there for the latter stage, works around the bug: "cp" doesn't create a hard link unless it's passed its "-l" flag, so the file that gets copied is no longer just a hard link to a file that doesn't get copied, and the bug isn't triggered.

from imagebuilder.

The number of files being copied isn't expected to make any difference beyond how long the command takes to complete, and I'd expect it to start from inside of /src/target/release at least. What error messages are you seeing when it fails?

from imagebuilder.

there were no error messages

from imagebuilder.

What do we get if it's run with a higher-than-default level of logging output, say -v 6?

from imagebuilder.

i may still have OSBS logs where it failed, shall i link it here?

from imagebuilder.

Yes, please. More information at this point can't hurt.

from imagebuilder.

i did the same build on my laptop, and with -v 9 , i get lots of logs. the relavant part is that the file in the COPY statement gets skipped

I0924 02:06:12.309614  613974 archive.go:45] Transform release/vector -> /usr/bin/vector: data=false ok=false skip=true err=<nil>

from imagebuilder.

i added you to the internal chat thread also

from imagebuilder.