Figure out how to remove platform specific files from docker images about containerd-wasm-shims HOT 8 OPEN

Discussed a bit at Runwasi office hours and offline with a few folks, A couple options we came up with (there might be more):

• Using a tool like tar-builder to build images in OCI image format
  • This works for Linux mostly. The ssl specific files wouldn't work well or would be in the image whether it was needed or not.
  • There was also a concern raise in containerd/runwasi#106 (comment) around the use of OCI image formats.
  • would require a windows specific snapshotter that doesn't have the file system requirements that it
• Using a tool to build OCI artifacts that contain WASM Modules.
  • There is a proposal for a storing wasm modules in a registry bytecodealliance/registry#87
    • The challenge with the proposal is that containerd doesn't necessarily know about OCI artifacts only OCI image manifests.
  • One solution to explore is using Stream Processors to translate the unknown media types to formats that Containerd does know: https://github.com/containerd/containerd/blob/main/docs/stream_processors.md
    • This possibly has a couple benefits:
      • work directly with modules instead of image formats (possibly address the concern around file systems mentioned above)
      • no need to write/adopt new snapshotters.
      • could potentially handle OS specific files via annotations
      • don't need to modify core contianerd, which means we can iterate faster than needing a containerd release atleast initially.

Prototyping the second solution using stream processors would be a good way to determine if it is feasible.

from containerd-wasm-shims.

@cpuguy83 @Mossaka Incase I am missing something

from containerd-wasm-shims.

So today we should be able to use stream processors, however it was suggested is to handle this at the "unpacker" level (via the transfer service) and essentially choose an unpacker passed on the image platform. So we could have a wasi/wasm unpacker and have different implementations of the unpacker when running on linux vs windows.

This (transfer service) is not wired up to CRI today, but it is being discussed.

from containerd-wasm-shims.

Is there any docs/issue/pr on the transfer service or unpacker that I could look at? What would be the differences and pros/cons of using stream processor vs transfer service?

from containerd-wasm-shims.

I think the main thing is you can select an unpacker based on image platform and not have to do anything special with new types.

Using a stream processor requires a new type, a backchannel to containerd to create the content in, and installing more binaries on a node.

from containerd-wasm-shims.

found containerd/containerd#7592 and containerd/containerd#8227

from containerd-wasm-shims.

The stream processor approach didn't work out since (see prototype)

• The stream processors don't get called on the manifest lists or manifests. They are used during "Apply" and so only called for each layers:
  https://github.com/containerd/containerd/blob/6a5b4c9c242a0278033b20f7f5eca4fdbbea231a/diff/windows/windows.go#L118
  https://github.com/containerd/containerd/blob/06e085c8b50a4953c6a9ea636b459ce3a18964e4/diff/apply/apply.go#L77

• Modifing the content streams creates a different Diff hash which fails

from containerd-wasm-shims.

I have since done a few prototypes and have a proposal in review at https://docs.google.com/document/d/11shgC3l6gplBjWF1VJCWvN_9do51otscAm0hBDGSSAc

from containerd-wasm-shims.