deislabs / containerd-wasm-shims Goto Github PK

Problem Description

Currently, the spin-shim imposes rigid preferences on the instantiation of the spin runtime. For instance, it starts spin on port 80 without any provision for configuration.

There exist several valuable spin up flags that could be applicable, even in Kubernetes environments. Some of these flags include:

Here are some flag options that are currently unavailable:

• --direct-mounts

  • Description: When dealing with local applications featuring directory mounts and no excluded files, this option allows for direct mounting instead of utilizing a temporary directory.
  • Current State: The direct-mounts feature is not active, which limits the functionality of certain sidecar scenarios (e.g., sharing ephemeral volumes). It should be made optional.

• --runtime-config-file <RUNTIME_CONFIG_FILE>

  • Description: This flag specifies a configuration file for config providers and Wasmtime config.
  • Current State: The absence of this option prevents the utilization of dynamic runtime configuration features of spin. Specifically, it hinders the use of Vault secrets or Redis as a substitute for the Spin-Internal KV-store.

• Additional intriguing spin up flags may include <FOLLOW_ID> or <KEY_VALUES>.

Feature Request

The request is to empower users to leverage spin's configuration options, ideally through the use of environment variables.

One particularly interesting option is <ENV>, as the shim would need to differentiate between environment variables intended for the runtime and those meant for the spin application.

• -e, --env <ENV>
  • Description: This flag allows for passing an environment variable (in the form of key=value) to all components of the application.

We therefore possibly would need to namespace env vars for runtime configuration.

I'm eager to hear about your opinion!

#121 adds the default executor to the spin shim to allow it run with Linux containers side-by-side in a pod. Now we need to add a demo for Spin + Dapr integration.

@0xE282B0 are you interested in taking this issue?

We will need to improve our integration test coverage on a few other popular use cases of slight and spin shims.

• Test a workload that uses more than one spin components, better if they are written in different langauges
• Test a workload that uses spin's outbound redis, pg, http capability
• Test the slight chat-app workload

Running through the README and getting the following error from the deployed pod:

Failed to create pod sandbox: rpc error: code = Unknown desc = RuntimeHandler "spin" not supported

Any tips to get around this would be appreciated.

Reproduce

Run the following commands on cgroup v1 Ubuntu 20.04.6 LTS:

sudo k3d cluster create wasm-cluster --image ghcr.io/deislabs/containerd-wasm-shims/examples/k3d:v0.9.2 -p "8081:80@loadbalancer" --agents 2
kubectl apply -f https://github.com/deislabs/containerd-wasm-shims/raw/main/deployments/workloads/runtime.yaml
kubectl apply -f https://github.com/deislabs/containerd-wasm-shims/raw/main/deployments/workloads/workload.yaml

Kubernetes Logs

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m25s                  default-scheduler  Successfully assigned default/wasm-spin-8649cf7566-jssf2 to k3d-wasm-cluster-agent-1
  Normal   Pulling    5m24s                  kubelet            Pulling image "ghcr.io/deislabs/containerd-wasm-shims/examples/spin-rust-hello:v0.9.2"
  Normal   Pulled     5m23s                  kubelet            Successfully pulled image "ghcr.io/deislabs/containerd-wasm-shims/examples/spin-rust-hello:v0.9.2" in 1.553587008s (1.553596909s including waiting)
  Normal   Pulled     3m41s (x4 over 5m20s)  kubelet            Container image "ghcr.io/deislabs/containerd-wasm-shims/examples/spin-rust-hello:v0.9.2" already present on machine
  Normal   Created    3m41s (x5 over 5m23s)  kubelet            Created container spin-hello
  Warning  Failed     3m41s (x5 over 5m21s)  kubelet            Error: failed to create containerd task: failed to create shim task: Others("failed to receive. \"waiting for init ready\". BrokenChannel"): unknown
  Warning  BackOff    17s (x25 over 5m19s)   kubelet            Back-off restarting failed container spin-hello in pod wasm-spin-8649cf7566-jssf2_default(567f5b2a-35ae-46ed-87c4-76b74b371238)

Containerd logs

time="2023-11-01T00:30:17.44428603Z" level=info msg="found manifest with WASM OCI image format."
time="2023-11-01T00:30:17.445159344Z" level=info msg="cgroup manager V1 will be used"
time="2023-11-01T00:30:17.45954827Z" level=error msg="failed to canonicalize "/sys/fs/cgroup/systemd/docker/8a32364d3653534991bb0b9dee5564982a08dae797ad10900d458942a9038b5e": No such file or directory (os error 2)"
time="2023-11-01T00:30:17.460004177Z" level=error msg="failed to mount Mount { destination: "/sys/fs/cgroup/systemd", typ: Some("bind"), source: Some("/sys/fs/cgroup/systemd/docker/8a32364d3653534991bb0b9dee5564982a08dae797ad10900d458942a9038b5e"), options: Some(["rw", "rbind"]) }: io error"
time="2023-11-01T00:30:17.460963692Z" level=error msg="failed to mount systemd cgroup hierarchy: io error"
time="2023-11-01T00:30:17.461164595Z" level=error msg="failed to mount cgroup v2: io error"
time="2023-11-01T00:30:17.461311097Z" level=error msg="failed to prepare rootfs err=Mount(Io(Os { code: 2, kind: NotFound, message: "No such file or directory" }))"
time="2023-11-01T00:30:17.461929807Z" level=error msg="failed to initialize container process: failed to prepare rootfs"
time="2023-11-01T00:30:17.462512216Z" level=error msg="failed to wait for init ready: failed to receive. "waiting for init ready". BrokenChannel"
time="2023-11-01T00:30:17.462560817Z" level=error msg="failed to run container process err=Channel(ReceiveError { msg: "waiting for init ready", source: BrokenChannel })"

This might be an issue upstream in runwasi.

The current integration tests for the spin shim is lacking. It only performs a GET request on the spin-hello-world image. We will want to add more integration tests for the spin shim.

• Spin outbound redis test #181
• Spin inbound redis (a.k.a redis trigger) test
• Spin OCI test
• Spin + Dapr with pod agonosity test

          **nit:** group `use` statements, or even better, use a `fmt` rule ;-)

Originally posted by @jprendes in #148 (comment)

The error below was failing in the build for PR #67.

Checking containerd-shim-slight-v1 v0.1.0 (/home/runner/work/containerd-wasm-shims/containerd-wasm-shims/containerd-shim-slight-v1)
[943](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:944)
error: variables can be used directly in the `format!` string
[944](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:945)
  --> src/main.rs:37:38
[945](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:946)
   |
[946](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:947)
37 |         .map_err(|err| Error::Others(format!("could not canonicalize rootfs: {}", err)))?;
[947](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:948)
   |                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[948](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:949)
   |
[949](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:950)
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
[950](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:951)
   = note: `-D clippy::uninlined-format-args` implied by `-D warnings`
[951](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:952)
help: change this to
[952](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:953)
   |
[953](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:954)
37 -         .map_err(|err| Error::Others(format!("could not canonicalize rootfs: {}", err)))?;
[954](https://github.com/deislabs/containerd-wasm-shims/actions/runs/4073348292/jobs/7028521089#step:5:955)
37 +         .map_err(|err| Error::Others(format!("could not canonicalize rootfs: {err}")))?;

The scenario is this. Right now, partly because we were trying to exercise the containerd shim and partly because there wasn't an oras-rs crate we could use from source our app hosts -- whether spin, slight, or future hosts -- can't give us a .toml because either we alter how the runtimes acquire their configs in order to integrate with k8s -- something that doesn't make a lot of sense. Even if we use volume mounts for the config files, you don't want the operational experience to ship separately. Instead, you want one reference to an immutable config of a host and a module. You want the joy of using the image value of the pod spec to point at "the entire thing that runs correctly" -- which includes more than one artifact.

The way we do that without OCI Artifacts is by building a scratch container and dropping both the module and the config for the runtime into it. That's fine, but we need to build a container in order to not use the container. :-) This is entirely a point-in-time thing, as an older friend used to say. No immediate hurries.

BUT: we do need to have a plan to a) support both this method and the OCI Artifact method (oras) and b) understand what that would mean for the yaml experience. We EITHER need to re-use the image key and take both and just do the right thing (check with oras and if not, use docker) OR we do a hard roll to oras and commit, re-using the image key for artifacts only. Alternatively, we could add an artifact key and have the runtime punt unless it was either/or but not both. (Problem with that scenario is that no one would recognize the artifact key as schema valid. :-( )

In any case, there is prior art we might be able to use. https://docs.rs/oras/latest/oras/struct.Client.html never got finished, but Sajay says he'd love to fund that. BUT... also Jacob LeGrone in the CNAB space did https://crates.io/crates/oras so I'll reach out to see where that codebase is. Seems to be private. But if he is willing, we can bring that up to date pretty rapidly.

If the rust spin image is compiled with symbol stripping the size can be decreased from 1.9MB to 132KB, using the command:

RUSTFLAGS='-C link-arg=-s' cargo build --target wasm32-wasi --release

this can also be done in the Cargo.toml file so that compiling in release mode strips symbols by default:

[profile.release]
strip = true

I tested to make sure it would work with the spin framework using spin up and it was functional.

I have the spin shim installed on my worker nodes:

And the spin containerd plugin configured on my worker nodes:

The runtime class configured on my cluster:

This is the deployment config for the hello world app I was trying to get working:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"wasm-spin","namespace":"default"},"spec":{"replicas":3,"selector":{"matchLabels":{"app":"wasm-spin"}},"template":{"metadata":{"labels":{"app":"wasm-spin"}},"spec":{"containers":[{"image":"ghcr.io/deislabs/containerd-wasm-shims/examples/spin-rust-hello:latest","name":"testwasm"}],"runtimeClassName":"wasmtime-spin-v1"}}}}
  creationTimestamp: "2023-10-11T01:42:09Z"
  generation: 1
  name: wasm-spin
  namespace: default
  resourceVersion: "75957"
  uid: 3104ba94-3ceb-496c-b7b3-23e6472500f3
spec:
  progressDeadlineSeconds: 600
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: wasm-spin
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: wasm-spin
    spec:
      containers:
      - image: ghcr.io/deislabs/containerd-wasm-shims/examples/spin-rust-hello:latest
        imagePullPolicy: Always
        name: testwasm
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      runtimeClassName: wasmtime-spin-v1
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  conditions:
  - lastTransitionTime: "2023-10-11T01:42:09Z"
    lastUpdateTime: "2023-10-11T01:42:09Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2023-10-11T01:42:09Z"
    lastUpdateTime: "2023-10-11T01:56:03Z"
    message: ReplicaSet "wasm-spin-58db6df759" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  observedGeneration: 1
  replicas: 3
  unavailableReplicas: 3
  updatedReplicas: 3

But this is what I get when trying to deploy any of the pods:

Did I miss something? I tried with the fermyon helm chart and binaries from here, same result. Any help is appreciated. Thanks!

I tried to deploy a slight wasm application into k8s cluster with kv redis capaiblity, however could not find a way to make REDIS_ADDRESS configurable, it is defined in slightfile.yaml, by the time shim starts, it needs this file in place, so looks like no way to dynamically configure my redis address.

Hello ,

any update to this quickstart?
https://github.com/deislabs/containerd-wasm-shims/blob/main/containerd-shim-spin-v1/quickstart.md

When I run: spin new qs-wasm-spin
I got:
Template qs-wasm-spin not found

When running make up from folder /deployments/k3d

...
docker buildx build --platform=wasi/wasm --load -t wasmtest_spin:latest ../../images/spin
[+] Building 0.5s (18/18) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                      0.0s
 => => transferring dockerfile: 726B                                                                                                                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/golang:1.19.5-bullseye                                                                                                                                                                                 0.4s
 => [internal] load metadata for docker.io/library/rust:1.67                                                                                                                                                                                              0.4s
 => [build-go 1/5] FROM docker.io/library/golang:1.19.5-bullseye@sha256:78b1726316ebfdec022fe7628f07547cafcd2d8a3934288187b87173f75925bc                                                                                                                  0.0s
 => [build 1/4] FROM docker.io/library/rust:1.67@sha256:aed98ad355d0e8303d057cc4894a7d331010980eb63c7dc46176de82670881dd                                                                                                                                  0.0s
 => [internal] load build context                                                                                                                                                                                                                         0.0s
 => => transferring context: 307B                                                                                                                                                                                                                         0.0s
 => CACHED [build 2/4] WORKDIR /opt/build                                                                                                                                                                                                                 0.0s
 => CACHED [build 3/4] COPY . .                                                                                                                                                                                                                           0.0s
 => CACHED [build 4/4] RUN rustup target add wasm32-wasi && cargo build --target wasm32-wasi --release                                                                                                                                                    0.0s
 => CACHED [stage-2 1/3] COPY --from=build /opt/build/target/wasm32-wasi/release/spin_rust_hello.wasm .                                                                                                                                                   0.0s
 => CACHED [stage-2 2/3] COPY --from=build /opt/build/spin.toml .                                                                                                                                                                                         0.0s
 => CACHED [build-go 2/5] WORKDIR /opt/build                                                                                                                                                                                                              0.0s
 => CACHED [build-go 3/5] COPY . .                                                                                                                                                                                                                        0.0s
 => CACHED [build-go 4/5] RUN curl -LO https://github.com/tinygo-org/tinygo/releases/download/v0.25.0/tinygo_0.25.0_amd64.deb && dpkg -i tinygo_0.25.0_amd64.deb                                                                                          0.0s
 => CACHED [build-go 5/5] RUN cd go-hello && tinygo build -wasm-abi=generic -target=wasi -gc=leaking -o spin_go_hello.wasm main.go                                                                                                                        0.0s
 => CACHED [stage-2 3/3] COPY --from=build-go /opt/build/go-hello/spin_go_hello.wasm .                                                                                                                                                                    0.0s
 => ERROR exporting to image                                                                                                                                                                                                                              0.0s
 => => exporting layers                                                                                                                                                                                                                                   0.0s
 => => writing image sha256:67311f8ddbc767ceb889f36a141e8333b1ccd0f40aa4144716967e6b2db2e07f                                                                                                                                                              0.0s
------
 > exporting to image:
------
ERROR: failed to solve: operating system is not supported
make: *** [Makefile:29: build-workload-images] Error 1

OS

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian

Docker

$ docker version
Client: Docker Engine - Community
 Version:           23.0.3
 API version:       1.42
 Go version:        go1.19.7
 Git commit:        3e7cbfd
 Built:             Tue Apr  4 22:03:23 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.3
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.7
  Git commit:       59118bf
  Built:            Tue Apr  4 22:03:23 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.20
  GitCommit:        2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc:
  Version:          1.1.5
  GitCommit:        v1.1.5-0-gf19387a
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker buildx version
github.com/docker/buildx v0.10.4 c513d34

Add an integration test harness that will do the following:

• Create a K3d cluster with each shim installed and containerd config updated for the shims
• Run a series of tests to verify
  • Pods start and reach a healthy state
  • Pods serve http traffic from a test application
• Destroy the cluster

warning: /home/kostya/containerd-wasm-shims/containerd-shim-slight-v1/Cargo.toml: dependency (openssl-sys) specified without providing a local path, Git repository, version, or workspace dependency to use. This will be considered an error in future versions
error: failed to create directory /target/release

Caused by:
Permission denied (os error 13)
make: *** [Makefile:78: build-slight-cross-x86_64-unknown-linux-musl] Error 101

Any help?

I have a Spin app running in k8s that works fine until I make an outgoing request to another resource. This using the latest 0.9.2.

When do I get a generic error from the spin's stdout:

Handler returned an error: Uncaught SyntaxError: unexpected token: 'Uncaught'                                                                                                                                        
    at <input>:1                                                                                                                                                                                                     
    at parse (native)                                                                                                                                                                                                
    at <anonymous> (script.js:397)                                                                                                                                                                                   
    at call (native)                                                                                                                                                                                                 
    at step (script.js:306)                                                                                                                                                                                          
    at <anonymous> (script.js)                                                                                                                                                                                       
    at fulfilled (script.js)   

But looking at the containerd logs, I see this more specific error:
Outbound HTTP connection error: URL https://dummyjson.com/products/1, error detail reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("dummyjson.com")), port: None, path: "/products/1", query: None, fragment: None }, source: hyper::Error(Connect, Ssl(Error { code: ErrorCode(5), cause: Some(Ssl(ErrorStack([Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 2147483650, library: "system library", function: "file_open", file: "providers/implementations/storemgmt/file_store.c", line: 267, data: "calling stat(/usr/local/ssl/certs)" }, Error { code: 167772294, library: "SSL routines", function: "tls_post_process_server_certificate", reason: "certificate verify failed", file: "ssl/statem/statem_clnt.c", line: 1889 }]))) }, X509VerifyResult { code: 20, error: "unable to get local issuer certificate" })) }

Any quick work arounds that I can look at to resolve?

I have been following the spin quickstart tutorial and executing the steps exactly as described. I'm getting this error when trying to run the qs-wasm-spin container.

Error: failed to start containerd task "testwasm": Cannot read manifest file from "/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/testwasm/rootfs/spin.toml": unknown                            

The only difference with the example is that I've updated the Dockerfile to use rust 1.68.1 instead of 1.59 as noticed some issues with restoring the spin-sdk package on 1.59.

The spin.toml file looks to be copied to the container just fine.

Strangely the example image works just fine:

ghcr.io/deislabs/containerd-wasm-shims/examples/spin-rust-hello:v0.5.1

This image does also seem to contain the spin_go_hello.wasm file though, but I'm assuming that doesn't matter.

Is the quickstart demo still up to date?
Thanks for any help on this!

Problem description:

With the 0.9.0 release, a K3d image has been created with two supported platforms, linux/amd64 and linux/arm64, both images contain the same shim binaries (x86-64). Using the linux/arm64 leads to this error when creating a container with those shims:

Failed to create pod sandbox: rpc error: code = 
  Unknown desc = failed  to create containerd task: 
    failed to start shim: start failed: : 
      fork/exec /bin/containerd-shim-spin-v1: exec format error: unknown 

Additional info:

root@kwasm-debug-6npz6:/# uname -m
aarch64

root@kwasm-debug-6npz6:/# file /mnt/node-root/bin/containerd-shim-spin-v1 
/mnt/node-root/bin/containerd-shim-spin-v1: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, with debug_info, not stripped

Just have a question on multiple runtimes on POD:

Currently, we have used some kind of sidecar (dapr or istio) tool on the Kubernetes cluster.

The sample file can be found below

apiVersion: apps/v1
kind: Deployment
metadata:
  name: product-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: product-api
  template:
    metadata:
      labels:
        app: product-api
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "product-api"
        dapr.io/app-port: "3000"
        dapr.io/enable-api-logging: "true"
    spec:
      runtimeClassName: wasmtime-spin-v1
      containers:
        - name: product-api
          image: ghcr.io/thangchung/dapr-labs/product-api-spin:1.0.0
          command: ["/"]
          ports:
          - containerPort: 3000
          env:
          - name: RUST_BACKTRACE
            value: "1"
          resources: # limit the resources to 128Mi of memory and 100m of CPU
            limits:
              cpu: 100m
              memory: 128Mi
            requests:
              cpu: 100m
              memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: product-api
spec:
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 5001
      targetPort: 3000
  selector:
    app: product-api

When we run it, it always throws the exception that the daprd didn't have spin.toml file, and we found out that because runtimeClassName: wasmtime-spin will scan all containers (include any sidecar in this pod). I know this is by the design of Kubernetes because I searched around and maybe we cannot run multiple runtimes on 1 pod.

Could anyone shed light for me on it? Thank you very much.

I tried to use the make command to build the program and encountered an error：

When I run make run_spin in the top directory, everything builds correctly. However, when it gets to the install step, I get this error:

   Compiling containerd-shim-slight-v1 v0.1.0 (/home/luke/Projects/WASM/containerd-wasm-shims/containerd-shim-slight-v1)
    Finished release [optimized] target(s) in 10m 01s
sudo install target/release/containerd-shim-*-v1 /usr/local/bin
[sudo] password for luke: 
install: cannot stat 'target/release/containerd-shim-*-v1': No such file or directory
make: *** [Makefile:37: install] Error 1

That path does not existing in the top directory, but it is in containerd-shim-spin-v1 and containerd-shim-slight-v1.

It looks like install should add in the dir for the different shims:

#29 introduces a workload example that uses both spin and slight. The documentation for the example will break due to this change and will need to be updated for the next release.

dinesh@INCHEL-JVQQ2Z2:/mnt/c/Users/c-dinkar/qs-wasm-spin$ docker buildx build --platform=wasi/wasm -t localhost:12345/qs-wasm-spin .
[+] Building 114.8s (8/10)
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 330B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/rust:1.59 11.6s
=> [build 1/4] FROM docker.io/library/rust:1.59@sha256:7c0ea3ee40fc4f12a43de1b9967bc5df1337bf469253c322a98952079493448b 0.0s
=> => resolve docker.io/library/rust:1.59@sha256:7c0ea3ee40fc4f12a43de1b9967bc5df1337bf469253c322a98952079493448b 0.0s
=> [internal] load build context 10.9s
=> => transferring context: 56.29kB 10.9s
=> CACHED [build 2/4] WORKDIR /opt/build 0.0s
=> CACHED [build 3/4] COPY . . 0.0s
=> ERROR [build 4/4] RUN rustup target add wasm32-wasi && cargo build --target wasm32-wasi --release 92.2s

[build 4/4] RUN rustup target add wasm32-wasi && cargo build --target wasm32-wasi --release:
#0 0.471 info: downloading component 'rust-std' for 'wasm32-wasi'
#0 2.629 info: installing component 'rust-std' for 'wasm32-wasi'
#0 4.720 Updating crates.io index
#0 83.69 Updating git repository https://github.com/fermyon/spin
#0 88.70 Updating git repository https://github.com/bytecodealliance/wit-bindgen
#0 92.01 error: no matching package named spin-sdk found
#0 92.01 location searched: https://github.com/fermyon/spin?tag=v1.1.0#28655d57
#0 92.01 required by package qs-wasm-spin v0.1.0 (/opt/build)

Dockerfile:4

2 | WORKDIR /opt/build
3 | COPY . .
4 | >>> RUN rustup target add wasm32-wasi && cargo build --target wasm32-wasi --release
5 |
6 | FROM scratch

ERROR: failed to solve: process "/bin/sh -c rustup target add wasm32-wasi && cargo build --target wasm32-wasi --release" did not complete successfully: exit code: 101
dinesh@INCHEL-JVQQ2Z2:/mnt/c/Users/c-dinkar/qs-wasm-spin$ cargo search spin-sdk
spin-sdk = "0.0.0" # Placeholder for Spin SDK
dinesh@INCHEL-JVQQ2Z2:/mnt/c/Users/c-dinkar/qs-wasm-spin$

Hi,
I noticed that with the release (at least 0.9.3 and 0.10.0) there are again no logs from within the module.
When I check out the 0.10.0 tag and do a make install the logs are working.

Expected output with the ghcr.io/deislabs/containerd-wasm-shims/example s/spin-rust-hello:latest image after calling localhost:80/hello:

Serving http://0.0.0.0:80
Available Routes:
  hello: http://0.0.0.0:80/hello
  go-hello: http://0.0.0.0:80/go-hello
Hello, world! You should see me in pod logs

Could it be something related to the binary stripping during release process?

Building from main from commit 5ca163a fails with the following on Ubuntu 22.04:

running 3 tests

curl http://localhost:8082/wws/hello
404 page not found

curl http://localhost:8082/spin/hello
curl http://localhost:8082/slight/hello
404 page not found

test wws_test ... ok
test spin_test ... ok
404 page not found

curl -X PUT http://localhost:8082/slight/set -d
404 page not found

thread 'slight_test' panicked at 'assertion failed: (left == right)
left: "404 page not found\n",
right: "wyyauejuBVjpz34MYmSnjpMNmbczsJ"', tests/integration_test.rs:48:5
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
test slight_test ... FAILED

failures:

failures:
slight_test

test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.26s

error: test failed, to rerun pass --test integration_test
make: *** [Makefile:23: integration-tests] Error 101

Hey,

Pod Status is CrashLoopBackOff when files variable is added under component to access static files.
Spin FileServer has same problem.

Hello 👋

I recently open a PR to add a new Wasm Workers Server shim. When adding this new feature, I notice the CI time is growing consistently with every new shim due compilations and other required steps.

Comparing the builds in the different repos, the wws adds around +10 minutes of compilation times. And this may grow with new shims in the future.

I don't have any proposal as I didn't check the CI pipelines in depth. For now, I created this issue for tracking purposes :)

Hi,
While testing the shims on the main branch I found that I can't see print from within the HTTP handler anymore.

Simple example:

use anyhow::Result;
use spin_sdk::{
    http::{Request, Response},
    http_component,
};

/// A simple Spin HTTP component.
#[http_component]
fn handle_hello_http(req: Request) -> Result<Response> {
    // #### This does not appear in log ####
    println!("{:?}", req.headers());
    // #############################
    Ok(http::Response::builder()
        .status(200)
        .header("foo", "bar")
        .body(Some("Hello, Fermyon".into()))?)
}

With v0.8.0 I can see the headers printed to the container log on every request. When using the shim from main there is no output. @Mossaka I guess it is related to the LibContainer change.

Btw. It's awesome that we have a Redis trigger now 😊

This seems like a problem with the v0.9.1 demo containers:

ctr image pull ghcr.io/deislabs/containerd-wasm-shims/examples/slight-rust-hello:v0.9.0
ghcr.io/deislabs/containerd-wasm-shims/examples/slight-rust-hello:v0.9.0:         resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:41d465c956cf1ba8ad2cd424a07a275f7b4531c444004029294b83095530a8ec: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:93c9ffabbf0dfcda9d9cae20da0318ab6b9961976a4a9386b568d9cbd1e67e42:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:ecefbfcf107e0dc2d00e0627b4ab4cf9a0c9a90f5e37b8fe0c1980340bb526e6:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:801005d9437a75d3cf663bb703775deeb1a037861141134c6d3d0303d434b3ba:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:fafaac574a1c0c1ca9531b5d8f283de2f6cc11671d371680d0caaed06997a86b:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 1.1 s                                                                    total:  2.0 Ki (1.8 KiB/s)          
                             
unpacking linux/amd64 sha256:41d465c956cf1ba8ad2cd424a07a275f7b4531c444004029294b83095530a8ec...
done: 168.623677ms

ctr image pull ghcr.io/deislabs/containerd-wasm-shims/examples/slight-rust-hello:v0.9.1
ghcr.io/deislabs/containerd-wasm-shims/examples/slight-rust-hello:v0.9.1:      resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:efa2f914708b51fc581b24efdf101dc73e71de70bffaf5cff332190d75d34326: done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.3 s                                                                 total:   0.0 B (0.0 B/s)               
                          
unpacking linux/amd64 sha256:efa2f914708b51fc581b24efdf101dc73e71de70bffaf5cff332190d75d34326...
ctr: no match for platform in manifest sha256:efa2f914708b51fc581b24efdf101dc73e71de70bffaf5cff332190d75d34326: not found

Same for the other runtime examples: all the v0.9.1 tags seem to be broken this way.

Hey 👋 I wasn't sure where to ask and this seemed like a better place that on the #krustlet channel in the K8s slack.

How do the shims decide what .wasm file(s?) to load when they get an image?

I see spin.toml in the example ghcr.io/deislabs/containerd-wasm-shims/examples/spin-rust-hello image specifies

[[component]]
id = "hello"
source = "spin_rust_hello.wasm"

...which leads me to wonder how the shims know to look for spin.toml in the image layer contents, and not credenza.toml or pteradactyl.toml. If I had all three, would only spin.toml be read by convention?

In wasimg I'm setting the .wasm module to run in the OCI Image Config's Entrypoint, but that doesn't seem to be read anywhere, and anyway you may end up moving to OCI Artifacts anyway, where there isn't necessarily a config (#43)

Can an Image/Artifact used by these shims contain multiple [[component]]s, and therefore multiple .wasm files? Can the shims read and consider multiple .toml files, potentially of different variants?

Trying to get an idea of how these images are expected to be laid out, so I can be sure I'm able to build images that are "conformant" with the "spec" (heavy quotes there 😆 ).

The slight shim expects the wasm file to be named as app.wasm which forces the image creater to use this exact file name when creating a slight application image. We should relax this restriction.

Proposal: Slight should add a new entry in slightfile to specify the path to the wasm app.

Ideally, WASM images should be built to contain pretty much only the wasm module and maybe a configuration file for the runtime (i.e. slightfile.toml) and not much else. This would keep the OCI packages cross platform with out relying on multi-arch manifests improving the ability to use them everywhere, without modifications.

This might be tricky for a few reasons:

• The current slight dockerfile contains linux specific logic for getting and install certs:

  containerd-wasm-shims/images/slight/Dockerfile

  Line 5 in a662b07

  RUN apt-get update && apt-get install ca-certificates -y

  . When porting to windows these files are not useful as is.
• For Windows there are some platform specific files (some files that are required by the Windows snapshotter in Contianerd) that are need but don't make sense on Linux. If we use buildx on Windows, then these files and a few other files get added to the image.
• other OS specific deps (please list below if you have come across any)

There is a crate in runwasi that helps with the creation of images like this without the need for buildx: https://github.com/containerd/runwasi/tree/main/crates/oci-tar-builder

There will need to be some investigation on how to provide things like ssl certs and likely other components as well. The end goal would be to have purely "wasm" based images but that might not be practical, in those cases how do we deal with OS specific deps?

containerd-shim-spin currently supports simple workloads only. It should support all outbound capabilities provided by Spin like Redis and PostgreSQL.

When deploying a Spin application that uses outbound-redis::publish currently results in the "pod" failing with:

Normal   Created  1s (x3 over 18s)  kubelet   Created container main
Warning  Failed   1s (x3 over 18s)  kubelet   Error: failed to start containerd task "main": unknown import: `outbound-redis::publish` has not been defined: unknown

Hi all,
Since the v0.9.0 release, cgroups v1 support has been added, making the shims compatible with more kubernetes distributions KWasm/kwasm-node-installer#43.

Could we do a v0.9.1 release where the artifacts can be downloaded from the release page?

As we are adding more shims (e.g. #122 ) to the repo, the CI pipeline becomes longer to finish (e.g. also reported in #91). Part of the reasons is that all the shims in this repo defines their own workspaces, so each shim build takes a trunk of time in the CI. The disk space pressure issue described #111 (comment) is also related to this problem.

I am thinking about

1. Create a workspace in root directory of the project for all the shims. This will, in theory, reduces the build time. However, the downside of it is that it will create a single shared Cargo.lock file. If I remember correctly, the slight shim and spin shim share different versions of wasmtime. Could we isolate this dependency to achieve this? I don't know.

2. Parallelize the pipeline to build shims and upload artifacts as described #121 (comment). Then create a second job to pull the artifacts for the integration tests. This is do-able and I might spend this afternoon on working towards it.

In the meantime, I will do a local test of the lunatic shim, and if everything works, I am fine merging it in. After we fix the infrastructure of running tests on the GH actions, I will do a release all together.

FYI @0xE282B0 @jprendes

When using spin shim running containers with executor = { type = "wagi" } in spin.toml fails with error:

Error: failed to start containerd task "rust-spin-audio-sine-wave": Error copying assets: 3 file(s) not copied: unknown

Hi,

I'm trying to run the new key-value storage example that comes with Spin v0.9.0 from this article in the K8s cluster provided by Docker Desktop.

To configure the K8s cluster to support running WASM workloads, I use a modified version of the daemonset image defined in this approach. Instead of kwasm/kwasm-node-installer, I use felipecruz/kwasm-node-installer containing Spin v0.9.0 as per:

• KWasm/kwasm-node-installer@main...felipecruz91:kwasm-node-installer:main

After deploying the daemonset and the key-value Go deployment resources to the K8s cluster (see /deploy folder to find the manifests in this repo), the Pod's status is RunContainerError.

Kubelet logs

kubelet Error: failed to start container "tinygo-key-value-spin": Error response from daemon: Cannot read manifest file from │ │ "/var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/tinygo-key-value-spin/rootfs/spin.toml": unknown

Containerd logs from Docker Desktop (containerd.log)

[INFO] server listen started
[INFO] server started
[INFO] Shim successfully started, waiting for exit signal...
[DEBUG] Got new client
[DEBUG] create: id: "0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67" bundle: "/var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67" rootfs {type: "overlay" source: "overlay" options: "workdir=/var/lib/containerd-stargz-grpc/snapshotter/snapshots/440/work" options: "upperdir=/var/lib/containerd-stargz-grpc/snapshotter/snapshots/440/fs" options: "lowerdir=/var/lib/containerd-stargz-grpc/snapshotter/snapshots/397/fs:/var/lib/containerd-stargz-grpc/snapshotter/snapshots/223/fs"} stdout: "/var/run/docker/containerd/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/init-stdout" stderr: "/var/run/docker/containerd/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/init-stderr"
[DEBUG] create done
[2023-02-26T10:16:57.286022615Z][containerd][I] time="2023-02-26T10:16:57.285763458Z" level=debug msg="event forwarded" ns=moby topic=/tasks/create type=containerd.events.TaskCreate
[DEBUG] connect: id: "0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67"
[DEBUG] start: id: "0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67"
[INFO]  >>> stdin: "", stdout: "/var/run/docker/containerd/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/init-stdout", stderr: "/var/run/docker/containerd/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/init-stderr"
[INFO]  >>> waiting for start notification
[INFO]  >>> loading module: /var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/rootfs/spin.toml
[INFO]  >>> working dir: /var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/rootfs
[INFO]  >>> starting spin
[INFO]  >>> building spin application
[INFO]  >>> error starting instance: Cannot read manifest file from "/var/run/desktop-containerd/daemon/io.containerd.runtime.v2.task/moby/0298b703c88281f40ff21341fc5c3c09b65ed672f274202a15c077492f301a67/rootfs/spin.toml"

From my findings, it seems the error happens when the line key_value_stores = ["default"] is defined in the spin.toml file and the shim fails right after this line:

/cc @rumpl

I moved the built containerd-shim-spin-v1 to /usr/local/bin/.

Then I try to modify the configuration file of containerd, the content is roughly as follows：

[metrics]
  address = ""
  grpc_histogram = false

[plugins.cri.containerd.runtimes.spin]
  runtime_type = "io.containerd.spin.v1"

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]

But I encountered an error when I finally started containerd, the information is as follows：

# containerd
containerd: failed to load TOML from /etc/containerd/config.toml: invalid plugin key URI "cri" expect io.containerd.x.vx

how do i solve this problem？

I installed the spin shim from this repo but it doesnt work without docker-desktop engine. how can i build a image for a spin app that can still be used with runtime spin without docker desktop?

I have 2 images, one build with

$ docker buildx build -f <dockerfile> -t ferrazarthur/spin-test <dir>

and other with

$ docker buildx build --platform wasi/wasm32 -f <dockerfile> -t ferrazarthur/spin-test:wasi/wasm <dir>

Running image

$ docker container run -i --rm --name spin-testes-arch -p 3000:80 -v ./spin-logs:/log --runtime io.containerd.spin.v1 ferrazarthur/spin-test up --log-dir /logs

The same behaviour occurs with

$ docker container run -i --rm --name speed-wasm --platform wasi/wasm32 --runtime io.containerd.spin.v1 ferrazarthur/spin-test:wasi-wasm up --log-dir /logs

While with docker-desktop everything works as it should.

Expected behaviour

To properly run a spin up and keep a server open listening on port 3000

Actual behaviour

Container exits immediately after start, without executing spin up.

Modification made in Makefile

As i only wanted the spin shim, i changed the first line of Makefile from:

SHIMS := slight spin wws lunatic

to

SHIMS := spin.

SETUP

SYSTEM:

ARCH="x86_64"
NAME="Linux Mint"
VERSION="21 (Vanessa)"
ID_LIKE="ubuntu debian"
UBUNTU_CODENAME=jammy

/etc/docker/daemon.json

{
  "debug": true,
  "log-level": "debug",
  "features": {
    "containerd-snapshotter": true
  }
}

usr/local/bin

$ ls -l /usr/local/bin | grep containerd
-rwxr-xr-x 1 root              root               58866944 out 18 08:56 containerd-shim-spin-v1
lrwxrwxrwx 1 root              root                     29 out 17 14:30 containerd-shim-wasmedged-v1 -> ./containerd-shim-wasmedge-v1
-rwxr-xr-x 1 root              root              302380880 out 17 14:30 containerd-shim-wasmedge-v1
lrwxrwxrwx 1 root              root                     27 out 17 14:30 containerd-shim-wasmerd-v1 -> ./containerd-shim-wasmer-v1
-rwxr-xr-x 1 root              root              535063368 out 17 14:30 containerd-shim-wasmer-v1
lrwxrwxrwx 1 root              root                     29 out 17 14:30 containerd-shim-wasmtimed-v1 -> ./containerd-shim-wasmtime-v1
-rwxr-xr-x 1 root              root              389835168 out 17 14:30 containerd-shim-wasmtime-v1
lrwxrwxrwx 1 root              root                     29 out 17 14:30 containerd-wasmedged -> ./containerd-shim-wasmedge-v1
lrwxrwxrwx 1 root              root                     27 out 17 14:30 containerd-wasmerd -> ./containerd-shim-wasmer-v1
lrwxrwxrwx 1 root              root                     29 out 17 14:30 containerd-wasmtimed -> ./containerd-shim-wasmtime-v1

DOCKER:

Client: Docker Engine - Community
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:31:44 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:31:44 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

SPIN:

spin 1.4.1 (e0bd911 2023-07-12)
also tried with spin 1.5.1 (8d4334e 2023-09-26)

RUST:

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/graphicslesstower/.rustup

installed targets for active toolchain

wasm32-unknown-unknown
wasm32-wasi
x86_64-unknown-linux-gnu
x86_64-unknown-linux-musl

active toolchain

stable-x86_64-unknown-linux-gnu (default)
rustc 1.73.0 (cc66ad468 2023-10-03)

It would be nice to have a --version arg for the shims to be able to determine the build version information. Currently, there is no way to easily determine the version of the shim bin.

          This can be a follow-up task. Could you please add at least one integration test for testing spin OCI images?

Originally posted by @Mossaka in #180 (comment)

Since kd3 is used we need a way to swap out contianerd version with the latest. I would assume kd3 will use latest containerd in the next release so that can be an option at the very least

Currently there is no way to individually release each shim. This issue tracks the work to support individual release in the release github action. Each shim with its own tag name shall be released independently of other shims.

One breaking thing from the user side is that they no longer can just do command: ["/"] in pod spec. This is because the youki libcontainer verifies that the first argument to ENTRYPOINT must be an executable bianry. So that to build a spin image, we have to do:

COPY --from=build --chmod=0755 /opt/build/target/wasm32-wasi/release/spin_rust_hello.wasm .
COPY --from=build /opt/build/spin.toml .
ENTRYPOINT [ "./spin_rust_hello.wasm" ]

This isn't super desired and I expect that this will go away once this issue closes.

Originally posted by @Mossaka in #119 (comment)

One way to do this is to add a third parameter to handle_run, which takes in a struct of pod IO paths. Then in slight, we redirect all the logging and wasi output to these IO paths.

This avoids the breaking changes in the images which require entrypoint to be a executable.

FYI @0xE282B0

When I want to build the ../containerd-shim-spin-v1 project by running
cargo build --manifest=./Cargo.toml
on Apple Silicon M1 Macbook I get the following error:

error[E0425]: cannot find function prctlin cratelibc --> /Users/marcelbindseil/.cargo/registry/src/index.crates.io-6f17d22bba15001f/caps-0.5.5/src/ambient.rs:39:15

xCode and rust target 'aarch64-apple-darwin' are installed.

Any idea how to solve it?