extism / js-pdk Goto Github PK

We want the JS-PDK to support host functions. They should appear as imports in the JS world. We could come up with some kind of convention like:

import {host_func1, host_func2} from "extism:host";

But the host won't be able to provide them unless they make it into the wasm binary in the imports table. We could try a similar trick to exports. What it does is at compile time, it parses the javascript and looks for the exports, then it generates some "shim" wasm instructions that eval the function by name: https://github.com/extism/js-pdk/blob/main/crates/cli/src/main.rs#L115

Then it adds the exports to the exports table pointing at the shims: https://github.com/extism/js-pdk/blob/main/crates/cli/src/main.rs#L115

We could try something similar with imports. What might make it tricky is, i think, the function indices in the module start with imports, so you can't really append to them without re-writing every function index.

At this point, we're basically writing and maintaining our own linker. So perhaps we should look into a way we can use a tool to statically link the wasm modules at compile time and handle all this merging and re-indexing of functions.

This work may collide with #19 if we decide to go down that path.

This project is based on an older, but simpler way of building a js toolchain. We are statically compiling everything we need into one wasm file though a series of steps.

When javy switched hands to the bytecode alliance, they moved to a model of linking at runtime. Switching to this model could provide us some benefits. We've already effectively created a poor-man's linker to get exports to work. Supporting host functions would take us deeper down that path. So we either need to make use of a compile time linker or leverage wasmtimes runtime linker. We need some more research before we make a decision. One concern would be supporting other non-wasmtime runtimes. Will it work?

• Add polyfills
• Support async and pending tasks
• Support ESM
• Upgrade quickjs dependencies
• Research runtime linking with core module

In attempting to follow along with the basic instructions for building the plugin I get the following error when building, altho a wasm file is produced:

warning: unknown name subsection with id 9 at 4579967

And when I run:

extism call plugin.wasm greet --input="foo" --wasi

I get the above error:

Error: module[extism:host/env] not instantiated

Any ideas what would cause this. I literally just copied and pasted the plugin.js and plugin.d.ts` files from the README.

I think polluting the global namespace is not the best decision for the API. And we should consider how this will work from typescript. I'll be doing some research and coming back with a document for feedback.

QuickJS does support ESM. Also the quickjs bindings give us a way to eval an ES Module https://docs.rs/quickjs-wasm-rs/3.0.0/quickjs_wasm_rs/struct.JSContextRef.html#method.eval_module

We should switch to ESM and maybe even remove CJS support. That will also require changing the README and some of the instructions. It will probably have an impact on the polyfill code too.

Right now all the test stuff is thrown in the root: https://github.com/extism/js-pdk/blob/main/bundle.js

Should create a subfolder for some different tests plugins and we should run them in CI

I'm trying to make a post request to a specific endpoint but it's not working.

Running install script failed on new macOS due to no /usr/local/bin folder:

sh install.sh
...
mv: rename extism-js-aarch64-macos-v1.0.0-rc11 to /usr/local/bin/extism-js: No such file or directory

Add basic support for Date() for simple timing. Need to consider possibility of exposing too much timing information.

I was looking into a build time error I was getting, which seems to be related to quickjs #261 which was fixed in a newer release of quickjs. During this investigation, I noticed that Javy has deprecated quickjs-wasm-rs in favor of rquickjs.

The reasons for the deprecation are outlined in javy #618.

I started a WIP branch to see what moving the pdk to rquickjs might look like. I've mostly moved through the logic for initializing the globals so far, but the migration seems feasible. It seems like rquickjs should let the pdk keep most of the same logic & API's, just moved around a bit.

Keeping the tooling similar to Javy and using rquickjs may also make it easier to implement #56 and #19 , but haven't looked into it too much.

Recognize that this pdk is in rc for 1.0, so a large crate change might not be the best timing. I am happy to take this migration farther if that's something y'all would be interested in, but didn't want to just drop a big migration PR on y'all without talking it through.

It might make sense to publish the whole package to npm and have the compiler published as a bin since that's how most people will be managing their other dependencies. We can also add it to the templates. There may be some trickiness around getting wasm-merge in place. Leaving this open for discussion and ideas on how to implement.

Currently WASI is required for JS plug-ins. This is for a number of reasons:

1. Some js code like Math.random() and new Date() requires randomness and clocks respectively
2. The way that the plug-in is "wizened" is by passing in the JS code through stdin and parsing it in the "compile" step. So WASI is needed during compile and the program is a continuation of that.

Ideally what would come of this would be a flag that you can pass specifying you want non-wasi (or the perhaps it could be non-wasi by default). We will probably need to compile 2 versions of the "core", one with the normal wasi stuff and one neutered from any syscalls. The flag can chose which core initialize at compile time. We should also find some other way to get the code in here: https://github.com/extism/js-pdk/blob/main/crates/core/src/lib.rs#L18

Some investigation into wizer will be needed. Or we may need to disable wizening for non-wasi plug-ins.

👋 Hello. I'm using Multipass for my development workspace (on a Mac M1), then

• the value of $OSTYPE is linux-gnu but I need to run the installer like this: bash install.sh (instead of sh install.sh otherwise I will get this message: unknown os
• then I get: unknown arch, the workaround was to change arm64*) ARCH="aarch64" ;; by arm64*|aarch64*) ARCH="aarch64" ;;

I believe the tools we use support windows, so we just need to add windows to our CI and publish workflows:

• https://github.com/extism/js-pdk/blob/main/.github/workflows/publish.yml
• https://github.com/extism/js-pdk/blob/main/.github/workflows/ci.yml

But testing may yield some other issues.

Say I have the following two exports defined:

// src/index.ts

export function start() {
  console.log('start');
}

export function end() {
  console.log('end');
}

// src/index.d.ts

declare module "main" {
  export function start(): void;
  export function end(): void;
}

I bundle src/index.ts to /dist/index.js with esbuild using cjs format and es2020 as target, then build the wasm using extism-js like with dist/index.js and src/index.d.ts.

If I then call the start wasm export, surprise: end will be logged:

❯ extism call ../bundled.wasm start --wasi --log-level debug
2024/04/10 18:00:35 Calling function : start
2024/04/10 18:00:35 end

Part of the reason in this case, is that esbuild sorts the exports alphabetically in the module.exports object. So end is suddenly defined first, but it still points to the end function nonetheless:

// dist/index.js

// snip...

var src_exports = {};
__export(src_exports, {
  end: () => end,
  start: () => start
});
module.exports = __toCommonJS(src_exports);
function start() {
  console.log("start");
}
function end() {
  console.log("end");
}

Now the order in the declaration doesn't match module.exports though, so something in the wasm building step seems to rely on the order being 1:1.

There seems to be a regression around the core engine handling exceptions. Given this test program:

function greet() {
  throw new Error('hello')
}

module.exports = { greet }

Output is:

2024/03/23 09:33:11 No runtime detected
2024/03/23 09:33:11 Calling function : greet
Error: wasm error: unreachable
wasm stack trace:
	.$1151(i32,i32,i32,i32,i32,i32)
	.$1290(i32,i32)
	.$1309(i32,i32,i32,i32,i32)
	.$1357(i32) i32
	.$1374() i32

We should see the exception

Running a js-pdk plugin with extism-wamr gives a warning:

[15:03:41:772 - 7F54D39B9740]: warning: a module with WASI apis should be either a command or a reactor

We may want to add an _initialize export to fulfill this expectation

Hi. I'm trying the example at https://extism.org/docs/write-a-plugin/js-pdk. When I compile the plugin, I get:

warning: unknown name subsection with id 9 at 3197029

This is using version 0.5.0 of the extism-js compiler, on Linux x64. (Ubuntu 22.04 on WSL)

It does produce a wasm file, but that file doesn't work in my Golang host. I get the following when attempting to load the plugin:

module[wasi_snapshot_preview1] not instantiated

However, if I use the precompiled wasm sample files from https://github.com/extism/extism/tree/main/wasm my host works fine.

As mentioned in the comments in #68, we do not conform to the full spec for console.log and the other methods on console. We should refactor it to be compliant. There are two paths:

1. write some more rust code to emulate how
2. expose the underlying IO functions and implement the console object in the js layer

For path number 2 you'd expose extism log functions as global js functions like this:

then you'd implement the console object and it's methods in javascript