wasmcloud / weld Goto Github PK

Test message serialization/deserialization between languages and also any supported language to itself.

The test should leverage a set of operations and data structures created for this test, and data fuzzing.

See also https://github.com/wasmCloud/actor-interfaces/pull/84

I had unintentionally imported std::fmt::Result into my actor and then things stopped compiling because the Actor derive macro uses bare Results as a return type. As it is common for libraries to define their own Result alias, we should make sure we fully qualify the types in the derive macro

Implement code generator to create AssemblyScript actor projects and interface libraries from a smithy model. Much of the rust code generator should be reusable.

This applies to actor (wasm32) code only. None of the provider-generation code needs to be ported.

As of rust 1.67 (possibly slightly earlier), code generated by weld doesn't satisfy default clippy warnings. The most common violation is https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args

Example, from the messaging interface:

❯ cargo clippy
   Compiling wasmcloud-interface-messaging v0.8.1 (/Users/connor/Documents/wasmCloud/interfaces/messaging/rust)
warning: variables can be used directly in the `format!` string
   --> messaging/rust/src/messaging.rs:484:50
    |
484 |                     .map_err(|e| RpcError::Deser(format!("'SubMessage': {}", e)))?;
    |                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#uninlined_format_args
    = note: `#[warn(clippy::uninlined_format_args)]` on by default
help: change this to
    |
484 -                     .map_err(|e| RpcError::Deser(format!("'SubMessage': {}", e)))?;
484 +                     .map_err(|e| RpcError::Deser(format!("'SubMessage': {e}")))?;
    |

This generates noise for everyone, and for users whose CI builds fail on warnings, this will cause failures

Capability providers should be able to provider their own logger, and looking at our init_logger https://github.com/wasmCloud/weld/blob/main/rpc-rs/src/channel_log.rs#L112 function it appears we return an error if it's not initialized properly. For tracing, custom loggers, etc we should just warn about this failure and assume the provider writer knows what they are doing.

There is a work-around, which is to install a nats leaf-node on localhost and let the capability-provider connect to that without tls.

I don't think the ratsio client supports TLS, so adding TLS support probably involves making a PR to ratsio.

I think I may have found an incompatibility when using the following versions together in a test suite:

• wasmcloud-test-util=0.6.4
• wasmbus-rpc=0.12.0

The error I get is:

$ make test
cargo test  -- --nocapture
   Compiling wasmcloud-search-meilisearch v0.1.0 (/home/path/provider/code)
warning: unused imports: `CreateIndexInput`, `SearchSender`, `Search`
  --> search-meilisearch/tests/op_create_index.rs:10:34
   |
10 | use wasmcloud_interface_search::{Search, SearchSender, CreateIndexInput};
   |                                  ^^^^^^  ^^^^^^^^^^^^  ^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

error[E0277]: the trait bound `wasmcloud_interface_testing::TestResult: From<(&str, Result<(), RpcError>)>` is not satisfied
  --> search-meilisearch/tests/op_create_index.rs:35:15
   |
35 |     let res = run_selected_spawn!(opts, health_check,);
   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |               |
   |               the trait `From<(&str, Result<(), RpcError>)>` is not implemented for `wasmcloud_interface_testing::TestResult`
   |               required by a bound introduced by this call
   |
   = help: the trait `From<(&'name str, Result<T, wasmbus_rpc::error::RpcError>)>` is implemented for `wasmcloud_interface_testing::TestResult`
   = note: required for `(&str, Result<(), RpcError>)` to implement `Into<wasmcloud_interface_testing::TestResult>`
   = note: this error originates in the macro `run_selected_spawn` (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try `rustc --explain E0277`.
warning: `wasmcloud-search-meilisearch` (test "op_create_index") generated 1 warning
error: could not compile `wasmcloud-search-meilisearch` due to 3 previous errors; 1 warning emitted
warning: build failed, waiting for other jobs to finish...
make: *** [Makefile:14: test] Error 101

It's not immediately clear why the types won't unify but it looks to be that lifetime maybe... comparing them:

From<(&str, Result<(), RpcError>)>
From<(&'name str, Result<T, wasmbus_rpc::error::RpcError>)>

When I downgrade to wasmbus-rpc=0.11.2, the test works properly.

Looking at the changes in between 0.11.2 and 0.12.0, it's not quite obvious what was introduced... maybe the broader requirements for Transport? Not terribly clear.

in codegen.toml, you can specify a smithy model dependency with either a file path or an http(s) url. A third and useful option would be to support a "git:" scheme to fetch a file from a github repo.

A point of friction with the current two options is that if you have a .smithy file in one repo and want to generate code from it in another repo - as you might have with distributed development teams - the smithy file has to be copied into the static space of an http(s) server, or you need to publish the repo as github pages.

Right now the host ID being placed onto the Invocation struct instance is String::new(). We need to allow the host ID to be passed in somewhere because all (valid) invocations must bear the host ID from which they originated. Capability providers know the host ID on which they're being hosted by virtue of the host data struct injected at start time.

https://github.com/wasmCloud/weld/blob/main/rpc-rs/src/rpc_client.rs#L245

build, run tests, run clippy checks, run security audit
test on ubuntu, mac, windows

change to wasmbus_core smithy model: add fields to structure

change to wasmbus-rpc: add fields to structure, and process link_definitions on startup.

Update documentation if necessary

The current set of smithy models were developed under wasmcloud/weld/examples/interface/, where they will live until the smithy interfaces are stabilized. After that, they should move somewhere else.

A more permanent home needs to be identified for

• core model files (wasmcloud-model & wasmcloud-core)
• interfaces that define capability providers, and
• example code

The destination for each of these types isn't necessarily the same.

Because the core model files are tightly connected to both codegen and rpc-rs, those model files will likely stay in this repo (unless this repo is also split up).

Add support for Timestamp data type

There's already a derive macro for rust actors to implement the dispatch between implemented Receiver traits, but not one for providers yet.
Implement the one for providers to cut 21 lines of boilerplate from each capability provider!.

The wasmcloud/models repo contains html files and downloadable smithy model files. This repo (weld, and/or wherever shared smithy files end up living) needs to have CI tools to copy changed smithy files into the correct locations in the models repo.

Depends on Issue #2

#93 added new functionality to gracefully ignore when a logger is already init-ed. However, because the error is a String we have to see what the string contains. This is brittle functionality and the init_logger function should be updated to have an enum error to differentiate between error cases

The smithy spec describes an optional @enum constraint attribute for string fields, that requires the string to be one of a fixed set of values.

This change can simplify apis that require enum fields, and the generated error checking will improve reliability and safety of apis that use them.

msgpack does not support enums, so enums have to be transmitted as either strings or numbers on the wire.

This would be fairly straightforward to add to the rust code generator, so that in Rust, instead of a String field, the code generator would generate an Enum, but the msgpack serializer would still transmit the value as a Strong. For example:

enum Direction {
  Asc,
  Desc,
}

struct Query {
   sort: Option<Direction>,
    // .. other fields omitted
|

from the declaration

structure Query {
  sort: Direction,
}

@enum([{ value: "Asc" }, { value: "Desc" } ])
string Direction

The @enum declaration also supports documentation for each of the variants, which should be copied into the source as the enum is generated.

On whether to use string or an int on the wire, there are a few advantages to starting with strings:

• Smithy's @enum is defined to work for strings only, so no new annotations need to be defined.
• Strings are easier to read when viewing a msgpack-serialized message in a debugger.
• Strings are less fragile (not dependent on order of declaration)
• Strings don't require additional annotation for the variant number.

The primary motivation for choosing an integer encoding on the wire, (such as a u8 for enums with < 256 variants) is a hypothetical performance improvement, but no measurements have been conducted to determine whether there would be a material difference in performance.

smithy files downloaded from a url are cached locally in ~/.cache, and the cache can be cleared with 'wash drain', but there isn't an automated way for cached smithy files to be downloaded if they are out of date.

One option is to try to use Last-Modified HTTP header (and a if-modified-since header for requests), and compare the header date to the file's timestamp, but I'm not sure if we'll be able to set the modified timestamp of files on all supported developer OS's.
A course but simpler option would be to force a re-download at some interval, such as 24 hours. That would still enable most of the benefits of caching, and still enable offline builds. The only OS requirement in that case is to be able to read the file's modified timestamp.

When you have a blob input on an operation, the generated code uses a borrowed Vec, which clippy doesn't like. This error should be ignored or it should be switched to use &[u8]

writing `&Vec<_>` instead of `&[_]` involves one more reference and cannot be used with non-Vec-based slices
   --> src/kubernetes_applier.rs:284:47
    |
284 |     async fn apply(&self, ctx: &Context, arg: &Vec<u8>) -> RpcResult<OperationResponse>;

I've updated the core model to add this field (https://github.com/wasmCloud/interfaces/blob/main/core/wasmcloud-core.smithy#L167). When attempting to generate code that uses this smithy file at the URL "https://wasmcloud.github.io/interfaces/idl/org.wasmcloud" , it still generates code without this field. I am not sure how this is even possible, but there's either a documentation issue happening or this "magic" URL (it produces a 404 on github) is incorrect?

Regardless of the root cause, I'm unable to generate code against this new field in github.

Optional string fields need to be checked for a non-empty value, and if empty, encode a WriteNil value instead of a string value. When the host receives a string value (to include an empty string), specifically for the below error field, it treats it as an error.

This is the edited code from the core tinygo module

// MEncode serializes a InvocationResponse using msgpack
func (o *InvocationResponse) MEncode(encoder msgpack.Writer) error {
	encoder.WriteMapSize(4)
	encoder.WriteString("msg")
	encoder.WriteByteArray(o.Msg)
	encoder.WriteString("invocation_id")
	encoder.WriteString(o.InvocationId)
	encoder.WriteString("error")
	if len(o.Error) > 0 {
		encoder.WriteString(o.Error)

	} else {
		encoder.WriteNil()
	}
	encoder.WriteString("content_length")
	encoder.WriteUint64(o.ContentLength)

	return encoder.CheckError()
}

In implementing the blobstore-fs provider I found that there are a few things that could be added to TimeStamp which would be helpful:

• Add a since_epoch method which returns the TimeStamp value since epoch. Basically now minus TimeStamp for epoch start
• Add a from_systemtime method which converts systemtime to TimeStamp
• Add a duration_since_epoch which takes a system time and calculates the duration since epoch
• Add duration method which calculates the TimeStamp value as duration between two timestamps
• Comparison (>=,<,==, !=) would be useful too.

Maybe there is already a library for this?

Describe the bug
Following the example in the documentation for Creating an Interface fails for no obvious reason.

To Reproduce
Steps to reproduce the behavior:

1. Install with the instructions for MacOS:

$ brew tap wasmcloud/wasmcloud
$ brew install wash [email protected]

2. Follow the instructions for Creating an Interface:
   $ wash new interface payments
3. Generate the code and build the Rust library:

$ make
Compiling wasmbus-rpc v0.13.0
error: failed to run custom build command for `wasmcloud-examples-payments v0.1.0 (/Users/vances/payments/rust)`
Caused by:
  process didn't exit successfully: `/Users/vances/payments/rust/target/debug/build/wasmcloud-examples-payments-2b0df543af719e9f/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-changed=/Users/vances/payments/codegen.toml
  cargo:rerun-if-changed=/Users/vances/payments/./payments.smithy
  cargo:rerun-if-changed=/Users/vances/Library/Caches/smithy/wasmcloud.github.io/wasmcloud-core.f2facc62766ec8c4.smithy
  cargo:rerun-if-changed=/Users/vances/Library/Caches/smithy/wasmcloud.github.io/wasmcloud-model.6e8584f5be28601b.smithy
  --- stderr
  Error: Formatter("failed to start: No such file or directory (os error 2)")

Expected behavior
The build completes successfully.

Environment

• macOS 13.3.1 (22E261)
• bash
• wash 0.17.2

Additional context
Installing rustfmt resolved the problem:
$ brew install rustfmt

The homebrew formula has a dependency on rust but not rustfmt:

$ brew deps wash     
ca-certificates
openssl@3
pkg-config
rust

When I call an actor that doesn't implement a certain message, I get the error

~/github.com/wasmcloud/weld/rpc-rs ➜ wash call MARGEE37MWZL2CY3WLUBQ7MMLZA3XTGKPPPPLGBFTAXVQU2C7PPZOFOW Tesst.PartOne '{}' -t 10000

deserialization: response to Tesst.PartOne: deserialization: invalid type: unit value, expected byte array

However, subscribing to NATS on *.> I can actually see that the error is that the method was not implemented, and that error string is what's failing to deserialize as an InvocationResponse. This is an error message that's misleading and can cause confusion

The intent behind https://github.com/wasmCloud/weld/blob/main/rpc-rs/src/provider_main.rs#L153 was to initialize capability providers with all of its link definitions, however, a malformed input or link definition that is placed that returns an error will forever prevent a provider from starting until it's deleted via the washboard / wash ctl links del

What I would love is a way for the provider to start with whatever feedback as is necessary to indicate that the linkdef failed to establish and even better would be to remove that link definition

cc @autodidaddict @stevelr

We have a field for enabling structured logging, but not one for setting the log level a provider should log at. The current workaround is to use env vars passed to the host, but this is sloppy, insecure, and complicated. Not all languages even have a convention for setting a global log level via an env var.

I propose the enum values should be:

• trace (maybe not? Not all languages support trace-level logs)
• debug
• info
• warn
• error

I started on implementing this and got caught up in recompiling the world of wash/weld so I could run wash gen so I could recompile providers so I could test this. With smithy-bindgen recently added I might be going about this wrong? I'm filing this issue so someone with more context than I have on the state of codegen can help

when compiling crates that depend on interfaces, it's not uncommon to get a trait error such as
"the trait bound ZZZ: MessageDispatch is not satisfied" for a class ZZZ when the trait bounds are satisfied, but there is a version difference between wasmbus-rpc dependency of the crate being compiled, and the wasmbus-rpc version used at the time the interface crate was published - even if there has been no api change in the trait.

If each interface crate re-exported the wasmbus-rpc traits that it depends on (this would be a simple change to weld-codegen), some of these errors may go away, or else they could be avoided by the downstream crate importing the traits from the interface crate instead of wasmbus-rpc.

After the change is made to weld-codegen, the interface crates would need to be regenerated with new versions.

Adapt the rust code generator to generate go-language code for capability provders
and port the rpc-rs runtime library to go.

The capability provider support needed is for the otp host (wasmcloud >0.18) with nats messaging.

@stevelr @thomastaylor312

After having upgraded to wasmbus-rpc = "0.10" I do not see any log outputs (used crate is log = "0.4").
Also, following instructions for changing log level, the following

Logger.configure(level: :debug)

fails with

iex([email protected])1> Logger.configure(level: :debug)
** (MatchError) no match of right hand side value: {:error, {:not_found, Logger}}
    (logger 1.12.3) lib/logger.ex:588: Logger.configure/1

P.s. crate tracing produces log outputs.

Capability providers based on wasmbus-rpc are using a nats regular subscription instead of a queue subscription (provider.rs line 264). This prevents using more than one provider on the same contract and link to increase capacity and redundancy.

Project files generated with "weld gen --create ..." are out of date.

The actor and interface code has evolved somewhat since the project generator was implemented, so the templates for those project types need to be brought up to date.

Capability-provider templates need to be written.

In the meantime, a work-around for new project creators is to copy a folder from under examples/actor, examples/interface, or examples/provider that is closest to what they want to create.

For the same reason that StructuredLogging is included...the provider knows what level to log at.

WASMCLOUD_STRUCTURED_LOG_LEVEL <- looking for this setting in HostData

Or am I missing it somewhere?

We cannot write async function in non async function. Sometimes we got no choice but to write non async function. The "futures::executor::block_on" in the actor causes panic.

This will require moving lint/validate code from wash into this crate.