This is a little experiment in building cross-platform components in rust, based on things we've learned in the mozilla/application-services project.

It's at the "very hand-wavy prototype" stage, so don't get your hopes up just yet ;-)

We're interested in building re-useable components for sync- and storage-related browser functionality - things like storing and syncing passwords, working with bookmarks and signing in to your Firefox Account.

We want to write the code for these components once, in Rust. We want to easily re-use these components from all the different languages and on all the different platforms for which we build browsers, which currently includes JavaScript for PCs, Kotlin for Android, and Swift for iOS.

And of course, we want to do this in a way that's convenient, maintainable, and difficult to mess up.

Our current approach to building shared components in rust involves writing a lot of boilerplate code by hand. Take the fxa-client component as an example, which contains:

• The core functionality of the component, as a Rust crate.
• A second Rust crate for the FFI layer, which flattens the Rust API into a set of functions and enums and opaque pointers that can be accessed from any language capable of binding to a C-style API.
• A Kotlin package which wraps that C-style FFI layer back into rich classes and methods and so-on, for use in Android applications.
• A Swift package which wraps that C-style FFI layer back into rich classes and methods and so-on,for use in iOS applications.
• A third Rust crate for exposing the core functionality to JavaScript via XPCOM (which doesn't go via the C-style FFI).

That's a lot of layers! We've developed some helpers to make it easier, but it's still a lot of repetitive similarly-shaped code, and a lot of opportunities for human error.

What if we didn't have to write all of that by hand?

In an aspirational world, we could get this kind of easy cross-language interop for free using wasm_bindgen and webassembly interface types - imagine writing an API in Rust, annotating it with some #[wasm_bindgen] macros, compiling it into a webassembly bundle, and being able to import and use that bundle from any target language, complete with a rich high-level API!

That kind of tooling is not available to shipping applications today, but that doesn't mean we can't take a small step in that general direction while the Rust and Wasm ecosystem continues to evolve.

• Specify the component API using an abstract Interface Definition Language.
• When implementing the component:
  • Process the IDL into some Rust code scaffolding to define the FFI, data classes, etc.
  • Have the component crate !include() the scaffolding and fill in the implementation.
• When using the component:
  • Process the IDL to produce FFI bindings in your language of choice
  • Use some runtime helpers to hook it up to the compiled FFI from the component crate.

This is all very experimental and incomplete, but we do have some basic examples working, implementing functions in Rust and calling them from Kotlin. Take a look in the ./examples/ directory to see them in action.

We'll abstractly specify the API of a component using the syntax of WebIDL, but without getting too caught up in matching its precise semantics. This choice is largely driven by the availability of quality tooling such as the weedle crate, general familiarity around Mozilla, and the desire to avoid bikeshedding any new syntax.

We'll model the semantics of a component's API loosely on the primitives defined by the Wasm Interface Types proposal (henceforth "WIT"). WIT aims to solve a very similarly-shaped problem to the one we're faced with here, and by organizing this work around similar concepts, we might make it easier to one day replace all of this with direct use of WIT tooling.

In the future, we may be able to generate the Interface Definition from annotations on the rust code (in the style of wasm_bindgen or perhaps the cxx crate) rather than from a separate IDL file. But it's much easier to get started using a separate file.

The prototype implementation of parsing an IDL file into an in-memory representation of the component's APIs is in ./src/types.rs. See arithmetic.idl for a simple example that actually works today, or see fxa-client.idl for an aspirational example of an interface for a real-world component.

We'll avoid WedIDL's sparse and JS-specific types and aim to provide similar primitive types to the WIT proposal: strings, bools, integers of various sizes and signedeness. We already know how to pass these around through a C-style FFI and the details don't seem very remarkable.

These all pass by copying (including strings, which get copied out of Rust and into the host language when transiting the FFI layer).

These are what they say on the tin - named callables that take typed arguments and return a typed result. In WebIDL these always live in a namespace, like so:

namespace MyFunctions {
  my_function();
  string concat(string s1, string s2);
};

In the FFI, these are extern "C" functions that know how to convert values to and from Rust and the host language. (WIT calls this "lifting" and "lowering" and we'll use the same terminology here).

These represent objects that you can instantiate, that have opaque internal state and methods that operate on that state. They're typically the "interesting" part of a component's API. We currently implement these by defining a Rust struct, putting instances of it in a ConcurrentHandleMap, and defining a bunch of extern "C" functions that can be used to call methods on it.

In WebIDL these would be an interface, like so:

interface MyObject {
  constructor(string foo, bool isBar);
  bool checkIfBar();
}

I don't think the WIT proposal has an equivalent to these types; they're kind of like an anyref I guess? We should investigate further...

In the FFI, instances are represented by an opaque u64 handle, and their methods become extern "C" functions that work just like plain functions, but take a handle as their first argument.

When generating component scaffolding, we'll rely on hand-written Rust code to provide a MyObject struct with apropriate methods. we'll transparently create a HandleMap to hold instances of this struct, and a suite of extern "C" functions that load handles into struct instances and delegate to their methods. Rust's strong typing will help us ensure that the generated scaffolding code fits together properly with the core component code.

When generating language-specific bindings, these becomes a class or equivalent. Each instance of the class will hold a handle to the corresponding instance on the Rust side, and its methods will call the exposed extern "C" functions from the FFI layer in order to delegate operations to the Rust code.

TODO:

• Can we use member attributes to annotate which methods require mutable vs shared access?
• Can we use member attributes to identify which methods may block, and hence should be turned into a deferred/promise/whatever.

These are structural types that are passed around by value and are typically only used for their data. In current hand-written components, we pass these between Rust and the host language by serializing into JSON or Protocol Buffers and deserializing on the other side.

In WebIDL this corresponds to the notion of a dictionary, which IMHO is not a great name for them in the context of our work here, but will do the job:

dictionary MyData {
  required string foo;
  u64 value = 0;
}

In the WIT proposal these are "records" and we use the same name here internally.

In the FFI layer, records do not show up explicitly. Functions that take or return a record will do so via an opaque byte buffer, with the calling side serializing the record into the buffer and the receiving side deserializing it. Buffers are always freed by the host language side (using a provided destructor function for buffers that originate from Rust).

When generating the component scaffolding, we'll turn the record description into a rust struct with appropriate fields, and helper methods for serializing/deserializing from a byte buffer.

When generating language-specific bindings, records become a "data class" or similar construct, again with field access and serialization helpers.

Since we are autogenerating the code on both sides of serializing/deserializing records, we will probably not use protocol buffers or JSON for this, but will instead use a simple bespoke encoding. We assume that both producer and consumer will be build from the same IDL file using the same version of uniffi. (Our current build tooling enforces this, and we'll try to build some simple hooks into the generated code to ensure it as well).

Both WebIDL and WIT have a builtin sequence type and we should use it verbatim.

interface MyObject {
    sequence<Foo> getAllTheFoos();
}

In current hand-written compoinents we use ad-hoc Protobuf messages for this, e.g. the fxa-client component has an AccountEvent record for a single event and an AccountEvents record for a list of them. Since we're auto-generating things we'll instead use a more generic, re-useable implementation.

In the FFI layer, these operate similarly to records, passing back and forth via an opque bytebuffer.

When generating the component scaffolding, we'll try to use Rust's rich iterator support to accept any iterable as a sequence return value. Sequence arguments will arrive as Vecs.

When generating language-specific bindings, sequences will show up as the native list/array/whatever type.

WebIDL as simple C-style enums, like this:

enum AccountEventType {
  "INCOMING_DEVICE_COMMAND",
  "PROFILE_UPDATED",
  "DEVICE_CONNECTED",
  "ACCOUNT_AUTH_STATE_CHANGED",
  "DEVICE_DISCONNECTED",
  "ACCOUNT_DESTROYED",
};

In the FFI layer these will be encoded into an unsigned integer type.

When generating the component scaffolding, these will become a Rust enum in the obvious fashion.

When generating language-specific bindings, these will show up however it's most obvious for an enum to show up in that language.

There is also more sophisticated stuff in there, like union types and nullable types. I haven't really thought about how to map those on to what we need.

WebIDL has support for these, and they probably have an obvious representation via Rust's Option type and the equivalent in host languages. But we haven't investiated these in any detail.

WebIDL has some support for these, and they're probably useful, but we haven't worked through any details of how they might show up in a sensible way on both sides of the generated API.

WebIDL has some syntax for them, but I haven't looked at this in any detail at all. It seems hard, but also extremely valuable because handling callbacks across the FFI boundary has been a pain point for us in the past.

Is still in its infancy, but we're working on it. The current implementation uses askama for templating because it seems to give nice integration with the Rust type system.

Currently a very hacky attempt in ./src/scaffolding.rs, and a generate_component_scaffolding(idl_file: &str) function that's intended to be used from the component's build file.

Currently a very very hacky attempt in ./src/bindings/kotlin/, and it's not yet clear exactly how we should expose this for consumers. As something done from the component's build script? As a standlone executable that can translate an IDL file into the bindings?

Totally unimplemented. If you're interested in having a go at it, try copying the Kotlin bindings generator and adapting it to your needs!

Totally unimplemented. If you're interested in having a go at it, try copying the Kotlin bindings generator and adapting it to your needs!

We haven't even tried it yet! It could be a fun experiment to try to generate some code that uses wasm-bindgen to expose a component to javascript.

Lots!

The complexity of maintaining all this tooling could be a greater burden then maintaining the manual bindings. We might isolate expertise in a small number of team members. We might spend more time working on this tooling than we'll ever hope to get back in time savings from the generated code.

By trying to define a one-size-fits-all API surface, we might end up with suboptimal APIs on every platform, and it could be harder to tweak them on a platform-by-platform basis.

The resulting autogenerated code might be a lot harder to debug when things go wrong.

It would be wonderful to get much or all of this for free from wasm-bindgen, but it exclusively targets JavaScript as a host language. The upcoming Wasm Interface Types proposal should help a lot with this, but that's still in its early stages.

We're not aware of any production-ready WebAssembly runtimes for Android or iOS (with nice integration with Kotlin and Swift respectively) which is a requirement for current consumers of our components.

But aspirationally, we'd be pretty happy to one day throw away much of the code in this crate in favour of tooling from the Wasm ecosystem.

SWIG is a great and venerable project in this broad domain, but it's designed for C/C++ as the implementation language rather than Rust, and at time of writing it doesn't appear to support generating Kotlin or Swift bindings. Either of these alone might not rule it out (e.g. we could conceivable use time spent on uniffi to instead write a Kotlin backgend for SWIG) but missing them both seems to make it a bad fit for our needs.

It targets C++ as the implementation language rather than rust, and it's been explicitly put into "maintenance only" mode by its authors.

Please suggest it by filing an issue! If there's existing tooling to meet our needs then you might spoil a bit of fun, but save us a whole bunch of work!