mozilla / toodle Goto Github PK

This project forked from fluffyemily/cross-platform-rust

INACTIVE - http://mzl.la/ghe-archive - Example project showing a not-very-real-world app with Rust storage running on iOS, on Android, and in a WebExtension. Please follow progress on Mozilla's next generation cross platform sync and storage solution at https://github.com/mozilla/mentat.

Java 16.16% C 9.86% Rust 39.16% Swift 15.85% Objective-C 0.40% Shell 0.10% JavaScript 15.67% HTML 0.24% CSS 2.56%

inactive unmaintained

toodle's Introduction

Toodle: cross-platform, Rust-powered TODO app

This is an example project demonstrating Rust-based generic storage and syncing for iOS, Android and Firefox's WebExtension.

iOS

Building

First, we have to install Xcode and then set up Xcode build tools. If you already have the build tools installed and they are up to date, you can skip this step.

xcode-select --install

Next, we need to ensure that Rust is installed and that we can cross compile to the iOS architectures. For this we will be using rustup. If you already have rustup installed, you can skip this step. Rustup installs Rust from the official release channels and enables you to easily switch between different release versions. It will be useful to you for all your future Rust development, not just here.

curl https://sh.rustup.rs -sSf | sh

Add the iOS architectures to rustup so we can use them during cross compilation.

rustup target add aarch64-apple-ios armv7-apple-ios armv7s-apple-ios x86_64-apple-ios i386-apple-ios

When you installed Rust, it also installed cargo, which is a package manager similar to pip, gems etc. Now we will use cargo to install cargo-lipo. This is a cargo subcommand which automatically creates a universal library for use with iOS. Without this crate, cross compiling Rust to work on iOS is infinitely harder.

cargo install cargo-lipo

We need to build our library against the iOS architectures using cargo-lipo. The built artifacts of will be placed in rust/target/. The universal iOS library that we are interested in can be found in rust/target/universal/release/libtoodle.a.

cd rust
cargo lipo --release

We are now using Carthage to import external dependencies. Install Carthage using the instructions on their site. Once installed, open a terminal and navigate to the iOS project root. To install the project dependencies, run the following command:

carthage update

Open ios/Toodle/Toodle.xcodeproj in Xcode. Select the Toodle project from the project navigator, and then ensure the Toodle target is selected. Open the General tab. Scroll down to the Linked Frameworks and Libraries section. Import your libtoodle_ffi.a library by either dragging it in from Finder, or clicking the + at the bottom of the list, clicking 'Add other…' and navigating to rust/target/universal/release/. Select libtoodle_ffi.a and then click Open.

You should now be able to build and run your iOS app.

Android

{TODO}

Configure Rust build environment as per https://mozilla.github.io/firefox-browser-architecture/experiments/2017-09-21-rust-on-android.html
Build for all architectures via helper script rust/android_build_all.sh
Load the android/Toodle project in Android Studio, and run it on an API26 emulator

WebExtension

WebExtensions can't directly bind to native code. Instead, we use the native messaging API to send requests to a helper application, or "bridge", that embeds the Toodle library. The bridge manages a single Toodle store, translates requests into method calls, and communicates with the WebExtension via length-prefixed JSON messages over standard input and output. The WebExtension runtime automatically launches and terminates the bridge.

To build the bridge:

cd toodlext
cargo update
cargo build

The WebExtension runtime requires a native messaging manifest to allow the WebExtension to talk to the bridge. There's a helper Node script that installs a native manifest for a debug build of the bridge on macOS. This only needs to be done once. Please note that Windows and Linux use different locations, but the install script doesn't handle them yet.

cd webextension
./install-native-manifest.js

Then, to launch the WebExtension in a dedicated Firefox profile:

npm install
npm run dev

This also watches for source file changes, rebuilds, and automatically reloads the extension.

toodle's People

Contributors

Stargazers

Watchers

Forkers

jprider63 grigoryk eoger thomcc yarwelp mhammond rdr8 cnevinc ncalexan qpdb

toodle's Issues

[demo][stretch goal] Add UI for label creation to iOS

Define view creation vocabulary

When creating a view we need to be able to specify:

a particular schema field should be kept in cache
a complex query should be used to create a snapshot of data for querying.
schema cache versions to be tied to schema snippit version
complex query versions should be linked to schema snippit versions and have versions themselves
complex query versions should be able to be defined as JSON, Cached or SQL tables.

Update TxObservers to filter by more complex criteria

Currently TxObservers only handle a set of matching attributes to filter whether it wants to be notified about a set of transactions. We need to be able to specify a set of filter criteria to further exclude notifications from an object.

For clients this might be - Providing an entid so a notification will only get fired for a specific record and not for all records that touch an attribute.

For materialized views this might be aggregation, negation or comparison logic, i.e. only notify me about transactions that have a value less than a constant, or that change number of times a url has been viewed.

Microsecond/nanosecond timestamp calculation is probably wrong

See #26.

[demo][stretch goal] Add UI for label creation to Android

Update cache on write

Before we write we should update any cached values.

Move memory mapped schema values to use cache

The caches held in Schema should be migrated to use the new caches.
EntidMap
IdentMap
AttributeMap

Enable Rust library to be built with Bitcode enabled on iOS

Currently, building on Xcode with bitcode enabled fails as the Rust library is not built Bitcode compatible. This issue tracks enabling the library to be build with Bitcode enabled so we don't force iOS developers to disable Bitcode on their apps.

Materialized View Listener

When materialized views are created they need to register for change notifications with the transaction log observer.

This issue involves the creation of a standard materialized view transaction listener that registers for changes on the attributes that are included in the view.
On notification, the view should be destroyed and recreated.
Deregister listener on removal of view.

Rename all of Toodle's subcrates so they are more easily identifiable

It's currently hard to tell toodle subcrates apart from external dependencies. The subcrates should be renamed so that they are all prefixed with toodle_, so toodle_ffi_utils and toodle_store etc.

Separate caching of unique and non-unique attributes

Currently everything is stored in a Vec regardless of whether the values are unique or non-unique, causing a slow down in fetching as we are constantly referencing a vector. This issue covers creating 2 caches inside EagerCache, one for unique and one for non-unique attributes such that we can always use the most efficient caching for our attributes.

Specify release or debug in `install-native-manifest.js` to

Currently when running install-native-manifest.js the path that is used is hardcoded to the debug build path. Add ability to pass in --release or --debug to the script to set the path according to which library you want to run.

Use vocabulary management interface instead of directly transacting schema datoms

Here's how that looks (docs + test):

https://github.com/mozilla/mentat/blob/master/src/vocabulary.rs#L12

Cross platform logging crate

There are plenty of platform specific logging crates out there, but we want to log once and have it use the correct logging for all of our target platforms.

This issue tracks investigations into discovering existing crates that allow us to do this, or wrapping existing logging crates for Android, iOS, C++ and JS into a single crate that does platform detection.

API for defining complex queries as materialized views

Update transaction parser to accept vocabulary for creating a view from a complex query.

Should use query parser to parse query
Should use transaction parser for everything else.

Also create programmatic API for creating materialized views without having to do query parsing

Swift compatible C Header generation

Currently Cheader and it's ilk do not generate Swift compatible C headers. This issue tracks contributing to an existing C header generation tool in order to autogenerate C headers from FFI for Swift projects.

Update toodle to use var macro

Create FFI boundary for vocabulary management

We want to be able to management vocabulary directly from native code, not just from Rust so we need to create an FFI interface for it.

Add cache command to cli

Documentation!

Beef up API documentation such that there are lots of useful, easy to understand resources for new developers to go to for reference.

Use programmatic write API (EntityBuilder) and transactions

At present the Toodle store layer builds transaction strings and calls conn.transact multiple times. This is inefficient in several ways: string parsing, multiple SQL interactions, multiple commits, multiple fsyncs.

It should instead use EntityBuilder to construct terms, and use begin_transaction to couple a SQLite connection to the Conn to get an InProgress, which we later commit.

Here's how that looks:

https://github.com/mozilla/mentat/blob/master/src/entity_builder.rs#L413-L446

Debugging!

On iOS and Android

This issue tracks investigating how to make debugging a Rust library inside Android Studio or Xcode possible.

This issue is less important for Mentat as we don't expect developers using the Mentat crate to need to debug it!

Add timer to CLI

"Today" widget for either iOS or Android to showcase materialized views

Add "How to" Guides for client deployment to the README

How to build and deploy for Android, iOS and WebExtension.

[demo][stretch goal] Update webextension to add labels to items

Swift Library

This involves creating a Swift framework wrapping Mentat's FFI. The framework should be compatible with SwiftPackageManager, Carthage and Cocoapods.

Demo: Create branch of Toodle without `due_date` functionality for demo purposes.

One of the things we want to demo is the ability to append new data types to an app, and have that sync without it breaking all the other clients. In order to do that we should take a field we currently have,create a version of the apps without that field and then we can "add" the field back in for the demo.

We are using a field that already exists as it means we can reuse already written code in the demo rather than having to write new code which will take longer.

Android footprint size.

Currently (13/3/2018) the Android app footprint size is huge (70mb). We need to ensure that we can build Android apps with a sensible footprint. This ticket tracks discovering build options and streamlining our imports to reduce our release Android library size.

Android Toodle crashes on new_toodle (Rust panic) call after recent changes

Materialized View versioning

We need to ensure that:

We version check against schema fragment versions and error if versions are not correct
We allow multiple versions of the same view to be present as long as their schema fragment versions match

Expose basic "sync now" via functionality over FFI

Applications should be able to trigger a sync. The most basic version of this looks like pull-to-refresh in Toodle - or a "sync now" menu button.

A more complicated version on Android, which we might want to explore at some point, looks like SyncAdaptor integration - letting OS kick-off and schedule our syncs whenever appropriate for the device.

Add FFI for generic API

This involves exposing both the datalog query and transact mechanisms, InProgress and the vocabulary management and entity building APIs.

All associated structs, such as NamespacedKeyword and Variable will need to be dealt with in some way.

Create cross platform logging crate

We want to call log::debug, log::info & log::error from inside our Rust code and have it automatically use the correct logging tool for the architecture we are built for.

C++ XPCom library

Same as Android/Swift/JS but this time C++/XPCOM compatible.

Create FFI boundary for programmatic write API and transactions

We want to be able to use the write and transaction API from inside native code, not just from inside Rust so we need to create an FFI interface for it.

Create SQL view when Complex Query View defined

When a complex query view transaction statement is executed, a snapshot of the defined query results should be created and stored in an SQL table.

Query used to create view needs to be stored somewhere for updating later

Transaction Log Observer

Observe transaction log.
Allow Listeners to to register for notification on attributes.
Call correct listeners when changes to the values associated with registered attributes occur & provide changeset.
No not call listeners that are not affected by changes.

Cross compilation tooling

We need a bunch of tools to make cross compilation easy. Ideally this would include a script that can be run passing in only the platforms we want to set ourselves up on that will involve setting up the cross platform environments, and another for performing cross platform compilation for requested environments.

Reverse lookup unique identity cache

When a cached attribute is marked as unique identity, we should create a reverse map to go from value to entity. That’s useful for applications, but will also be useful for the transactor itself.

API for caching attribute values

Allow the caching of attribute data.

Create a mechanism by which a client can pass a keyword into mentat and allow that values against that keyword to be cached in memory.

Things to think about:

Do we want 2 levels of caching, eager and lazy? Or just one, in which case do we want either eager or lazy caching?
Do we want a cache per connection? If so, where should this go and how do we ensure that we don't have multiple copies of the same cache? If not, what is the best way of sharing that cache between threads?

Allow clients to fetched cached data
How do we include an in memory value in a filter?

Create FFI for caching

Caching needs to be usable from native code, not just Rust code. An FFI interface needs to be created.