shekohex / allo-isolate Goto Github PK

View Code? Open in Web Editor NEW

116.0 116.0 18.0 110 KB

Run Multithreaded Rust along with Dart VM (in isolate) 🌀

License: Apache License 2.0

Rust 97.34% Shell 0.34% Nix 2.32%

dart ffi flutter rust-lang

allo-isolate's People

Contributors

Stargazers

Watchers

Forkers

cytrinox hoangpq rustworks jerel icota fzyzcjy raphaelrobert trueb2 sagudev trobanga ohir lattice0 wooden-worm meta-flutter desdaemon iota9star nightfeather0615

allo-isolate's Issues

when zero-copy is enabled, shall we mark `ZeroCopyBuffer` as deprecated, and provide a NonZeroCopyBuffer

Just some nits that comes to my mind (not of high priority)!

shall we mark ZeroCopyBuffer as deprecated - otherwise it is confusing as it is no-op
provide a NonZeroCopyBuffer - otherwise there is no way to test/use non-zero copy

bug: DartCObject does not handle cases except Int8/Uint8, making memory leak

Hi thanks for the lib. I see DartCObject does not handle cases except Int8/Uint8, making memory leak. It can be clearly seen from the code:

https://github.com/sunshine-protocol/allo-isolate/blob/d10c582aba2a41f65889f1270f009b4c9a4c899b/src/ffi.rs#L161

possibly, a panic can be made instead of no-op.

Publish a new version?

Hi thanks for this lib! It has been days since some PRs I made ;) So I wonder when will we have the new version? Seems that I cannot use the git dependency, otherwise the following error happens when publishing to crate:

error: all dependencies must have a version specified when publishing.
dependency allo-isolate does not specify a version
Note: The published dependency will use the version from crates.io,
the git specification will be removed from the dependency declaration.

Is memory copy involved when sending `Uint8List` from Dart to Rust?

Hello, hope this project is doing well. I wanted to ask a question in discussions, but since I couldn't create a thread I write here.

I had a simple question just like the title says. I thought that, Uint8List would be copied before being sent to Rust as Vec<u8> because mutating the bytes in Rust should be safe, even if the Uint8List at the Dart side is not garbage collected yet. DId I get it right?

Is there any way to run a Rust function on Dart VM shutdown?

I have some resources such as databases on the Rust side. When Dart VM gets shut down, perhaps because the user closed the Flutter app, the Rust finalizer functions that destroy Rust resources should be run.

Is this possible with allo-isolate, or can be made possible with a PR?

Passing structs through an isolate.post()

@shekohex I've looked through dart-bindgen, allo_isolate, and the flutterrust projects and one example that I can't find is that of passing a struct to an isolate. Is that currently possible?

Example:

struct Location {
  lat: f64,
  lng: f64,
  name: String,
}

let location = Location {
  lat: 20.1,
  lng: 52.2,
  name: String::from("test"),
};

let isolate = Isolate::new(port);
isolate.post(location);

Custom types from library result in: `Cell<bool>` cannot be shared between threads safely

Description:
I'm trying to implement the iota/identity library into flutter+rust. Most of the functions are asynchronous so I'm using allo-isolate. But some custom types of the IOTA library cannot be shared between threads safely.

My async function which mostly uses custom types:

pub async fn create_did_document() -> Result<String> {
    // Generate a new DID Document and public/private key pair.
    // The generated document will have an authentication key associated with
    // the keypair.
    let client = &Client::new().await?;
    let keypair: KeyPair = KeyPair::new_ed25519()?;
    let mut document: IotaDocument = IotaDocument::from_keypair(&keypair)?;

    // Sign the DID Document with the default authentication key.
    document.sign(keypair.secret())?;

    // Use the Client to publish the DID Document to the Tangle.
    document.publish(client).await?;

    println!("DID document: {}", document);
    println!("Transaktion link: {}", document.message_id());
    
    // Return document and keypair.
    Ok(document.to_json().unwrap())
}

Exposing the async create_did_document function:

lazy_static! {
    static ref RUNTIME: io::Result<Runtime> = Builder::new()
        .threaded_scheduler()
        .enable_all()
        .core_threads(4)
        .thread_name("flutterust")
        .build();
}

/// Simple Macro to help getting the value of the runtime.
macro_rules! runtime {
    () => {
        match RUNTIME.as_ref() {
            Ok(rt) => rt,
            Err(_) => {
                return 0;
            }
        }
    };
}

#[no_mangle]
pub extern "C" fn create_did_document(port: i64) -> i32 {
    let rt = runtime!();
    let t = Isolate::new(port).task(iota_did::create_did_document()); // not satisfied
    rt.spawn(t);
    1
}

Full error message:

`Cell<bool>` cannot be shared between threads safely
within `identity_iota::did::doc::iota_document::IotaDocument`, the trait `Sync` is not implemented for `Cell<bool>`
required because it appears within the type `identity_core::crypto::signature::signature::Signature`
required because it appears within the type `Option<identity_core::crypto::signature::signature::Signature>`
required because it appears within the type `identity_did::verifiable::properties::Properties<identity_iota::did::doc::properties::Properties>`
required because it appears within the type `identity_did::document::core_document::CoreDocument<identity_did::verifiable::properties::Properties<identity_iota::did::doc::properties::Properties>>`
required because it appears within the type `identity_iota::did::doc::iota_document::IotaDocument`
required because of the requirements on the impl of `Send` for `&identity_iota::did::doc::iota_document::IotaDocument`
required because it appears within the type `for<'r, 's, 't0, 't1, 't2, 't3> {ResumeTy, &'r mut identity_iota::did::doc::iota_document::IotaDocument, &identity_iota::tangle::client::Client, identity_iota::tangle::network::Network, Option<&'s identity_iota::tangle::client::Client>, &'t0 identity_iota::tangle::client::Client, &'t1 identity_iota::did::doc::iota_document::IotaDocument, impl std::future::Future, (), impl std::future::Future, Result<identity_iota::tangle::client::Client, identity_iota::error::Error>, identity_iota::tangle::client::Client}`
required because it appears within the type `[static generator@identity_iota::did::doc::iota_document::IotaDocument::publish<'_, &identity_iota::tangle::client::Client>::{closure#0} for<'r, 's, 't0, 't1, 't2, 't3> {ResumeTy, &'r mut identity_iota::did::doc::iota_document::IotaDocument, &identity_iota::tangle::client::Client, identity_iota::tangle::network::Network, Option<&'s identity_iota::tangle::client::Client>, &'t0 identity_iota::tangle::client::Client, &'t1 identity_iota::did::doc::iota_document::IotaDocument, impl std::future::Future, (), impl std::future::Future, Result<identity_iota::tangle::client::Client, identity_iota::error::Error>, identity_iota::tangle::client::Client}]`
required because it appears within the type `from_generator::GenFuture<[static generator@identity_iota::did::doc::iota_document::IotaDocument::publish<'_, &identity_iota::tangle::client::Client>::{closure#0} for<'r, 's, 't0, 't1, 't2, 't3> {ResumeTy, &'r mut identity_iota::did::doc::iota_document::IotaDocument, &identity_iota::tangle::client::Client, identity_iota::tangle::network::Network, Option<&'s identity_iota::tangle::client::Client>, &'t0 identity_iota::tangle::client::Client, &'t1 identity_iota::did::doc::iota_document::IotaDocument, impl std::future::Future, (), impl std::future::Future, Result<identity_iota::tangle::client::Client, identity_iota::error::Error>, identity_iota::tangle::client::Client}]>`
required because it appears within the type `impl std::future::Future`
required because it appears within the type `impl std::future::Future`
required because it appears within the type `for<'r, 's, 't0, 't1, 't2, 't3> {ResumeTy, impl std::future::Future, (), identity_iota::tangle::client::Client, &'r identity_iota::tangle::client::Client, identity_core::crypto::key::pair::KeyPair, identity_iota::did::doc::iota_document::IotaDocument, &'s mut identity_iota::did::doc::iota_document::IotaDocument, impl std::future::Future}`
required because it appears within the type `[static generator@iota_did::create_did_document::{closure#0} for<'r, 's, 't0, 't1, 't2, 't3> {ResumeTy, impl std::future::Future, (), identity_iota::tangle::client::Client, &'r identity_iota::tangle::client::Client, identity_core::crypto::key::pair::KeyPair, identity_iota::did::doc::iota_document::IotaDocument, &'s mut identity_iota::did::doc::iota_document::IotaDocument, impl std::future::Future}]`
required because it appears within the type `from_generator::GenFuture<[static generator@iota_did::create_did_document::{closure#0} for<'r, 's, 't0, 't1, 't2, 't3> {ResumeTy, impl std::future::Future, (), identity_iota::tangle::client::Client, &'r identity_iota::tangle::client::Client, identity_core::crypto::key::pair::KeyPair, identity_iota::did::doc::iota_document::IotaDocument, &'s mut identity_iota::did::doc::iota_document::IotaDocument, impl std::future::Future}]>`
required because it appears within the type `impl std::future::Future`
required because it appears within the type `impl std::future::Future`

Full Code:
This is the full repo of my project with the corresponding rust implementation of iota-identity. my_project

Implicit zero copy on `Vec<u8>`

Hi, thank you very much for this library. This post is an idea rather than an issue.

I've been wondering why we would need to copy Vec<u8> when we have the ability to 'zero copy' the bytes array. What if we could just zero copy Vec<u8> types by default without explicit ZeroCopyBuffer? If I can contribute, would it be okay to work on this idea?

bug: Cannot use allo-isolate in two libraries with static linking because of symbol name duplication

When a user is using two packages having allo-isolate, and doing static linking, the symbol store_dart_post_cobject is used in two Rust crates, and thus it seems that the linker will throw away one symbol completely. Then, when calling store_dart_post_cobject, only one of the two Rust crates will actually store that pointer. The other crate will never store it, and thus allo-isolate cannot work at all.

/cc @AlexV525

bug: Memory leak using Dart_PostCObject because need to consider the return value is true or false

Hi thanks for the lib! I suspect there is memory leak when using Dart_PostCObject. Please refer to: dart-lang/sdk#47270 (comment)

Sanitizer errors

RUSTFLAGS="-Z sanitizer=memory" cargo bench --bench uuid --features uuid --target x86_64-unknown-linux-gnu

Generates

==645733==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x561346a35b2d in std::sys_common::once::futex::Once::call::ha1b7ce84c30f3f69 (/b/allo-isolate/target/x86_64-unknown-linux-gnu/release/deps/uuid-057c435e97cecd96+0x2cb2d) (BuildId: 8f7ad90da4aa620babb32b2e687b840b40299b6b)
    #1 0x561346b8dc8b in _$LT$criterion..Criterion$u20$as$u20$core..default..Default$GT$::default::h496c49941cd756f4 (/b/allo-isolate/target/x86_64-unknown-linux-gnu/release/deps/uuid-057c435e97cecd96+0x184c8b) (BuildId: 8f7ad90da4aa620babb32b2e687b840b40299b6b)
    #2 0x561346ada341 in uuid::main::hd193fa060419cf1a (/b/allo-isolate/target/x86_64-unknown-linux-gnu/release/deps/uuid-057c435e97cecd96+0xd1341) (BuildId: 8f7ad90da4aa620babb32b2e687b840b40299b6b)
    #3 0x561346aed462 in std::sys_common::backtrace::__rust_begin_short_backtrace::h5647b6428466359c (/b/allo-isolate/target/x86_64-unknown-linux-gnu/release/deps/uuid-057c435e97cecd96+0xe4462) (BuildId: 8f7ad90da4aa620babb32b2e687b840b40299b6b)
    #4 0x561346b2650b in std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h5d19866072bdbf27 (.llvm.14944981500331809237) (/b/allo-isolate/target/x86_64-unknown-linux-gnu/release/deps/uuid-057c435e97cecd96+0x11d50b) (BuildId: 8f7ad90da4aa620babb32b2e687b840b40299b6b)
    #5 0x561346f3f6ea in core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::hcf0f452e35b42714 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/core/src/ops/function.rs:286:13
    #6 0x561346f3f6ea in std::panicking::try::do_call::h9076463b86fc98c2 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panicking.rs:464:40
    #7 0x561346f3f6ea in std::panicking::try::h59a38c38fab81132 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panicking.rs:428:19
    #8 0x561346f3f6ea in std::panic::catch_unwind::h024c7f29e5fae637 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panic.rs:137:14
    #9 0x561346f3f6ea in std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::h1bc9c4aefbf9b583 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/rt.rs:148:48
    #10 0x561346f3f6ea in std::panicking::try::do_call::hf630db2c23823f3c /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panicking.rs:464:40
    #11 0x561346f3f6ea in std::panicking::try::h5c6c892f05382366 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panicking.rs:428:19
    #12 0x561346f3f6ea in std::panic::catch_unwind::h893ebad4c00d9e60 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panic.rs:137:14
    #13 0x561346f3f6ea in std::rt::lang_start_internal::h2fa3ed5bcd048bb3 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/rt.rs:148:20
    #14 0x561346adadf7 in main (/b/allo-isolate/target/x86_64-unknown-linux-gnu/release/deps/uuid-057c435e97cecd96+0xd1df7) (BuildId: 8f7ad90da4aa620babb32b2e687b840b40299b6b)
    #15 0x7f3cd0fcfb74 in __libc_start_main (/lib64/libc.so.6+0x27b74) (BuildId: 08df60634339b221bb854d4e10b7278cafde70c4)
    #16 0x561346a5cbdd in _start (/b/allo-isolate/target/x86_64-unknown-linux-gnu/release/deps/uuid-057c435e97cecd96+0x53bdd) (BuildId: 8f7ad90da4aa620babb32b2e687b840b40299b6b)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/b/allo-isolate/target/x86_64-unknown-linux-gnu/release/deps/uuid-057c435e97cecd96+0x2cb2d) (BuildId: 8f7ad90da4aa620babb32b2e687b840b40299b6b) in std::sys_common::once::futex::Once::call::ha1b7ce84c30f3f69
Exiting

gnuRUSTFLAGS="-Z sanitizer=memory" cargo test --target x86_64-unknown-linux-gnu

Generates

     Running unittests src/lib.rs (target/x86_64-unknown-linux-gnu/debug/deps/allo_isolate-e78f1fbba8eee840)
Uninitialized bytes in MemcmpInterceptorCommon at offset 0 inside [0x708000000081, 3)
==650316==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x557aff1fb68d in bcmp /rustc/llvm/src/llvm-project/compiler-rt/lib/msan/../sanitizer_common/sanitizer_common_interceptors.inc:978:10
    #1 0x557aff2b1ca0 in _$LT$std..path..Component$u20$as$u20$core..cmp..PartialEq$GT$::eq::hca69560419389308 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/path.rs
    #2 0x557aff2ad6c7 in core::cmp::impls::_$LT$impl$u20$core..cmp..PartialEq$LT$$RF$B$GT$$u20$for$u20$$RF$A$GT$::eq::h85b3b6b1f9b934b5 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/path.rs:292:43
    #3 0x557aff2ad6c7 in std::path::iter_after::h9743549dd819bdf2 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/path.rs:292:43
    #4 0x557aff2ad6c7 in std::path::Path::_starts_with::h1e354f9bbbd5797a /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/path.rs:2307:9
    #5 0x557aff2a6fb7 in std::path::Path::starts_with::h15703652269aa2f9 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/path.rs:2303:9
    #6 0x557aff2a6fb7 in std::sys::unix::thread::cgroups::quota_v2::he02b771439b16a7b /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/sys/unix/thread.rs:481:19
    #7 0x557aff2a6fb7 in std::sys::unix::thread::cgroups::quota::h3f141d9dbf7bcfda /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/sys/unix/thread.rs:452:31
    #8 0x557aff2a6fb7 in std::sys::unix::thread::available_parallelism::h194b1eac50ae29db /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/sys/unix/thread.rs:294:29
    #9 0x557aff2a6fb7 in std::thread::available_parallelism::h2eebd5ecaf5da702 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/thread/mod.rs:1655:5
    #10 0x557aff271fa3 in test::helpers::concurrency::get_concurrency::h87ad303fc8483364 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/test/src/helpers/concurrency.rs:12:9
    #11 0x557aff2683f4 in core::ops::function::FnOnce::call_once::hbde7220a563542ac /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/core/src/ops/function.rs:251:5
    #12 0x557aff2683f4 in core::option::Option$LT$T$GT$::unwrap_or_else::hb273e30f102261ab /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/test/src/console.rs:269:28
    #13 0x557aff2683f4 in test::console::run_tests_console::h9a3d180b51e649a8 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/test/src/console.rs:269:28
    #14 0x557aff27d68c in test::test_main::h65a820c41c08cb2f /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/test/src/lib.rs:114:15
    #15 0x557aff27e653 in test::test_main_static::he901575514289151 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/test/src/lib.rs:133:5
    #16 0x557aff2521a1 in allo_isolate::main::h60aeada2def2916a /b/allo-isolate/src/lib.rs:1:1
    #17 0x557aff2527ec in core::ops::function::FnOnce::call_once::h7f928e1e99b1730d /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/core/src/ops/function.rs:251:5
    #18 0x557aff252372 in std::sys_common::backtrace::__rust_begin_short_backtrace::h2ed475e7e98b249d /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/sys_common/backtrace.rs:122:18
    #19 0x557aff2525c1 in std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::hfc48b97e61d99b42 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/rt.rs:166:18
    #20 0x557aff2a53da in core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::hcf0f452e35b42714 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/core/src/ops/function.rs:286:13
    #21 0x557aff2a53da in std::panicking::try::do_call::h9076463b86fc98c2 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panicking.rs:464:40
    #22 0x557aff2a53da in std::panicking::try::h59a38c38fab81132 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panicking.rs:428:19
    #23 0x557aff2a53da in std::panic::catch_unwind::h024c7f29e5fae637 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panic.rs:137:14
    #24 0x557aff2a53da in std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::h1bc9c4aefbf9b583 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/rt.rs:148:48
    #25 0x557aff2a53da in std::panicking::try::do_call::hf630db2c23823f3c /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panicking.rs:464:40
    #26 0x557aff2a53da in std::panicking::try::h5c6c892f05382366 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panicking.rs:428:19
    #27 0x557aff2a53da in std::panic::catch_unwind::h893ebad4c00d9e60 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/panic.rs:137:14
    #28 0x557aff2a53da in std::rt::lang_start_internal::h2fa3ed5bcd048bb3 /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/rt.rs:148:20
    #29 0x557aff252495 in std::rt::lang_start::h2717c68b38bf1baa /rustc/8796e7a9cfd4c5c4f1de15ec1c53994ddf288665/library/std/src/rt.rs:165:17
    #30 0x557aff252209 in main (/b/allo-isolate/target/x86_64-unknown-linux-gnu/debug/deps/allo_isolate-e78f1fbba8eee840+0x81209) (BuildId: 8278091c6966f0ea2a3174ad8efd3574c1f327a9)
    #31 0x7f74a3936b74 in __libc_start_main (/lib64/libc.so.6+0x27b74) (BuildId: 08df60634339b221bb854d4e10b7278cafde70c4)
    #32 0x557aff1ec73d in _start (/b/allo-isolate/target/x86_64-unknown-linux-gnu/debug/deps/allo_isolate-e78f1fbba8eee840+0x1b73d) (BuildId: 8278091c6966f0ea2a3174ad8efd3574c1f327a9)

SUMMARY: MemorySanitizer: use-of-uninitialized-value /rustc/llvm/src/llvm-project/compiler-rt/lib/msan/../sanitizer_common/sanitizer_common_interceptors.inc:978:10 in bcmp
Exiting

Support Vec<Box<XXX>>

This issue is related to flutter_rust_bridge

The official website, Sunshine.foundation, says that domain is being sold...

Hi when looking around, I see the official website linked from here https://github.com/sunshine-protocol. But it says it is being sold. So maybe the link needs to be updated ;)

Directly find out Dart_PostCObject function pointer, instead of manually letting Dart to call Rust's store_dart_post_cobject?

Hi thanks for the lib! Looking at https://github.com/dart-lang/sdk/blob/master/runtime/include/dart_api_dl.h , I guess a C++ programmer will simply #include this header and then use the extern Dart_PostCObject_Type Dart_PostCObject_DL; directly. So what about removing store_dart_post_cobject and directly do that?

I can make a PR.

related: dart-lang/sdk#47356

Support `Float32List`? (Can make a PR)

I need Float32List for things like drawRawAtlas; I can make a PR for this.

Streaming data

I am trying to stream data from Rust to Dart.
I am listening to CAN frames and when I receive one, I post it into Isolate, but the problem is that data is not streamed to Dart, instead it is sent in batch after the function is completed. Because of this kind of behavior, I am trying to use a workaround on Dart's side where I call the function over and over again after I receive data from CAN, but it leads to missing some CAN frames which are really important. Here is the code on the Rust side:

extern crate mio;
extern crate mio_serial;

use allo_isolate::Isolate;
use futures_util::stream::StreamExt;
use tokio_socketcan::{CANSocket, Error};

#[no_mangle]
pub extern "C" fn read_data(isolate_port: i64) {
    read_can(isolate_port);
}

#[tokio::main]
async fn read_can(isolate_port: i64) -> Result<(), Error> {
    let isolate = Isolate::new(isolate_port);

    let mut socket_rx = CANSocket::open("vcan0")?;

    while let Some(Ok(frame)) = socket_rx.next().await {
        let data = frame.data();
        /*
         Here I perform some data manipulation
       */
        isolate.post(format!("{:?}", can_frame_map));
        break; // I had to explicitly add break here since it was not posting any values until function finishes
    }
    Ok(())
}

And here is the data listener class:

class DataListener {
  static setup() {
    native.store_dart_post_cobject(NativeApi.postCObject);
    print("Port listener Setup Done");
  }

  Stream listenData() {
    print('Creating receive port');
    final receivePort = ReceivePort();
    print('Starting listen port');
    native.listen_port(
      receivePort.sendPort.nativePort,
    );
    print('Returning stream');
    return receivePort; //It doesn't matter if I return receivePort and listen to it later or start listening straight away in the function. The behaviour is still the same
  }
}

This is the function on the Dart's side which calls for the listenData function:

static void readData(SendPort sendPort) async {
    DataListener.setup();
    while (true) {
      Stream<dynamic> canStream = DataListener().listenData();
      sendPort.send(await canStream.first); //Sending to another isolate for processing
     // Even without waiting for the first element of the stream, the behavior is the same. It is pretty much just a silly workaround
    }
  }

So I basically want to create an infinite loop for streaming data from Rust to Dart. Or is it even possible to do with Allo isolate?

How is the library tested? (I cannot run it via Valgrind + real Dart)

Since this lib interacts with Dart, it is very important to test it with real Dart. Otherwise, you know, memory problems can exist and cause big troubles. I have tried to use Valgrind on a (real) Dart + rust but failed. So how do you guys to it? Thanks

bug: critical DOUBLE-FREE problem for all calls to post()

Please correct me if I am wrong. Suppose we call post with some vec of u8.

Looking at

https://github.com/sunshine-protocol/allo-isolate/blob/d10c582aba2a41f65889f1270f009b4c9a4c899b/src/lib.rs#L117-L125
the logic is:

create raw pointer of that DartCObject (containing a big array)
call Dart_PostCObject function
use Rust's logic to free the Box of that DartCObject

When drop(boxed_obj) it will call the following:

https://github.com/sunshine-protocol/allo-isolate/blob/d10c582aba2a41f65889f1270f009b4c9a4c899b/src/ffi.rs#L152-L184
So, since it is a DartTypedData, will call let _ = unsafe { Vec::from_raw_parts(...) }. Then when the _ variable leaves the scope, Rust will drop that Vec. In other words, the big array in memory is dropped (once) here.

However, on the other hand, consider what Dart_PostCObject says: dart-lang/sdk#47270 (comment)

If true is returned, the message was enqueued, and finalizers for external

typed data will eventually run, even if the receiving isolate shuts down

before processing the message. If false is returned, the message was not

enqueued and ownership of external typed data in the message remains with the

caller.

To begin with, there do definitely exist a possible issue here: #6

However, more critical issue is double free. Notice that, the callback tied with typed data will finally be called (if Dart_PostCObject returns true - which is most of the cases). And we do https://github.com/sunshine-protocol/allo-isolate/blob/d10c582aba2a41f65889f1270f009b4c9a4c899b/src/into_dart.rs#L205 . In other words, we call deallocate_rust_buffer which is drop(Vec::from_raw_parts(ptr, len, len));. Hey, this is another free of the big array!

So we have double free! Dangerous!

Possibly related: #3
@jerel @shekohex