rusterlium / erlang_nif-sys Goto Github PK

View Code? Open in Web Editor NEW

89.0 14.0 20.0 861 KB

Low level bindings to Erlang NIF API for Rust

License: Apache License 2.0

Rust 57.39% Erlang 41.97% C 0.64%

rust erlang nif ffi

erlang_nif-sys's People

Contributors

Stargazers

Watchers

Forkers

linearregression hansihe tatsuya6502 agelin tsloughter ortuna id3as cybernetlab jorendorff zeam-vm robulik philipstears collision02 torifaye filmor otolonetworks elbow-jason davidalphafox

erlang_nif-sys's Issues

update NIF API to erlang 19.0

There were changes in the NIF API from 19.0-rc2 to 19.0. Erlang_nif-sys is only up to 19.0-rc2, so update to 19.0

With Erlang/OTP 17.5 from Homebrew and Rust 1.0.0 installing using their install script, building ruster_unsafe is fine, but building the ruster_unsafe_demo does not work (Daniel suggested I file this issue here rather than the demo).

$ cargo build --verbose
   Compiling ruster_unsafe v0.1.0
     Running `rustc /Users/nfiedler/.cargo/registry/src/github.com-1ecc6299db9ec823/ruster_unsafe-0.1.0/build.rs --crate-name build_script_build --crate-type bin -C prefer-dynamic -g --out-dir /Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/build/ruster_unsafe-421b52893b529f99 --emit=dep-info,link -L dependency=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps -L dependency=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps -Awarnings`
   Compiling libc v0.1.8
     Running `rustc /Users/nfiedler/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.1.8/rust/src/liblibc/lib.rs --crate-name libc --crate-type lib -g --cfg feature="cargo-build" --cfg feature="default" -C metadata=2eda841eb12a3090 -C extra-filename=-2eda841eb12a3090 --out-dir /Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps --emit=dep-info,link -L dependency=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps -L dependency=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps -Awarnings`
     Running `/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/build/ruster_unsafe-421b52893b529f99/build-script-build`
     Running `rustc /Users/nfiedler/.cargo/registry/src/github.com-1ecc6299db9ec823/ruster_unsafe-0.1.0/src/lib.rs --crate-name ruster_unsafe --crate-type lib -g -C metadata=421b52893b529f99 -C extra-filename=-421b52893b529f99 --out-dir /Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps --emit=dep-info,link -L dependency=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps -L dependency=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps --extern libc=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps/liblibc-2eda841eb12a3090.rlib -Awarnings`
   Compiling ruster_unsafe_demo v0.0.1 (file:///Users/nfiedler/Downloads/ruster_unsafe_demo)
     Running `rustc src/lib.rs --crate-name ruster_unsafe_demo --crate-type dylib -g --out-dir /Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug --emit=dep-info,link -L dependency=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug -L dependency=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps --extern ruster_unsafe=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps/libruster_unsafe-421b52893b529f99.rlib`
error: linking with `cc` failed: exit code: 1
note: "cc" "-m64" "-L" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib" "-o" "/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/libruster_unsafe_demo.dylib" "/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/ruster_unsafe_demo.o" "-Wl,-force_load,/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libmorestack.a" "/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/ruster_unsafe_demo.metadata.o" "-Wl,-dead_strip" "-nodefaultlibs" "/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps/libruster_unsafe-421b52893b529f99.rlib" "/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps/liblibc-2eda841eb12a3090.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libstd-4e7c5e5c.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libcollections-4e7c5e5c.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libunicode-4e7c5e5c.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/librand-4e7c5e5c.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/liballoc-4e7c5e5c.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/liblibc-4e7c5e5c.rlib" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib/libcore-4e7c5e5c.rlib" "-L" "/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug" "-L" "/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps" "-L" "/usr/local/lib/rustlib/x86_64-apple-darwin/lib" "-L" "/Users/nfiedler/Downloads/ruster_unsafe_demo/.rust/lib/x86_64-apple-darwin" "-L" "/Users/nfiedler/Downloads/ruster_unsafe_demo/lib/x86_64-apple-darwin" "-lc" "-lm" "-lSystem" "-lpthread" "-lc" "-lm" "-dynamiclib" "-Wl,-dylib" "-lcompiler-rt"
note: ld: warning: directory not found for option '-L/Users/nfiedler/Downloads/ruster_unsafe_demo/.rust/lib/x86_64-apple-darwin'
ld: warning: directory not found for option '-L/Users/nfiedler/Downloads/ruster_unsafe_demo/lib/x86_64-apple-darwin'
Undefined symbols for architecture x86_64:
  "_enif_get_tuple", referenced from:
      tuple_add::__rust_abi in ruster_unsafe_demo.o
  "_enif_get_int", referenced from:
      native_add::__rust_abi in ruster_unsafe_demo.o
      tuple_add::__rust_abi in ruster_unsafe_demo.o
  "_enif_make_int", referenced from:
      native_add::__rust_abi in ruster_unsafe_demo.o
      tuple_add::__rust_abi in ruster_unsafe_demo.o
  "_enif_make_badarg", referenced from:
      native_add::__rust_abi in ruster_unsafe_demo.o
      tuple_add::__rust_abi in ruster_unsafe_demo.o
  "_enif_make_atom", referenced from:
      load::__rust_abi in ruster_unsafe_demo.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

error: aborting due to previous error
Could not compile `ruster_unsafe_demo`.

Caused by:
  Process didn't exit successfully: `rustc src/lib.rs --crate-name ruster_unsafe_demo --crate-type dylib -g --out-dir /Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug --emit=dep-info,link -L dependency=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug -L dependency=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps --extern ruster_unsafe=/Users/nfiedler/Downloads/ruster_unsafe_demo/target/debug/deps/libruster_unsafe-421b52893b529f99.rlib` (exit code: 101)

$ cat /usr/local/lib/erlang/releases/17/OTP_VERSION
17.5

$ rustc --version
rustc 1.0.0 (a59de37e9 2015-05-13) (built 2015-05-14)

$ uname -a
Darwin mac-nfiedler 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64 i386 MacPro5,1 Darwin

Please let me know if there is any other information I can provide. As time allows, I will continue trying to debug this and report anything I find. Thanks.

ignore

`enif_make_map_from_arrays` not in documentation

The recently added function enif_make_map_from_arrays does not appear in the documentation.

rebar3_rust link not found

where is gone this plugin? https://github.com/sdwolf/rebar3_rust return a not_found error now.

Unsupported Erlang version (erl 20)

i've got this error using rustler crate:

--- stdout
Unsupported Erlang version.

my erlang version:

Erlang/OTP 20 [RELEASE CANDIDATE 2] [erts-9.0] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V9.0  (abort with ^G)

erlang_nif-sys version from Cargo.lock

[[package]]
name = "erlang_nif-sys"
version = "0.6.1"

better build errors

Give useful build errors when escript or cc are not available.

ErlNifEnv should be an empty type

It is supposed to be opaque. It should at least be of size 0.

erlang_nif-sys is not up to date

I received the following error:

error: failed to run custom build command for erlang_nif-sys v0.6.4

Unsupported Erland version.

Is the erlang_nif-sys version up to date in the Cargo.toml?
Does 'cargo update' fix it?
If not please report at https://github.com/goertzenator/erlang_nif-sys.

cargo update does not fix this.

Erlang 22 support

Erlang 22 is out (and in my case, my linux distro has updated), and with it comes NIF version 2.15. This means that NIFs won't build even with the latest erlang_nif-sys.

ignore

Compile error when building with Cargo

error: failed to run custom build command for `erlang_nif-sys v0.6.4`

Caused by:
  process didn't exit successfully: `/Users/julia.boeira/Documents/bercode/native/io/target/debug/build/erlang_nif-sys-1d4db100cb37e287/build-script-build` (exit code: 101)
  --- stdout
  Unsupported Erlang version.

  Is the erlang_nif-sys version up to date in the Cargo.toml?
  Does 'cargo update' fix it?
  If not please report at https://github.com/goertzenator/erlang_nif-sys.

  --- stderr
  thread 'main' panicked at 'gen_api.erl encountered an error.', /Users/julia.boeira/.cargo/registry/src/github.com-1ecc6299db9ec823/erlang_nif-sys-0.6.4/build.rs:28:22
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

Windows API optimization

Consider using unchecked_unwrap from https://crates.io/crates/unreachable to optimized Windows API calls.

reexport libc types

Reexport libc types as per this reddit discussion.

0.6.5 not found in the crates.io, and neither the github repo tag.

improve version config scheme

After PR #6, it is now evident that the version selection scheme in gen_api.erl is going to start looking ugly before too many more NIF revisions.

Quick ideas to think more about later when I have time...

Use when X>Y logic to select APIs.
Add an intermediate layer that explodes a NIF version number into a set of flags for selecting APIs.

wrap optional pointer parameters with Option

Optional pointer parameters should be wrapped with Option. This is an API breaking change, so disposition is to wait until there are additional reasons to break the API before making this change.

Known instances:

tried parameter in enif_open_resource_type().

ErlNifBinary.data should be mut u8 instead of const u8 (?)

I am trying to port vinoski/bitwise NIF examples to Rust, but I found that Rust compiler does not let me update the contents of the data field of ErlNifBinary.

I am not totally sure as I just started to play with Rust a few days ago, but it looks like the data field should be defined as *mut u8 instead of *const u8?

ruster_unsafe/src/lib.rs

pub struct ErlNifBinary {
    pub size: size_t,
    pub data: *const u8,
    bin_term: ERL_NIF_TERM,
    ref_bin: *mut c_void,
}

Here is relevant code from my program. (Ported from bitwise_nif.c#L14-L38)

/// Erlang: -spec exor(binary(), byte::0..255) -> binary().
///
/// exor misbehaves on a regular scheduler thread when the incomng binary is
/// large because it blocks the thread for too long. But it works fine on a
/// dirty scheduler.
extern "C" fn exor(env: *mut ErlNifEnv,
                   argc: c_int,
                   args: *const ERL_NIF_TERM) -> ERL_NIF_TERM {
    unsafe {
        let mut bin: ErlNifBinary = uninitialized();
        let mut outbin: ErlNifBinary = uninitialized();
        let mut val: c_uint = uninitialized();

        if argc != 2
            || 0 == enif_inspect_binary(env, *args, &mut bin)
            || 0 == enif_get_uint(env, *args.offset(1), &mut val)
            || val > 255 {
            return enif_make_badarg(env);
        }
        if bin.size == 0 {
            return *args;
        }
        enif_alloc_binary(bin.size, &mut outbin);
        do_xor(bin.data, bin.size, outbin.data, val as u8);  // <-- This line does not compile.

        // @TODO: Implement enif_make_tuple2() and friends in ruster_unsafe
        // enif_make_tuple2(env,
        //                  enif_make_binary(env, &mut outbin),
        //                  enif_make_int(env, 0))
        enif_make_binary(env, &mut outbin)
    }
}

fn do_xor(bin_data: *const u8, length: usize, outbin_data: *mut u8, byte: u8) {
    let bin_slice: &[u8] = unsafe {slice::from_raw_parts(bin_data, length)};
    let outbin_slice: &mut[u8] = unsafe {slice::from_raw_parts_mut(outbin_data, length)};

    for (b, ob) in bin_slice.iter().zip(outbin_slice.iter_mut()) {
        (*ob) = (*b) ^ byte;
    }
}

And the compile error:

src/lib.rs:75:36: 75:47 error: mismatched types:
 expected `*mut u8`,
    found `*const u8`
(values differ in mutability) [E0308]
src/lib.rs:75         do_xor(bin.data, bin.size, outbin.data, val as u8);
                                                 ^~~~~~~~~~~
src/lib.rs:75:36: 75:47 help: run `rustc --explain E0308` to see a detailed explanation
error: aborting due to previous error
Could not compile `bitwise_rust_erlang`.

If I change the data field to *mut u8, it will compile.

% cargo build --release
   Compiling libc v0.2.4
   Compiling ruster_unsafe v0.2.0 (https://github.com/tatsuya6502/ruster_unsafe/?rev=nif-2.9#2f68285f)
   Compiling bitwise_rust_erlang v0.1.0 (file:///usr/home/tatsuya/workhub/dev/bitwise-rust-erlang)
% ls -lh target/release/lib*.so
-rwxr-xr-x  1 tatsuya  tatsuya   2.0M Jan  7 21:36 target/release/libbitwise_nif.so

ErlNifEnv mutability changes

This case proposes two changes around ErlNifEnv:

Remove ErlNifEnv’s Sync trait. ErlNifEnv is an opaque type and it’s initial naive implementation happens to be Sync. The NIF documentation indicates it is not thread safe, therefore it should not have Sync. Send is okay though.
This one is more radical: Change usages of *mut ErlNifEnv to *const ErlNifEnv in all API functions. Both Ruster and Rustler use const references in their env types because it better represents the allowed usages of ErlNifEnv. Changing the mutability of the underlying pointer in erlang_nif-sys would give other users a better cue of how to use that type, and allow less transmuting/wrapping between erlang_nif-sys and Ruster/Rustler. I should add that ErlNifEnv is “exteriorly immutable”; there are no operations on it that visibly modify it. I should also point out that removal of the Sync trait tightens any looseness const references may have had in the past.

A quick look and Rustler shows that a tweak to the NIF_ENV is all that is needed to adapt to this change. @hansihe and @jorendorff , please advise if you object to this change or if you think I've misunderstood something. Thanks.

Fails to run custom build command with latest Erlang

After updating to Erlang/OTP 20 [RELEASE CANDIDATE 2], this dependency fails to build

   Compiling void v1.0.2
   Compiling quote v0.3.15
   Compiling unicode-xid v0.0.4
   Compiling erlang_nif-sys v0.6.1
   Compiling lazy_static v0.1.16
   Compiling synom v0.11.3
   Compiling lazy_static v0.2.8
   Compiling unreachable v0.1.1
   Compiling syn v0.11.11
error: failed to run custom build command for `erlang_nif-sys v0.6.1`
process didn't exit successfully: `../debug/build/erlang_nif-sys-06da930bd64b6054/build-script-build` (exit code: 101)
--- stdout
Unsupported Erlang version.

Is the erlang_nif-sys version up to date in the Cargo.toml?
Does 'cargo update' fix it?
If not please report at https://github.com/goertzenator/erlang_nif-sys.

--- stderr
thread 'main' panicked at 'gen_api.erl encountered an error.'../.cargo/registry/src/github.com-1ecc6299db9ec823/erlang_nif-sys-0.6.1/build.rs:28
note: Run with `RUST_BACKTRACE=1` for a backtrace.

Build failed, waiting for other jobs to finish...
error: build failed
** (RuntimeError) Rust NIF compile error (rustc exit code 101)
    lib/mix/tasks/compile.rustler.ex:58: Mix.Tasks.Compile.Rustler.compile_crate/1
    (elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
    lib/mix/tasks/compile.rustler.ex:12: Mix.Tasks.Compile.Rustler.run/1
    (mix) lib/mix/task.ex:300: Mix.Task.run_task/3
    (elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
    (mix) lib/mix/tasks/compile.all.ex:19: anonymous fn/1 in Mix.Tasks.Compile.All.run/1
    (mix) lib/mix/tasks/compile.all.ex:37: Mix.Tasks.Compile.All.with_logger_app/1
    (mix) lib/mix/task.ex:300: Mix.Task.run_task/3

appveyor erlang version

Appveyor ignores the downloaded erlang installation and uses the outdated version included on the VM image.

Updated to Erlang/OPT 19.0 and dirty nifs appear broken.

Using Homebrew, I updated my Erlang/OTP installation to 19.0 and now I'm encountering an error with the dirty scheduler support.

Error in process <0.159.0> with exit value:
{{badmatch,{error,{bad_lib,"NIF emagick_rs:image_fit/3 requires a runtime with dirty scheduler support."}}},
 [{emagick_rs,init,0,
              [{file,"/Users/nfiedler/Downloads/emagick.rs/_build/test/lib/emagick_rs/src/emagick_rs.erl"},
               {line,51}]},
  {code_server,'-handle_on_load/5-fun-0-',1,
               [{file,"code_server.erl"},{line,1327}]}]}

Same code works with Erlang/OTP 18.3, so I'm guessing something more dramatic changed than was indicated in the 19.0 release notes for OTP-13123.

Any ideas?

Erlang 23 Support

Erlang has reached version 23. We should update.

Add this usage example?

Hello,

I've created this repo with an example of using erlang_nif-sys with a simple erlang project: https://github.com/sdwolf/rustfromerl

It features:

a Dockerfile for setting up the environment with the latest stable versions of erlang and rust.
a rebar3 project
automatic compilation of rust nifs with the help of this rebar3 plugin: https://github.com/sdwolf/rebar3_rust
a matrix multiplication module with functions written in both languages to show performance difference.

Would you be interested in adding this to the examples? If yes I can create a PR.

Cheers!

resource mutability

While working on Ruster, I've discovered that the resource functions in erlang_nif-sys may have the wrong mutability:

fn enif_alloc_resource(_type: *mut ErlNifResourceType, size: size_t) -> *mut c_void
fn enif_release_resource(obj: *mut c_void)
fn enif_get_resource(arg1: *mut ErlNifEnv, term: ERL_NIF_TERM, _type: *mut ErlNifResourceType, objp: *mut *mut c_void) -> c_int
fn enif_keep_resource(obj: *mut c_void)

I think the object pointers in all except enif_alloc_resource should be changed to *const c_void. Consider the scenario where two Erlang processes have the same resource term and simultaneously call Nifs that manipulate that resource. Having a mutable pointer from enif_get_resource will enable concurrent mutation (bad!). The user can be steered in a safer direction by providing a const pointer. Interior mutability can still be achieved with Cell, RefCell, and locks.

The underlying C NIF API is compelled to use mutable pointers here because resource refcounts need to be mutated. C can't hide these things under interior mutability while staying exteriorly immutable. With this Rust wrapper we have an opportunity to hide such implementation details.

@hansihe, would this change cause breakage for Rustler?

Update for nif version 2.8

Along with OTP 18, the nif api version number was bumped up to 2.8. According to the version history in the file erl_nif.h, this new version added one new function, enif_has_pending_exception.

Adding support for this one function should be trivial, and I would absolutely be willing to submit a pull request. I am however slightly unsure how you would handle multiple different versions of the nif api in the codebase. Would you like to keep compatibility with 2.7, or would a new release remove support for this version in favor of 2.8?

What about enif_make_pid?

Hi! I can't find enif_make_pid function in your project. Is it possible to add it? Also I've try to add it in my code as a workaround:

extern "C" {
    fn enif_make_pid(env: *mut ErlNifEnv, pid: *const ErlNifPid) -> ERL_NIF_TERM;
}

but it crashes when nif library loading into VM with message Failed to load NIF library: '/.../mylib.so: undefined symbol: enif_make_pid'. Looks like it is a new feature? This is my erl banner: Erlang/OTP 19 [erts-8.1] [source] [64-bit] [smp:4:4] [async-threads:10] [kernel-poll:false].

Regards