snipsco / snips-nlu-rs Goto Github PK

View Code? Open in Web Editor NEW

337.0 34.0 58.0 24.43 MB

Snips NLU rust implementation

Home Page: https://snips.ai

License: Other

C 2.92% Kotlin 4.30% Python 3.76% Rust 81.86% Shell 2.01% Swift 5.05% Objective-C 0.11%

rust nlu inference

snips-nlu-rs's Introduction

Snips NLU Rust

Installation

Add it to your Cargo.toml:

[dependencies]
snips-nlu-lib = { git = "https://github.com/snipsco/snips-nlu-rs", branch = "master" }

Add extern crate snips_nlu_lib to your crate root and you are good to go!

Intent Parsing with Snips NLU

The purpose of the main crate of this repository, snips-nlu-lib, is to perform an information extraction task called intent parsing.

Let’s take an example to illustrate the main purpose of this lib, and consider the following sentence:

"What will be the weather in paris at 9pm?"

Properly trained, the Snips NLU engine will be able to extract structured data such as:

{
   "intent": {
      "intentName": "searchWeatherForecast",
      "confidenceScore": 0.95
   },
   "slots": [
      {
         "value": "paris",
         "entity": "locality",
         "slotName": "forecast_locality"
      },
      {
         "value": {
            "kind": "InstantTime",
            "value": "2018-02-08 20:00:00 +00:00"
         },
         "entity": "snips/datetime",
         "slotName": "forecast_start_datetime"
      }
   ]
}

In order to achieve such a result, the NLU engine needs to be fed with a trained model (json file). This repository only contains the inference part, in order to produce trained models please check the Snips NLU python library.

Example and API Usage

The interactive parsing CLI is a good example of to how to use snips-nlu-rs.

Here is how you can run the CLI example:

$ git clone https://github.com/snipsco/snips-nlu-rs
$ cd snips-nlu-rs
$ cargo run --example interactive_parsing_cli data/tests/models/nlu_engine

Here we used a sample trained engine, which consists in two intents: MakeCoffee and MakeTea. Thus, it will be able to parse queries like "Make me two cups of coffee please" or "I'd like a hot tea".

As mentioned in the previous section, you can train your own nlu engine with the Snips NLU python library.

License

Licensed under either of

Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT license (LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

snips-nlu-rs's People

Contributors

Stargazers

Watchers

snips-nlu-rs's Issues

Unable to run rust example on ubuntu

I am running cargo on Ubuntu 16.04 LTS. Running the command

cargo run --verbose --example weather examples "What will be the weather in London tomorrow at 8am?"

results in the following error:

note: /home/raghu/snips-nlu-rs/snips-nlu-lib/target/debug/deps/libsnips_nlu_lib-db0d3b8725811e50.rlib(snips_nlu_lib-db0d3b8725811e50.g4ocvkiolag6no4.rcgu.o): In function `crfsuite::Tagger::set::hf3ed06d04cff7be0':
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:145: undefined reference to `crfsuite_instance_init_n'
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:156: undefined reference to `crfsuite_item_init'
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:164: undefined reference to `crfsuite_attribute_set'
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:165: undefined reference to `crfsuite_item_append_attribute'
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:176: undefined reference to `crfsuite_instance_finish'
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:180: undefined reference to `crfsuite_instance_finish'
          /home/raghu/snips-nlu-rs/snips-nlu-lib/target/debug/deps/libcrfsuite-7428f11a847b8a45.rlib(crfsuite-7428f11a847b8a45.crfsuite4.rcgu.o): In function `crfsuite::Tagger::create_from_memory::h1ceffb08ed3d9cb3':
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:76: undefined reference to `crfsuite_create_instance_from_memory'
          collect2: error: ld returned 1 exit status

error: aborting due to previous error

error: Could not compile `snips-nlu-lib`.

I have also included the weather_example_run.log, which contains the complete output from the terminal window.

Failed to run interactive example: multiple rlib candidates for `snips_nlu_lib` found

Hi, I come from the Python Snips NLU package and am not familiar with Rust. Any help would be greatly appreciated! PS: Can I compile the engine to WASM and run inference on the browser with a model trained in Python?

build failures due to snips-nlu-ontology-parsers

I'm trying to run the example as per the readme (with an engine already built) and getting this failure:

> cargo run -p snips-nlu-lib --example interactive_parsing_cli ../snlu/engine
error: no matching package named `snips-nlu-ontology-parsers` found
location searched: https://github.com/snipsco/snips-nlu-ontology?branch=develop
required by package `snips-nlu-lib v0.63.0-SNAPSHOT (/Users/naitik/workspace/snips-nlu-rs/snips-nlu-lib)`

It seemed like this was the fix:

-snips-nlu-ontology-parsers = { git = "https://github.com/snipsco/snips-nlu-ontology", branch = "develop" }
+snips-nlu-parsers = { git = "https://github.com/snipsco/snips-nlu-parsers", branch = "develop" }

But still fails:

error: no matching package named `snips-nlu-ontology-parsers` found
location searched: https://github.com/snipsco/snips-nlu-ontology?branch=develop
required by package `snips-nlu-lib v0.63.0-SNAPSHOT (https://github.com/snipsco/snips-nlu-rs?branch=develop#2d1f58b6)`
    ... which is depended on by `snips-nlu-ffi v0.63.0-SNAPSHOT (https://github.com/snipsco/snips-nlu-rs?branch=develop#2d1f58b6)`
    ... which is depended on by `snips-nlu-python-ffi v0.63.0-SNAPSHOT (/Users/naitik/workspace/snips-nlu-rs/snips-nlu-ffi/python/snips-nlu-python-ffi)`

I think it's because the dependency is pulling in the upstream snips-nlu-rs without my local fix? Any help would be appreciated!

crf_slot_filler.rs

I have no idea why these errors occured

Support for wasm target?

Is there any intention to support wasm32-unknown-unknown target? I've tried to target web assembly but there are current old or deprecated dependencies that don't support wasm32 looking at the cargo tree. How hard would it be updating or replacing some of the dependencies that use libraries like time rustc-serialize memchr and support wasm?

NLU Configuration filename should not be hardcoded

Description
At the moment, FileBasedConfiguration and ZipBasedConfiguration construction logic relies on an hardcoded filename (which is trained_assistant.json).
This seems wrong in terms of testing, code clarity and future-proof-ness.

Instead, these objects should be initialized by passing an additional filename parameter.

.gitmodules file on master branch is out-of-date

https://github.com/snipsco/snips-nlu-rs/blob/master/.gitmodules has

[submodule "snips-nlu-resources/snips-nlu-resources"]
	path = snips-nlu-resources/snips-nlu-resources
url = ssh://[email protected]/snipsco/snips-nlu-resources

The url should use https and not ssh otherwise Cargo can't fetch it for anonymous users. The .gitmodules on the develop branch has the correct url but the instructions in README.rst say to use the master branch with Cargo.

Installation

Add it to your Cargo.toml:

[dependencies]
snips-nlu-lib = { git = "https://github.com/snipsco/snips-nlu-rs", branch = "master" }

You might want to commit the updated version to master or change the instructions to say to use the develop branch. Using the develop branch in Cargo I was able to build the project.

trained_assistant.json file downloaded from the website (in zip) seems to break things

Hey!

I've been hacking around for a while now using the example in this repo to get up and running and
had an issue earlier where it would break when I tried to parse a query.
More specifically
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidLength', libcore/result.rs:945:5

I then ran a backtrace and tracked it down to lib/src/intent_parser/probabilistic_intent_parser.rs:50.
Afterwards, I decided to try the trained_assistant.json file in the examples directory and everything worked fine which leads me to believe the trained_assistant.json file in the zip file, which I've downloaded from snips.ai, was faulty.

I also spotted these older version numbers in the zip file version which are different to the example versions.
"training_package_version": "0.11.2", "model_version": "0.12.0"

See this gist which shows my code
Not sure if that helps at all?

Like I mentioned, I'm up and running now using the example file, just thought I'd mention it here.

I already bombarded your Discord NLU channel too 🎉

Adding snips-nlu-lib causes errror

Hey guys,
I would like to compile an android lib with rust, jni and your snips-nlu-lib. Without the snips-nlu-lib my android lib compiles as expected. My cargo config looks like:

[target.armv7-linux-androideabi]
ar = "/Users/marcelkaufmann/Library/Android/sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-ar"
linker = "/Users/marcelkaufmann/Library/Android/sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi21-clang++"

I am not using the standalone android toolchain but the standard ndk as mentioned in https://developer.android.com/ndk/guides/other_build_systems because of my ndk version r19. The lib as well as the JNI connection works:

Finished release [optimized] target(s) in 0.14s

After adding

snips-nlu-lib = { git = "https://github.com/snipsco/snips-nlu-rs", branch = "master" }

I find

Compiling backtrace-sys v0.1.28
   Compiling bzip2-sys v0.1.7
   Compiling clap v2.32.0
   Compiling rmp v0.8.7
   Compiling clang-sys v0.23.0
   Compiling enum_primitive v0.1.1
   Compiling chrono v0.4.6
error: failed to run custom build command for `backtrace-sys v0.1.28`
process didn't exit successfully: `/Users/marcelkaufmann/Projekte/MeineProjekte/rust-nlp/cargo/target/release/build/backtrace-sys-5917014bf132b9c5/build-script-build` (exit code: 101)
--- stdout
cargo:rustc-cfg=rbt
TARGET = Some("armv7-linux-androideabi")
OPT_LEVEL = Some("3")
HOST = Some("x86_64-apple-darwin")
CC_armv7-linux-androideabi = None
CC_armv7_linux_androideabi = None
TARGET_CC = None
CC = None
CFLAGS_armv7-linux-androideabi = None
CFLAGS_armv7_linux_androideabi = None
TARGET_CFLAGS = None
CFLAGS = None
running: "arm-linux-androideabi-clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=armv7-linux-androideabi" "-I" "src/libbacktrace" "-I" "/Users/marcelkaufmann/Projekte/MeineProjekte/rust-nlp/cargo/target/armv7-linux-androideabi/release/build/backtrace-sys-27dbfe46cbd61a97/out" "-fvisibility=hidden" "-DBACKTRACE_ELF_SIZE=32" "-DBACKTRACE_SUPPORTED=1" "-DBACKTRACE_USES_MALLOC=1" "-DBACKTRACE_SUPPORTS_THREADS=0" "-DBACKTRACE_SUPPORTS_DATA=0" "-D_GNU_SOURCE=1" "-D_LARGE_FILES=1" "-Dbacktrace_full=__rbt_backtrace_full" "-Dbacktrace_dwarf_add=__rbt_backtrace_dwarf_add" "-Dbacktrace_initialize=__rbt_backtrace_initialize" "-Dbacktrace_pcinfo=__rbt_backtrace_pcinfo" "-Dbacktrace_syminfo=__rbt_backtrace_syminfo" "-Dbacktrace_get_view=__rbt_backtrace_get_view" "-Dbacktrace_release_view=__rbt_backtrace_release_view" "-Dbacktrace_alloc=__rbt_backtrace_alloc" "-Dbacktrace_free=__rbt_backtrace_free" "-Dbacktrace_vector_finish=__rbt_backtrace_vector_finish" "-Dbacktrace_vector_grow=__rbt_backtrace_vector_grow" "-Dbacktrace_vector_release=__rbt_backtrace_vector_release" "-Dbacktrace_close=__rbt_backtrace_close" "-Dbacktrace_open=__rbt_backtrace_open" "-Dbacktrace_print=__rbt_backtrace_print" "-Dbacktrace_simple=__rbt_backtrace_simple" "-Dbacktrace_qsort=__rbt_backtrace_qsort" "-Dbacktrace_create_state=__rbt_backtrace_create_state" "-Dbacktrace_uncompress_zdebug=__rbt_backtrace_uncompress_zdebug" "-o" "/Users/marcelkaufmann/Projekte/MeineProjekte/rust-nlp/cargo/target/armv7-linux-androideabi/release/build/backtrace-sys-27dbfe46cbd61a97/out/src/libbacktrace/alloc.o" "-c" "src/libbacktrace/alloc.c"

--- stderr
thread 'main' panicked at '

Internal error occurred: Failed to find tool. Is `arm-linux-androideabi-clang` installed?

', /Users/marcelkaufmann/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.29/src/lib.rs:2314:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::continue_panic_fmt
   6: std::panicking::begin_panic_fmt
   7: cc::fail
   8: cc::Build::compile
   9: build_script_build::main
  10: std::rt::lang_start::{{closure}}
  11: std::panicking::try::do_call
  12: __rust_maybe_catch_panic
  13: std::rt::lang_start_internal
  14: main

warning: build failed, waiting for other jobs to finish...
error: failed to run custom build command for `bzip2-sys v0.1.7`
process didn't exit successfully: `/Users/marcelkaufmann/Projekte/MeineProjekte/rust-nlp/cargo/target/release/build/bzip2-sys-338438d94e50ff86/build-script-build` (exit code: 101)
--- stdout
TARGET = Some("armv7-linux-androideabi")
OPT_LEVEL = Some("3")
HOST = Some("x86_64-apple-darwin")
CC_armv7-linux-androideabi = None
CC_armv7_linux_androideabi = None
TARGET_CC = None
CC = None
CFLAGS_armv7-linux-androideabi = None
CFLAGS_armv7_linux_androideabi = None
TARGET_CFLAGS = None
CFLAGS = None
DEBUG = Some("false")
running: "arm-linux-androideabi-clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=armv7-linux-androideabi" "-I" "bzip2-1.0.6" "-D_FILE_OFFSET_BITS=64" "-DBZ_NO_STDIO" "-o" "/Users/marcelkaufmann/Projekte/MeineProjekte/rust-nlp/cargo/target/armv7-linux-androideabi/release/build/bzip2-sys-4a4802aaf443cba8/out/lib/bzip2-1.0.6/blocksort.o" "-c" "bzip2-1.0.6/blocksort.c"

--- stderr
thread 'main' panicked at '

Internal error occurred: Failed to find tool. Is `arm-linux-androideabi-clang` installed?

', /Users/marcelkaufmann/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.29/src/lib.rs:2314:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::continue_panic_fmt
   6: std::panicking::begin_panic_fmt
   7: cc::fail
   8: cc::Build::compile
   9: build_script_build::main
  10: std::rt::lang_start::{{closure}}
  11: std::panicking::try::do_call
  12: __rust_maybe_catch_panic
  13: std::rt::lang_start_internal
  14: main

warning: build failed, waiting for other jobs to finish...
error: build failed
------------------------------------------------------------

Do you have any suggestion what the problem is? Why does the compiler search for arm-linux-androideabi-clang and where? And why after adding your library?
Thanks :)

0.65.0 build failing for android

0.65.0 build is failing for android when compiling the crfsuite-rs
NDK Version = android-ndk-r20

running: "aarch64-linux-android-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-I" "c/include" "-Wall" "-Wextra" "-o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crfsuite.o" "-c" "c/crf/crfsuite.c"
cargo:warning=c/crf/crfsuite.c:322:14: warning: unused function 'safe_strncpy' [-Wunused-function]
cargo:warning=static char *safe_strncpy(char *dst, const char *src, size_t n)
cargo:warning= ^
cargo:warning=1 warning generated.
exit code: 0
AR_aarch64-linux-android = None
AR_aarch64_linux_android = None
TARGET_AR = None
AR = None
running: "aarch64-linux-android-ar" "crs" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/libcrfsuite.a" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/lbfgs/lbfgs.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/cqdb/lookup3.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/cqdb/cqdb.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/dictionary.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/logging.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/params.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/quark.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/rumavl.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/dataset.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/holdout.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/train_arow.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/train_averaged_perceptron.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/train_l2sgd.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/train_lbfgs.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/train_passive_aggressive.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crf1d_context.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crf1d_model.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crf1d_feature.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crf1d_encode.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crf1d_tag.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crfsuite_train.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crfsuite.o"
exit code: 0
cargo:rustc-link-lib=static=crfsuite
cargo:rustc-link-search=native=/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out

--- stderr
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
clang80: error: no input files
thread 'main' panicked at 'called Result::unwrap() on an Err value: Error(Msg("You must either define a TARGET_SYSROOT or use Dinghy to build your project."), State { next_error: None })', src/libcore/result.rs:997:5
stack backtrace:
0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
1: std::sys_common::backtrace::_print
at src/libstd/sys_common/backtrace.rs:71
2: std::panicking::default_hook::{{closure}}
at src/libstd/sys_common/backtrace.rs:59
at src/libstd/panicking.rs:197
3: std::panicking::default_hook
at src/libstd/panicking.rs:211
4: std::panicking::rust_panic_with_hook
at src/libstd/panicking.rs:474
5: std::panicking::continue_panic_fmt
at src/libstd/panicking.rs:381
6: rust_begin_unwind
at src/libstd/panicking.rs:308
7: core::panicking::panic_fmt
at src/libcore/panicking.rs:85
8: core::result::unwrap_failed
at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/macros.rs:18
9: core::result::Result<T,E>::unwrap
at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/result.rs:798
10: build_script_build::main
at ./build.rs:55
11: std::rt::lang_start::{{closure}}
at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/rt.rs:64
12: std::panicking::try::do_call
at src/libstd/rt.rs:49
at src/libstd/panicking.rs:293
13: __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:87
14: std::rt::lang_start_internal
at src/libstd/panicking.rs:272
at src/libstd/panic.rs:388
at src/libstd/rt.rs:48
15: std::rt::lang_start
at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/rt.rs:64
16: main
17: __libc_start_main
18: _start

[Build error] Could not find auto-linked library 'snips_nlu_ffi'

Basically the same as #53, but when trying to build for macos it gives the following error: Could not find auto-linked library 'snips_nlu_ffi'. This is because the build script tries to copy libsnips_nlu_ffi.a in install_local_core:

cp: /Users/christian/workspace/soundreply/snips/snips-nlu-rs/snips-nlu-ffi/swift/SnipsNlu/../../..//target//debug/libsnips_nlu_ffi.a

which gives No such file or directory.

undefined reference to `crfsuite_instance_init_n'

I am trying to compile on an AWS EC2 Instance, and the code fails to compile with the following error:

/home/ec2-user/snips-nlu-rs/snips-nlu-lib/target/debug/deps/libsnips_nlu_lib-81442700a8c7b248.rlib(snips_nlu_lib-81442700a8c7b248.g4ocvkiolag6no4.rcgu.o): In function crfsuite::Tagger::set::h2f0968d9babb3c13': /home/ec2-user/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:145: undefined reference to crfsuite_instance_init_n'
/home/ec2-user/snips-nlu-rs/snips-nlu-lib/target/debug/deps/libsnips_nlu_lib-81442700a8c7b248.rlib(snips_nlu_lib-81442700a8c7b248.g4ocvkiolag6no4.rcgu.o): In function crfsuite::Tagger::set::h2f0968d9babb3c13': g4ocvkiolag6no4-c308c9baf1e779a490d159873856ad7a.rs:(.text._ZN8crfsuite6Tagger3set17h2f0968d9babb3c13E+0x320): undefined reference to crfsuite_item_init'
g4ocvkiolag6no4-c308c9baf1e779a490d159873856ad7a.rs:(.text._ZN8crfsuite6Tagger3set17h2f0968d9babb3c13E+0x647): undefined reference to crfsuite_attribute_set' g4ocvkiolag6no4-c308c9baf1e779a490d159873856ad7a.rs:(.text._ZN8crfsuite6Tagger3set17h2f0968d9babb3c13E+0x659): undefined reference to crfsuite_item_append_attribute'
g4ocvkiolag6no4-c308c9baf1e779a490d159873856ad7a.rs:(.text._ZN8crfsuite6Tagger3set17h2f0968d9babb3c13E+0x6c0): undefined reference to crfsuite_instance_finish' g4ocvkiolag6no4-c308c9baf1e779a490d159873856ad7a.rs:(.text._ZN8crfsuite6Tagger3set17h2f0968d9babb3c13E+0x6ce): undefined reference to crfsuite_instance_finish'
/home/ec2-user/snips-nlu-rs/snips-nlu-lib/target/debug/deps/libcrfsuite-ad8b4b3fb39b3a5b.rlib(crfsuite-ad8b4b3fb39b3a5b.crfsuite0.rcgu.o): In function crfsuite::Tagger::create_from_memory::hf43200afd689451e': /home/ec2-user/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:76: undefined reference to crfsuite_create_instance_from_memory'
collect2: error: ld returned 1 exit status

error: aborting due to previous error

error: Could not compile snips-nlu-lib.

error: Could not compile `snips-nlu-resources-packed`

I'm not terribly experienced with Rust, so I apologize if the information here is lacking.

I have a project with the following Cargo.toml

[dependencies]
snips-nlu-lib = { git = "https://github.com/talkative-ai/snips-nlu-rs", branch = "master" }
redis = "0.8.0"
hyper = "0.11"
futures = "0.1.14"

When I execute cargo run --verbose eventually the compilation fails:

error: Could not compile `snips-nlu-resources-packed`

The information on the error is as follows:

Caused by: process didn't exit successfully: `rustc --crate-name snips_nlu_resources_packed /usr/local/cargo/git/checkouts/snips-nlu-rs-c3854c7c83110319/85787f9/snips-nlu-resources-packed/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=cdca8d510a223171 -C extra-filename=-cdca8d510a223171 --out-dir /tmp/project/target/debug/deps -L dependency=/tmp/project/target/debug/deps --extern snips_nlu_ontology=/tmp/project/target/debug/deps/libsnips_nlu_ontology-64e186b4f4ebb163.rlib --extern phf=/tmp/project/target/debug/deps/libphf-f0c00edcacda1116.rlib --extern failure=/tmp/project/target/debug/deps/libfailure-4e47e324962fe3f2.rlib --cap-lints allow -L native=/tmp/project/target/debug/build/backtrace-sys-7833d5b313c46d32/out/.libs`

rustc and cargo info:
rustc 1.23.0 (766bd11c8 2018-01-01)
cargo 0.24.0 (45043115c 2017-12-05)

Any ideas why this is happening?

Loading of the model takes a lot of time

We have around 40 intents. The model loading takes long time. Is there any way to compress the model so that it can be loaded faster.

S3 Access Denied Error

Hi, on trying to build the Swift library in snips-nlu-ffi/swift the build script tries to download a file via curl -s https://s3.amazonaws.com/snips/snips-nlu-dev/snips-nlu-ios.0.58.0-SNAPSHOT.tgz. This then fails on trying to extract the archive.

When trying to manually debug this and download the archive, I get the following response.

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>B0F1298F8121AAEE</RequestId><HostId>g899i0XBMKQWq3iEp0nS9zyiTgpc6gv8fPMnPsnczAzAqbCbIUmYFTXOfrMOMoQQHFxlm3ORMSA=</HostId></Error>%

Can't install snips-nlu-lib due to authentication error fetching snips-nlu-resources module

This is my Cargo.toml dependencies section:

[dependencies]
snips-nlu-lib = { git = "https://github.com/snipsco/snips-nlu-rs", branch = "master" }

This is the error I get when I try to build it:

C:\Projects\hello_cargo>cargo build
  Updating git repository `https://github.com/snipsco/snips-nlu-rs`
error: failed to load source for a dependency on `snips-nlu-lib`

Caused by:
 Unable to update https://github.com/snipsco/snips-nlu-rs

Caused by:
 failed to update submodule `snips-nlu-resources/snips-nlu-resources`

Caused by:
 failed to fetch submodule `snips-nlu-resources/snips-nlu-resources` from ssh://[email protected]/snipsco/snips-nlu-resources

Caused by:
 failed to authenticate when downloading repository
attempted ssh-agent authentication, but none of the usernames `git` succeeded

Caused by:
 error authenticating: failed connecting agent; class=Ssh (23)

It seems that there is an authentication issue with anonymous users accessing the snips-nlu-resources git repository.

How to train CRFSlotFiller in Rust?

How to train CRFSlotFiller in Rust? thanks!

Can one engine instance be used across mnany threads?

I want to use the native snips_nlu_ffi library binary in a multi-threaded app. Is it safe to share one engine pointer across many threads and call functions like snips_nlu_engine_run_get_intents concurrently?

Many intents

Hello,

I'm looking for an NLU engine for a chatbot project. Firstly, thanks for opensourcing this rust lib !

One sentence seems to be linked to only one intent, is it a kind of limitation, isn't it ?

As an example : "Can you open the door and put light on ?".
Here I've two intents : DoorOpening and PutLightsOn.

Any way to achieve this one ?

Maybe it's easy for Deterministic intent parser but not for probabilistic one, maybe returning all intents with probability and let the app pick one or many (as an example, a strategy may be to take all intents with probability over 90% ?)

Thanks,
Gaël

Failed building wheel for snips-nlu-rust

On raspbian stretch (RPi 3 B), trying to install snips-nlu-rust without success
not familiar with rust, don't know what I'm doing wrong

sudo apt-get install python3-venv 
mkdir snips-nlu && cd snips-nlu
python3 -m venv .env 
source .env/bin/activate 
sudo apt install rustc cargo
pip install setuptools_rust==0.8.4
pip install wheel
pip install snips-nlu-rust

getting:

error: native library `clang` is being linked to by more than one version of the same package, but it can only be linked once; try updating or pinning your dependencies to ensure that this package only shows up once
  
    clang-sys v0.26.4
    clang-sys v0.22.0
  error: cargo failed with code: 101
    
  ----------------------------------------
  Failed building wheel for snips-nlu-rust

Complete output:

Installing collected packages: snips-nlu-rust
  Running setup.py install for snips-nlu-rust ... error
    Complete output from command /home/admin/snips-nlu/.env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-w7nn52sk/snips-nlu-rust/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-mundmd59-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/admin/snips-nlu/.env/include/site/python3.5/snips-nlu-rust:
    /home/admin/snips-nlu/.env/lib/python3.5/site-packages/setuptools/dist.py:333: UserWarning: Normalizing '0.57.0
    ' to '0.57.0'
      normalized_version,
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-armv7l-3.5
    creating build/lib.linux-armv7l-3.5/snips_nlu_rust
    copying snips_nlu_rust/nlu_engine.py -> build/lib.linux-armv7l-3.5/snips_nlu_rust
    copying snips_nlu_rust/utils.py -> build/lib.linux-armv7l-3.5/snips_nlu_rust
    copying snips_nlu_rust/__init__.py -> build/lib.linux-armv7l-3.5/snips_nlu_rust
    running egg_info
    writing top-level names to snips_nlu_rust.egg-info/top_level.txt
    writing dependency_links to snips_nlu_rust.egg-info/dependency_links.txt
    writing snips_nlu_rust.egg-info/PKG-INFO
    writing requirements to snips_nlu_rust.egg-info/requires.txt
    reading manifest file 'snips_nlu_rust.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files matching '*' found under directory 'snips-nlu-python-ffi/target/'
    warning: no previously-included files matching '__pycache__' found anywhere in distribution
    warning: no previously-included files matching '*.py[cod]' found anywhere in distribution
    writing manifest file 'snips_nlu_rust.egg-info/SOURCES.txt'
    copying snips_nlu_rust/__version__ -> build/lib.linux-armv7l-3.5/snips_nlu_rust
    creating build/lib.linux-armv7l-3.5/snips_nlu_rust/dylib
    copying snips_nlu_rust/dylib/.gitignore -> build/lib.linux-armv7l-3.5/snips_nlu_rust/dylib
    running build_ext
    running build_rust
    cargo rustc --lib --manifest-path /tmp/pip-build-w7nn52sk/snips-nlu-rust/snips-nlu-python-ffi/Cargo.toml --release -- --crate-type cdylib
    error: native library `clang` is being linked to by more than one version of the same package, but it can only be linked once; try updating or pinning your dependencies to ensure that this package only shows up once
    
      clang-sys v0.26.4
      clang-sys v0.22.0
    error: cargo failed with code: 101
    
    
    ----------------------------------------
Command "/home/admin/snips-nlu/.env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-w7nn52sk/snips-nlu-rust/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-mundmd59-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/admin/snips-nlu/.env/include/site/python3.5/snips-nlu-rust" failed with error code 1 in /tmp/pip-build-w7nn52sk/snips-nlu-rust/

Any help would be appreciate; really appreciate snips-nlu on my laptop by the way
Thanks