Public Rust API about sudachi.rs HOT 4 OPEN

Names and semantics should be close to Java version as possible.
(comment from Takaoka-san)

from sudachi.rs.

TL:DR

Nice API with multiple sentences is currently blocked in stable Rust by in-progress GATs feature, also see http://lukaskalbertodt.github.io/2018/08/03/solving-the-generalized-streaming-iterator-problem-without-gats.html.

Scratch prototype impl

Want to have:

• Analyzer having internal mutable state for storing one sentence (and not more) because of performance reasons
• Iterator over analyzed sentences needs to borrow analyzer mutably to enforce that it is impossible to access the next sentence before consuming the current one

Problems:

• Current Rust iterators can't borrow from &self without GATs which would probably introduce new-ish Iterator API as well

What to do

• Provide non-allocating API for
  • Sentence splitting, returning Iterator of &str
  • Provide sentence-based analysis API
• Optionally provide allocating combined API, which copies needed information (mostly POS) from analyzer
• Another option would be to implement Iterator-like pattern without using standard library traits to iterate over analyzed sentences (for consuming in while loop).

from sudachi.rs.

Splitting API into sentence splitter / analysis

for sentence in analyzer.split_sentences(line)? {
 let result = analyzer.analyze_sentence(sentence)?
 for token in result.tokens() {
   // process token
 }
}

from sudachi.rs.

Morpheme's part_of_speech should not return option of POS array, it should panic when given invalid POS id instead.

from sudachi.rs.