AnalyzedSentence Design about sudachi.rs HOT 4 CLOSED

Considering to add MorphemeList following java/python:

struct MorphemeList<'a, T> {
    dictionary: T,
    input_text: &'a str,
    path: Vec<lattice::Node>
}

And let Morpheme have reference to MorhemeList:

struct Morpheme<'a, T> {
    list: &MorphemeList<'a, T>,
    index: usize,
    word_info: Option<WordInfo>, // for lazy load
}

MorhemeList<'a, T> will be returned from tokenizer<T>.tokenize(input: &'a str).
We may change the struct for the path field later.

from sudachi.rs.

In this formulation will it be usable in Pyo3? Even with wrappers.

from sudachi.rs.

For python side I considered to create a variant with no generic parameter:

struct PyMorphemeList {
    dictionary: Arc<JapaneseDictionary>,
    input_text: String,
    path: Vec<lattice::Node>
}

struct PyMorpheme {
    list: Arc<PyMorphemeList>,
    index: usize,
    word_info: Option<PyWordInfo>
}

from sudachi.rs.

It would be nice if the Rust version of MorphemeList would be somehow reusable allocation-wise. At least in theory and not in the initial implementation.

For the Python that may be impossible (and Python FFI requires making copies for basically everything).

from sudachi.rs.