The ve-rs from jannisbecker

ve-rs's Introduction

Ve for Rust

This is a port of the ruby gem https://github.com/Kimtaro/ve to Rust. 🦀

The Rust version is meant to be used with https://github.com/daac-tools/vibrato/, a great and blazingly fast mecab-compatible tokenizer, and an IPADIC dictionary which can be found in the same repo (under Releases).

(Support for other tokenizer crates like Lindera planned as well, should be easy)

I'm trying to mostly stay close to the original codebase, and use Rust ways and idioms where applicable.

In the future I'm planning to add tests comparing outputs from the ruby and rust version, making sure there's no unexpected differences in logic.

Getting started

You can play around with the library as-is simply by cloning the repo and cargo running it. This will tokenize an example string using vibrato and then postprocess the tokens to return a more meaningful array of words.

The example code also shows in a simple way how to use this crate in your own application, provided that you're working with vibrato for tokenization.

ve-rs's People

Contributors

Stargazers

Watchers

ve-rs's Issues

Condition in `sanitize_asterisk` is incorrect resulting in lemma being discarded

Hi, firstly thank you for this crate, it has been useful for something I'm working on :)

I just wanted to point out that I believe the condition in the sanitize_asterisk function is the wrong way around.

The current code is as follows:

fn sanitize_asterisk(value: &str) -> Option<String> {
    if value.is_empty() || value == "*" {
        Some(value.into())
    } else {
        None
    }
}

To my understanding, an '*' indicates a lack of a relevant value in the MeCab format, which we would presumably want to represent as None. If not an empty string or '*', then presumably we want to keep the lemma. Instead, this code does the opposite.

I changed it to this (the same but shorter and with the condition is inverted):

fn sanitize_asterisk(value: &str) -> Option<String> {
    (!value.is_empty() && value != "*").then(|| value.to_string())
}

Now for example entering the input ’難しかった' results in a Word instance where lemma is Some("難しい".to_string()) whereas before my change it was just None.

Please let me know if I'm missing something, I'm not super familiar with MeCab and such.

Recommend Projects

jannisbecker / ve-rs Goto Github PK

ve-rs's Introduction

Ve for Rust

Getting started

ve-rs's People

Contributors

Stargazers

Watchers

ve-rs's Issues

Condition in `sanitize_asterisk` is incorrect resulting in lemma being discarded

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent