word normalize support for mecab tokenizer about konoha HOT 4 CLOSED

@sobamchan

I'm wondering if base_form would be the thing you want.
(I think normalized_form is the normalization of base_form in Sudachi)

from konoha.

Cool, I didn't know the token instance has base_form property.
This is exactly what I wanted, I already wrote a parser myself but will update with this.
Thanks!

from konoha.

Thank you for your interest!
The output of normalized_form is the unique feature of Sudachi.
As far as I understand, MeCab doesn't provide this information.
For more detail, please refer to the documentation. 🙏

https://github.com/WorksApplications/Sudachi#mecab--kuromoji-%E3%81%A8%E3%81%AE%E6%AF%94%E8%BC%83

from konoha.

Ah, ok my bad.
I thought normalized_form is equivalent to lemmatize/stemming (which I wanted for my project).

Thanks for your time!

from konoha.