Combining Unicode characters not correctly handled by LanguageTool about languagetool HOT 4 OPEN

Thinking about it further, maybe the column being 7 instead of 6 is not a bug after all. It depends on what is meant by "column". LT counts "Code Units" in UTF-16 I think, and I expected that it counted "Grapheme clusters" (see definition of those terms here http://www.unicode.org/faq/char_combmark.html). However, this needs to be clarified somewhere in the documentation. Right now meaning of "column" is rather ambiguous in the output in command line or in the output of the --api when it comes to:

• combining characters
• surrogate characters (not fitting in a single 16-bits code unit)

But in any case:

• the fact that command line underlines at the wrong place is a definitely a bug
• and the fact that words with combining chars are not recognized as correct words is also a bug. Hunspell in command line recognizes words with combining chars so this is a bug with LT and not with Hunspell. In fact the French Hunspell file languagetool-language-modules/fr/target/classes/org/languagetool/resource/fr/hunspell contains a rule for such combining char (ICONV é é) but it seems ignored by LT rule HUNSPELL_NO_SUGGEST_RULE.

from languagetool.

The following picture illustrates the 2 issues with Unicode characters caused by LanguageTool inside Vim:

• First line does not use combining char and everything looks fine, incorrect word Foobar is highlighted at correction location (Line 1, column 6)
• Second line uses combining char (U+0065 + U+0301) for the e with acute accent and notice that the first word is not recognized, and the 2nd word Foobar is highlighted at the wrong location (line 2, column 7), it should be (line 2, column 6) i.e. same column as first line.

from languagetool.

We have input and output conversion already in MorfologikSpeller, so we could apply these conversions for ligatures etc. But I think it makes sense to normalize UTF-8 using standard Java code anyway, so that LT would treat everything the same.

from languagetool.

Is there any workaround for unicode output being underlined incorrectly on the command line output?

from languagetool.