ANY language has weird UTF8 bug!? about link-grammar HOT 7 CLOSED

Actually, I guess there are two errors here: first, relex is scrambling the utf8 input and turning it into the junk � which is probably not a valid utf8 char. Next, link-grammar trips over this invalid char ...

from link-grammar.

this bug is the ultimate victim: opencog/opencog#1689

from link-grammar.

It looks as if it behaves as planned.

Each of these words contain from some reason an invalid utf-8 character:

$ od -xc
Ph� T�n.
0000000    6850    bfef    20bd    ef54    bdbf    2e6e    000a
          P   h 357 277 275       T 357 277 275   n   .  \n
0000015

It cannot match anything in any/4.0.regex since all the regex'es there require that all the letters are particular ones:

ANY-WORD:  /^[[:alnum:]_'-]+$/
ANY-PUNCT:  /^[[:punct:]]+$/

JUNK: /^[^[:punct:]][^[:space:]]+[^[:punct:]]$/

So the final result is that these words are notifies as not in the dict, as "needed"...

It is possible to "fix" that by adding UNKNOWN-WORD to any/4.0.dict:

UNKNOWN-WORD: ();

And now we get:

$ link-parser any
link-grammar: Info: Dictionary found at ../any/4.0.dict
link-grammar: Info: Using locale en_US.utf8.
link-grammar: Info: Dictionary version 5.1.0.
link-grammar: Info: Library version link-grammar-5.3.0. Enter "!help" for help.
linkparser> !m
Display word morphology turned on.
linkparser> Ph� T�n.
No complete linkages found.
Found 4 linkages (4 had no P.P. violations) at null count 1
    Linkage 1, cost vector = (UNUSED=1 DIS= 0.00 LEN=1)

    +-------ANY------+----ANY----+
    |                |           |
LEFT-WALL [Ph�] T�n[!JUNK] .[!ANY-PUNCT] 

Press RETURN for the next linkage.
linkparser> 

In addition, sentences with invalid Unicode in them should be just rejected (else the program can segfault, or get a security problem). I can try to add such a check.
But this may also cause Relex to get stuck due to the same reason, so Relex should be fixed anyway.

from link-grammar.

So looks like ANY sometimes has unused words. How add, I thought it did not do that. Maybe this is a side-effect of the regex?

from link-grammar.

The problem is that the JUNK regex does't really match any word that doesn't match ANY-WORD and ANY-PUNC (by design).
So there exist words that don't match any regex in any/4.0.regex, and since any/4.0.dict contains only regex labels, such words are "not in the dict".

It is possible to define after JUNK:

INVALID: /^/
and define it in the dict as
INVALID: ();
but this is just like using UNKNOWN-WORD.

from link-grammar.

closing; the bug is not in the relex/link-grammar toolchain. See opencog/opencog#1689 for the debug log establishing this.

from link-grammar.

opencog guile implemetation was a missing per-thread utf8 initialization; threads were defaulting to an iso-8859-1 encoding :-(

from link-grammar.