No langauges output despite isReliable=True about cld2 HOT 5 CLOSED

Working as intended.
The inner call of CLD2 returns the total number of bytes of text found, a list 
of three languages, their three percentages of the total text bytes, and a 
reliability Boolean.

There are several degenerate cases possible
- UNKNOWN_LANGUAGE is a valid language and may show up in the list (some 
webcruft strings such as "http" or "jpg" may deliberately match 
UNKNOWN_LANGUAGE to prevent them from falsely indicating Somali or somesuch).
- The three percentages in general will total less than 100%, implying that the 
remainder of the text is UNKNOWN_LANGUAGE.
- The percentage of the top language might be small but non-zero, meaning that 
any other detected languages are a smaller percentage and the rest is unknown.
- The percentage of the top language might be 0%, meaning 100% unknown.
- Several languages are detected but they differ so slightly or they score much 
too low or much too high compared to real text in each language, so the 
reliability Boolean is set to false.

In your particular example, only four letter groups score:
  fogr fik_ _über_ spie

Other letter groups such as 
  _inte tive_ _info _vide deos 
occur in so many different languages that they are ignored. The letter sequence 
"_über_" is strongly German, but not much else is, so the German language 
score is too low for a normal 49 bytes of German. (And 49 bytes is too low for 
CLD2 to do well -- two sentences is a more reasonable amount of input; CLD2's 
design center is real text from web pages, not 1-4 word fragments from searches 
or Twitter or suchlike.) 

The letter sequence "spie" occurs about equally in German and Latvian, so the 
overall score separation between those two ends up too low. In the end, both 
languages are dropped entirely with too few useful table hits, leaving 100% 
"other". The reliability bit is essentially over the null set of returned 
languages in this case.

I haven't looked carefully at the Python wrapper, but Mike may want to expose 
the percentages or set the reliability bool to false in more of the degenerate 
cases above. /dick

OK I fixed the Python bindings to always return 3 languages even when some of 
them are UNKNOWN (previously I would skip UNKNOWN), and added a test case.

I'm off on vacation in upstate Wisconsin for a week, back on the 20th. At
that time, I plan to tweak CLD2 to return unreliable if the top language is
less than 2% of the total text -- this will also cover the all-unknown case.

On Friday, August 9, 2013, wrote:

Updated to return is_reliable=false if top language is UNKNOWN_LANGUAGE.