jmhodges / rchardet Goto Github PK

Character encoding auto-detection in Ruby. As smart as your browser. Open source.

Home Page: https://rubygems.org/gems/rchardet

License: GNU Lesser General Public License v2.1

Ruby 100.00%

rchardet's Introduction

Encoding detection for ruby.

gem install rchardet

Usage

require 'rubygems'
require 'rchardet'

cd = CharDet.detect(some_data)
encoding = cd['encoding']
confidence = cd['confidence'] # 0.0 <= confidence <= 1.0

Contributors

Maintained by Michael Grosser
[email protected]
License: LGPL
Made for rFeedParser

rchardet's People

Contributors

Stargazers

Watchers

rchardet's Issues

valid input encoded as windows-1252 is detected with nil encoding

I have an input file encoded in windows-1252 with the following (anonymized) contents:
1234567890,ASDF,JKL,123 WHEREVER AVE,SOMEWHERE TOWN,UBERLâNDIA,[email protected]
The case has been preserved; note the lowercased accented 'â'.
When running the contents of this file through CharDet.detect, I get back a nil encoding with 0 confidence. Running the same input through uchardet and chardetect (command line wrappers for the C++ and Python implementations of chardet, respectively) both report windows-1252.
I understand mixed case like this might be considered "weird" input, but getting a nil back as the encoding doesn't sound right to me. At the very least, rchardet appears to be doing something inconsistent with the other implementations.

Exception

Hi.
Exception:
unknown exception: undefined local variable or method jp2CharContext' for #<CharDet::SJISContextAnalysis:0x000000021c4998> /usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet/jpcntx.rb:159:infeed'
/usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet/sjisprober.rb:66:in block in feed' /usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet/sjisprober.rb:50:ineach'
/usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet/sjisprober.rb:50:in feed' /usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet/charsetgroupprober.rb:66:inblock in feed'
/usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet/charsetgroupprober.rb:63:in each' /usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet/charsetgroupprober.rb:63:infeed'
/usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet/universaldetector.rb:124:in block in feed' /usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet/universaldetector.rb:123:ineach'
/usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet/universaldetector.rb:123:in feed' /usr/local/lib/ruby/gems/1.9.1/gems/rchardet19-1.3.7/lib/rchardet19.rb:68:indetect'

I'm using:
ruby 1.9.3p547 (2014-05-14 revision 45962) [x86_64-linux]
rchardet19 (1.3.7)

Invalid Multibyte escape, Ruby 2.1.2

Upgrading to ruby 2.1.2 from 1.9.3, which works, and am receiving this error:

irb(main):001:0> require 'rchardet'
SyntaxError: /Users/ayers/.rbenv/versions/2.1.2/gemsets/workers/gems/rchardet-1.3.1/lib/rchardet/universaldetector.rb:39: invalid multibyte escape: /[\x80-\xFF]/

undefined method `-' for "\xC5":String

Stacktrace:

rchardet19-1.3.5/lib/rchardet/jpcntx.rb:198:in `get_order'
rchardet19-1.3.5/lib/rchardet/jpcntx.rb:147:in `feed'
rchardet19-1.3.5/lib/rchardet/sjisprober.rb:66:in `block in feed'
rchardet19-1.3.5/lib/rchardet/sjisprober.rb:50:in `each'
rchardet19-1.3.5/lib/rchardet/sjisprober.rb:50:in `feed'
rchardet19-1.3.5/lib/rchardet/charsetgroupprober.rb:66:in `block in feed'
rchardet19-1.3.5/lib/rchardet/charsetgroupprober.rb:63:in `each'
rchardet19-1.3.5/lib/rchardet/charsetgroupprober.rb:63:in `feed'
rchardet19-1.3.5/lib/rchardet/universaldetector.rb:124:in `block in feed'
rchardet19-1.3.5/lib/rchardet/universaldetector.rb:123:in `each'
rchardet19-1.3.5/lib/rchardet/universaldetector.rb:123:in `feed'
rchardet19-1.3.5/lib/rchardet19.rb:68:in `detect'

Sample:

"<div><div> <p> Follow FUT: </p> <div> </div> <div> Help: <a href=\"http://help.ea.com/uk/article/fifa-ultimate-team-14-faq\">FUT FAQÂť</a> </div> </div></div>"

I detect encoding for html because Sanitize gem failed with encoding issues

UTF-8 is detected with inconsistent case

Texts are usually returned as 'utf-8', but when they have a BOM they are returned as 'UTF-8'

CharDet.detect("UTF-8¡¡¡") # "encoding"=>"utf-8"
CharDet.detect("\xEF\xBB\xBF BOM¡¡¡") # "encoding"=>"UTF-8"

If it's intentional behaviour, it should be documented.
If not, it should probably return things in a consistent case, but that might be a breaking change.

ISO-8859-2 detected instead of MACROMAN for Excel CSV Mac format exported files

>> d=CharDet.detect (csv=File.read('/home/braulio/Downloads/listateste.csv'))
=> {"encoding"=>"ISO-8859-2", "confidence"=>0.2136145055663291}
>> csv
=> "Produtor;PRODUTO;Unidade;Valor\rS\xEATIO UWADA;ALFACE AMERICANA ; Unidade ;2,2\rS\xEATIO UWADA;ALFACE CRESPA ; Unidade ;2\rS\xEATIO UWADA;ALFACE LISA ; Unidade ;2\rS\xEATIO UWADA;ALFACE MIMOSA; Unidade ;2\rS\xEATIO UWADA;ALFACE ROMANA; Unidade ;2\rS\xEATIO UWADA;ALFACE ROXA; Unidade ;2\rS\xEATIO UWADA;ALMEIR\xCCO FINO ; ma\x8Do ;2\rS\xEATIO UWADA;ALMEIR\xCCO P\xCCO DE A\x82\xF2CAR; ma\x8Do ;2\rS\xEATIO UWADA;CATALONHA; ma\x8Do ;2,5\rS\xEATIO UWADA;ESCAROLA; Unidade ;2\rS\xEATIO UWADA;ESPINAFRE ; ma\x8Do ;3\rS\xEATIO UWADA;MIX SALSA/MANJERIC\xCCO ; ma\x8Do ;1,8\rS\xEATIO UWADA;NABO ; Unidade ;2\rS\xEATIO UWADA;SALSA ; ma\x8Do ;1,8\rS\xEATIO SOL NASCENTE;ALHO POR\xEE ; Unidade ;4\rS\xEATIO SOL NASCENTE;ACELGA CHINESA; Unidade ;2,5\rS\xEATIO SOL NASCENTE;COUVE MANTEIGA ; ma\x8Do ;2\rS\xEATIO SOL NASCENTE;ERVA DOCE ; Unidade ;3\rS\xEATIO SOL NASCENTE;REPOLHO VERDE ; Unidade ;3\rS\xEATIO SOL NASCENTE;REPOLHO ROXO ; Unidade ;3\rS\xEATIO SOL NASCENTE;INHAME - 500G; pacote  ;3\rS\xEATIO SOL NASCENTE;RADICCHIO ; Unidade ;2,5\r\n"
>> csv.encode 'UTF-8'
=> "Produtor;PRODUTO;Unidade;Valor\rS\xEATIO UWADA;ALFACE AMERICANA ; Unidade ;2,2\rS\xEATIO UWADA;ALFACE CRESPA ; Unidade ;2\rS\xEATIO UWADA;ALFACE LISA ; Unidade ;2\rS\xEATIO UWADA;ALFACE MIMOSA; Unidade ;2\rS\xEATIO UWADA;ALFACE ROMANA; Unidade ;2\rS\xEATIO UWADA;ALFACE ROXA; Unidade ;2\rS\xEATIO UWADA;ALMEIR\xCCO FINO ; ma\x8Do ;2\rS\xEATIO UWADA;ALMEIR\xCCO P\xCCO DE A\x82\xF2CAR; ma\x8Do ;2\rS\xEATIO UWADA;CATALONHA; ma\x8Do ;2,5\rS\xEATIO UWADA;ESCAROLA; Unidade ;2\rS\xEATIO UWADA;ESPINAFRE ; ma\x8Do ;3\rS\xEATIO UWADA;MIX SALSA/MANJERIC\xCCO ; ma\x8Do ;1,8\rS\xEATIO UWADA;NABO ; Unidade ;2\rS\xEATIO UWADA;SALSA ; ma\x8Do ;1,8\rS\xEATIO SOL NASCENTE;ALHO POR\xEE ; Unidade ;4\rS\xEATIO SOL NASCENTE;ACELGA CHINESA; Unidade ;2,5\rS\xEATIO SOL NASCENTE;COUVE MANTEIGA ; ma\x8Do ;2\rS\xEATIO SOL NASCENTE;ERVA DOCE ; Unidade ;3\rS\xEATIO SOL NASCENTE;REPOLHO VERDE ; Unidade ;3\rS\xEATIO SOL NASCENTE;REPOLHO ROXO ; Unidade ;3\rS\xEATIO SOL NASCENTE;INHAME - 500G; pacote  ;3\rS\xEATIO SOL NASCENTE;RADICCHIO ; Unidade ;2,5\r\n"
>> csv.force_encoding Encoding::MACROMAN
=> "Produtor;PRODUTO;Unidade;Valor\rS\xEATIO UWADA;ALFACE AMERICANA ; Unidade ;2,2\rS\xEATIO UWADA;ALFACE CRESPA ; Unidade ;2\rS\xEATIO UWADA;ALFACE LISA ; Unidade ;2\rS\xEATIO UWADA;ALFACE MIMOSA; Unidade ;2\rS\xEATIO UWADA;ALFACE ROMANA; Unidade ;2\rS\xEATIO UWADA;ALFACE ROXA; Unidade ;2\rS\xEATIO UWADA;ALMEIR\xCCO FINO ; ma\x8Do ;2\rS\xEATIO UWADA;ALMEIR\xCCO P\xCCO DE A\x82\xF2CAR; ma\x8Do ;2\rS\xEATIO UWADA;CATALONHA; ma\x8Do ;2,5\rS\xEATIO UWADA;ESCAROLA; Unidade ;2\rS\xEATIO UWADA;ESPINAFRE ; ma\x8Do ;3\rS\xEATIO UWADA;MIX SALSA/MANJERIC\xCCO ; ma\x8Do ;1,8\rS\xEATIO UWADA;NABO ; Unidade ;2\rS\xEATIO UWADA;SALSA ; ma\x8Do ;1,8\rS\xEATIO SOL NASCENTE;ALHO POR\xEE ; Unidade ;4\rS\xEATIO SOL NASCENTE;ACELGA CHINESA; Unidade ;2,5\rS\xEATIO SOL NASCENTE;COUVE MANTEIGA ; ma\x8Do ;2\rS\xEATIO SOL NASCENTE;ERVA DOCE ; Unidade ;3\rS\xEATIO SOL NASCENTE;REPOLHO VERDE ; Unidade ;3\rS\xEATIO SOL NASCENTE;REPOLHO ROXO ; Unidade ;3\rS\xEATIO SOL NASCENTE;INHAME - 500G; pacote  ;3\rS\xEATIO SOL NASCENTE;RADICCHIO ; Unidade ;2,5\r\n"
>> csv.encode 'UTF-8'
=> "Produtor;PRODUTO;Unidade;Valor\rSÍTIO UWADA;ALFACE AMERICANA ; Unidade ;2,2\rSÍTIO UWADA;ALFACE CRESPA ; Unidade ;2\rSÍTIO UWADA;ALFACE LISA ; Unidade ;2\rSÍTIO UWADA;ALFACE MIMOSA; Unidade ;2\rSÍTIO UWADA;ALFACE ROMANA; Unidade ;2\rSÍTIO UWADA;ALFACE ROXA; Unidade ;2\rSÍTIO UWADA;ALMEIRÃO FINO ; maço ;2\rSÍTIO UWADA;ALMEIRÃO PÃO DE AÇÚCAR; maço ;2\rSÍTIO UWADA;CATALONHA; maço ;2,5\rSÍTIO UWADA;ESCAROLA; Unidade ;2\rSÍTIO UWADA;ESPINAFRE ; maço ;3\rSÍTIO UWADA;MIX SALSA/MANJERICÃO ; maço ;1,8\rSÍTIO UWADA;NABO ; Unidade ;2\rSÍTIO UWADA;SALSA ; maço ;1,8\rSÍTIO SOL NASCENTE;ALHO PORÓ ; Unidade ;4\rSÍTIO SOL NASCENTE;ACELGA CHINESA; Unidade ;2,5\rSÍTIO SOL NASCENTE;COUVE MANTEIGA ; maço ;2\rSÍTIO SOL NASCENTE;ERVA DOCE ; Unidade ;3\rSÍTIO SOL NASCENTE;REPOLHO VERDE ; Unidade ;3\rSÍTIO SOL NASCENTE;REPOLHO ROXO ; Unidade ;3\rSÍTIO SOL NASCENTE;INHAME - 500G; pacote  ;3\rSÍTIO SOL NASCENTE;RADICCHIO ; Unidade ;2,5\r\n"

rchardet19 upgrade

Is there any specific dependency on rchardet19 gem or could it be upgraded to latest rchardet?

https://github.com/jmhodges/rchardet

If we use e.g. git gem which has this as dependency, there is conflict in some constants with rchardet19 version.

gems/rchardet19-1.3.7/lib/rchardet19.rb:59: warning: already initialized constant CharDet::VERSION
gems/rchardet-1.8.0/lib/rchardet/version.rb:2: warning: previous definition of VERSION was here

I know it's only warning, but I'd like to resolve it

Doesn't detect UTF-16 without BOM.

I have this file. I created it by converting some UTF-8 from stdin to UTF-16LE with iconv (which didn't produce a BOM). When I try to detect the encoding with rchardet, I get this result:

>>> CharDet.detect File.read('text_without_bom.txt')
=> {"encoding"=>"ascii", "confidence"=>1.0}

What I'd like to see is the actual UTF-16LE encoding.

Error after require 'rchardet'

Got an error:

/usr/local/rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require': /usr/local/rvm/gems/ruby-1.9.2-p180/gems/rchardet-1.3/lib/rchardet/chardistribution.rb:60: syntax error, unexpected ':', expecting keyword_then or ';' or '\n' (SyntaxError) /usr/local/rvm/gems/ruby-1.9.2-p180/gems/rchardet-1.3/lib/rchardet/chardistribution.rb:61: syntax error, unexpected ':', expecting keyword_then or ';' or '\n' /usr/local/rvm/gems/ruby-1.9.2-p180/gems/rchardet-1.3/lib/rchardet/chardistribution.rb:75: syntax error, unexpected ':', expecting keyword_then or ';' or '\n' /usr/local/rvm/gems/ruby-1.9.2-p180/gems/rchardet-1.3/lib/rchardet/chardistribution.rb:84: syntax error, unexpected keyword_end, expecting $end from /usr/local/rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
from /usr/local/rvm/gems/ruby-1.9.2-p180/gems/rchardet-1.3/lib/rchardet.rb:25:in <top (required)>' from /usr/local/rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:58:inrequire'
from /usr/local/rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:58:in rescue in require' from /usr/local/rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:35:inrequire'
from ./infotorg.rb:18:in `

Doesn't seem to track chardet

We are getting very different results from rchardet compared to the Python package, where the Python version is noticeably more reliable.

Simple example:
File 1

Column 1,Column 2,Column 3,Column 4,Column 5
1,López,López,2,2

rchardet result:

irb(main):001:0> require 'rchardet'
=> true
irb(main):002:0> cd = CharDet.detect(File.new(file1).read)
=> {"encoding"=>"GB18030", "confidence"=>0.99}

Python chardet result:

$ pip install chardet
Successfully installed chardet-3.0.4
$ chardetect file1
file1: utf-8 with confidence 0.7525

Using the rchardet encoding is incorrect, even though it has extremely high confidence:

$ iconv -f GB18030 -t utf8 file1
Column 1,Column 2,Column 3,Column 4,Column 5
1,L贸pez,L贸pez,2,2

Wrong encoding detected for UTF-8 file

Using rchardet 1.4.2 with the following file

abcdefg abcdefg abcdef
abcdefg abcdefg abcdef ◎abc

the wrong encoding is detected

require 'rchardet'
CharDet.detect(File.open('a.txt', 'rb').read)
# => {"encoding"=>"ISO-8859-2", "confidence"=>0.5207664687035084}

Note the unicode character "◎". Removing some characters returns the correct encoding (but it depends on which characters one removes, I found no logic).

Exception `NoMethodError' at /var/lib/gems/1.8/gems/rchardet-1.3/lib/rchardet/escprober.rb:46 - undefined method `active=' for #<CharDet::CodingStateMachine:0x7fad8ba855f8>

I'm receiving this exception:

Exception `NoMethodError' at /var/lib/gems/1.8/gems/rchardet-1.3/lib/rchardet/escprober.rb:46 - undefined method `active=' for #<CharDet::CodingStateMachine:0x7fad8ba855f8>

Using the following code:

    begin
      charset = CharDet.detect(message)
      i = Iconv.new('UTF-8','LATIN1')
      message = i.iconv(message) if charset['confidence'] >= 0.5 && charset['encoding'] != 'utf-8'      
    rescue Exception => e
    end

jpcntx.rb:159 - undefined local variable

Exception NameError' at /var/lib/gems/1.8/gems/rchardet-1.3/lib/rchardet/jpcntx.rb:159 - undefined local variable or methodjp2CharContext' for #CharDet::SJISContextAnalysis:0xb73b5bac

This error was received while trying the following:

require 'rubygems'
require 'open-uri'
require 'rchardet'

body= open("http://1rk.net").read
cd = CharDet.detect(body)

I'm using rchardet-1.3 and ruby 1.8.7 (2010-01-10 patchlevel 249) [i486-linux]
Sorry I don't have time to learn rchardet and provide a patch.

Bug?: "Encoding::CompatibilityError: incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)"

Here's the error, using the latest version of the gem:

me@adlp101-deb:~$ irb
2.1.5 :001 > require 'rchardet'
 => true 
2.1.5 :002 > CharDet.detect "Paco"
 => {"encoding"=>"ascii", "confidence"=>1.0} 
2.1.5 :003 > CharDet.detect "nombre;Last_name;matricula;code;GIFT\nJosé Rodrigo;Serrano Mateos;1234DFG;SDFG45654RRGG;REGALO30\nRamón;Ramone;9876ZXC;453DFDD;CINE10\nÁfrica;Nuñez;5656AAA;fffffff3333;regalo50\nFrançesc;Plá;3456ASD;GGGFFdddd;regalo50"
Encoding::CompatibilityError: incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string)
from /home/rserrano/.rvm/gems/ruby-2.1.5/gems/rchardet-1.4.2/lib/rchardet/universaldetector.rb:101:in `=~'
from /home/rserrano/.rvm/gems/ruby-2.1.5/gems/rchardet-1.4.2/lib/rchardet/universaldetector.rb:101:in `feed'
from /home/rserrano/.rvm/gems/ruby-2.1.5/gems/rchardet-1.4.2/lib/rchardet.rb:63:in `detect'
from (irb):3
from /home/rserrano/.rvm/rubies/ruby-2.1.5/bin/irb:11:in `<main>'
2.1.5 :004 >

Any idea? I'm a newbie, so even though I will try to find the reason for this bug (?) by myself, it might take some days ...

Charset ks_c_5601-1987 is not supported

mikel/mail#436
As per the comments, I believe this can be mapped to EUC-KR

Currently CharDet doesn't support this encoding.
Thoughts?

string is incorrectly identifed as EUC-TW; regression from upstream

I have a string, Se̱ora (hex bytes 53 65 cc b1 6f 72 61). It is intended to read "Señora"; never mind the invalid byte sequence (cc b1) for now. CharDet.detect(s) returns EUC-TW with 99% confidence. This doesn't sound right at all; I would expect UTF-8 or at the very least an ISO-8859-*. Other chardet implementations (uchardet and pychardet) both report UTF-8.

Taking a closer look at the code, I see there's a bit of logic that differs between uchardet and rchardet:
in uchardet, CharDistribution.cpp:

  if (mTotalChars <= 0 || mFreqChars <= mDataThreshold) {
    return SURE_NO;
  }

in rchardet, chardistribution.rb:

      if @totalChars <= 0
        return SURE_NO
      end

It appears that the second part of the check involving @freqChars was dropped somehow. Adding it back in makes rchardet work consistently again (plus specs pass).

Got ArgumentError: invalid byte sequence in UTF-8 when reading a 'ISO-8859-1' encoded csv file.

+1 The description is the same as the subject. It occurs under ruby 1.9.3

cd = CharDet.detect(content)
ArgumentError: invalid byte sequence in UTF-8
from /Users/boti/.rvm/gems/ruby-1.9.3-p327@search_server/gems/rchardet-1.3.1/lib/rchardet/universaldetector.rb:99:in =~' from /Users/boti/.rvm/gems/ruby-1.9.3-p327@search_server/gems/rchardet-1.3.1/lib/rchardet/universaldetector.rb:99:infeed'
from /Users/boti/.rvm/gems/ruby-1.9.3-p327@search_server/gems/rchardet-1.3.1/lib/rchardet.rb:63:in `detect'

Chardet exception with badly encoded strings

I came across an exception when using chardet to detect email encoding - managed to reproduce with the example below. I've worked round this in my code, but wanted to raise this in case it's helpful.

Clearly the string is invalid - but it would be nice if chardet still worked - and suggested an encoding.

1.9.3-p194 :001 > require 'rchardet'
 => true 
1.9.3-p194 :002 > CharDet.detect("bad encoding: \xc3\x28")
ArgumentError: invalid byte sequence in UTF-8
    from /Users/greg/.rvm/gems/ruby-1.9.3-p194/gems/rchardet-1.3.1/lib/rchardet/universaldetector.rb:99:in `=~'
    from /Users/greg/.rvm/gems/ruby-1.9.3-p194/gems/rchardet-1.3.1/lib/rchardet/universaldetector.rb:99:in `feed'
    from /Users/greg/.rvm/gems/ruby-1.9.3-p194/gems/rchardet-1.3.1/lib/rchardet.rb:63:in `detect'
    from (irb):2
    from /Users/greg/.rvm/rubies/ruby-1.9.3-p194/bin/irb:12:in `<main>'
1.9.3-p194 :003 >

CharDet.detect changes the encoding of the string

This call mutate string, but there is no indication of this behavior.
In method feed of UniversalDetector used force_encoding, but this method dosn't use clone string and dosn't change encoding back.

jmhodges / rchardet Goto Github PK

rchardet's Introduction

Usage

rchardet's People

Contributors

Stargazers

Watchers

Forkers

rchardet's Issues

Recommend Projects

Recommend Topics

Recommend Org