aglyzov / charsetdetect Goto Github PK

This is Go bindings for the C-library libdetectcharset which is itself a C-wrapper for the C++ library Universal Charset Detector by Mozilla.

I was interested in comparing performance of several available charset detectors and created this bindings for this purpose.

I bencharked four charset detectors available in Go:

• saintfish/chardet - Go port of ICU algorithms
• goodsign/icu – libICU wrapper
• endevit/enca – libenca wrapper
• aglyzov/charsetdetect – libdetectcharset wrapper

The test was conducted on a MacBook Air 2013 (1.3 GHz Intel Core i5) using 1 CPU core. Test data – 4 HTML pages:

• 106 KB – UTF-8
• 105 KB – incorrect UTF-8
• 521 KB - CP1251
• 61 KB - KOI8-R

endevit/enca was a clear winner both in speed and quality. aglyzov/charsetdetect was coming the second. So personally I recommend using endevit/enca instead.