Unicode normalization for string comparison. about vtk-dicom HOT 2 OPEN

Implementation ideas: Name comparisons should be case-folded and normalized. The data file DerivedNormalizationProps.txt provides a table with the NFKC_CF property for each code point, which can be incorporated into my existing code. For example, my CaseFoldUnicode() function could take an option to tell it whether to simply case fold (as it does now), or case fold and normalize for compatibility.

For each case-folded character that is returned, "D117 Canonical Composition Algorithm" from Unicode chapter 3.11 must be applied: the returned character must be combined with a character that is already present in the string if and only if such a combination is possible. Detailed information is given in tr15 "1.3 Description of the Normalization Process". Note that combination must be done in a certain order, so it is necessary to push all combining characters onto a queue and sort them into the correct order before combining them with the previous non-combining character.

from vtk-dicom.

Another option is to throw away all diacritical marks, resulting in a very loose comparison that should be well suited for names. This simplifies the code, since the "combining" part of the algorithm is no longer needed. This is basically conversion via NFKC_CF (DerivedNormalizationProps.txt) followed by decomposition (UnicodeData.txt) followed by deletion of combining diacritics. It should be very suitable for loose string comparisons.

from vtk-dicom.