`maxLength` constraint checking seems to not code (Hindi) unicode characters properly about ajv HOT 7 CLOSED

Hi there and thanks for reaching out.

'विकी मेड मेडिकल इनसाइक्लोपीडिया हिंदी में'.length reports as 41 which I think is due to the multibyte characters required to write Hindi. I don't think that this is something that should be supported by the AJV core library but given the extensibility of AJV you could write your own keywords to correctly handle this text the way you think it should.

edit: actually it is not about unicode pair characters (which are counted as a single character by AJV, it seems to be related to how multiple characters, particularly accents, are grouped together in Hindi?

For example, look at the result of 'विकी मेड मेडिकल इनसाइक्लोपीडिया हिंदी में'.split('')

(41) ['व', 'ि', 'क', 'ी', ' ', 'म', 'े', 'ड', ' ', 
'म', 'े', 'ड', 'ि', 'क', 'ल', ' ', 'इ', 'न', 'स', 'ा', 
'इ', 'क', '्', 'ल', 'ो', 'प', 'ी', 'ड', 'ि', 'य', 
'ा', ' ', 'ह', 'ि', 'ं', 'द', 'ी', ' ', 'म', 'े', 'ं']

from ajv.

there is unicode option (deprecated, probably) that determines how length is computed.

from ajv.

https://github.com/ajv-validator/ajv/blob/master/lib/vocabularies/validation/limitLength.ts#L25

from ajv.

it's on by default (it does not use length), and if it's not working correctly, it needs fixing

https://github.com/ajv-validator/ajv/blob/master/lib/runtime/ucs2length.ts

from ajv.

Ok, I will have a look

from ajv.

Hi @kelson42 after discussing with EP we have decided that this is not something that we will be fixing within the core AJV library.

This problem is due to the multi-glyph characters that make up this Devanagari charset and no doubt many other languages. A single character like वि is actually made up of multiple characters व and 'ि (notice the dotted line circle that shows how this character interacts with others). These are called grapheme clusters.

From just inspecting the characters there is no metadata that will tell us which chars are part of a grapheme cluster and should therefore be counted as 1. For this reason we cannot put this logic into AJV as it would require a lot of bespoke code to cover all multi-glyph charsets.

This doesn't stop you from solving this problem yourself using custom keywords, you could even publish the solution for others, but it doesn't belong in the AJV code base.

I will however document this issue and I thank you again for bringing it to our attention.

edit: to add a link to the spec on grapheme clusters

from ajv.

@jasoniangreen @epoberezkin Thank you for considering my issue and for your advices. For the record, here how I have fixed the problem.

from ajv.