Rethink string representation about luaparse HOT 5 OPEN

I will probably add a switch to toggle between these modes:

• no interpretation for string literals at all; extended identifiers not mangled
• pseudo-ISO-8859-1/x-user-defined (option 0)
• UTF-8 (either current behaviour or option 3/4)

from luaparse.

Got some WIP code that implements an encodingMode option, allowing to switch between:

• current behaviour
• no mangling for identifiers, .value of string literal nodes is null
• ISO-8859-1
• x-user-defined.

A 'true UTF-8' mode (option 3 or 4) would be considerably mode involved, and perhaps not worth it. Still considering it, though.

from luaparse.

I changed my mind; I won't keep current behaviour as the default, maybe I won't even keep it as an option; nobody seems to expect or desire it anyway. The default will be no mangling and no literal interpretation; this allows users to parse Unicode source code without hassle, while those interested in string literals can choose some other mode that ensures a coherent interpretation.

I implemented UTF-8 modes too, but they're a little hacky. I also still need to document the option.

from luaparse.

Finally committed as fstirlitz:2b04739...fstirlitz:10666c7.

Leaving out UTF-8 modes for the moment; I may add them later. I’m leaving this issue open until I make a decision, but either way it goes, it’s not a release blocker.

from luaparse.

I'd still be interested in UTF-8. I've tried reading up on x-user-defined but did not come away with an understanding where it would break down -- I am interested in literal strings as I want to use luaparse to change lua source into JS.

from luaparse.