Unicode variable names about jou HOT 5 OPEN

We could try this again, because we no longer have a fuzzer. Non-ASCII variable names can lead to weird bugs where the variable name isn't what you think it is, but on the other hand, forcing everyone to program in English doesn't seem like a great idea either.

That said, I think we should do something more clever than just allowing all non-ASCII characters in variable names. For example, if you use non-breaking spaces, you can have working code that looks like this:

my awesome variable = 1

Here the "spaces" between words would be non-breaking spaces, not normal ASCII space characters, so if we just allow all non-ASCII characters in variable names, the compiler would sees the left side of = as one variable name. (Apparently GitHub replaces non-breaking spaces with normal spaces, so I can't actually write the above code example with them.)

It might be enough to ban all Unicode space characters in variable names. There's not that many, and it is relatively easy to find them with Python:

>>> [x for x in range(0x110000) if chr(x).isspace()]
[9, 10, 11, 12, 13, 28, 29, 30, 31, 32, 133, 160, 5760, 8192, 8193, 8194, 8195, 8196, 8197, 8198, 8199, 8200, 8201, 8202, 8232, 8233, 8239, 8287, 12288]

I am using range(0x110000) because I get this error message if I try to pass a big number to chr():

>>> chr(1000000000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: chr() arg not in range(0x110000)

from jou.

Idea: Allow everything non-ASCII except characters in categories Cf, Cc, Zs, Zl, Zp. This is a total of 220 non-ASCII characters to ban.

from jou.

For now I think I will allow everything non-ASCII, but it could be better to emit errors for using e.g. non-breaking space as a variable name, or using an encoding other than UTF-8.

from jou.

Moosems: makes a newline into a variable.
Akuli: The..... What?

from jou.

So will the test fail again with non-ASCII variable names?

from jou.