Add support for Unicode identifiers about asteval HOT 2 CLOSED

How do you feel about using the stdlib tokenize module? It should be pretty trivial to implement this change using it.

If that's fine, then the question is: do we use the old method with this new method, or do we use it as a fallback to double-check non-ASCII strings that have failed the first test? If the new method is slow, it might merit the second approach, but otherwise a simple replacement might be the better option.

from asteval.

@natezb I don't have a strong preference about this. That is, I think it would generally be a bad idea to use non-ASCII as identifiers in code, but I don't know why we should prevent someone from doing that.

I don't know whether valid_symbol_name() is the only place the "ASCII only" assumption is made, but it would definitely be the place to start. One approach might be to have valid_symbol_name() short-circuit to always return True and figure out where downstream code fails.

Does tokenize help? I think it would be OK to use anything from the standard library, but I'm not seeing immediately how that module would help.

from asteval.