Nodes that come from f-string parents lack token information about asttokens HOT 4 OPEN

The new asttokens.ASTText class now supports nodes inside f-strings for Python 3.9.7+ with the .get_text* methods. But there's still no token information.

from asttokens.

@LiraNuna thanks for keeping this issue open and for the explanations in #4 about what's going on 👍

The original Python issue 29051 was split into two - and the first one https://bugs.python.org/issue30465 "FormattedValue expressions have wrong lineno and col_offset information" is now resolved. Does this mean that this bug can be solved properly now?

For reference, I'm about to wire in a work around to fix this here: jamescooke/flake8-aaa#101 - probably something to prevent the error being raised when the linter tries to access the token information on an f-string.

from asttokens.

It's been a while and I have to admit I forgot most of the details.

from asttokens.

I'm not really up for a PR right now, but here's an idea for an algorithm:

1. Get the range of tokens (string literals) representing the JoinedString
2. For each FormattedValue (iterating through the string tokens in parallel):
   1. Find the starting { in the current string token.
   2. Determine which character is the end of the main expression: }, !, or :, based on the conversion and format specifier values in the node.
   3. For each such character after the starting position:
      1. Try parsing the text between start and end.
      2. If successful, compare the resulting tree to the FormattedValue tree.
      3. If equal, mark tokens on the subtree, then readjust positions based on the starting position of the string.
   4. Repeat for format specifier if present.

from asttokens.