failures with cpython master about asttokens HOT 5 CLOSED

Thanks @tacaswell for the report!

OK the first problem I see is that slices are now expressions and location information has been added which has fixed behaviour in both asttokens and ast:

import ast

from asttokens import ASTTokens

source = """
x[1:2]
"""

tree = ast.parse(source)
print(ast.dump(tree))

node = tree.body[0].value.slice
print(ast.dump(node))
print(f"{ast.get_source_segment(source, node)=}")

atok = ASTTokens(source, tree=tree)
print(f"{atok.get_text(node)=}")

print(f"{ast.Slice.__base__=}")

In Python 3.8:

ast.get_source_segment(source, node)=None
atok.get_text(node)='2'
ast.Slice.__base__=<class '_ast.slice'>

(the exact result of atok.get_text(node) varies quite a bit with different slices)

In 3.9:

ast.get_source_segment(source, node)=':2'
atok.get_text(node)=':2'
ast.Slice.__base__=<class 'ast.expr'>

Because they're now expressions, the tests try to verify the slice nodes and parse source code like ':2' which of course is a syntax error.

@dsagal how should we handle this? By far the easiest course is to just ignore slices in tests, as has done before.

from asttokens.

Oh and ExtSlice is now represented by Tuple. See https://bugs.python.org/issue34822

from asttokens.

Overall, there are 5 astroid failures and 10 regular failures.

If I ignore slices in verify_all_nodes, then there are 0 astroid failures and 7 regular failures.

The other problem is that keyword nodes have changed: https://bugs.python.org/issue40141

This simple script fails:

import asttokens

asttokens.ASTTokens("""
foo(x=y)
""", parse=1)

If in addition to ignoring slices I remove MarkTokens.visit_keyword, then there are 2 astroid failures and 0 regular failures!

That's all I have time for now, will look at this again later.

from asttokens.

Thank you @alexmojaki !

from asttokens.

Awesome! Thank you for the quick fix (even if I only just noticed today)!

from asttokens.