gristlabs / asttokens Goto Github PK

View Code? Open in Web Editor NEW

160.0 160.0 33.0 328 KB

Annotate Python AST trees with source text and token information

License: Apache License 2.0

Python 99.54% Makefile 0.46%

asttokens's People

Contributors

Stargazers

Watchers

asttokens's Issues

get_text returns wrong result for `(lambda: 0)()`

First of all, great library! I'm using it for this and it's worked wonders for me.

Consider this script:

import asttokens

tokens = asttokens.ASTTokens('(lambda: 0)()', parse=True)
call = tokens.tree.body[0].value
print(tokens.get_text(call))

The expected output is (lambda: 0)(), but the actual output is (lambda: 0).

Strangely, there isn't a problem if you add an argument, e.g. (lambda: 0)(1).

Of course this is a ridiculous way to use lambdas and if this is the only place this problem occurs I don't think it's worth fixing. I only came across this while playing with some nonsensical code. But I thought I should point it out in case there's an underlying problem that manifests in more serious ways.

Have "exact_type" attribute for tokens

With tokenize module:

>>> next(tokenize.generate_tokens(iter(['()\n']).__next__)).type == token.OP
True
>>> next(tokenize.generate_tokens(iter(['()\n']).__next__)).exact_type == token.LPAR
True

With asttokens:

>>> asttokens.ASTTokens('()\n', True).tokens[0].type == token.OP
True
>>> asttokens.ASTTokens('()\n', True).tokens[0].exact_type == token.LPAR
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Token' object has no attribute 'exact_type'

The exact type gives a more detailed type when type is generic OP value.

Get node from token?

Hello,

I have a line number and a column. I can use that to get the the token.

How can I get the node from the token?

currently I'm doing this, but perhaps there is a way built in to asttokens?

def find_node_by_position(node, line, column):
    if isinstance(node, (nodes.ClassDef, nodes.FunctionDef)):
        node_line = node.fromlineno
    else:
        node_line = node.lineno

    node_column = node.col_offset

    if node_line == line and node_column == column:
        return node

    for child in node.get_children():
        result = find_node_by_position(child, line, column)
        if result:
            return result

    return None

2.0.5: pytest is failing in few `astroid` units

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

python3 -sBm build -w --no-isolation
because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
install .whl file in </install/prefix>
run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

I'm using astroid 2.11.2.

ASTText astroid support

So that Grist can fully support f-strings!

Include parentheses around expressions in token range

I am trying to use asttokens for source-level transformations in a code editor. For example, I would like to position the cursor on a binary operation, press a key, and the operands would be transposed: (a+b) * c → c * (a+b).

The way I’m trying to do that is:

Parse a small portion of the file into an asttokens.ASTTokens. (The smaller the better because the whole file cannot be guaranteed to be syntactically correct all the time. For now, I operate at the level of function definitions.)
Find the target node by the cursor position. Check if it’s a BinOp.
Determine the character ranges corresponding to the left and right operands, and swap them. (This is to preserve as much of the original code formatting as possible. For instance, re-generating the whole function using astor will strip all comments and possibly mess up spacing and line structure.)

However, I find that at step 3, if either operand is enclosed in parentheses, they do not count towards the token range:

import asttokens

tokens = asttokens.ASTTokens('(a + b) * c', parse=True)
print('\n'.join(
    '# {} at {}: {!r}'.format(node.__class__.name,
                              tokens.get_text_range(node),
                              tokens.get_text(node))
    for node in asttokens.util.walk(tokens.tree))

# Module at (0, 11): '(a + b) * c'
# Expr at (0, 11): '(a + b) * c'
# BinOp at (0, 11): '(a + b) * c'
# BinOp at (1, 6): 'a + b'
# Name at (1, 2): 'a'
# Name at (5, 6): 'b'
# Name at (10, 11): 'c'

so if I swap the corresponding character ranges, I get (c) * a + b which is wrong.

It would be nice if any parentheses that are not part of the parent node’s syntax were included in the token range of the child nodes:

# Module at (0, 11): '(a + b) * c'
# Expr at (0, 11): '(a + b) * c'
# BinOp at (0, 11): '(a + b) * c'
# BinOp at (0, 7): '(a + b)'
# Name at (1, 2): 'a'
# Name at (5, 6): 'b'
# Name at (10, 11): 'c'

(Or am I overlooking some other Obvious Way to Do It?)

Drop Python 2.7?

Continuing the discussion in #110 (comment)

I've just noticed that GHA still has PyPy 2.7, which is actually probably enough to say that the library is still being tested on 2.7 and is thus expected to work. Dropping support (i.e. the PyPI classifier) is only valuable if it saves us from doing work to have that confidence.

On the other hand, I don't want people doing (or thinking that they have to do) extra work in the future for the sake of Python 2, e.g. I feel bad about the effort @palfrey went through with mypy and Python 2.

If we do go forward with this, #67 would help.

ValueError on list comprehension with Python 3.8

Hi!
The following program causes a ValueError when run with Python 3.8 and asttokens 1.1.15. The problem did not arise with Python 3.6.

import asttokens

text = '''@invariant(lambda self: all([" " not in n for n in self.parts]))
def dummy(): pass
'''
asttokens.ASTTokens(text, parse=True)

The error is:

Traceback (most recent call last):
  File "/home/marko/workspace/parquery/icontract/deleteme.py", line 6, in <module>
    asttokens.ASTTokens(text, parse=True)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/util.py", line 192, in visit_tree
    ret = postvisit(current, par_value, value)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 175, in visit_listcomp
    return self.handle_comp('[', node, first_token, last_token)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 171, in handle_comp
    util.expect_token(before, token.OP, open_brace)
  File "/home/marko/workspace/parquery/icontract/venv/lib/python3.8/site-packages/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token OP:'[', got OP:'(' on line 1 col 28

Test new syntax in 3.10

https://docs.python.org/3/whatsnew/3.10.html

Pattern matching. One quick way is to add this so that test_sys_modules tests it:

      try:
        import test.test_patma
      except ImportError:
        pass

This works, which suggests that no fixes are needed. Slightly less crude would be making a proper fixture file. Obviously a targeted test would be ideal.

Parenthesised context managers.

Wrong token range given for list with trailing comma

Hello,

Great library! I've been using it to implement this pre-commit hook that ensures trailing commas exist.

Here is a simple repro case to illustrate an issue I'm facing:

import ast, asttokens


class TestcaseVisitor(ast.NodeVisitor):
    def __init__(self, atok):
        self.atok = atok

    def visit_List(self, node):
        super().generic_visit(node)

        tokens = self.atok.token_range(node.first_token, node.last_token, include_extra=True)
        for token in tokens:
            print(token)


atok = asttokens.ASTTokens("""
[
    1,
]
""", parse=True)
visitor = TestcaseVisitor(atok)
visitor.visit(atok.tree)

This will output

OP:'['
NL:'\n'
NUMBER:'1'

Where the expected result should be

OP:'['
NL:'\n'
NUMBER:'1'
OP:','
NL:'\n'
OP:']'

Please note that removing the trailing comma will not reproduce the bug.
This issue is not limited to a single param - as long as there's a trailing comma. The following will produce similar a issue:

[
    1,
    2,
]

[
    1,
    2,
    3,
]

and so on

Wrong tokens for ImportFrom.names

The following sample:

import asttokens

sample = """from foo import bar"""

astt = asttokens.ASTTokens(sample, parse=True)
print(astt.tree.body[0].names[0].first_token)

displays from but it should really display bar

Followup to astroid import fix

#87 had to be rushed through to fix #85. Followup tasks required:

Fix the coverage
Somehow test automatically that asttokens still works when astroid isn't installed.

Astroid 2.2 and Python 3.8 test failures

The test suite fails with Python 3.8.0a3:

On c8697dc with:

diff --git a/tox.ini b/tox.ini
index b320505..30652a5 100644
--- a/tox.ini
+++ b/tox.ini
@@ -4,7 +4,7 @@
 # and then run "tox" from this directory.
 
 [tox]
-envlist = py{27,35,36,37}
+envlist = py{27,35,36,37,38}
 
 [testenv]
 commands = nosetests
@@ -13,4 +13,4 @@ deps =
     coverage
     py{27}: astroid
     py{35,36}: astroid<2
-    py{37}: astroid>=2.dev
+    py{37,38}: astroid>=2.dev

$ tox -e py38
GLOB sdist-make: .../asttokens/setup.py
py38 inst-nodeps: .../asttokens/.tox/.tmp/package/1/asttokens-1.1.13.zip
py38 installed: astroid==2.2.5,asttokens==1.1.13,coverage==4.5.3,lazy-object-proxy==1.3.1,nose==1.3.7,six==1.12.0,typed-ast==1.3.1,wrapt==1.11.1
py38 run-test-pre: PYTHONHASHSEED='707348335'
py38 runtests: commands[0] | nosetests
..E...S..E.FF..F.FEEE....E.EEE.....F.......FEEFFFF.E........EEE.FFEE.EEEF....F.
======================================================================
ERROR: test_calling_lambdas (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 353, in test_calling_lambdas
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 294, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 385, in visit_call
    newnode.postinit(self.visit(node.func, newnode), args, keywords)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 699, in visit_lambda
    newnode.postinit(self.visit(node.args, newnode), self.visit(node.body, newnode))
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 332, in visit_binop
    self._bin_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_fixture10 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 147, in test_fixture10
    def test_fixture10(self): self.verify_fixture_file('astroid/noendingnewline.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 652, in visit_if
    self.visit(node.test, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 438, in visit_compare
    [
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 439, in <listcomp>
    (self._cmp_op_classes[op.__class__], self.visit(expr, newnode))
KeyError: <class '_ast.Eq'>

======================================================================
ERROR: test_fixture7 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 144, in test_fixture7
    def test_fixture7(self): self.verify_fixture_file('astroid/format.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 294, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 716, in visit_listcomp
    [self.visit(child, newnode) for child in node.generators],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 716, in <listcomp>
    [self.visit(child, newnode) for child in node.generators],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 451, in visit_comprehension
    [self.visit(child, newnode) for child in node.ifs],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 451, in <listcomp>
    [self.visit(child, newnode) for child in node.ifs],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 332, in visit_binop
    self._bin_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.Mod'>

======================================================================
ERROR: test_fixture8 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 145, in test_fixture8
    def test_fixture8(self): self.verify_fixture_file('astroid/module.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 1031, in visit_classdef
    return super(TreeRebuilder3, self).visit_classdef(
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 404, in visit_classdef
    [self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 404, in <listcomp>
    [self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 607, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 597, in _visit_functiondef
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 597, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 982, in visit_try
    body = [self.visit(child, newnode) for child in node.body]
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 982, in <listcomp>
    body = [self.visit(child, newnode) for child in node.body]
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 652, in visit_if
    self.visit(node.test, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 438, in visit_compare
    [
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 439, in <listcomp>
    (self._cmp_op_classes[op.__class__], self.visit(expr, newnode))
KeyError: <class '_ast.In'>

======================================================================
ERROR: test_fixture9 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 146, in test_fixture9
    def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 294, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 332, in visit_binop
    self._bin_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.BitOr'>

======================================================================
ERROR: test_nonascii (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 264, in test_nonascii
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 294, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 332, in visit_binop
    self._bin_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_parens_around_func (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 515, in test_parens_around_func
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 294, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 385, in visit_call
    newnode.postinit(self.visit(node.func, newnode), args, keywords)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 342, in visit_boolop
    self._bool_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.Or'>

======================================================================
ERROR: test_print_function (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 250, in test_print_function
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 986, in visit_try
    return self.visit_tryexcept(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 847, in visit_tryexcept
    [self.visit(child, newnode) for child in node.handlers],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 847, in <listcomp>
    [self.visit(child, newnode) for child in node.handlers],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 942, in visit_excepthandler
    [self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 942, in <listcomp>
    [self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 607, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 597, in _visit_functiondef
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 597, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 560, in visit_for
    return self._visit_for(nodes.For, node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 553, in _visit_for
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 553, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 313, in visit_augassign
    self._bin_op_classes[type(node.op)] + "=",
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_return_annotation (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 403, in test_return_annotation
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 607, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 597, in _visit_functiondef
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 597, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 803, in visit_return
    newnode.postinit(self.visit(node.value, newnode))
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 332, in visit_binop
    self._bin_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_calling_lambdas (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 353, in test_calling_lambdas
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 294, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 385, in visit_call
    newnode.postinit(self.visit(node.func, newnode), args, keywords)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 699, in visit_lambda
    newnode.postinit(self.visit(node.args, newnode), self.visit(node.body, newnode))
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 332, in visit_binop
    self._bin_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_comprehensions (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 369, in test_comprehensions
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_mark_tokens.py", line 21, in create_mark_checker
    return tools.MarkChecker(source, parse=True)
  File ".../asttokens/tests/tools.py", line 82, in __init__
    self.atok = asttokens.ASTTokens(source, parse=parse, tree=tree)
  File ".../asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File ".../asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File ".../asttokens/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File ".../asttokens/asttokens/util.py", line 184, in visit_tree
    ret = postvisit(current, par_value, value)
  File ".../asttokens/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File ".../asttokens/asttokens/mark_tokens.py", line 175, in visit_listcomp
    return self.handle_comp('[', node, first_token, last_token)
  File ".../asttokens/asttokens/mark_tokens.py", line 171, in handle_comp
    util.expect_token(before, token.OP, open_brace)
  File ".../asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token OP:'[', got ENDMARKER:'' on line 2 col 1

======================================================================
ERROR: test_fixture10 (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 147, in test_fixture10
    def test_fixture10(self): self.verify_fixture_file('astroid/noendingnewline.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 652, in visit_if
    self.visit(node.test, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 438, in visit_compare
    [
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 439, in <listcomp>
    (self._cmp_op_classes[op.__class__], self.visit(expr, newnode))
KeyError: <class '_ast.Eq'>

======================================================================
ERROR: test_fixture7 (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 144, in test_fixture7
    def test_fixture7(self): self.verify_fixture_file('astroid/format.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_mark_tokens.py", line 21, in create_mark_checker
    return tools.MarkChecker(source, parse=True)
  File ".../asttokens/tests/tools.py", line 82, in __init__
    self.atok = asttokens.ASTTokens(source, parse=parse, tree=tree)
  File ".../asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File ".../asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File ".../asttokens/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File ".../asttokens/asttokens/util.py", line 184, in visit_tree
    ret = postvisit(current, par_value, value)
  File ".../asttokens/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File ".../asttokens/asttokens/mark_tokens.py", line 175, in visit_listcomp
    return self.handle_comp('[', node, first_token, last_token)
  File ".../asttokens/asttokens/mark_tokens.py", line 171, in handle_comp
    util.expect_token(before, token.OP, open_brace)
  File ".../asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token OP:'[', got OP:'=' on line 7 col 3

======================================================================
ERROR: test_fixture8 (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 145, in test_fixture8
    def test_fixture8(self): self.verify_fixture_file('astroid/module.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_mark_tokens.py", line 21, in create_mark_checker
    return tools.MarkChecker(source, parse=True)
  File ".../asttokens/tests/tools.py", line 82, in __init__
    self.atok = asttokens.ASTTokens(source, parse=parse, tree=tree)
  File ".../asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File ".../asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File ".../asttokens/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File ".../asttokens/asttokens/util.py", line 184, in visit_tree
    ret = postvisit(current, par_value, value)
  File ".../asttokens/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File ".../asttokens/asttokens/mark_tokens.py", line 175, in visit_listcomp
    return self.handle_comp('[', node, first_token, last_token)
  File ".../asttokens/asttokens/mark_tokens.py", line 171, in handle_comp
    util.expect_token(before, token.OP, open_brace)
  File ".../asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token OP:'[', got OP:'=' on line 53 col 19

======================================================================
ERROR: test_fixture9 (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 146, in test_fixture9
    def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 294, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 332, in visit_binop
    self._bin_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.BitOr'>

======================================================================
ERROR: test_mark_tokens_simple (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 47, in test_mark_tokens_simple
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_mark_tokens.py", line 21, in create_mark_checker
    return tools.MarkChecker(source, parse=True)
  File ".../asttokens/tests/tools.py", line 82, in __init__
    self.atok = asttokens.ASTTokens(source, parse=parse, tree=tree)
  File ".../asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File ".../asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File ".../asttokens/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File ".../asttokens/asttokens/util.py", line 184, in visit_tree
    ret = postvisit(current, par_value, value)
  File ".../asttokens/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File ".../asttokens/asttokens/mark_tokens.py", line 175, in visit_listcomp
    return self.handle_comp('[', node, first_token, last_token)
  File ".../asttokens/asttokens/mark_tokens.py", line 171, in handle_comp
    util.expect_token(before, token.OP, open_brace)
  File ".../asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token OP:'[', got OP:'=' on line 53 col 19

======================================================================
ERROR: test_nonascii (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 264, in test_nonascii
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 294, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 332, in visit_binop
    self._bin_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_parens_around_func (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 515, in test_parens_around_func
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 294, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 385, in visit_call
    newnode.postinit(self.visit(node.func, newnode), args, keywords)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 342, in visit_boolop
    self._bool_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.Or'>

======================================================================
ERROR: test_print_function (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 250, in test_print_function
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 986, in visit_try
    return self.visit_tryexcept(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 847, in visit_tryexcept
    [self.visit(child, newnode) for child in node.handlers],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 847, in <listcomp>
    [self.visit(child, newnode) for child in node.handlers],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 942, in visit_excepthandler
    [self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 942, in <listcomp>
    [self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 607, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 597, in _visit_functiondef
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 597, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 560, in visit_for
    return self._visit_for(nodes.For, node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 553, in _visit_for
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 553, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 313, in visit_augassign
    self._bin_op_classes[type(node.op)] + "=",
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_return_annotation (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 403, in test_return_annotation
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    left = to_source(rebuilt_node)
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 607, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 597, in _visit_functiondef
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 597, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 803, in visit_return
    newnode.postinit(self.visit(node.value, newnode))
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py38/lib/python3.8/site-packages/astroid/rebuilder.py", line 332, in visit_binop
    self._bin_op_classes[type(node.op)], node.lineno, node.col_offset, parent
KeyError: <class '_ast.Add'>

======================================================================
FAIL: test_fixture12 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 149, in test_fixture12
    def test_fixture12(self): self.verify_fixture_file('astroid/recursion.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 131, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: "' For issue #25 '\n\n\nclass Base(object):\n    pass\n\n\n" != '""" For issue #25 """\n\n\n\nclass Base(object):\n    pass\n\n\n'
- ' For issue #25 '
? ^               ^
+ """ For issue #25 """
? ^^^               ^^^
+ 
  
  
  class Base(object):
      pass
  
  


======================================================================
FAIL: test_fixture13 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 150, in test_fixture13
    def test_fixture13(self): self.verify_fixture_file('astroid/suppliermodule_test.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 131, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: "' file suppliermodule.py '\n\n\nclass No[229 chars]\n\n" != '""" file suppliermodule.py """\n\n\n\ncl[235 chars]\n\n'
- ' file suppliermodule.py '
? ^                        ^
+ """ file suppliermodule.py """
? ^^^                        ^^^
+ 
  
  
  class NotImplemented(Exception):
      pass
  
  
  
  class Interface:
      
      def get_value(self):
          raise NotImplemented()
      
      def set_value(self, value):
          raise NotImplemented()
  
  
  
  class DoNothing:
      pass
  
  


======================================================================
FAIL: test_fixture4 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 141, in test_fixture4
    def test_fixture4(self): self.verify_fixture_file('astroid/clientmodule_test.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 131, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: '\' docstring for file clientmodule.py \'\[819 chars]\n\n' != '""" docstring for file clientmodule.py ""[827 chars]\n\n'
Diff is 1219 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_fixture6 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 143, in test_fixture6
    def test_fixture6(self): self.verify_fixture_file('astroid/email.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 131, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: '"fake email module to test absolute import doesn\'t grab this one"\n\n' != '"""fake email module to test absolute import doesn\'t grab this one"""\n\n\n\n'
- "fake email module to test absolute import doesn't grab this one"
+ """fake email module to test absolute import doesn't grab this one"""
? ++                                                                 ++
  
+ 
+ 


======================================================================
FAIL: test_to_source (tests.test_asttokens.TestASTTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_asttokens.py", line 91, in test_to_source
    self.assertEqual(tools.to_source(root).strip(), source)
AssertionError: "def foo():\n    'xxx'\n    None" != 'def foo():\n    """xxx"""\n    None'
  def foo():
-     'xxx'
+     """xxx"""
      None

======================================================================
FAIL: test_adjacent_strings (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 229, in test_adjacent_strings
    self.assertEqual(m.view_nodes_at(2, 6), {
AssertionError: Items in the first set but not the second:
"Constant:'x y z'"
Items in the second set but not the first:
'Str:\'x y z\' \\\n\'\'\'a b c\'\'\' "u v w"'

======================================================================
FAIL: test_conditional_expr (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 342, in test_conditional_expr
    self.assertEqual(m.view_nodes_at(1, 4),
AssertionError: Items in the first set but not the second:
'Constant:True'
Items in the second set but not the first:
'NameConstant:True'

======================================================================
FAIL: test_decorators (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 449, in test_decorators
    self.assertIn('FunctionDef:@deco1\ndef f():\n  pass', m.view_nodes_at(2, 0))
AssertionError: 'FunctionDef:@deco1\ndef f():\n  pass' not found in set()

======================================================================
FAIL: test_deep_recursion (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 183, in test_deep_recursion
    self.assertEqual(m.view_node(m.all_nodes[-1]),
AssertionError: "Constant:'F1akOFFiRIgPHTZksKBAgMCLGTdGNIAAQgKfDAcgZbj0odOnUA8GBAA7'" != "Str:'F1akOFFiRIgPHTZksKBAgMCLGTdGNIAAQgKfDAcgZbj0odOnUA8GBAA7'"
- Constant:'F1akOFFiRIgPHTZksKBAgMCLGTdGNIAAQgKfDAcgZbj0odOnUA8GBAA7'
? ^^^^ ^^^
+ Str:'F1akOFFiRIgPHTZksKBAgMCLGTdGNIAAQgKfDAcgZbj0odOnUA8GBAA7'
? ^ ^


======================================================================
FAIL: test_del_dict (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 391, in test_del_dict
    self.assertEqual(m.view_nodes_at(1, 5), {'Num:4'})
AssertionError: Items in the first set but not the second:
'Constant:4'
Items in the second set but not the first:
'Num:4'

======================================================================
FAIL: test_keyword_arg_only (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 431, in test_keyword_arg_only
    self.assertEqual(m.view_nodes_at(1, 4), {'Num:1'})
AssertionError: Items in the first set but not the second:
'Constant:1'
Items in the second set but not the first:
'Num:1'

======================================================================
FAIL: test_mark_tokens_multiline (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 95, in test_mark_tokens_multiline
    self.assertEqual(all_text, {
AssertionError: Items in the second set but not the first:
'a,      # line2\nb +     # line3\n  c +   # line4\n  d'

======================================================================
FAIL: test_slices (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 205, in test_slices
    self.assertEqual(m.view_nodes_at(1, 1),
AssertionError: Items in the second set but not the first:
'Tuple:foo.Area_Code, str(foo.Phone)[:3], str(foo.Phone)[3:], foo[:], bar[::, :]'

======================================================================
FAIL: test_walk_ast (tests.test_util.TestUtil)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_util.py", line 43, in test_walk_ast
    self.assertEqual(scan, [
AssertionError: Lists differ: ["Mod[202 chars]2', 'Constant:1', 'Constant:2', "BinOp:'hello'[95 chars]ld'"] != ["Mod[202 chars]2', 'Num:1', 'Num:2', "BinOp:'hello' + ', ' + [70 chars]ld'"]

First differing element 7:
'Constant:1'
'Num:1'

  ["Module:foo(bar(1 + 2), 'hello' + ', ' + 'world')",
   "Expr:foo(bar(1 + 2), 'hello' + ', ' + 'world')",
   "Call:foo(bar(1 + 2), 'hello' + ', ' + 'world')",
   'Name:foo',
   'Call:bar(1 + 2)',
   'Name:bar',
   'BinOp:1 + 2',
-  'Constant:1',
-  'Constant:2',
+  'Num:1',
+  'Num:2',
   "BinOp:'hello' + ', ' + 'world'",
   "BinOp:'hello' + ', '",
-  "Constant:'hello'",
-  "Constant:', '",
-  "Constant:'world'"]
+  "Str:'hello'",
+  "Str:', '",
+  "Str:'world'"]

----------------------------------------------------------------------
Ran 79 tests in 0.833s

FAILED (SKIP=1, errors=20, failures=14)
ERROR: InvocationError for command '.../asttokens/.tox/py38/bin/nosetests' (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py38: commands failed

mypy reports `error: Name "asttokens.ASTTokens" is not defined`

Hi!
I updated to the latest version of asttokens 2.0.7 on my package icontract during a CI run, and the mypy started complaining that asttokenst.ASTTokens is missing.

Please see this error: https://github.com/Parquery/icontract/runs/7819826733?check_suite_focus=true

Here's a relevant excerpt:

Collecting mypy==0.950
Collecting asttokens<3,>=2
  Downloading asttokens-2.0.7-py2.py3-none-any.whl (23 kB)

...

icontract/_represent.py:41: error: Name "asttokens.ASTTokens" is not defined
icontract/_represent.py:188: error: Name "asttokens.ASTTokens" is not defined
icontract/_represent.py:210: error: Name "asttokens.ASTTokens" is not defined
icontract/_represent.py:264: error: Module has no attribute "ASTTokens"

I fixed the issue by importing asttokens.asttokens and writing out asttokens.asttokens.ASTTokens instead of importing asttokens and using asttokens.ASTTokens. Please see this pull request: Parquery/icontract#252

I tried to figure out what caused this mypy error. Honestly, I couldn't really understand where the culprit is. I installed the latest mypy version (0.971) and that didn't help either. I see that you haven't changed asttokens/__init__.py in 6 years so that can't be a reason either.

Here is a minimal code snippet if you want to reproduce the issue (I called the file deleteme.py):

import asttokens

_ = asttokens.ASTTokens("True", parse=True)

Then, after installing mypy==0.971 and asttokens==2.0.7, run:

mypy --strict deleteme.py

I get the error message:

deleteme.py:3: error: Module has no attribute "ASTTokens"
Found 1 error in 1 file (checked 1 source file)

Tests fail with astroid 2.0.0 (needed for Python 3.7)

Astroid 2.0.0devX is the first astroid to support Python 3.7 at all, so the only way to use asttokens on Python 3.7 is with astroid 2.0.0, but that doesn't work:

$ tox --pre -r -e py36

GLOB sdist-make: .../asttokens/setup.py
py36 recreate: .../asttokens/.tox/py36
py36 installdeps: nose, astroid, coverage
py36 inst: .../asttokens/.tox/dist/asttokens-1.1.10.zip
py36 installed: astroid==2.0.0.dev1,asttokens==1.1.10,coverage==4.5.1,lazy-object-proxy==1.3.1,nose==1.3.7,six==1.11.0,typed-ast==1.1.0,wrapt==1.10.11
py36 runtests: PYTHONHASHSEED='3250051827'
py36 runtests: commands[0] | nosetests
F.E...E..E.FF..F.FEEEE...E.EE....F........E......E........EEE....E.EE.......
======================================================================
ERROR: test_calling_lambdas (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 345, in test_calling_lambdas
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 268, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 345, in visit_call
    newnode.postinit(self.visit(node.func, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 623, in visit_lambda
    self.visit(node.body, newnode))
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 296, in visit_binop
    newnode = nodes.BinOp(self._bin_op_classes[type(node.op)],
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_deep_recursion (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 160, in test_deep_recursion
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_astroid.py", line 14, in create_mark_checker
    tree = builder.string_build(source)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/builder.py", line 154, in string_build
    module = self._data_build(data, modname, path)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/builder.py", line 195, in _data_build
    module = builder.visit_module(node, modname, node_file, package)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
    ...snip...
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 297, in visit_binop
    node.lineno, node.col_offset, parent)
RecursionError: maximum recursion depth exceeded

======================================================================
ERROR: test_fixture10 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 146, in test_fixture10
    def test_fixture10(self): self.verify_fixture_file('astroid/noendingnewline.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 581, in visit_if
    newnode.postinit(self.visit(node.test, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 391, in visit_compare
    for (op, expr) in zip(node.ops, node.comparators)])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 391, in <listcomp>
    for (op, expr) in zip(node.ops, node.comparators)])
KeyError: <class '_ast.Eq'>

======================================================================
ERROR: test_fixture7 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 143, in test_fixture7
    def test_fixture7(self): self.verify_fixture_file('astroid/format.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 268, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 642, in visit_listcomp
    for child in node.generators])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 642, in <listcomp>
    for child in node.generators])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 400, in visit_comprehension
    for child in node.ifs],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 400, in <listcomp>
    for child in node.ifs],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 296, in visit_binop
    newnode = nodes.BinOp(self._bin_op_classes[type(node.op)],
KeyError: <class '_ast.Mod'>

======================================================================
ERROR: test_fixture8 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 144, in test_fixture8
    def test_fixture8(self): self.verify_fixture_file('astroid/module.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 919, in visit_classdef
    newstyle=newstyle)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 367, in visit_classdef
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 367, in <listcomp>
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 539, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in _visit_functiondef
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in <listcomp>
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 870, in visit_try
    for child in node.body]
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 870, in <listcomp>
    for child in node.body]
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 581, in visit_if
    newnode.postinit(self.visit(node.test, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 391, in visit_compare
    for (op, expr) in zip(node.ops, node.comparators)])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 391, in <listcomp>
    for (op, expr) in zip(node.ops, node.comparators)])
KeyError: <class '_ast.In'>

======================================================================
ERROR: test_fixture9 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 145, in test_fixture9
    def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 268, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 296, in visit_binop
    newnode = nodes.BinOp(self._bin_op_classes[type(node.op)],
KeyError: <class '_ast.BitOr'>

======================================================================
ERROR: test_fstrings (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 276, in test_fstrings
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_astroid.py", line 15, in create_mark_checker
    return tools.MarkChecker(source, tree=tree)
  File ".../asttokens/tests/tools.py", line 82, in __init__
    self.atok = asttokens.ASTTokens(source, parse=parse, tree=tree)
  File ".../asttokens/asttokens/asttokens.py", line 62, in __init__
    self.mark_tokens(self._tree)
  File ".../asttokens/asttokens/asttokens.py", line 73, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File ".../asttokens/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File ".../asttokens/asttokens/util.py", line 170, in visit_tree
    ret = postvisit(current, par_value, value)
  File ".../asttokens/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File ".../asttokens/asttokens/mark_tokens.py", line 264, in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
  File ".../asttokens/asttokens/util.py", line 58, in expect_token
    token.start[0], token.start[1] + 1))
ValueError: Expected token NAME:'kwarg', got STRING:"f'{function(kwarg=24)}'" on line 1 col 1

======================================================================
ERROR: test_nonascii (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 256, in test_nonascii
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 268, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 296, in visit_binop
    newnode = nodes.BinOp(self._bin_op_classes[type(node.op)],
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_print_function (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 242, in test_print_function
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 876, in visit_try
    return self.visit_tryexcept(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 747, in visit_tryexcept
    for child in node.handlers],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 747, in <listcomp>
    for child in node.handlers],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 836, in visit_excepthandler
    for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 836, in <listcomp>
    for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 539, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in _visit_functiondef
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in <listcomp>
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 505, in visit_for
    return self._visit_for(nodes.For, node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 498, in _visit_for
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 498, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 282, in visit_augassign
    newnode = nodes.AugAssign(self._bin_op_classes[type(node.op)] + "=",
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_return_annotation (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 395, in test_return_annotation
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 539, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in _visit_functiondef
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in <listcomp>
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 704, in visit_return
    newnode.postinit(self.visit(node.value, newnode))
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 296, in visit_binop
    newnode = nodes.BinOp(self._bin_op_classes[type(node.op)],
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_calling_lambdas (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 345, in test_calling_lambdas
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 268, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 345, in visit_call
    newnode.postinit(self.visit(node.func, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 623, in visit_lambda
    self.visit(node.body, newnode))
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 296, in visit_binop
    newnode = nodes.BinOp(self._bin_op_classes[type(node.op)],
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_fixture10 (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 146, in test_fixture10
    def test_fixture10(self): self.verify_fixture_file('astroid/noendingnewline.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 581, in visit_if
    newnode.postinit(self.visit(node.test, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 391, in visit_compare
    for (op, expr) in zip(node.ops, node.comparators)])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 391, in <listcomp>
    for (op, expr) in zip(node.ops, node.comparators)])
KeyError: <class '_ast.Eq'>

======================================================================
ERROR: test_fixture7 (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 143, in test_fixture7
    def test_fixture7(self): self.verify_fixture_file('astroid/format.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 268, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 642, in visit_listcomp
    for child in node.generators])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 642, in <listcomp>
    for child in node.generators])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 400, in visit_comprehension
    for child in node.ifs],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 400, in <listcomp>
    for child in node.ifs],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 296, in visit_binop
    newnode = nodes.BinOp(self._bin_op_classes[type(node.op)],
KeyError: <class '_ast.Mod'>

======================================================================
ERROR: test_fixture8 (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 144, in test_fixture8
    def test_fixture8(self): self.verify_fixture_file('astroid/module.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 919, in visit_classdef
    newstyle=newstyle)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 367, in visit_classdef
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 367, in <listcomp>
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 539, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in _visit_functiondef
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in <listcomp>
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 870, in visit_try
    for child in node.body]
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 870, in <listcomp>
    for child in node.body]
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 581, in visit_if
    newnode.postinit(self.visit(node.test, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 391, in visit_compare
    for (op, expr) in zip(node.ops, node.comparators)])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 391, in <listcomp>
    for (op, expr) in zip(node.ops, node.comparators)])
KeyError: <class '_ast.In'>

======================================================================
ERROR: test_fixture9 (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 145, in test_fixture9
    def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 268, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 296, in visit_binop
    newnode = nodes.BinOp(self._bin_op_classes[type(node.op)],
KeyError: <class '_ast.BitOr'>

======================================================================
ERROR: test_nonascii (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 256, in test_nonascii
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 268, in visit_assign
    value=self.visit(node.value, newnode),
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 296, in visit_binop
    newnode = nodes.BinOp(self._bin_op_classes[type(node.op)],
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_print_function (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 242, in test_print_function
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 876, in visit_try
    return self.visit_tryexcept(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 747, in visit_tryexcept
    for child in node.handlers],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 747, in <listcomp>
    for child in node.handlers],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 836, in visit_excepthandler
    for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 836, in <listcomp>
    for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 539, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in _visit_functiondef
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in <listcomp>
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 505, in visit_for
    return self._visit_for(nodes.For, node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 498, in _visit_for
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 498, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 282, in visit_augassign
    newnode = nodes.AugAssign(self._bin_op_classes[type(node.op)] + "=",
KeyError: <class '_ast.Add'>

======================================================================
ERROR: test_return_annotation (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 395, in test_return_annotation
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
  File ".../asttokens/tests/tools.py", line 59, in to_source
    anode = builder.visit_module(node_copy, '', '', '')
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 539, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in _visit_functiondef
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 533, in <listcomp>
    for child in node.body],
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 704, in visit_return
    newnode.postinit(self.visit(node.value, newnode))
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
  File ".../asttokens/.tox/py36/lib/python3.6/site-packages/astroid/rebuilder.py", line 296, in visit_binop
    newnode = nodes.BinOp(self._bin_op_classes[type(node.op)],
KeyError: <class '_ast.Add'>

======================================================================
FAIL: test_adjacent_joined_strings (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 290, in test_adjacent_joined_strings
    "JoinedStr:f'x y z' \\\nf'''a b c''' f\"u v w\""
AssertionError: Items in the first set but not the second:
'Const:f\'x y z\' \\\nf\'\'\'a b c\'\'\' f"u v w"'
Items in the second set but not the first:
'JoinedStr:f\'x y z\' \\\nf\'\'\'a b c\'\'\' f"u v w"'

======================================================================
FAIL: test_fixture12 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 148, in test_fixture12
    def test_fixture12(self): self.verify_fixture_file('astroid/recursion.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
AssertionError: "' For issue #25 '\n\n\nclass Base(object):\n    pass\n\n\n" != '""" For issue #25 """\n\n\n\nclass Base(object):\n    pass\n\n\n'
- ' For issue #25 '
? ^               ^
+ """ For issue #25 """
? ^^^               ^^^
+ 
  
  
  class Base(object):
      pass
  
  


======================================================================
FAIL: test_fixture13 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 149, in test_fixture13
    def test_fixture13(self): self.verify_fixture_file('astroid/suppliermodule_test.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
AssertionError: "' file suppliermodule.py '\n\n\nclass No[229 chars]\n\n" != '""" file suppliermodule.py """\n\n\n\ncl[235 chars]\n\n'
- ' file suppliermodule.py '
? ^                        ^
+ """ file suppliermodule.py """
? ^^^                        ^^^
+ 
  
  
  class NotImplemented(Exception):
      pass
  
  
  
  class Interface:
      
      def get_value(self):
          raise NotImplemented()
      
      def set_value(self, value):
          raise NotImplemented()
  
  
  
  class DoNothing:
      pass
  
  


======================================================================
FAIL: test_fixture4 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 140, in test_fixture4
    def test_fixture4(self): self.verify_fixture_file('astroid/clientmodule_test.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
AssertionError: '\' docstring for file clientmodule.py \'\[819 chars]\n\n' != '""" docstring for file clientmodule.py ""[827 chars]\n\n'
Diff is 1219 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_fixture6 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 142, in test_fixture6
    def test_fixture6(self): self.verify_fixture_file('astroid/email.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 129, in verify_all_nodes
    test_case.assertEqual(to_source(rebuilt_node), to_source(node))
AssertionError: '"fake email module to test absolute import doesn\'t grab this one"\n\n' != '"""fake email module to test absolute import doesn\'t grab this one"""\n\n\n\n'
- "fake email module to test absolute import doesn't grab this one"
+ """fake email module to test absolute import doesn't grab this one"""
? ++                                                                 ++
  
+ 
+ 


======================================================================
FAIL: test_to_source (tests.test_asttokens.TestASTTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_asttokens.py", line 91, in test_to_source
    self.assertEqual(tools.to_source(root).strip(), source)
AssertionError: "def foo():\n    'xxx'\n    None" != 'def foo():\n    """xxx"""\n    None'
  def foo():
-     'xxx'
+     """xxx"""
      None

----------------------------------------------------------------------
Ran 76 tests in 0.917s

FAILED (errors=18, failures=6)
ERROR: InvocationError: '.../asttokens/.tox/py36/bin/nosetests'
___________________________________ summary ____________________________________
ERROR:   py36: commands failed

$ tox --pre -r -e py37

GLOB sdist-make: .../asttokens/setup.py
py37 recreate: .../asttokens/.tox/py37
py37 installdeps: nose, astroid, coverage
py37 inst: .../asttokens/.tox/dist/asttokens-1.1.10.zip
py37 installed: astroid==2.0.0.dev1,asttokens==1.1.10,coverage==4.5.1,lazy-object-proxy==1.3.1,nose==1.3.7,six==1.11.0,wrapt==1.10.11
py37 runtests: PYTHONHASHSEED='1682714241'
py37 runtests: commands[0] | nosetests
FF....E...........................F.....FF..................................
======================================================================
ERROR: test_deep_recursion (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 160, in test_deep_recursion
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_astroid.py", line 14, in create_mark_checker
    tree = builder.string_build(source)
  File ".../asttokens/.tox/py37/lib/python3.7/site-packages/astroid/builder.py", line 154, in string_build
    module = self._data_build(data, modname, path)
  File ".../asttokens/.tox/py37/lib/python3.7/site-packages/astroid/builder.py", line 195, in _data_build
    module = builder.visit_module(node, modname, node_file, package)
  File ".../asttokens/.tox/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 152, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 152, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File ".../asttokens/.tox/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 164, in visit
    return visit_method(node, parent)
    ...snip...
  File ".../asttokens/.tox/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 297, in visit_binop
    node.lineno, node.col_offset, parent)
RecursionError: maximum recursion depth exceeded

======================================================================
FAIL: test_adjacent_joined_strings (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 293, in test_adjacent_joined_strings
    "JoinedStr:'x y z'   # comment2\n       'a b c'   # comment3\n       f'u v w'"
AssertionError: Items in the first set but not the second:
"JoinedStr:'x y z'"
Items in the second set but not the first:
"JoinedStr:'x y z'   # comment2\n       'a b c'   # comment3\n       f'u v w'"

======================================================================
FAIL: test_adjacent_strings (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 225, in test_adjacent_strings
    node_name + ":'x y z'   # comment2\n       'a b c'   # comment3\n       'u v w'"
AssertionError: Items in the first set but not the second:
"Const:'x y z'"
Items in the second set but not the first:
"Const:'x y z'   # comment2\n       'a b c'   # comment3\n       'u v w'"

======================================================================
FAIL: test_token_methods (tests.test_asttokens.TestASTTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_asttokens.py", line 46, in test_token_methods
    self.assertEqual(atok.prev_token(atok.tokens[5]), atok.tokens[3])
AssertionError: Token(type=56, string='\n', start=(2, 0), end=(2, 1)[40 chars]s=22) != Token(type=4, string='\n', start=(1, 20), end=(1, 21[61 chars]s=21)

======================================================================
FAIL: test_adjacent_joined_strings (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 293, in test_adjacent_joined_strings
    "JoinedStr:'x y z'   # comment2\n       'a b c'   # comment3\n       f'u v w'"
AssertionError: Items in the first set but not the second:
"JoinedStr:'x y z'"
Items in the second set but not the first:
"JoinedStr:'x y z'   # comment2\n       'a b c'   # comment3\n       f'u v w'"

======================================================================
FAIL: test_adjacent_strings (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 225, in test_adjacent_strings
    node_name + ":'x y z'   # comment2\n       'a b c'   # comment3\n       'u v w'"
AssertionError: Items in the first set but not the second:
"Str:'x y z'"
Items in the second set but not the first:
"Str:'x y z'   # comment2\n       'a b c'   # comment3\n       'u v w'"

----------------------------------------------------------------------
Ran 76 tests in 1.659s

FAILED (errors=1, failures=5)
ERROR: InvocationError: '.../asttokens/.tox/py37/bin/nosetests'
___________________________________ summary ____________________________________
ERROR:   py37: commands failed

IndexError in asttokens.ASTTokens

The following program raises an uncaught exception:

import sys
import asttokens, ast
import atheris

def TestOneInput(data):
  fdp = atheris.FuzzedDataProvider(data)
  source_to_parse = fdp.ConsumeUnicodeNoSurrogates(4196)
  try:
    ast.parse(source_to_parse)
  except:
    # Avoid anything that throws any issues in ast.parse.
    return
  try:
    atok = asttokens.ASTTokens(source_to_parse, parse=True)
  except SyntaxError:
    pass

data = (b"\x79\x0a\x79\x0a\x79\x0d\x79\x0a\x0a\x79\x0a\x79\x0a\x79\x0a\x79\x0a\x79\x0a\x79\x79\x0a\x0a\x79\x0a\x79\x0a\x79\x0a\x79\x0a\x79\xae\x79\x0a\x78\x0a\x79\x0a\x79\x0a\x79\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\xc5\x0a")
TestOneInput(data)

Where the atheris module refers to https://pypi.org/project/atheris/

The program is a derivative of the fuzzer here https://github.com/google/oss-fuzz/blob/master/projects/asttokens/fuzz_asttokens.py

The following program is a shortened version of above, without fuzzing-related logic:

import asttokens, ast

def TestOneInput():
  source_to_parse = "\x0a\x79\x0a\x79\x0d\x79\x0a\x0a\x79\x0a\x79\x0a\x79\x0a\x79\x0a\x79\x0a\x79\x79\x0a\x0a\x79\x0a\x79\x0a\x79\x0a\x79\x0a\x79\x2e\x79\x0a\x78\x0a\x79\x0a\x79\x0a\x79\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x45\x0a"

  try:
    ast.parse(source_to_parse)
  except:
    # Avoid anything that throws any issues in ast.parse.
    return
  try:
    atok = asttokens.ASTTokens(source_to_parse, parse=True)
  except SyntaxError:
    pass

TestOneInput()

This produces the stack trace:

# python3 ./reproducer.py 
Traceback (most recent call last):
  File "./reproducer.py", line 29, in <module>
    TestOneInput()
  File "./reproducer.py", line 26, in TestOneInput
    atok = asttokens.ASTTokens(source_to_parse, parse=True)
  File "/usr/local/lib/python3.8/site-packages/asttokens/asttokens.py", line 127, in __init__
    self.mark_tokens(self._tree)
  File "/usr/local/lib/python3.8/site-packages/asttokens/asttokens.py", line 139, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/usr/local/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 61, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/usr/local/lib/python3.8/site-packages/asttokens/util.py", line 273, in visit_tree
    ret = postvisit(current, par_value, cast(Optional[Token], value))
  File "/usr/local/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 109, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/usr/local/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 220, in handle_attr
    name = self._code.next_token(dot)
  File "/usr/local/lib/python3.8/site-packages/asttokens/asttokens.py", line 210, in next_token
    while is_non_coding_token(self._tokens[i].type):
IndexError: list index out of range

This was found by way of OSS-Fuzz and the set up here: https://github.com/google/oss-fuzz/tree/master/projects/asttokens If you find this issue helpful then it would be great to have maintainer emails in the project.yaml to receive notifications of bug reports, which contain all details similar to what I posted above -- namely they contain the stacktrace, crashing input and identification of the fuzzer.

ASTTokens won't parse strings with coding declarations in Python 2

Python 2:

>>> asttokens.ASTTokens("# coding: ascii\n1", parse=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/google/home/devinj/.local/lib/python2.7/site-packages/asttokens/asttokens.py", line 50, in __init__
    self._tree = ast.parse(source_text, filename) if parse else tree
  File "/usr/lib/python2.7/ast.py", line 37, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 0
SyntaxError: encoding declaration in Unicode string

In Python 3 it works just fine.

I expect failures if the string is not UTF-8 (e.g. if it had coding:latin1 and some invalid utf-8 in there), but I had somewhat hoped that it would also just work to pass in the raw bytes.

I think the easiest workaround for end users is to parse it manually by hand:

>>> s = "# coding: ascii\n1"
>>> asttokens.ASTTokens(s, tree=ast.parse(s))
<asttokens.asttokens.ASTTokens object at 0x7f35d3b76550>

asttokens could also do that internally itself -- parse the tree before decoding, instead of after.

failures with cpython master

There have been changes to the AST upstream in cpython are breaking asttokens

====================================================================== short test summary info ======================================================================
FAILED tests/test_astroid.py::TestAstroid::test_complex_slice_and_parens - astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
FAILED tests/test_astroid.py::TestAstroid::test_fixture9 - AssertionError: 'Slice(\n   lower=None,\n   upper=None,\n   step=None)' != "Name(name='d')"
FAILED tests/test_astroid.py::TestAstroid::test_nonascii - astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
FAILED tests/test_astroid.py::TestAstroid::test_slices - AssertionError: 'Slice(\n   lower=None,\n   upper=Const(value=3),\n   step=None)' != 'Const(value=3)'
FAILED tests/test_astroid.py::TestAstroid::test_sys_modules - AssertionError: "Slice(\n   lower=None,\n   upper=BinOp(\n[245 chars]one)" != "BinOp(\n   op='-',\n ...
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_assignment_expressions - ValueError: Expected token NAME:'x', got NAME:'y0' on line 17 col 1
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_complex_slice_and_parens -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_decorators - ValueError: Expected token NAME:'a', got DEDENT:'' on line 12 col 1
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_fixture8 - ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_fixture9 - ValueError: Expected token NAME:'file', got NAME:'e' on line 98 col 1
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_keyword_arg_only - ValueError: Expected token NAME:'x', got NEWLINE:'' on line 2 col 15
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_mark_tokens_simple - ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_nonascii -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_slices -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_sys_modules - ValueError: Expected token NAME:'file', got NAME:'message' on line 235 col 13

In addition there seems to have been changes that break asttokens in a way that breaks stack-data which breaks IPython that don't look like they are capture in the tests:

bleeding) ✘  ~/source/other_source/asttokens [master {origin/master}|✔ ]
jupiter@17:52  ➤  ipython
Python 3.9.0a5+ (heads/master:799d7d61a9, Apr  6 2020, 17:38:49) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.0.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: 1/0                                                                                                                                                          
Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-9e1622b385b6>", line 1, in <module>
    1/0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:
<snipped>
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 81, in __init__
    self.asttokens()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 336, in asttokens
    return ASTTokens(
  File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 49, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 186, in visit_tree
    ret = postvisit(current, par_value, value)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 92, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 334, in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token NAME:'co_flags', got NAME:'new_code' on line 144 col 9

Full pytest output:

(bleeding) ✘  ~/source/other_source/asttokens [master {origin/master}|✔ ]
jupiter@17:43  ➤  pytest
======================================================================== test session starts ========================================================================
platform linux -- Python 3.9.0a5+, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/tcaswell/source/other_source/asttokens, inifile: setup.cfg
plugins: cov-2.8.1, rerunfailures-9.0, timeout-1.3.4, forked-1.1.3, xdist-1.31.0, benchmark-3.2.3
collected 107 items                                                                                                                                                 

tests/test_astroid.py .........F...s...............F....F.....F..F...                                                                                         [ 43%]
tests/test_asttokens.py ......                                                                                                                                [ 49%]
tests/test_line_numbers.py ...                                                                                                                                [ 52%]
tests/test_mark_tokens.py ...F.....F..F...............FF.F.FF.....F..F...                                                                                     [ 96%]
tests/test_util.py ....                                                                                                                                       [100%]

============================================================================= FAILURES ==============================================================================
_____________________________________________________________ TestAstroid.test_complex_slice_and_parens _____________________________________________________________

self = <tests.test_astroid.TestAstroid testMethod=test_complex_slice_and_parens>

    def test_complex_slice_and_parens(self):
      source = 'f((x)[:, 0])'
>     self.create_mark_checker(source)

tests/test_mark_tokens.py:597: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:90: in verify_all_nodes    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:275: in parse
    return builder.string_build(code, modname=module_name, path=path)
../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:140: in string_build
    module = self._data_build(data, modname, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <astroid.builder.AstroidBuilder object at 0x7fc24d0dc7c0>, data = '_\n(:, 0)', modname = '', path = None

    def _data_build(self, data, modname, path):
        """Build tree node from data and add some informations"""
        try:
            node = _parse(data + "\n")
        except (TypeError, ValueError, SyntaxError) as exc:
>           raise exceptions.AstroidSyntaxError(
                "Parsing Python code failed:\n{error}",
                source=data,
                modname=modname,
                path=path,
                error=exc,
            ) from exc
E           astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E           invalid syntax (<unknown>, line 2)

../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:168: AstroidSyntaxError
_____________________________________________________________________ TestAstroid.test_fixture9 _____________________________________________________________________

self = <tests.test_astroid.TestAstroid testMethod=test_fixture9>

>   def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')

tests/test_mark_tokens.py:172: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:134: in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
tests/tools.py:97: in verify_all_nodes
    test_case.assertEqual(
E   AssertionError: 'Slice(\n   lower=None,\n   upper=None,\n   step=None)' != "Name(name='d')"
E   + Name(name='d')- Slice(
E   -    lower=None,
E   -    upper=None,
E   -    step=None)
_____________________________________________________________________ TestAstroid.test_nonascii _____________________________________________________________________

self = <tests.test_astroid.TestAstroid testMethod=test_nonascii>

    def test_nonascii(self):
      # Test of https://bitbucket.org/plas/thonny/issues/162/weird-range-marker-crash-with-non-ascii
      # Only on PY3 because Py2 doesn't support unicode identifiers.
      for source in (
        "sünnikuupäev=str((18+int(isikukood[0:1])-1)//2)+isikukood[1:3]",
        "sünnikuupaev=str((18+int(isikukood[0:1])-1)//2)+isikukood[1:3]"):
>       m = self.create_mark_checker(source)

tests/test_mark_tokens.py:286: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:90: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:275: in parse
    return builder.string_build(code, modname=module_name, path=path)
../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:140: in string_build
    module = self._data_build(data, modname, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <astroid.builder.AstroidBuilder object at 0x7fc24c4d2670>, data = '_\n(0:1)', modname = '', path = None

    def _data_build(self, data, modname, path):
        """Build tree node from data and add some informations"""
        try:
            node = _parse(data + "\n")
        except (TypeError, ValueError, SyntaxError) as exc:
>           raise exceptions.AstroidSyntaxError(
                "Parsing Python code failed:\n{error}",
                source=data,
                modname=modname,
                path=path,
                error=exc,
            ) from exc
E           astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E           invalid syntax (<unknown>, line 2)

../../../.virtualenvs/bleeding/lib/python3.9/site-packages/astroid/builder.py:168: AstroidSyntaxError
______________________________________________________________________ TestAstroid.test_slices ______________________________________________________________________

self = <tests.test_astroid.TestAstroid testMethod=test_slices>

    def test_slices(self):
      # Make sure we don't fail on parsing slices of the form `foo[4:]`.
      source = "(foo.Area_Code, str(foo.Phone)[:3], str(foo.Phone)[3:], foo[:], bar[::2, :], [a[:]][::-1])"
>     m = self.create_mark_checker(source)

tests/test_mark_tokens.py:230: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:97: in verify_all_nodes
    test_case.assertEqual(
E   AssertionError: 'Slice(\n   lower=None,\n   upper=Const(value=3),\n   step=None)' != 'Const(value=3)'
E   + Const(value=3)- Slice(
E   -    lower=None,
E   -    upper=Const(value=3),
E   -    step=None)
___________________________________________________________________ TestAstroid.test_sys_modules ____________________________________________________________________

self = <tests.test_astroid.TestAstroid testMethod=test_sys_modules>

    def test_sys_modules(self):
      """
      Verify all nodes on source files obtained from sys.modules.
      This can take a long time as there are many modules,
      so it only tests all modules if the environment variable
      ASTTOKENS_SLOW_TESTS has been set.
      """
      modules = list(sys.modules.values())
      if not os.environ.get('ASTTOKENS_SLOW_TESTS'):
        modules = modules[:20]
    
      start = time()
      for module in modules:
        # Don't let this test (which runs twice) take longer than 13 minutes
        # to avoid the travis build time limit of 30 minutes
        if time() - start > 13 * 60:
          break
    
        try:
          filename = inspect.getsourcefile(module)
        except TypeError:
          continue
    
        if not filename:
          continue
    
        filename = os.path.abspath(filename)
        print(filename)
        try:
          with io.open(filename) as f:
            source = f.read()
        except OSError:
          continue
    
        # Astroid fails with a syntax error if a type comment is on its own line
        if self.is_astroid_test and re.search(r'^\s*# type: ', source, re.MULTILINE):
          print('Skipping', filename)
          continue
    
>       self.create_mark_checker(source)

tests/test_mark_tokens.py:639: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:97: in verify_all_nodes
    test_case.assertEqual(
E   AssertionError: "Slice(\n   lower=None,\n   upper=BinOp(\n[245 chars]one)" != "BinOp(\n   op='-',\n   left=Call(\n      [172 chars]f'))"
E   + BinOp(
E   - Slice(
E   -    lower=None,
E   -    upper=BinOp(
E   -       op='-',
E   ? ---
E   +    op='-',
E   -       left=Call(
E   ? ---
E   +    left=Call(
E   -          func=Name(name='len'),
E   ? ---
E   +       func=Name(name='len'),
E   -          args=[Attribute(
E   ? ---
E   +       args=[Attribute(
E   -                attrname='__name__',
E   ? ---
E   +             attrname='__name__',
E   -                expr=Name(name='module'))],
E   ? ---
E   +             expr=Name(name='module'))],
E   -          keywords=None),
E   ? ---
E   +       keywords=None),
E   -       right=Name(name='cut_off')),
E   ? ---                              --
E   +    right=Name(name='cut_off'))-    step=None)
----------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------
/home/tcaswell/.pybuild/bleeding/lib/python3.9/importlib/_bootstrap.py
____________________________________________________________ TestMarkTokens.test_assignment_expressions _____________________________________________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_assignment_expressions>

        def test_assignment_expressions(self):
          # From https://www.python.org/dev/peps/pep-0572/
>         self.create_mark_checker("""
    # Handle a matched regex
    if (match := pattern.search(data)) is not None:
        # Do something with match
        pass
    
    # A loop that can't be trivially rewritten using 2-arg iter()
    while chunk := file.read(8192):
       process(chunk)
    
    # Reuse a value that's expensive to compute
    [y := f(x), y**2, y**3]
    
    # Share a subexpression between a comprehension filter clause and its output
    filtered_data = [y for x in data if (y := f(x)) is not None]
    
    y0 = (y1 := f(x))  # Valid, though discouraged
    
    foo(x=(y := f(x)))  # Valid, though probably confusing
    
    def foo(answer=(p := 42)):  # Valid, though not great style
        ...
    
    def foo(answer: (p := 42) = 5):  # Valid, but probably never useful
        ...
    
    lambda: (x := 1) # Valid, but unlikely to be useful
    
    (x := lambda: 1) # Valid
    
    lambda line: (m := re.match(pattern, line)) and m.group(1) # Valid
    
    if any((comment := line).startswith('#') for line in lines):
        print("First comment:", comment)
    
    if all((nonblank := line).strip() == '' for line in lines):
        print("All lines are blank")
    
    partial_sums = [total := total + v for v in values]
    """)

tests/test_mark_tokens.py:678: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:32: in create_mark_checker
    atok = self.create_asttokens(source)
tests/test_mark_tokens.py:48: in create_asttokens
    return ASTTokens(source, parse=True)
asttokens/asttokens.py:65: in __init__
    self.mark_tokens(self._tree)
asttokens/asttokens.py:76: in mark_tokens
    MarkTokens(self).visit_tree(root_node)
asttokens/mark_tokens.py:49: in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
asttokens/util.py:186: in visit_tree
    ret = postvisit(current, par_value, value)
asttokens/mark_tokens.py:92: in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
asttokens/mark_tokens.py:334: in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

token = Token(type=1, string='y0', start=(17, 0), end=(17, 2), line='y0 = (y1 := f(x))  # Valid, though discouraged\n', index=92, startpos=435, endpos=437)
tok_type = 1, tok_str = 'x'

    def expect_token(token, tok_type, tok_str=None):
      """
      Verifies that the given token is of the expected type. If tok_str is given, the token string
      is verified too. If the token doesn't match, raises an informative ValueError.
      """
      if not match_token(token, tok_type, tok_str):
>       raise ValueError("Expected token %s, got %s on line %s col %s" % (
          token_repr(tok_type, tok_str), str(token),
          token.start[0], token.start[1] + 1))
E       ValueError: Expected token NAME:'x', got NAME:'y0' on line 17 col 1

asttokens/util.py:56: ValueError
___________________________________________________________ TestMarkTokens.test_complex_slice_and_parens ____________________________________________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_complex_slice_and_parens>

    def test_complex_slice_and_parens(self):
      source = 'f((x)[:, 0])'
>     self.create_mark_checker(source)

tests/test_mark_tokens.py:597: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:90: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

source = '_\n(:, 0)', filename = '<unknown>', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            assert major == 3
            feature_version = minor
        elif feature_version is None:
            feature_version = -1
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           (:, 0)
E            ^
E       SyntaxError: invalid syntax

../../../.pybuild/bleeding/lib/python3.9/ast.py:50: SyntaxError
__________________________________________________________________ TestMarkTokens.test_decorators ___________________________________________________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_decorators>

    def test_decorators(self):
      # See https://bitbucket.org/plas/thonny/issues/49/range-marker-fails-with-decorators
      source = textwrap.dedent("""
        @deco1
        def f():
          pass
        @deco2(a=1)
        def g(x):
          pass
    
        @deco3()
        def g(x):
          pass
      """)
>     m = self.create_mark_checker(source)

tests/test_mark_tokens.py:491: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:32: in create_mark_checker
    atok = self.create_asttokens(source)
tests/test_mark_tokens.py:48: in create_asttokens
    return ASTTokens(source, parse=True)
asttokens/asttokens.py:65: in __init__
    self.mark_tokens(self._tree)
asttokens/asttokens.py:76: in mark_tokens
    MarkTokens(self).visit_tree(root_node)
asttokens/mark_tokens.py:49: in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
asttokens/util.py:186: in visit_tree
    ret = postvisit(current, par_value, value)
asttokens/mark_tokens.py:92: in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
asttokens/mark_tokens.py:334: in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

token = Token(type=6, string='', start=(12, 0), end=(12, 0), line='', index=49, startpos=80, endpos=80), tok_type = 1, tok_str = 'a'

    def expect_token(token, tok_type, tok_str=None):
      """
      Verifies that the given token is of the expected type. If tok_str is given, the token string
      is verified too. If the token doesn't match, raises an informative ValueError.
      """
      if not match_token(token, tok_type, tok_str):
>       raise ValueError("Expected token %s, got %s on line %s col %s" % (
          token_repr(tok_type, tok_str), str(token),
          token.start[0], token.start[1] + 1))
E       ValueError: Expected token NAME:'a', got DEDENT:'' on line 12 col 1

asttokens/util.py:56: ValueError
___________________________________________________________________ TestMarkTokens.test_fixture8 ____________________________________________________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_fixture8>

>   def test_fixture8(self): self.verify_fixture_file('astroid/module.py')

tests/test_mark_tokens.py:171: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:133: in verify_fixture_file
    m = self.create_mark_checker(source, verify=False)
tests/test_mark_tokens.py:32: in create_mark_checker
    atok = self.create_asttokens(source)
tests/test_mark_tokens.py:48: in create_asttokens
    return ASTTokens(source, parse=True)
asttokens/asttokens.py:65: in __init__
    self.mark_tokens(self._tree)
asttokens/asttokens.py:76: in mark_tokens
    MarkTokens(self).visit_tree(root_node)
asttokens/mark_tokens.py:49: in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
asttokens/util.py:186: in visit_tree
    ret = postvisit(current, par_value, value)
asttokens/mark_tokens.py:92: in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
asttokens/mark_tokens.py:334: in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

token = Token(type=1, string='autre', start=(53, 12), end=(53, 17), line='            autre = [a for (a, b) in MY_DICT if b]\n', index=224, startpos=967, endpos=972)
tok_type = 1, tok_str = 'val'

    def expect_token(token, tok_type, tok_str=None):
      """
      Verifies that the given token is of the expected type. If tok_str is given, the token string
      is verified too. If the token doesn't match, raises an informative ValueError.
      """
      if not match_token(token, tok_type, tok_str):
>       raise ValueError("Expected token %s, got %s on line %s col %s" % (
          token_repr(tok_type, tok_str), str(token),
          token.start[0], token.start[1] + 1))
E       ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13

asttokens/util.py:56: ValueError
___________________________________________________________________ TestMarkTokens.test_fixture9 ____________________________________________________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_fixture9>

>   def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')

tests/test_mark_tokens.py:172: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:133: in verify_fixture_file
    m = self.create_mark_checker(source, verify=False)
tests/test_mark_tokens.py:32: in create_mark_checker
    atok = self.create_asttokens(source)
tests/test_mark_tokens.py:48: in create_asttokens
    return ASTTokens(source, parse=True)
asttokens/asttokens.py:65: in __init__
    self.mark_tokens(self._tree)
asttokens/asttokens.py:76: in mark_tokens
    MarkTokens(self).visit_tree(root_node)
asttokens/mark_tokens.py:49: in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
asttokens/util.py:186: in visit_tree
    ret = postvisit(current, par_value, value)
asttokens/mark_tokens.py:92: in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
asttokens/mark_tokens.py:334: in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

token = Token(type=1, string='e', start=(98, 0), end=(98, 1), line='e = d[a:b:c]\n', index=476, startpos=1233, endpos=1234), tok_type = 1, tok_str = 'file'

    def expect_token(token, tok_type, tok_str=None):
      """
      Verifies that the given token is of the expected type. If tok_str is given, the token string
      is verified too. If the token doesn't match, raises an informative ValueError.
      """
      if not match_token(token, tok_type, tok_str):
>       raise ValueError("Expected token %s, got %s on line %s col %s" % (
          token_repr(tok_type, tok_str), str(token),
          token.start[0], token.start[1] + 1))
E       ValueError: Expected token NAME:'file', got NAME:'e' on line 98 col 1

asttokens/util.py:56: ValueError
_______________________________________________________________ TestMarkTokens.test_keyword_arg_only ________________________________________________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_keyword_arg_only>

    def test_keyword_arg_only(self):
      # See https://bitbucket.org/plas/thonny/issues/52/range-marker-fails-with-ridastrip-split
      source = "f(x=1)\ng(a=(x),b=[y])"
>     m = self.create_mark_checker(source)

tests/test_mark_tokens.py:460: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:32: in create_mark_checker
    atok = self.create_asttokens(source)
tests/test_mark_tokens.py:48: in create_asttokens
    return ASTTokens(source, parse=True)
asttokens/asttokens.py:65: in __init__
    self.mark_tokens(self._tree)
asttokens/asttokens.py:76: in mark_tokens
    MarkTokens(self).visit_tree(root_node)
asttokens/mark_tokens.py:49: in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
asttokens/util.py:186: in visit_tree
    ret = postvisit(current, par_value, value)
asttokens/mark_tokens.py:92: in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
asttokens/mark_tokens.py:334: in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

token = Token(type=4, string='', start=(2, 14), end=(2, 15), line='', index=21, startpos=21, endpos=21), tok_type = 1, tok_str = 'x'

    def expect_token(token, tok_type, tok_str=None):
      """
      Verifies that the given token is of the expected type. If tok_str is given, the token string
      is verified too. If the token doesn't match, raises an informative ValueError.
      """
      if not match_token(token, tok_type, tok_str):
>       raise ValueError("Expected token %s, got %s on line %s col %s" % (
          token_repr(tok_type, tok_str), str(token),
          token.start[0], token.start[1] + 1))
E       ValueError: Expected token NAME:'x', got NEWLINE:'' on line 2 col 15

asttokens/util.py:56: ValueError
______________________________________________________________ TestMarkTokens.test_mark_tokens_simple _______________________________________________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_mark_tokens_simple>

    def test_mark_tokens_simple(self):
      source = tools.read_fixture('astroid', 'module.py')
>     m = self.create_mark_checker(source)

tests/test_mark_tokens.py:73: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:32: in create_mark_checker
    atok = self.create_asttokens(source)
tests/test_mark_tokens.py:48: in create_asttokens
    return ASTTokens(source, parse=True)
asttokens/asttokens.py:65: in __init__
    self.mark_tokens(self._tree)
asttokens/asttokens.py:76: in mark_tokens
    MarkTokens(self).visit_tree(root_node)
asttokens/mark_tokens.py:49: in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
asttokens/util.py:186: in visit_tree
    ret = postvisit(current, par_value, value)
asttokens/mark_tokens.py:92: in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
asttokens/mark_tokens.py:334: in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

token = Token(type=1, string='autre', start=(53, 12), end=(53, 17), line='            autre = [a for (a, b) in MY_DICT if b]\n', index=224, startpos=967, endpos=972)
tok_type = 1, tok_str = 'val'

    def expect_token(token, tok_type, tok_str=None):
      """
      Verifies that the given token is of the expected type. If tok_str is given, the token string
      is verified too. If the token doesn't match, raises an informative ValueError.
      """
      if not match_token(token, tok_type, tok_str):
>       raise ValueError("Expected token %s, got %s on line %s col %s" % (
          token_repr(tok_type, tok_str), str(token),
          token.start[0], token.start[1] + 1))
E       ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13

asttokens/util.py:56: ValueError
___________________________________________________________________ TestMarkTokens.test_nonascii ____________________________________________________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_nonascii>

    def test_nonascii(self):
      # Test of https://bitbucket.org/plas/thonny/issues/162/weird-range-marker-crash-with-non-ascii
      # Only on PY3 because Py2 doesn't support unicode identifiers.
      for source in (
        "sünnikuupäev=str((18+int(isikukood[0:1])-1)//2)+isikukood[1:3]",
        "sünnikuupaev=str((18+int(isikukood[0:1])-1)//2)+isikukood[1:3]"):
>       m = self.create_mark_checker(source)

tests/test_mark_tokens.py:286: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:90: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

source = '_\n(0:1)', filename = '<unknown>', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            assert major == 3
            feature_version = minor
        elif feature_version is None:
            feature_version = -1
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           (0:1)
E             ^
E       SyntaxError: invalid syntax

../../../.pybuild/bleeding/lib/python3.9/ast.py:50: SyntaxError
____________________________________________________________________ TestMarkTokens.test_slices _____________________________________________________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_slices>

    def test_slices(self):
      # Make sure we don't fail on parsing slices of the form `foo[4:]`.
      source = "(foo.Area_Code, str(foo.Phone)[:3], str(foo.Phone)[3:], foo[:], bar[::2, :], [a[:]][::-1])"
>     m = self.create_mark_checker(source)

tests/test_mark_tokens.py:230: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:90: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

source = '_\n(:3)', filename = '<unknown>', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            assert major == 3
            feature_version = minor
        elif feature_version is None:
            feature_version = -1
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           (:3)
E            ^
E       SyntaxError: invalid syntax

../../../.pybuild/bleeding/lib/python3.9/ast.py:50: SyntaxError
__________________________________________________________________ TestMarkTokens.test_sys_modules __________________________________________________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_sys_modules>

    def test_sys_modules(self):
      """
      Verify all nodes on source files obtained from sys.modules.
      This can take a long time as there are many modules,
      so it only tests all modules if the environment variable
      ASTTOKENS_SLOW_TESTS has been set.
      """
      modules = list(sys.modules.values())
      if not os.environ.get('ASTTOKENS_SLOW_TESTS'):
        modules = modules[:20]
    
      start = time()
      for module in modules:
        # Don't let this test (which runs twice) take longer than 13 minutes
        # to avoid the travis build time limit of 30 minutes
        if time() - start > 13 * 60:
          break
    
        try:
          filename = inspect.getsourcefile(module)
        except TypeError:
          continue
    
        if not filename:
          continue
    
        filename = os.path.abspath(filename)
        print(filename)
        try:
          with io.open(filename) as f:
            source = f.read()
        except OSError:
          continue
    
        # Astroid fails with a syntax error if a type comment is on its own line
        if self.is_astroid_test and re.search(r'^\s*# type: ', source, re.MULTILINE):
          print('Skipping', filename)
          continue
    
>       self.create_mark_checker(source)

tests/test_mark_tokens.py:639: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_mark_tokens.py:32: in create_mark_checker
    atok = self.create_asttokens(source)
tests/test_mark_tokens.py:48: in create_asttokens
    return ASTTokens(source, parse=True)
asttokens/asttokens.py:65: in __init__
    self.mark_tokens(self._tree)
asttokens/asttokens.py:76: in mark_tokens
    MarkTokens(self).visit_tree(root_node)
asttokens/mark_tokens.py:49: in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
asttokens/util.py:186: in visit_tree
    ret = postvisit(current, par_value, value)
asttokens/mark_tokens.py:92: in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
asttokens/mark_tokens.py:334: in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

token = Token(type=1, string='message', start=(235, 12), end=(235, 19), line="            message = '# ' + message\n", index=1087, startpos=7481, endpos=7488)
tok_type = 1, tok_str = 'file'

    def expect_token(token, tok_type, tok_str=None):
      """
      Verifies that the given token is of the expected type. If tok_str is given, the token string
      is verified too. If the token doesn't match, raises an informative ValueError.
      """
      if not match_token(token, tok_type, tok_str):
>       raise ValueError("Expected token %s, got %s on line %s col %s" % (
          token_repr(tok_type, tok_str), str(token),
          token.start[0], token.start[1] + 1))
E       ValueError: Expected token NAME:'file', got NAME:'message' on line 235 col 13

asttokens/util.py:56: ValueError
----------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------
/home/tcaswell/.pybuild/bleeding/lib/python3.9/importlib/_bootstrap.py
====================================================================== short test summary info ======================================================================
FAILED tests/test_astroid.py::TestAstroid::test_complex_slice_and_parens - astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
FAILED tests/test_astroid.py::TestAstroid::test_fixture9 - AssertionError: 'Slice(\n   lower=None,\n   upper=None,\n   step=None)' != "Name(name='d')"
FAILED tests/test_astroid.py::TestAstroid::test_nonascii - astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
FAILED tests/test_astroid.py::TestAstroid::test_slices - AssertionError: 'Slice(\n   lower=None,\n   upper=Const(value=3),\n   step=None)' != 'Const(value=3)'
FAILED tests/test_astroid.py::TestAstroid::test_sys_modules - AssertionError: "Slice(\n   lower=None,\n   upper=BinOp(\n[245 chars]one)" != "BinOp(\n   op='-',\n ...
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_assignment_expressions - ValueError: Expected token NAME:'x', got NAME:'y0' on line 17 col 1
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_complex_slice_and_parens -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_decorators - ValueError: Expected token NAME:'a', got DEDENT:'' on line 12 col 1
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_fixture8 - ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_fixture9 - ValueError: Expected token NAME:'file', got NAME:'e' on line 98 col 1
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_keyword_arg_only - ValueError: Expected token NAME:'x', got NEWLINE:'' on line 2 col 15
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_mark_tokens_simple - ValueError: Expected token NAME:'val', got NAME:'autre' on line 53 col 13
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_nonascii -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_slices -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_sys_modules - ValueError: Expected token NAME:'file', got NAME:'message' on line 235 col 13
======================================================== 15 failed, 91 passed, 1 skipped, 1 warning in 6.23s ========================================================

full IPython traceback

(bleeding) ✘  ~/source/other_source/asttokens [master {origin/master}|✔ ]
jupiter@17:52  ➤  ipython
Python 3.9.0a5+ (heads/master:799d7d61a9, Apr  6 2020, 17:38:49) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.0.0.dev -- An enhanced Interactive Python. Type '?' for help.

In [1]: 1/0                                                                                                                                                          
Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-9e1622b385b6>", line 1, in <module>
    1/0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'ZeroDivisionError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 261, in executing
    args = executing_cache[key]
KeyError: (<code object run_code at 0x7faaade683a0, file "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3293>, 140371038733216, 142)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3254, in run_ast_nodes
    if (await self.run_code(code, result,  async_=asy)):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3348, in run_code
    self.showtraceback(running_compiled_code=True)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2046, in showtraceback
    stb = self.InteractiveTB.structured_traceback(etype,
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 977, in structured_traceback
    return FormattedTB.structured_traceback(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 877, in structured_traceback
    return VerboseTB.structured_traceback(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 734, in structured_traceback
    formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 700, in format_exception_as_a_whole
    records = self.get_records(etb, number_of_lines_of_context, tb_offset)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 728, in get_records
    return list(stack_data.FrameInfo.stack_data(etb, options=options))[tb_offset:]
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 543, in stack_data
    yield from collapse_repeated(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/utils.py", line 80, in collapse_repeated
    yield from map(mapper, original_group)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 533, in mapper
    return cls(f, options)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 498, in __init__
    self.executing = Source.executing(frame_or_tb)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 263, in executing
    source = cls.for_frame(frame)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 212, in for_frame
    return cls.for_filename(frame.f_code.co_filename, frame.f_globals or {})
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 225, in for_filename
    result = source_cache[filename] = cls(filename, lines)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 81, in __init__
    self.asttokens()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 336, in asttokens
    return ASTTokens(
  File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 49, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 186, in visit_tree
    ret = postvisit(current, par_value, value)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 92, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 334, in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token NAME:'co_flags', got NAME:'new_code' on line 144 col 9

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'ValueError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 261, in executing
    args = executing_cache[key]
KeyError: (<code object run_ast_nodes at 0x7faaade68240, file "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3142>, 140371038732864, 690)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2886, in _run_cell
    return runner(coro)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/async_helpers.py", line 68, in _pseudo_sync_runner
    coro.send(None)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3062, in run_cell_async
    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3273, in run_ast_nodes
    self.showtraceback()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2046, in showtraceback
    stb = self.InteractiveTB.structured_traceback(etype,
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 977, in structured_traceback
    return FormattedTB.structured_traceback(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 877, in structured_traceback
    return VerboseTB.structured_traceback(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 734, in structured_traceback
    formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 700, in format_exception_as_a_whole
    records = self.get_records(etb, number_of_lines_of_context, tb_offset)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 728, in get_records
    return list(stack_data.FrameInfo.stack_data(etb, options=options))[tb_offset:]
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 543, in stack_data
    yield from collapse_repeated(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/utils.py", line 80, in collapse_repeated
    yield from map(mapper, original_group)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 533, in mapper
    return cls(f, options)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 498, in __init__
    self.executing = Source.executing(frame_or_tb)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 263, in executing
    source = cls.for_frame(frame)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 212, in for_frame
    return cls.for_filename(frame.f_code.co_filename, frame.f_globals or {})
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 225, in for_filename
    result = source_cache[filename] = cls(filename, lines)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 81, in __init__
    self.asttokens()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 336, in asttokens
    return ASTTokens(
  File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 49, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 186, in visit_tree
    ret = postvisit(current, par_value, value)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 92, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 334, in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token NAME:'co_flags', got NAME:'new_code' on line 144 col 9

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'ValueError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 261, in executing
    args = executing_cache[key]
KeyError: (<code object _run_cell at 0x7faaade60a80, file "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2865>, 140371038702208, 60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tcaswell/.virtualenvs/bleeding/bin/ipython", line 8, in <module>
    sys.exit(start_ipython())
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/__init__.py", line 126, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/terminal/ipapp.py", line 356, in start
    self.shell.mainloop()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/terminal/interactiveshell.py", line 559, in mainloop
    self.interact()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/terminal/interactiveshell.py", line 550, in interact
    self.run_cell(code, store_history=True)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2857, in run_cell
    result = self._run_cell(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2891, in _run_cell
    self.showtraceback(running_compiled_code=True)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 2046, in showtraceback
    stb = self.InteractiveTB.structured_traceback(etype,
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 977, in structured_traceback
    return FormattedTB.structured_traceback(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 877, in structured_traceback
    return VerboseTB.structured_traceback(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 734, in structured_traceback
    formatted_exception = self.format_exception_as_a_whole(etype, evalue, etb, number_of_lines_of_context,
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 700, in format_exception_as_a_whole
    records = self.get_records(etb, number_of_lines_of_context, tb_offset)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/IPython/core/ultratb.py", line 728, in get_records
    return list(stack_data.FrameInfo.stack_data(etb, options=options))[tb_offset:]
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 543, in stack_data
    yield from collapse_repeated(
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/utils.py", line 80, in collapse_repeated
    yield from map(mapper, original_group)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 533, in mapper
    return cls(f, options)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 498, in __init__
    self.executing = Source.executing(frame_or_tb)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 263, in executing
    source = cls.for_frame(frame)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 212, in for_frame
    return cls.for_filename(frame.f_code.co_filename, frame.f_globals or {})
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 225, in for_filename
    result = source_cache[filename] = cls(filename, lines)
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/stack_data/core.py", line 81, in __init__
    self.asttokens()
  File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.9/site-packages/executing.py", line 336, in asttokens
    return ASTTokens(
  File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 49, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 186, in visit_tree
    ret = postvisit(current, par_value, value)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 92, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/mark_tokens.py", line 334, in visit_keyword
    util.expect_token(name, token.NAME, node.arg)
  File "/home/tcaswell/source/other_source/asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token NAME:'co_flags', got NAME:'new_code' on line 144 col 9

If you suspect this is an IPython 8.0.0.dev bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at [email protected]

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

attn @alexmojaki @Carreau

xonsh shell tree

Hello! Thank you for asttokens!

I'm wondering can I use asttokens with xonsh shell syntax tree that is a superset of Python?

For example I replaced tree to xonsh and it works with pure python:

# To run this code just do `pip install xonsh`, run `xonsh` and copy-paste this code
import ast, asttokens
st='''
def greet(a):
  return b
'''
atok = asttokens.ASTTokens(
                  source_text=st, 
                  parse=False,
                  tree=__xonsh__.execer.parse(st, ctx=__xonsh__.ctx) # ast.Expression
                 )


for node in ast.walk(atok.tree):
  if hasattr(node, 'lineno'):
    print(atok.get_text_range(node), node.__class__.__name__, atok.get_text(node))

Result:

(1, 25) FunctionDef def greet(a):
  return b
(17, 25) Return return b
(11, 12) arg a
(24, 25) Name b

The token positions - it's what I need.

But when I set:

st="echo 1" # subprocess command

I've got traceback:

# Run `$XONSH_SHOW_TRACEBACK = True` in xonsh to show traceback
Traceback (most recent call last):
  File "/opt/miniconda/lib/python3.8/site-packages/xonsh/base_shell.py", line 362, in default
    run_compiled_code(code, self.ctx, None, "single")
  File "/opt/miniconda/lib/python3.8/site-packages/xonsh/codecache.py", line 67, in run_compiled_code
    func(code, glb, loc)
  File "<xonsh-code>", line 3, in <module>
  File "/opt/miniconda/lib/python3.8/site-packages/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "/opt/miniconda/lib/python3.8/site-packages/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "/opt/miniconda/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 49, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "/opt/miniconda/lib/python3.8/site-packages/asttokens/util.py", line 199, in visit_tree
    ret = postvisit(current, par_value, value)
  File "/opt/miniconda/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 92, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File "/opt/miniconda/lib/python3.8/site-packages/asttokens/mark_tokens.py", line 189, in handle_attr
    name = self._code.next_token(dot)
  File "/opt/miniconda/lib/python3.8/site-packages/asttokens/asttokens.py", line 141, in next_token
    while is_non_coding_token(self._tokens[i].type):
IndexError: list index out of range

Could you please help or advice can I achieve token positions using asttokens and xonsh shell ast parser? If no could you point out to another way?

Thanks!

`asttokens` 2.0.6 has undeclared dependency on `astroid`

Attempting to import asttokens when the astroid package is not installed results an a ModuleNotFoundError:

$ python3 -c "import asttokens"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/asttokens-issue/venv/lib/python3.8/site-packages/asttokens/__init__.py", line 22, in <module>
    from .asttokens import ASTTokens
  File "/tmp/asttokens-issue/venv/lib/python3.8/site-packages/asttokens/asttokens.py", line 20, in <module>
    from .util import Token, match_token, is_non_coding_token, AstNode, patched_generate_tokens
  File "/tmp/asttokens-issue/venv/lib/python3.8/site-packages/asttokens/util.py", line 26, in <module>
    from astroid.node_classes import NodeNG  # type: ignore[import]
ModuleNotFoundError: No module named 'astroid'

astroid is not listed as a dependency in setup.cfg, only six and typing, so it does not get installed when asttokens is:

asttokens/setup.cfg

Lines 36 to 38 in beb8a4e

 install_requires = 

 six 

 typing; python_version < "3.5"

Not work properly on non-ascii source code

Consider a clearer optional-functionality API

Currently ASTTokens serves a number of different roles depending on what arguments its passed. With #93 this is growing further and the workarounds to type checking being added in that PR perhaps suggest that the limits of the current approach are being reached. #95 will add another construction, though not another role.
I understand the goals of these different roles are around avoiding redundant parsing/tree-building work where not needed, which is a reasonable goal. I'm not suggesting a removal of functionality here, rather that the functionality be rearranged for clarity and to aid type checking.

There are at least the following modes of use:

atok = ASTTokens(source, parse=True)
assert atok.tree, "Placate type check even though we know this is always valid"

atok = ASTTokens(source, tree=existing_tree)
assert atok.tree, "Placate type check even though we know this is always valid"

There is also currently ASTTokens(source, tree=None, parse=False), though it's not clear what use that has (#94 (comment)).

With #93 there's also this case:

atok = ASTTokens(source, init_tokens=False)
atok.get_token_from_offset(42)  # Fail at runtime, but no type-check warnings

These cases are somewhat "obvious", however if the construction of the ASTTokens instance is far from the point of use they could easily be very non-obvious.

We can observe that ASTTokens contains broadly three classes of functionality:

tokens-only
tree-only
both together

With the goal of enabling efficient use of the related functionalities I can see a couple of paths forwards:

lazy evaluation of the relevant structured data from the source
failures when using functionality with missing data

For the former I would advocate for moving all the public-member types to being non-optional and using something like https://pypi.org/project/cached-property/ (or functools.cached_property in Python 3.8+) to hide the evaluations. This would still support users passing in pre-computed values if they want but means that all usages will "just work":

atok = ASTTokens(source)
tree: ast.AST = atok.tree  # no error now that `.tree` is non-optional

atok = ASTTokens(source, init_tokens=False)  # request lazy tokens initialisation
atok.get_token_from_offset(42)  # works due to lazy parsing

I can see that this approach may be undesirable in some cases, perhaps where there are reasons to prefer not doing the parsing at all.

This leads to the other approach (errors for missing functionality), for which I propose using separate classes:

# ASTSource is a thin wrapper around LineNumbers. (Very open to better names.)
# Does not take a tree or tokens at all, but works with un-marked AST nodes
asrc = ASTSource(source)
asrc.tree  # type check error: no such member
asrc.get_text(node)  # type check error: no such member
asrc.get_token_from_offset(42)  # type check error: no such member
asrc.get_text_unmarked(node)  # works fine

# TokensSource is a new type which contains the tokens/source utils currently in ASTTokens
# It wraps LineNumbers similarly to ASTSource, but is otherwise unrelated
# It does not have a tree
tsrc = TokensSource(source)  # parses tokens if not passed
tsrc.tree  # type check error: no such member
tsrc.get_text(node)  # type check error: no such member
tsrc.get_token_from_offset(42)  # works fine
tsrc.get_text_unmarked(node)  # type check error: no such member

# ASTTokens now always has a tree and tokens, creating them both if not required
# It probably inherits from TokensSource and drops the `get_text*_unmarked` methods as they no longer make sense here
atok = ASTTokens(source)
tree: ast.AST = atok.tree  # no error now that `.tree` is non-optional
atok.get_text(node)  # works fine
atok.get_token_from_offset(42)  # also works fine
atok.get_text_unmarked(node)  # type check error: no such member

Aside from the question of whether ASTTokens keeps its get_text*_unmarked methods, the usages here are essentially compatible with existing code. The constructor signature might change, though keeping the existing arguments for compatibility is probably possible; if so then ASTTokens(source, tree=None, parse=False) would now be at least a runtime error (with @overloads it could be a type error too).

Note also that this approach is compatible with another class, perhaps LazyASTTokens which looks like ASTTokens but has lazy evaluation. This could easily be added later (or not at all) depending on whether it ends up being useful.

The main gain here is that each of these specialised classes addresses a specific use-case and data-requirements, which remove the need for conditional presence checks (or asserts) throughout both the library code and client code. This will make the code clearer, easier to maintain and (due to the type checker support) more robust as well as offering the performance gains desired.

Debian package

Hi!

First, thank you for this nice library! I wrote similar (but much more buggy) solution for my project (https://thonny.org), but now I'd like to replace my implementation with asttokens. At the same time I'd like to continue providing Debian package for my project and for this, all my dependencies should be also available in Debian repo.

Have you considered publishing asttokens on Debian? If you don't have time for this, then I could prepare the package myself and try to get it accepted. With luck it may be even possible to get it into upcoming Debian 10.

IndexError on f((x)[:, 0])

I've had this for a while but only now found the time to boil down to a minimal reproducing case. :X

Reproducer:

import asttokens
asttokens.ASTTokens('f((x)[:, 0])', parse=True)

Running this file gives:

Traceback (most recent call last):
  File "/tmp/reproducer.py", line 2, in <module>
    asttokens.ASTTokens('f((x)[:, 0])', parse=True)
  File "asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File "asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "asttokens/util.py", line 192, in visit_tree
    ret = postvisit(current, par_value, value)
  File "asttokens/mark_tokens.py", line 87, in _visit_after_children
    first, last = self._expand_to_matching_pairs(first, last, node)
  File "asttokens/mark_tokens.py", line 142, in _expand_to_matching_pairs
    last = self._code.next_token(last_token)
  File "asttokens/asttokens.py", line 141, in next_token
    while is_non_coding_token(self._tokens[i].type):
IndexError: list index out of range

find_token() skips comments

I'm trying to find the comment related to an AST node of the python source code I am analysing:

x = 1  # my comment

I tried

import tokenize
atok.find_token(node.last_token, tokenize.COMMENT)

where node is an AST node e.g. an AST Name node for 'x'

The find never works. Looking at the source code of asttokens, I think it is because when find_token() calls next_token() to iterate through the tokens, it never passes through True to the include_extra parameter of next_token().

Any chance of adding an include_extra parameter to find_token() and passing that through to next_token() ? You seem to have that parameter everywhere else!

Compatibility with Python 3.12 and unreleased astroid 3.0.0

Hello,

I'd like to ask for help, in Fedora, we are rebuilding all Python packages with the upcoming Python 3.12.0. I've tested astroid from the main branch (last commit pylint-dev/astroid@8d57ce2 at this time) and it works with 3.12, but asttokens fails with following failures. Thank you.

=================================== FAILURES ===================================
___________________ TestAstroid.test_adjacent_joined_strings ___________________

self = <astroid.builder.AstroidBuilder object at 0x7f28a8c00ad0>
data = "_\n(f')", modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
>           node, parser_module = _parse_string(data, type_comments=True)

/usr/lib/python3.12/site-packages/astroid/builder.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = "_\n(f')", type_comments = True

    def _parse_string(
        data: str, type_comments: bool = True
    ) -> tuple[ast.Module, ParserModule]:
        parser_module = get_parser_module(type_comments=type_comments)
        try:
>           parsed = parser_module.parse(data + "\n", type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/builder.py:482: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ParserModule(unary_op_classes={<class 'ast.UAdd'>: '+', <class 'ast.USub'>: '-', <class 'ast.Not'>: 'not', <class 'ast...<class 'ast.Store'>: <Context.Store: 2>, <class 'ast.Del'>: <Context.Del: 3>, <class 'ast.Param'>: <Context.Store: 2>})
string = "_\n(f')\n", type_comments = True

    def parse(self, string: str, type_comments: bool = True) -> ast.Module:
>       return ast.parse(string, type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/_ast.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

source = "_\n(f')\n", filename = '<unknown>', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if feature_version is None:
            feature_version = -1
        elif isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            if major != 3:
                raise ValueError(f"Unsupported major version: {major}")
            feature_version = minor
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           (f')
E            ^
E       SyntaxError: unterminated f-string literal (detected at line 2)

/usr/lib64/python3.12/ast.py:52: SyntaxError

The above exception was the direct cause of the following exception:

self = <tests.test_astroid.TestAstroid testMethod=test_adjacent_joined_strings>

        def test_adjacent_joined_strings(self):
            source = """
    foo = f'x y z' \\
    f'''a b c''' f"u v w"
    bar = ('x y z'   # comment2
           'a b c'   # comment3
           f'u v w'
          )
    """
>           m = self.create_mark_checker(source)

tests/test_mark_tokens.py:327: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:49: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:107: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:801: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
/usr/lib/python3.12/site-packages/astroid/builder.py:305: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.12/site-packages/astroid/builder.py:146: in string_build
    module, builder = self._data_build(data, modname, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astroid.builder.AstroidBuilder object at 0x7f28a8c00ad0>
data = "_\n(f')", modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
            node, parser_module = _parse_string(data, type_comments=True)
        except (TypeError, ValueError, SyntaxError) as exc:
>           raise AstroidSyntaxError(
                "Parsing Python code failed:\n{error}",
                source=data,
                modname=modname,
                path=path,
                error=exc,
            ) from exc
E           astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E           unterminated f-string literal (detected at line 2) (<unknown>, line 2)

/usr/lib/python3.12/site-packages/astroid/builder.py:178: AstroidSyntaxError
__________________________ TestAstroid.test_fixture9 ___________________________

self = <astroid.builder.AstroidBuilder object at 0x7f28a80692e0>
data = '_\nclass Aaaa(base):', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
>           node, parser_module = _parse_string(data, type_comments=True)

/usr/lib/python3.12/site-packages/astroid/builder.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = '_\nclass Aaaa(base):', type_comments = True

    def _parse_string(
        data: str, type_comments: bool = True
    ) -> tuple[ast.Module, ParserModule]:
        parser_module = get_parser_module(type_comments=type_comments)
        try:
>           parsed = parser_module.parse(data + "\n", type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/builder.py:482: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ParserModule(unary_op_classes={<class 'ast.UAdd'>: '+', <class 'ast.USub'>: '-', <class 'ast.Not'>: 'not', <class 'ast...<class 'ast.Store'>: <Context.Store: 2>, <class 'ast.Del'>: <Context.Del: 3>, <class 'ast.Param'>: <Context.Store: 2>})
string = '_\nclass Aaaa(base):\n', type_comments = True

    def parse(self, string: str, type_comments: bool = True) -> ast.Module:
>       return ast.parse(string, type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/_ast.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

source = '_\nclass Aaaa(base):\n', filename = '<unknown>', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if feature_version is None:
            feature_version = -1
        elif isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            if major != 3:
                raise ValueError(f"Unsupported major version: {major}")
            feature_version = minor
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           class Aaaa(base):
E                            ^
E       IndentationError: expected an indented block after class definition on line 2

/usr/lib64/python3.12/ast.py:52: IndentationError

The above exception was the direct cause of the following exception:

self = <tests.test_astroid.TestAstroid testMethod=test_fixture9>

>   def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')

tests/test_mark_tokens.py:181: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:140: in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
tests/tools.py:107: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:804: in parse_snippet
    return self.module.parse('_\n' + text).body[1]
/usr/lib/python3.12/site-packages/astroid/builder.py:305: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.12/site-packages/astroid/builder.py:146: in string_build
    module, builder = self._data_build(data, modname, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astroid.builder.AstroidBuilder object at 0x7f28a80692e0>
data = '_\nclass Aaaa(base):', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
            node, parser_module = _parse_string(data, type_comments=True)
        except (TypeError, ValueError, SyntaxError) as exc:
>           raise AstroidSyntaxError(
                "Parsing Python code failed:\n{error}",
                source=data,
                modname=modname,
                path=path,
                error=exc,
            ) from exc
E           astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E           expected an indented block after class definition on line 2 (<unknown>, line 2)

/usr/lib/python3.12/site-packages/astroid/builder.py:178: AstroidSyntaxError
__________________________ TestAstroid.test_fstrings ___________________________

self = <astroid.builder.AstroidBuilder object at 0x7f28a806bbf0>
data = '_\n((f")', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
>           node, parser_module = _parse_string(data, type_comments=True)

/usr/lib/python3.12/site-packages/astroid/builder.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = '_\n((f")', type_comments = True

    def _parse_string(
        data: str, type_comments: bool = True
    ) -> tuple[ast.Module, ParserModule]:
        parser_module = get_parser_module(type_comments=type_comments)
        try:
>           parsed = parser_module.parse(data + "\n", type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/builder.py:482: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ParserModule(unary_op_classes={<class 'ast.UAdd'>: '+', <class 'ast.USub'>: '-', <class 'ast.Not'>: 'not', <class 'ast...<class 'ast.Store'>: <Context.Store: 2>, <class 'ast.Del'>: <Context.Del: 3>, <class 'ast.Param'>: <Context.Store: 2>})
string = '_\n((f")\n', type_comments = True

    def parse(self, string: str, type_comments: bool = True) -> ast.Module:
>       return ast.parse(string, type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/_ast.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

source = '_\n((f")\n', filename = '<unknown>', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if feature_version is None:
            feature_version = -1
        elif isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            if major != 3:
                raise ValueError(f"Unsupported major version: {major}")
            feature_version = minor
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           ((f")
E             ^
E       SyntaxError: unterminated f-string literal (detected at line 2)

/usr/lib64/python3.12/ast.py:52: SyntaxError

The above exception was the direct cause of the following exception:

self = <tests.test_astroid.TestAstroid testMethod=test_fstrings>

    def test_fstrings(self):
      for source in (
        '(f"He said his name is {name!r}.",)',
        "f'{function(kwarg=24)}'",
        'a = f"""result: {value:{width}.{precision}}"""',
        """[f"abc {a['x']} def"]""",
        "def t():\n  return f'{function(kwarg=24)}'"):
>       self.create_mark_checker(source)

tests/test_mark_tokens.py:316: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:49: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:107: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:801: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
/usr/lib/python3.12/site-packages/astroid/builder.py:305: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.12/site-packages/astroid/builder.py:146: in string_build
    module, builder = self._data_build(data, modname, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astroid.builder.AstroidBuilder object at 0x7f28a806bbf0>
data = '_\n((f")', modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
            node, parser_module = _parse_string(data, type_comments=True)
        except (TypeError, ValueError, SyntaxError) as exc:
>           raise AstroidSyntaxError(
                "Parsing Python code failed:\n{error}",
                source=data,
                modname=modname,
                path=path,
                error=exc,
            ) from exc
E           astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E           unterminated f-string literal (detected at line 2) (<unknown>, line 2)

/usr/lib/python3.12/site-packages/astroid/builder.py:178: AstroidSyntaxError
_________________________ TestAstroid.test_sys_modules _________________________

self = <astroid.builder.AstroidBuilder object at 0x7f28a750fcb0>
data = "_\n(_DeadlockError(f')", modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
>           node, parser_module = _parse_string(data, type_comments=True)

/usr/lib/python3.12/site-packages/astroid/builder.py:176: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = "_\n(_DeadlockError(f')", type_comments = True

    def _parse_string(
        data: str, type_comments: bool = True
    ) -> tuple[ast.Module, ParserModule]:
        parser_module = get_parser_module(type_comments=type_comments)
        try:
>           parsed = parser_module.parse(data + "\n", type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/builder.py:482: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = ParserModule(unary_op_classes={<class 'ast.UAdd'>: '+', <class 'ast.USub'>: '-', <class 'ast.Not'>: 'not', <class 'ast...<class 'ast.Store'>: <Context.Store: 2>, <class 'ast.Del'>: <Context.Del: 3>, <class 'ast.Param'>: <Context.Store: 2>})
string = "_\n(_DeadlockError(f')\n", type_comments = True

    def parse(self, string: str, type_comments: bool = True) -> ast.Module:
>       return ast.parse(string, type_comments=type_comments)

/usr/lib/python3.12/site-packages/astroid/_ast.py:26: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

source = "_\n(_DeadlockError(f')\n", filename = '<unknown>', mode = 'exec'

    def parse(source, filename='<unknown>', mode='exec', *,
              type_comments=False, feature_version=None):
        """
        Parse the source into an AST node.
        Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
        Pass type_comments=True to get back type comments where the syntax allows.
        """
        flags = PyCF_ONLY_AST
        if type_comments:
            flags |= PyCF_TYPE_COMMENTS
        if feature_version is None:
            feature_version = -1
        elif isinstance(feature_version, tuple):
            major, minor = feature_version  # Should be a 2-tuple.
            if major != 3:
                raise ValueError(f"Unsupported major version: {major}")
            feature_version = minor
        # Else it should be an int giving the minor version for 3.x.
>       return compile(source, filename, mode, flags,
                       _feature_version=feature_version)
E         File "<unknown>", line 2
E           (_DeadlockError(f')
E                           ^
E       SyntaxError: unterminated f-string literal (detected at line 2)

/usr/lib64/python3.12/ast.py:52: SyntaxError

The above exception was the direct cause of the following exception:

self = <tests.test_astroid.TestAstroid testMethod=test_sys_modules>

    def test_sys_modules(self):
      """
      Verify all nodes on source files obtained from sys.modules.
      This can take a long time as there are many modules,
      so it only tests all modules if the environment variable
      ASTTOKENS_SLOW_TESTS has been set.
      """
      modules = list(sys.modules.values())
      if not os.environ.get('ASTTOKENS_SLOW_TESTS'):
        modules = modules[:20]
    
      start = time()
      for module in modules:
        # Don't let this test (which runs twice) take longer than 13 minutes
        # to avoid the travis build time limit of 30 minutes
        if time() - start > 13 * 60:
          break
    
        try:
          filename = inspect.getsourcefile(module)
        except TypeError:
          continue
    
        if not filename:
          continue
    
        filename = os.path.abspath(filename)
        print(filename)
        try:
          with io.open(filename) as f:
            source = f.read()
        except OSError:
          continue
    
        if self.is_astroid_test and (
            # Astroid fails with a syntax error if a type comment is on its own line
            re.search(r'^\s*# type: ', source, re.MULTILINE)
            # Astroid can fail on this file, specifically raising an exception at this line of code:
            #     lambda node: node.name == "NamedTuple" and node.parent.name == "typing"
            # with the error:
            #     AttributeError: 'If' object has no attribute 'name'
            # See https://github.com/gristlabs/asttokens/runs/7602147792
            # I think the code that causes the problem is:
            #     if sys.version_info >= (3, 11):
            #         NamedTuple = typing.NamedTuple
            or filename.endswith("typing_extensions.py")
        ):
          print('Skipping', filename)
          continue
    
>       self.create_mark_checker(source)

tests/test_mark_tokens.py:673: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:49: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:107: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:801: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
/usr/lib/python3.12/site-packages/astroid/builder.py:305: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.12/site-packages/astroid/builder.py:146: in string_build
    module, builder = self._data_build(data, modname, path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astroid.builder.AstroidBuilder object at 0x7f28a750fcb0>
data = "_\n(_DeadlockError(f')", modname = '', path = None

    def _data_build(
        self, data: str, modname: str, path: str | None
    ) -> tuple[nodes.Module, rebuilder.TreeRebuilder]:
        """Build tree node from data and add some informations."""
        try:
            node, parser_module = _parse_string(data, type_comments=True)
        except (TypeError, ValueError, SyntaxError) as exc:
>           raise AstroidSyntaxError(
                "Parsing Python code failed:\n{error}",
                source=data,
                modname=modname,
                path=path,
                error=exc,
            ) from exc
E           astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E           unterminated f-string literal (detected at line 2) (<unknown>, line 2)

/usr/lib/python3.12/site-packages/astroid/builder.py:178: AstroidSyntaxError
----------------------------- Captured stdout call -----------------------------
/usr/lib64/python3.12/importlib/_bootstrap.py
_________________ TestMarkTokens.test_adjacent_joined_strings __________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_adjacent_joined_strings>

        def test_adjacent_joined_strings(self):
            source = """
    foo = f'x y z' \\
    f'''a b c''' f"u v w"
    bar = ('x y z'   # comment2
           'a b c'   # comment3
           f'u v w'
          )
    """
>           m = self.create_mark_checker(source)

tests/test_mark_tokens.py:327: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:49: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:84: in verify_all_nodes
    self.check_get_text_tokenless(node, test_case, text)
tests/tools.py:142: in check_get_text_tokenless
    test_case.assertEqual(text, text_tokenless, ast.dump(node))
E   AssertionError: "f'" != 'f\'x y z\' \\\nf\'\'\'a b c\'\'\' f"u v w"'
E   - f'
E   + f'x y z' \
E   + f'''a b c''' f"u v w"
E    : JoinedStr(values=[Constant(value='x y za b cu v w')])
_________________________ TestMarkTokens.test_fstrings _________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_fstrings>

    def test_fstrings(self):
      for source in (
        '(f"He said his name is {name!r}.",)',
        "f'{function(kwarg=24)}'",
        'a = f"""result: {value:{width}.{precision}}"""',
        """[f"abc {a['x']} def"]""",
        "def t():\n  return f'{function(kwarg=24)}'"):
>       self.create_mark_checker(source)

tests/test_mark_tokens.py:316: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:49: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:84: in verify_all_nodes
    self.check_get_text_tokenless(node, test_case, text)
tests/tools.py:142: in check_get_text_tokenless
    test_case.assertEqual(text, text_tokenless, ast.dump(node))
E   AssertionError: '(f"' != '(f"He said his name is {name!r}.",)'
E   - (f"
E   + (f"He said his name is {name!r}.",)
E    : Tuple(elts=[JoinedStr(values=[Constant(value='He said his name is '), FormattedValue(value=Name(id='name', ctx=Load()), conversion=114), Constant(value='.')])], ctx=Load())
_______________________ TestMarkTokens.test_sys_modules ________________________

self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_sys_modules>

    def test_sys_modules(self):
      """
      Verify all nodes on source files obtained from sys.modules.
      This can take a long time as there are many modules,
      so it only tests all modules if the environment variable
      ASTTOKENS_SLOW_TESTS has been set.
      """
      modules = list(sys.modules.values())
      if not os.environ.get('ASTTOKENS_SLOW_TESTS'):
        modules = modules[:20]
    
      start = time()
      for module in modules:
        # Don't let this test (which runs twice) take longer than 13 minutes
        # to avoid the travis build time limit of 30 minutes
        if time() - start > 13 * 60:
          break
    
        try:
          filename = inspect.getsourcefile(module)
        except TypeError:
          continue
    
        if not filename:
          continue
    
        filename = os.path.abspath(filename)
        print(filename)
        try:
          with io.open(filename) as f:
            source = f.read()
        except OSError:
          continue
    
        if self.is_astroid_test and (
            # Astroid fails with a syntax error if a type comment is on its own line
            re.search(r'^\s*# type: ', source, re.MULTILINE)
            # Astroid can fail on this file, specifically raising an exception at this line of code:
            #     lambda node: node.name == "NamedTuple" and node.parent.name == "typing"
            # with the error:
            #     AttributeError: 'If' object has no attribute 'name'
            # See https://github.com/gristlabs/asttokens/runs/7602147792
            # I think the code that causes the problem is:
            #     if sys.version_info >= (3, 11):
            #         NamedTuple = typing.NamedTuple
            or filename.endswith("typing_extensions.py")
        ):
          print('Skipping', filename)
          continue
    
>       self.create_mark_checker(source)

tests/test_mark_tokens.py:673: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_mark_tokens.py:49: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:84: in verify_all_nodes
    self.check_get_text_tokenless(node, test_case, text)
tests/tools.py:142: in check_get_text_tokenless
    test_case.assertEqual(text, text_tokenless, ast.dump(node))
E   AssertionError: "_DeadlockError(f'" != "_DeadlockError(f'deadlock detected by {self!r}')"
E   - _DeadlockError(f'
E   + _DeadlockError(f'deadlock detected by {self!r}')
E    : Call(func=Name(id='_DeadlockError', ctx=Load()), args=[JoinedStr(values=[Constant(value='deadlock detected by '), FormattedValue(value=Name(id='self', ctx=Load()), conversion=114)])], keywords=[])
----------------------------- Captured stdout call -----------------------------
/usr/lib64/python3.12/importlib/_bootstrap.py
____________________ TestTokenless.test_get_text_tokenless _____________________

self = <tests.test_tokenless.TestTokenless testMethod=test_get_text_tokenless>

    def test_get_text_tokenless(self):
      atok = ASTText(source)
    
      for node in ast.walk(atok.tree):
        if not isinstance(node, (ast.arguments, ast.arg)):
>         self.check_node(atok, node)

tests/test_tokenless.py:70: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_tokenless.py:108: in check_node
    self.assertEqual(atok_text, ast_text, node)
E   AssertionError: '' != '{xx + 22}'
E   + {xx + 22}
E    : <ast.FormattedValue object at 0x7f28a95a4ed0>
_____________________________ test_combine_tokens ______________________________

    def test_combine_tokens():
      from tokenize import TokenInfo, generate_tokens, ERRORTOKEN, OP, NUMBER, NAME
      from asttokens.util import combine_tokens, patched_generate_tokens
    
      text = "â„˜Â·2=1"
      original_tokens = list(generate_tokens(io.StringIO(text).readline))[:4]
>     assert original_tokens == [
        TokenInfo(ERRORTOKEN, string='â„˜', start=(1, 0), end=(1, 1), line='â„˜Â·2=1'),
        TokenInfo(ERRORTOKEN, string='Â·', start=(1, 1), end=(1, 2), line='â„˜Â·2=1'),
        TokenInfo(NUMBER, string='2', start=(1, 2), end=(1, 3), line='â„˜Â·2=1'),
        TokenInfo(OP, string='=', start=(1, 3), end=(1, 4), line='â„˜Â·2=1'),
      ]
E     AssertionError: assert [TokenInfo(ty...line='â„˜Â·2=1')] == [TokenInfo(ty...line='â„˜Â·2=1')]
E       At index 0 diff: TokenInfo(type=1 (NAME), string='â„˜Â·2', start=(1, 0), end=(1, 3), line='â„˜Â·2=1') != TokenInfo(type=66 (ERRORTOKEN), string='â„˜', start=(1, 0), end=(1, 1), line='â„˜Â·2=1')
E       Full diff:
E         [
E       -  TokenInfo(type=66 (ERRORTOKEN), string='â„˜', start=(1, 0), end=(1, 1), line='â„˜Â·2=1'),
E       -  TokenInfo(type=66 (ERRORTOKEN), string='Â·', start=(1, 1), end=(1, 2), line='â„˜Â·2=1'),
E       -  TokenInfo(type=2 (NUMBER), string='2', start=(1, 2), end=(1, 3), line='â„˜Â·2=1'),
E       ?                 ^   ^ - -                         ^...
E       
E       ...Full output truncated (6 lines hidden), use '-vv' to show

tests/test_util.py:126: AssertionError
=========================== short test summary info ============================
FAILED tests/test_astroid.py::TestAstroid::test_adjacent_joined_strings - ast...
FAILED tests/test_astroid.py::TestAstroid::test_fixture9 - astroid.exceptions...
FAILED tests/test_astroid.py::TestAstroid::test_fstrings - astroid.exceptions...
FAILED tests/test_astroid.py::TestAstroid::test_sys_modules - astroid.excepti...
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_adjacent_joined_strings
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_fstrings - AssertionEr...
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_sys_modules - Assertio...
FAILED tests/test_tokenless.py::TestTokenless::test_get_text_tokenless - Asse...
FAILED tests/test_util.py::test_combine_tokens - AssertionError: assert [Toke...
============ 9 failed, 106 passed, 1 skipped, 25 warnings in 4.10s =============

Statements with semicolons

Putting multiple statements on a line with semicolons causes get_text to return the full line when given a statement, e.g:

import ast

import asttokens

source = """
a; b
"""
atok = asttokens.ASTTokens(source, parse=True)
stmt = atok.tree.body[0]
print(ast.dump(stmt))         # Expr(value=Name(id='a', ctx=Load()))
print(atok.get_text(stmt))    # a; b

Docs outdated

The published docs at https://asttokens.readthedocs.io/en/latest/index.html appear to be somewhat outdated, having been built against c8697dc.
I'm not sure exactly what's changed in the API since then, however it would be great if they could reflect the current version.

Token spans for non-AST nodes

The main pain point I have with asttokens is when there's a non-AST object inside of the AST that semantically has a token span, but which asttokens does not expose (to the best of my knowledge). Two examples:

If you match foo.bar, what is the token span for bar? Unfortunately, the bar is a string object in the AST (Attribute.attr) and gets no first_token / last_token attributes. (Similar issues exist for alias nodes; see #27, which describes an attempted workaround) and other parts of the AST.) This makes it harder to replace one attribute with another, or change one import to another, things like that, without replacing the entire expr/stmt.
If you match [], and want to insert a new element to the list literal, you need to manually do the math, as there is no declared token span for the elts attribute, which is just an empty list. (For nonempty lists, you can use the token spans of the members of the list.)

Feature request: I would like to define a method on ASTTokens, something like get_span(node: AST, attr: str) -> Optional[(Token, Token)], which returns the first_token / last_token of an attribute, instead of of the node itself, as a best effort attempt. These would be either written down explicitly in mark_tokens off to the side (as e.g. a different attribute), or deduced on the fly.

Also, I would like some way to mark these inclusive/exclusive -- for example, in the empty list case, the token span for elts should be identical to the token span for the container, but exclusive of the [ and ] etc. Maybe an inclusive attribute on Token, or maybe just an extra pair of bool return values for get_span, something like this.

Context links: (1) is the main motivator for this limitation in my asttokens-using tool, and (2) will become a problem in ssbr/refex#6 (support for globs, like [$x...], which includes the empty list).

(If this sounds good, I can volunteer to do at least enough to make this work for the empty list case and probably the function call case, which I will need, and maybe some trivial easy ones like Attribute.attr. I don't want to try to solve it for everything though. :))

Support for constructing around a list of existing tokens

Thanks for this library, I've found it really useful for building source manipulation tools.

I'm currently building a flake8 plugin and it would be great if there was a way to build an ASTTokens instance from an existing list of (stdlib) tokens. This would complement the ability to pass in an existing AST tree and allow cases which already have both available to avoid re-parsing the source code. (Flake8 makes both an existing AST and tokens list available to plugins).

If this is something you'd be open to I'd be happy to put together a PR. (The change seems simple enough, though possibly I'm missing something)

DeprecationWarning: The 'astroid.node_classes' module is deprecated

asttokens 2.0.8 raises a DeprecationWarning in astroid-2.12.11:

import asttokens
source = "Robot('blue').walk(steps=10*n)"
atok = asttokens.ASTTokens(source, parse=True)

# ./lib/site-packages/astroid/node_classes.py:94: DeprecationWarning: 
# The 'astroid.node_classes' module is deprecated and will be replaced by
# 'astroid.nodes' in astroid 3.0.0

https://github.com/PyCQA/astroid/blob/7ed8c6db48faaab02eb57c94659aa54e5c9c3aa1/astroid/node_classes.py#L94-L98

By default Python doesn't show deprecation warnings. To see the warning the PYTHONWARNINGS=default env variable needs to be set or the above code run with python -Wd

https://docs.python.org/3/library/warnings.html#updating-code-for-new-versions-of-dependencies

Strange behavior with tokens of import nodes

Hi!

I tried the following code to parse import statements :

source = 'import asttokens as at'
atok = asttokens.ASTTokens(source, parse=True)

I get the following AST:

Module
  Import
    alias

Which is OK. My purpose is to gather the position of the identifiers asttokens and at in the code. Therefore I was trying to get the tokens composing the alias node, which sounds logic because asttokens as at is and alias definition. However when I ask for the tokens composing the alias node, using list(atok.get_tokens(atok.tree.body[0].names[0])) I get the following result :

[Token(type=1, string=u'import', start=(1, 0), end=(1, 6), line=u'import asttokens as at', index=0, startpos=0, endpos=6)]

Which is was not what I was expecting, I was expecting to have the ['asttokens', 'as', 'at'] tokens.

Is there something I am missing?

Best regards.

2.0.5 tag missing

We use tags to fetch releases from GitHub. Could you please push the missing 2.0.5 tag?

tests fail again on Fedora 30 with python 3.7.1

======================================================================
FAIL: test_unicode_offsets (tests.test_asttokens.TestASTTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/asttokens-1.1.10/tests/test_asttokens.py", line 108, in test_unicode_offsets
    "ENDMARKER:''"
AssertionError: Lists differ: ["NAM[61 chars], "OP:','", "NAME:'b'", "OP:')'", "NEWLINE:''", "ENDMARKER:''"] != ["NAM[61 chars], "OP:','", "NAME:'b'", "OP:')'", "ENDMARKER:''"]

First differing element 8:
"NEWLINE:''"
"ENDMARKER:''"

First list contains 1 additional elements.
First extra element 9:
"ENDMARKER:''"

  ["NAME:'foo'",
   "OP:'('",
   'STRING:"\'фыва\'"',
   "OP:','",
   "NAME:'a'",
   "OP:','",
   "NAME:'b'",
   "OP:')'",
-  "NEWLINE:''",
   "ENDMARKER:''"]

----------------------------------------------------------------------
Ran 76 tests in 2.071s

FAILED (SKIP=1, failures=1)

Frankly, I don't understand where the NEWLINE token comes from. The tests pass with python3-3.7.0-10.fc30.x86_64 and fail with python3-3.7.1-1.fc30.

Official support for 3.9 and maybe 3.10

#56 added testing for 3.9 but this should be made official in setup.cfg.

3.10 should also be tested.

This is based on aroberge/friendly#96 thinking that only 3.8 is supported.

Additional test failures with python 3.7

New stuff that wasn't listed in #16:

======================================================================
FAIL: test_adjacent_strings (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/asttokens-1.1.10/tests/test_mark_tokens.py", line 183, in test_adjacent_strings
    node_name + ":'x y z'   # comment2\n       'a b c'   # comment3\n       'u v w'"
AssertionError: Items in the first set but not the second:
"Const:'x y z'"
Items in the second set but not the first:
"Const:'x y z'   # comment2\n       'a b c'   # comment3\n       'u v w'"

======================================================================
FAIL: test_token_methods (tests.test_asttokens.TestASTTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/asttokens-1.1.10/tests/test_asttokens.py", line 46, in test_token_methods
    self.assertEqual(atok.prev_token(atok.tokens[5]), atok.tokens[3])
AssertionError: Token(type=56, string='\n', start=(2, 0), end=(2, 1)[40 chars]s=22) != Token(type=4, string='\n', start=(1, 20), end=(1, 21[61 chars]s=21)

======================================================================
FAIL: test_adjacent_strings (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/asttokens-1.1.10/tests/test_mark_tokens.py", line 183, in test_adjacent_strings
    node_name + ":'x y z'   # comment2\n       'a b c'   # comment3\n       'u v w'"
AssertionError: Items in the first set but not the second:
"Str:'x y z'"
Items in the second set but not the first:
"Str:'x y z'   # comment2\n       'a b c'   # comment3\n       'u v w'"

This is with
python3-3.7.0-1.fc29.x86_64
python3-astroid-2.0.0-0.4dev3.fc29.noarch

ValueError when parsing

When parsing this valid code:

[
    [
        1,
    ] for n in ()
]

A ValueError exception is raised:

ValueError: Expected token OP:'[', got ENDMARKER:'' on line 6 col 1

I believe it's related to #1 since removing the trailing comma does not throw, however I decided to file a separate issue just in case it's not.

Misannotation of comma token type as 54 not 53 when running Python 3.8

Hi there, thanks so much for writing this library 😄 I've used it within my library mvdef to get per-token column/line locations for tasks like identifying exactly where names and commas are in import statements while walking the AST of a program, to then be able to rearrange, rewrite, and move those import statements and function definitions between files in a well-organised way.

While working with your library though (in fact, just at the point of testing it on the original application that made me set out to write the library in the first place – I didn't have any trouble whatsoever with asttokens before now), I encountered a bug, which I've managed to reproduce in Python 3.8 (not sure what might be causing that I'm afraid, the code in question being tokenised is fully 3.7 compliant).

It's not catastrophic, but when I processed the AST of a generated line of code, its tokenised representation had commas as type 54 not 53. I simply added a check in case the type was not 53 to look for type 54 and just continued with the code, like I say it's not a catastrophic error.

asttokens.asttokens.token.tok_name identifies these two different numbers as indicating:

53: 'OP'
54: 'ERRORTOKEN'

From my attempts to reproduce a minimal example, I think this only occurs with Python 3.8

Here is a sample run in a Python 3.7 conda env (i.e. note that tko.tokens[4].type is 53)

>>> import asttokens
>>> mystr = "from A import B, C"
>>> tko = asttokens.ASTTokens(mystr)
>>> for t in tko.tokens: t
...
Token(type=1, string='from', start=(1, 0), end=(1, 4), line='from A import B, C', index=0, startpos=0, endpos=4)
Token(type=1, string='A', start=(1, 5), end=(1, 6), line='from A import B, C', index=1, startpos=5, endpos=6)
Token(type=1, string='import', start=(1, 7), end=(1, 13), line='from A import B, C', index=2, startpos=7, endpos=13)
Token(type=1, string='B', start=(1, 14), end=(1, 15), line='from A import B, C', index=3, startpos=14, endpos=15)
Token(type=53, string=',', start=(1, 15), end=(1, 16), line='from A import B, C', index=4, startpos=15, endpos=16)
Token(type=1, string='C', start=(1, 17), end=(1, 18), line='from A import B, C', index=5, startpos=17, endpos=18)
Token(type=4, string='', start=(1, 18), end=(1, 19), line='', index=6, startpos=18, endpos=18)
Token(type=0, string='', start=(2, 0), end=(2, 0), line='', index=7, startpos=18, endpos=18)

And here is the same code run in a Python 3.8 conda env (i.e. note that tko.tokens[4].type is 54)

>>> import asttokens
>>> mystr = "from A import B, C"
>>> tko = asttokens.ASTTokens(mystr)
>>> for t in tko.tokens: t
... 
Token(type=1, string='from', start=(1, 0), end=(1, 4), line='from A import B, C', index=0, startpos=0, endpos=4)
Token(type=1, string='A', start=(1, 5), end=(1, 6), line='from A import B, C', index=1, startpos=5, endpos=6)
Token(type=1, string='import', start=(1, 7), end=(1, 13), line='from A import B, C', index=2, startpos=7, endpos=13)
Token(type=1, string='B', start=(1, 14), end=(1, 15), line='from A import B, C', index=3, startpos=14, endpos=15)
Token(type=54, string=',', start=(1, 15), end=(1, 16), line='from A import B, C', index=4, startpos=15, endpos=16)
Token(type=1, string='C', start=(1, 17), end=(1, 18), line='from A import B, C', index=5, startpos=17, endpos=18)
Token(type=4, string='', start=(1, 18), end=(1, 19), line='', index=6, startpos=18, endpos=18)
Token(type=0, string='', start=(2, 0), end=(2, 0), line='', index=7, startpos=18, endpos=18)

I hope this is enough info for you to investigate, and do let me know if I can supply any more info.

Thanks again for developing this library, I've been recommending it to other devs 👍

Comma misannotation bug

I was reviewing some code I wrote almost 2 years ago and noted that I forgot to file a bug report (as far as I can see?), and instead simply worked around it, as documented in this commit:

fix bug relating to asttokens misannotating comma tokens' type as 54 (ERRORTOKEN) rather than 53
(OP), by just checking for comma tokens of type 54 matching the string ',', will inform asttokens devs

I think I forgot to do the "inform asttoken devs" part!

This was my workaround:

comma_tok = tko.find_token(al_as_tok, tok_type=53, tok_str=",")
if comma_tok.type == 0:
    # Due to an error in asttokens, sometimes tok_type is given as 54
    # although this should be an error (the failure tok_type is 0)
    comma_tok = tko.find_token(al_n_tok, tok_type=54, tok_str=",")

In other words, a non-error token for a comma is being annotated as an error (I think indicating parsing failure), when in fact no error took place in the parsing of the AST (it appears to have just been miscoded in this library and presumably as easily fixed).

I hope this is clear, at the time I found it fairly easy to show that this was what was happening but forgot to file it properly here.

Parser does not handle Python3.6 f-strings

An exception is raised when trying to parse python3.6's new f-strings.

  File "ensure_trailing_commas/trailing_comma_finder.py", line 118, in find_missing_trailing_commas
    atok = asttokens.ASTTokens(source_code, parse=True)
  File "asttokens/asttokens/asttokens.py", line 61, in __init__
    self.mark_tokens(self._tree)
  File "asttokens/asttokens/asttokens.py", line 72, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File "asttokens/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File "asttokens/asttokens/util.py", line 148, in visit_tree
    pvalue, post_value = previsit(current, par_value)
  File "asttokens/asttokens/mark_tokens.py", line 51, in _visit_before_children
    token = self._code.get_token_from_utf8(node.lineno, col) if col is not None else None
  File "asttokens/asttokens/asttokens.py", line 123, in get_token_from_utf8
    return self.get_token(lineno, self._line_numbers.from_utf8_col(lineno, col_offset))
  File "asttokens/asttokens/line_numbers.py", line 48, in from_utf8_col
    return offsets[max(0, min(len(offsets), utf8_column))]
IndexError: list index out of range

Fix travis build

https://travis-ci.org/github/gristlabs/asttokens is now marked as not working any more since their migration, and https://app.travis-ci.com/github/gristlabs/asttokens looks vaguely active, but not actually running anything. The latter probably needs switching on?

asttokens doesn't include the trailing paren in a call like `f({1: (2), 3: 4}, object())`

>>> def printbad(text):
  astt = asttokens.ASTTokens(text, parse=True)
  n = astt.tree.body[0].value  # the Call node
  print(astt.text[n.first_token.startpos: n.last_token.endpos])
... 

>>> printbad('f({1: (2), 3: 4}, object())')  # bad
f({1: (2), 3: 4}, object()

>>> printbad('f({1: (2), 3: 4}, object)')    # good
f({1: (2), 3: 4}, object)

Something about having a dict literal and parentheses inside the dict literal confuses asttokens, so that it is wrong about the end of the call expression and grabs the wrong parenthesis token.

(I attempted to minimize further, but without much luck.)

Timelines around support for legacy Python

Just wondering what the plans were for keeping supporting running this package under legacy Python?

As demonstrated by #106, doing so isn't free (even if it is relatively low effort), so it might be a good idea to consider dropping Python < 3.7 at some point.

It's worth noting that IPython, possibly the biggest package which depends in some form on asttokens hasn't supported < 3.5 for over 5 years now, which hopefully means that that isn't the driving reason to keep supporting the older versions.

Nodes that come from f-string parents lack token information

Because of https://bugs.python.org/issue29051, a temporary solution (See #4) was placed to avoid exceptions.

This issue is to track this workaround's life time so it could be fixed once the above python issue is fixed.

Test failure with py3.8 and astroid-2.6+: tests/test_astroid.py::TestAstroid::test_slices - AssertionError: Items in the first set but not the second:

I'm not sure if this is a problem in asttokens or astroid itself. If you believe it's the latter, please lemme know if I should file a bug there.

When running the test suite under Python 3.8 (3.9+ is fine, astroid<2.6 is fine), I get the following failure:

$ tox -e py38
GLOB sdist-make: /tmp/asttokens/setup.py
py38 create: /tmp/asttokens/.tox/py38
py38 installdeps: .[test]
py38 inst: /tmp/asttokens/.tox/.tmp/package/1/asttokens-2.0.5.zip
py38 installed: astroid==2.6.2,asttokens @ file:///tmp/asttokens/.tox/.tmp/package/1/asttokens-2.0.5.zip,attrs==21.2.0,iniconfig==1.1.1,lazy-object-proxy==1.6.0,packaging==21.0,pluggy==0.13.1,py==1.10.0,pyparsing==2.4.7,pytest==6.2.4,six==1.16.0,toml==0.10.2,wrapt==1.12.1
py38 run-test-pre: PYTHONHASHSEED='2657976898'
py38 run-test: commands[0] | pytest
========================================================= test session starts =========================================================
platform linux -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
cachedir: .tox/py38/.pytest_cache
rootdir: /tmp/asttokens, configfile: setup.cfg
collected 107 items                                                                                                                   

tests/test_astroid.py .............s..........................F......                                                           [ 43%]
tests/test_asttokens.py ......                                                                                                  [ 49%]
tests/test_line_numbers.py ...                                                                                                  [ 52%]
tests/test_mark_tokens.py ...............................................                                                       [ 96%]
tests/test_util.py ....                                                                                                         [100%]

============================================================== FAILURES ===============================================================
_______________________________________________________ TestAstroid.test_slices _______________________________________________________

self = <tests.test_astroid.TestAstroid testMethod=test_slices>

    def test_slices(self):
      # Make sure we don't fail on parsing slices of the form `foo[4:]`.
      source = "(foo.Area_Code, str(foo.Phone)[:3], str(foo.Phone)[3:], foo[:], bar[::2, :], [a[:]][::-1])"
      m = self.create_mark_checker(source)
      self.assertIn("Tuple:" + source, m.view_nodes_at(1, 0))
      self.assertEqual(m.view_nodes_at(1, 1),
                       { "Attribute:foo.Area_Code", "Name:foo" })
      self.assertEqual(m.view_nodes_at(1, 16),
                       { "Subscript:str(foo.Phone)[:3]", "Call:str(foo.Phone)", "Name:str"})
      self.assertEqual(m.view_nodes_at(1, 36),
                       { "Subscript:str(foo.Phone)[3:]", "Call:str(foo.Phone)", "Name:str"})
      # Slice and ExtSlice nodes are wrong, and in particular placed with parents. They are not very
      # important, so we skip them here.
      self.assertEqual({n for n in m.view_nodes_at(1, 56) if 'Slice:' not in n},
                       { "Subscript:foo[:]", "Name:foo" })
>     self.assertEqual({n for n in m.view_nodes_at(1, 64) if 'Slice:' not in n},
                       { "Subscript:bar[::2, :]", "Name:bar" })
E     AssertionError: Items in the first set but not the second:
E     'Tuple:bar[::2, :]'

tests/test_mark_tokens.py:242: AssertionError
======================================================= short test summary info =======================================================
FAILED tests/test_astroid.py::TestAstroid::test_slices - AssertionError: Items in the first set but not the second:
============================================== 1 failed, 105 passed, 1 skipped in 12.13s ==============================================
ERROR: InvocationError for command /tmp/asttokens/.tox/py38/bin/pytest (exited with code 1)
_______________________________________________________________ summary _______________________________________________________________
ERROR:   py38: commands failed

Inability to set source filename

For tooling, it is important to have the file names in order to print warnings/errors.

Currently, asttokens.ASTTokens does not allow to pass in the file name it's parsing.

It should be possible to do that here: https://github.com/gristlabs/asttokens/blob/master/asttokens/asttokens.py#L49

Replace deprecated imp module with importlib

This library uses the imp module which has been deprecated since Python 3.4 and set for removal in 3.12:

Raised PendingDeprecationWarning since 3.4 (2014)
Raised DeprecationWarning since 3.5 (2015)
Updated DeprecationWarning to say removal in 3.12 since 3.10 (2021)
Removal planned for 3.12 (2023)

Python 3.12 is set for release on 2023-10-02 and this library is one of the top 5,000 most-downloaded from PyPI.

Please could you upgrade to use importlib? The imp docs have suggestions on what to use to replace each function and constant.

List comprehensions with numpy-like indexing raises ValueError

I am trying to parse the following minimal example:

import numpy as np

xs = np.arange(8).reshape(2, 2, 2)
steps = 2

def get_steps(xs, steps):
    return [xs[:, step, :] for step in range(steps)]

print(get_steps(xs, steps))

I am using the following code

import ast
import asttokens

code="""
import numpy as np

xs = np.arange(8).reshape(2, 2, 2)
steps = 2

def get_steps(xs, steps):
    return [xs[:, step, :] for step in range(steps)]

print(get_steps(xs, steps))
"""

def parse_ast(code):
    tree = ast.parse(code)
    return tree

def parse_asttokens(code):
    tree = asttokens.ASTTokens(code, parse=True).tree
    return tree


parse_ast(code)
parse_asttokens(code)

This fails with a ValueError: Expected token OP:'[', got NAME:'return' on line 8 col 5

I have tried looking at the internals of the the library but did not manage to understand why the before tokens falls out of the list comprehension

test failures in Fedora rawhide with python3.9

This is with python3-3.9.0~b1-4.fc33.x86_64, python3-astroid-2.4.1-2.gita672051.fc33.noarch and asttokens=2.0.4:

=================================================== FAILURES ====================================================
___________________________________________ TestAstroid.test_fixture9 ___________________________________________
/usr/lib/python3.9/site-packages/astroid/builder.py:168: in _data_build
    node, parser_module = _parse_string(data, type_comments=True)
/usr/lib/python3.9/site-packages/astroid/builder.py:445: in _parse_string
    parsed = parser_module.parse(data + "\n", type_comments=type_comments)
/usr/lib/python3.9/site-packages/astroid/_ast.py:48: in parse
    return parse_func(string)
/usr/lib64/python3.9/ast.py:50: in parse
    return compile(source, filename, mode, flags,
E     File "<unknown>", line 2
E       (*args)
E              ^
E   SyntaxError: invalid syntax

The above exception was the direct cause of the following exception:
tests/test_mark_tokens.py:172: in test_fixture9
    def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')
tests/test_mark_tokens.py:134: in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
tests/tools.py:96: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
/usr/lib/python3.9/site-packages/astroid/builder.py:279: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.9/site-packages/astroid/builder.py:142: in string_build
    module = self._data_build(data, modname, path)
/usr/lib/python3.9/site-packages/astroid/builder.py:170: in _data_build
    raise exceptions.AstroidSyntaxError(
E   astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E   invalid syntax (<unknown>, line 2)
____________________________________________ TestAstroid.test_splat _____________________________________________
/usr/lib/python3.9/site-packages/astroid/builder.py:168: in _data_build
    node, parser_module = _parse_string(data, type_comments=True)
/usr/lib/python3.9/site-packages/astroid/builder.py:445: in _parse_string
    parsed = parser_module.parse(data + "\n", type_comments=type_comments)
/usr/lib/python3.9/site-packages/astroid/_ast.py:48: in parse
    return parse_func(string)
/usr/lib64/python3.9/ast.py:50: in parse
    return compile(source, filename, mode, flags,
E     File "<unknown>", line 2
E       (*arr)
E             ^
E   SyntaxError: invalid syntax

The above exception was the direct cause of the following exception:
tests/test_mark_tokens.py:334: in test_splat
    m = self.create_mark_checker(source)
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:96: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
/usr/lib/python3.9/site-packages/astroid/builder.py:279: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.9/site-packages/astroid/builder.py:142: in string_build
    module = self._data_build(data, modname, path)
/usr/lib/python3.9/site-packages/astroid/builder.py:170: in _data_build
    raise exceptions.AstroidSyntaxError(
E   astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E   invalid syntax (<unknown>, line 2)
_________________________________________ TestAstroid.test_sys_modules __________________________________________
/usr/lib/python3.9/site-packages/astroid/builder.py:168: in _data_build
    node, parser_module = _parse_string(data, type_comments=True)
/usr/lib/python3.9/site-packages/astroid/builder.py:445: in _parse_string
    parsed = parser_module.parse(data + "\n", type_comments=type_comments)
/usr/lib/python3.9/site-packages/astroid/_ast.py:48: in parse
    return parse_func(string)
/usr/lib64/python3.9/ast.py:50: in parse
    return compile(source, filename, mode, flags,
E     File "<unknown>", line 2
E       (*args)
E              ^
E   SyntaxError: invalid syntax

The above exception was the direct cause of the following exception:
tests/test_mark_tokens.py:639: in test_sys_modules
    self.create_mark_checker(source)
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:96: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
/usr/lib/python3.9/site-packages/astroid/builder.py:279: in parse
    return builder.string_build(code, modname=module_name, path=path)
/usr/lib/python3.9/site-packages/astroid/builder.py:142: in string_build
    module = self._data_build(data, modname, path)
/usr/lib/python3.9/site-packages/astroid/builder.py:170: in _data_build
    raise exceptions.AstroidSyntaxError(
E   astroid.exceptions.AstroidSyntaxError: Parsing Python code failed:
E   invalid syntax (<unknown>, line 2)
--------------------------------------------- Captured stdout call ----------------------------------------------
/usr/lib64/python3.9/importlib/_bootstrap.py
_________________________________________ TestMarkTokens.test_fixture9 __________________________________________
tests/test_mark_tokens.py:172: in test_fixture9
    def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')
tests/test_mark_tokens.py:134: in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
tests/tools.py:96: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
/usr/lib64/python3.9/ast.py:50: in parse
    return compile(source, filename, mode, flags,
E     File "<unknown>", line 2
E       (*args)
E              ^
E   SyntaxError: invalid syntax
___________________________________________ TestMarkTokens.test_splat ___________________________________________
tests/test_mark_tokens.py:334: in test_splat
    m = self.create_mark_checker(source)
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:96: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
/usr/lib64/python3.9/ast.py:50: in parse
    return compile(source, filename, mode, flags,
E     File "<unknown>", line 2
E       (*arr)
E             ^
E   SyntaxError: invalid syntax
________________________________________ TestMarkTokens.test_sys_modules ________________________________________
tests/test_mark_tokens.py:639: in test_sys_modules
    self.create_mark_checker(source)
tests/test_mark_tokens.py:43: in create_mark_checker
    checker.verify_all_nodes(self)
tests/tools.py:96: in verify_all_nodes
    rebuilt_node = test_case.parse_snippet(text, node)
tests/test_mark_tokens.py:735: in parse_snippet
    return self.module.parse('_\n(' + text + ')').body[1].value
/usr/lib64/python3.9/ast.py:50: in parse
    return compile(source, filename, mode, flags,
E     File "<unknown>", line 2
E       (*args)
E              ^
E   SyntaxError: invalid syntax
--------------------------------------------- Captured stdout call ----------------------------------------------
/usr/lib64/python3.9/importlib/_bootstrap.py
============================================ short test summary info ============================================
FAILED tests/test_astroid.py::TestAstroid::test_fixture9 - astroid.exceptions.AstroidSyntaxError: Parsing Pyth...
FAILED tests/test_astroid.py::TestAstroid::test_splat - astroid.exceptions.AstroidSyntaxError: Parsing Python ...
FAILED tests/test_astroid.py::TestAstroid::test_sys_modules - astroid.exceptions.AstroidSyntaxError: Parsing P...
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_fixture9 -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_splat -   File "<unknown>", line 2
FAILED tests/test_mark_tokens.py::TestMarkTokens::test_sys_modules -   File "<unknown>", line 2
============================== 6 failed, 100 passed, 1 skipped, 1 warning in 6.34s ==============================

And it seems to all stem from trying to parse this snippet:

(Pdb) p data
'_\n(*args)'

elif node will start in elif keyword in next Python versions

I noticed this change https://bugs.python.org/issue39031. Not sure how it affects asttokens, but I thought I'll notify you just in case.

gristlabs / asttokens Goto Github PK

asttokens's People

Contributors

Stargazers

Watchers

Forkers

asttokens's Issues

Recommend Projects

Recommend Topics

Recommend Org