berkerpeksag / astor Goto Github PK

Hello,

There seems to be an error turning an ast back into the correct form when expanding tuples. For example, take the following code.

a = (1, 2)
b = (*a, 3)

Running rtrip on this code fragment leads to the following

a = 1, 2
b = *a, 3

which unfortunately is not valid python.

Not sure why and no time to debug right now.

sudo python setup.py install will appear to work under anaconda, but will install into system python directories.

All that setup stuff is a complete mystery to me.

Hi guys!

Found an issue with astor when working with non printable chars in string. For example code below yields an error:

old_source="""
def write(self, param):
    if param=='\n':
        return;
"""
tree = ast.parse(old_source)
new_source = to_source(tree)
print(new_source)

Error:
if param=='
^
SyntaxError: EOL while scanning string literal.

And thanks for astor, it's really useful!

I can take a stab at this when I get some time, but posting this now in case someone else is interested.

Sometimes I pass in an incorrect AST to an astor method, e.g. astor.to_source(ast) (e.g. ast contains a raw int instead of wrapping it an ast.Num). This leads to tricky errors internally in astor that can be hard to understand for the user (this particular error resulted in a failed assertion in precedence_setter.set_precedence).

I propose that astor validates the input AST for any user facing API functions. This would allow any internal logic to assume that the AST is valid and for users to be immediately notified that the input AST is wrong, rather than having to parse errors or exceptions from the internal logic.

Hi, our builds started failing with AttributeError: module 'astor' has no attribute 'codegen':

https://travis-ci.org/google/tangent/jobs/300437272

I wonder if it's related to the new 0.6.1 release?

The following code won't work in Python 3.7:

Full list:

astor/code_gen.py:311:    def visit_FunctionDef(self, node, async=False):
astor/code_gen.py:312:        prefix = 'async ' if async else ''
astor/code_gen.py:325:        self.visit_FunctionDef(node, async=True)
astor/code_gen.py:367:    def visit_For(self, node, async=False):
astor/code_gen.py:369:        prefix = 'async ' if async else ''
astor/code_gen.py:376:        self.visit_For(node, async=True)
astor/code_gen.py:383:    def visit_With(self, node, async=False):
astor/code_gen.py:384:        prefix = 'async ' if async else ''
astor/code_gen.py:395:        self.visit_With(node, async=True)

Not sure if it's introduced by Python 3.5, though. Hope it helps:

.....E.
======================================================================
ERROR: test_try_expect (tests.test_codegen.CodegenTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-astor/src/astor/tests/test_codegen.py", line 47, in test_try_expect
    self.assertAstSourceEqual(source)
  File "/build/python-astor/src/astor/tests/test_codegen.py", line 24, in assertAstSourceEqual
    self.assertEqual(astor.to_source(ast.parse(source)), source)
  File "/build/python-astor/src/astor/astor/codegen.py", line 40, in to_source
    generator.visit(node)
  File "/build/python-astor/src/astor/astor/misc.py", line 161, in visit
    return visitor(node)
  File "/build/python-astor/src/astor/astor/codegen.py", line 493, in visit_Module
    self.visit(stmt)
  File "/build/python-astor/src/astor/astor/misc.py", line 161, in visit
    return visitor(node)
  File "/build/python-astor/src/astor/astor/codegen.py", line 288, in visit_Try
    self.visit(handler)
  File "/build/python-astor/src/astor/astor/misc.py", line 161, in visit
    return visitor(node)
  File "/build/python-astor/src/astor/astor/codegen.py", line 300, in visit_ExceptHandler
    self.body(node.body)
  File "/build/python-astor/src/astor/astor/codegen.py", line 99, in body
    self.visit(stmt)
  File "/build/python-astor/src/astor/astor/misc.py", line 161, in visit
    return visitor(node)
  File "/build/python-astor/src/astor/astor/codegen.py", line 179, in visit_Expr
    self.generic_visit(node)
  File "/usr/lib/python3.5/ast.py", line 255, in generic_visit
    self.visit(value)
  File "/build/python-astor/src/astor/astor/misc.py", line 161, in visit
    return visitor(node)
  File "/build/python-astor/src/astor/astor/codegen.py", line 366, in visit_Call
    self.conditional_write(write_comma, '*', node.starargs)
AttributeError: 'Call' object has no attribute 'starargs'

----------------------------------------------------------------------
Ran 7 tests in 0.012s

FAILED (errors=1)

Hi!

I have the following example:

import astor
import ast

class CallWrapper(ast.NodeTransformer):
    def visit_Call(self, node):
        if node.col_offset == 0:
            return None
        return node

src = open('test.py', 'r').read()
p = ast.parse(src)
CallWrapper().visit(p)
ast.fix_missing_locations(p)
new_src = astor.to_source(p)

print(new_src)

It works with Python 2.7, but raise the following error on Python3.5

Traceback (most recent call last):
  File "easy.py", line 15, in <module>
    new_src = astor.to_source(p)
  File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/code_gen.py", line 51, in to_source
    generator.visit(node)
  File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/node_util.py", line 137, in visit
    return visitor(node)
  File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/code_gen.py", line 662, in visit_Module
    self.write(*node.body)
  File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/code_gen.py", line 156, in write
    self.visit(item)
  File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/node_util.py", line 137, in visit
    return visitor(node)
  File "/usr/lib/python3.5/site-packages/astor-0.6-py3.5.egg/astor/code_gen.py", line 260, in visit_Expr
    set_precedence(node, node.value)
AttributeError: 'Expr' object has no attribute 'value'

Am I missing something? Is it a bug? If so, please if possible, point me some direction that I'd love to create a PR.

Thank you

Right now, codegen.py just says "distributed under the BSD license", but there is no such thing as "the BSD license". Is it 2-clause? 3-clause? 4-clause?

Maybe @treo (author of the original gist) can clarify what the original license was supposed to be?

See also https://en.wikipedia.org/wiki/BSD_licenses for reference.

Note also that the rest of the astor codebase is under the BSD 2-clause (https://en.wikipedia.org/wiki/BSD_licenses#2-clause_license_.28.22Simplified_BSD_License.22_or_.22FreeBSD_License.22.29).

It was added in f87ae3b by @radomirbosak. We need to document it in docs/index.rst and docs/changelog.rst.

However, I don't consider this an urgent issue and it may be better to wait at least one release in order to avoid potential backwards-compatibility issues. Or we could just add a note in docs/changelog.rst and declare it as 'provisional'.

From https://travis-ci.org/berkerpeksag/astor/jobs/357390920:

======================================================================
FAIL: test_convert_stdlib (tests.test_rtrip.RtripTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/berkerpeksag/astor/tests/test_rtrip.py", line 24, in test_convert_stdlib
    self.assertEqual(result, [])
AssertionError: Lists differ: ['/home/travis/build/berkerpeksag/astor/.t[35785 chars].py'] != []
First list contains 344 additional elements.
First extra element 0:
'/home/travis/build/berkerpeksag/astor/.tox/py37/lib/python3.7/os.py'

These are not present in 0.6, but present in 0.6.1 and latest 0.6.2 release:

======================================================================
ERROR: test_codegen_as_submodule (tests.test_misc.PublicAPITestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-astor/src/astor-0.6.2-py2/tests/test_misc.py", line 39, in test_codegen_as_submodule
    with self.assertWarns(DeprecationWarning) as cm:
AttributeError: 'PublicAPITestCase' object has no attribute 'assertWarns'

======================================================================
ERROR: test_codegen_from_root (tests.test_misc.PublicAPITestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-astor/src/astor-0.6.2-py2/tests/test_misc.py", line 28, in test_codegen_from_root
    with self.assertWarns(DeprecationWarning) as cm:
AttributeError: 'PublicAPITestCase' object has no attribute 'assertWarns'

----------------------------------------------------------------------

• c479d88
• 12cdde6

Related to #82 and #85, ast.Nums containing non-finite complex numbers will be dumped as their reprs instead of legal Python expressions that evaluate to the right floating-point constructs.

import ast, astor

x = (1e1000 - 1e1000) - 1e1000j

print(x)
print(astor.code_gen.to_source(ast.Module([ast.Expr(ast.Num(x))])))

Both lines print as (nan-infj).

node.docstring has just removed from 3.7.0.b5 due to backwards compatibility reasons. See python/cpython#7121 for details.

By looking at the pull request, reverting 524d525 might be enough.

@radomirbosak would you like to submit a pull request?

The new f-strings are only single-quoted for now, which should work fine, but which will look ugly in some cases.

Adding the ability to fix those will involve code changes to code_gen.py and string_repr.py.

This is a science fair project I am unwilling to undertake at the current time.

Somewhat related to #82, the codegen of astor 0.5 would produce nan if given a Num node that contained NaN. In astor 0.6, the following raises an AssertionError in code_gen.py:

import ast, astor

nan = 1e1000 - 1e1000

print(astor.code_gen.to_source(
         ast.Module([ast.Expr(ast.BinOp(
             ast.Num(1.0),
             ast.Add(),
             ast.Num(nan)))])))

While Python has no NaN literal, you could represent ast.Num(nan) as an expression (such as 1e1000 - 1e1000) if you want to generate real Python. See hylang/hy#1447.

Looks like a test-only failure, though.

======================================================================
FAIL: test_convert_stdlib (tests.test_rtrip.RtripTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/python-astor/src/astor-0.6/tests/test_rtrip.py", line 24, in test_convert_stdlib
    self.assertEqual(result, [])
AssertionError: Lists differ: ['/usr/lib/python3.6/test/test_fstring.py'[34 chars].py'] != []

First list contains 2 additional elements.
First extra element 0:
'/usr/lib/python3.6/test/test_fstring.py'

+ []
- ['/usr/lib/python3.6/test/test_fstring.py',
-  '/usr/lib/python3.6/idlelib/grep.py']

Although the AST does not include a Comment node, it is frequently a good idea to insert comments into generated code. It's possible to wrap textual comments in a string instead, but it doesn't look as nice.

I've been experimenting and it looks like it should be feasible to add a custom Comment node (inheriting from ast.Expr, possibly).

The really interesting bit is when you allow the Comment's value to be not just strings, but other AST nodes. In this case you could generate syntactically-valid and properly indented commented-out code.

I've got this working for single line statements and I think I can do it for multiple line blocks like function definitions as well. However, before I go further I want to be sure something like this would be accepted into the project.

The profiler shows some low-hanging fruit for speedups. I'll spend a couple of hours looking at what I can do without going overboard.

I will be committing a branch with tests to deal with these:

1. Keyword-only should arguments come before '**'

2. 'from .. import' with no trailing module name did not work.

3. try/finally needs to come after else

4. Sometimes integers have attributes, e.g. (0).real We need the parentheses in this case, and did not have them.

5. When yield is an expression rather than a statement, it can be a syntax error if it is not enclosed in parentheses.

Here is an example in the 3.5 std library. Perhaps we can find/reuse the std library code for handling the file encoding descriptor?

>>> import astor
>>> astor.parsefile('/usr/local/lib/python3.5/sqlite3/test/dbapi.py')
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/pat/projects/opensource/astor/astor/misc.py", line 166, in parsefile
    fstr = f.read()
  File "/usr/local/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 120: invalid continuation byte

See hylang/hy#824 (comment) for more info.

This really is not an issue but is there a way to add a pretty print format to codegen that will handle outputing docstrings of this form

"""
This
is a pretty
docstring!
"""

I have installed the 0.5 version of astor with pip install astor, I can see it in my virtual environement when I do pip list. However, if I try to import astor then ModuleNotFound is thrown. What am I doing wrong?

I'd like to make a new Hy release soon (before August), and Hy now depends on new features and bug fixes in astor, particularly for Python 3.7 compatibility.

Running in Python 3.6.4

f"Hello " \
" {name}"

# printing the above yields
# "Hello {name}"

src dump

Module(body=[Expr(value=JoinedStr(values=[Str(s='Hello  {name}')]))])

dst dump:

Module(
    body=[
        Expr(
            value=JoinedStr(
                values=[Str(s='Hello  '), FormattedValue(value=Name(id='name'), conversion=-1, format_spec=None)]))])

Will investigate why this happens

I think it would be useful to have https://github.com/berkerpeksag/astor/blob/master/astor/misc.py#L185-190 as an helper in astor.

Move its content to astor.utils or something else.

Unfortunately, the Sphinx guys (ab)used the docutils class:: keyword for Python classes.

That means that the doc looks fine here but the class names are missing here.

A couple of years ago, I submitted an issue and PR to github to be able to support such things. They finally accepted it last month; the result is that you can make a doc that displays nicely under Sphinx, and reasonably at github, and also explains you should view it elsewhere. That is all documented here

So the right answer is to update index.rst accordingly.

When creating a FunctionDef ast, if you have a Call as part of the body, the Call will not be on a new line.

def foo():
    bar()

will appear as

def foo():bar()

Someone who knows sphinx better than me (e.g. at all) should look into this.

changelog.rst will have to be modified.

Look at modifications to index.rst to see how to make it still display properly at github.

python -c 'import ast, astor; print(astor.code_gen.to_source(ast.Set([])))'
{}

Since Python has no literal syntax for the empty set, and we can't use set() because the name set might be rebound, I think the right representation is {1}.__class__().

I get the following backtrace using astor 0.5 on python 3.5

  File "/Users/sborini/Work/github/simphony/simphony-remote/venv/lib/python3.5/site-packages/traitlet_documenter/util.py", line 62, in get_trait_definition
    return astor.to_source(node.value).strip()
  File "/Users/sborini/Work/github/simphony/simphony-remote/venv/lib/python3.5/site-packages/astor/codegen.py", line 40, in to_source
    generator.visit(node)
  File "/Users/sborini/Work/github/simphony/simphony-remote/venv/lib/python3.5/site-packages/astor/misc.py", line 161, in visit
    return visitor(node)
  File "/Users/sborini/Work/github/simphony/simphony-remote/venv/lib/python3.5/site-packages/astor/codegen.py", line 366, in visit_Call
    self.conditional_write(write_comma, '*', node.starargs)
AttributeError: 'Call' object has no attribute 'starargs'

The last release on PyPI is from 2015-04-18. There have been quite a lot of commits since then. It would be great not having to install directly from Github.

• astor.codegen
• astor.treewalk
• astor.misc

I ran into an issue where I defined a custom AST node, and when I tried rendering the source with astor.to_source(node), it crashed saying No defined handler for node of type xxx.

I understand that astor would not know how to render some unknown AST node, but it would be nice if we could register a visit function, such that when astor encounters a unknown AST node, instead of crashing it checks the registry for the visit function.

There are at least two reasons for having a custom AST class:

• A user defined a truly new AST class.
• The user created a derived class of some known AST class, e.g. a Attribute. Because obj.__class__.__name__ is used, even though it inherits from some known AST class, astor does not know what to do with it.

I understand why the first use case would not be convincing. But number two should be common enough that allowing to register one AST class name as an alias for another, or alternatively allowing to register an explicit visit function would be useful (or both).

My use case is as follows: I do the tree walk and then replace some nodes with a proxy node. The proxy node refers to the original node. However, later at my convenience, I replace the node the proxy refers to with another node. This allows me to switch out nodes without having to walk the tree. Currently, I monkey patched astor to be able to handle this as follows:

def visit_ASTNodeRef(self, node, *largs, **kwargs):
    return self.visit(node.ref_node, *largs, **kwargs)
SourceGenerator.visit_ASTNodeRef = visit_ASTNodeRef

But, it would be nice if there was an official way of doing this. I can work on a PR if that sounds acceptable.

Passing a method to code_to_ast causes the following error.

>>> import astor
>>> from requests.auth import HTTPDigestAuth
>>> node = astor.code_to_ast(HTTPDigestAuth.build_digest_header)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\miau\Documents\repo\astor\astor\file_util.py", line 102, in __call__
    return cache[key]
KeyError: ('C:\\ProgramData\\Anaconda3\\lib\\site-packages\\requests\\auth.py', 128)

It seems that astor doesn't look into ast.ClassDef.

Python 3.4 introduced a new node of type NameConstant for True,False & None.

Python 3.4.1rc1 (default, May  5 2014, 14:28:34) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import astor
>>> import ast
>>> astor.to_source(ast.parse("None"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/codegen.py", line 40, in to_source
    generator.visit(node)
  File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/misc.py", line 155, in visit
    return visitor(node)
  File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/codegen.py", line 490, in visit_Module
    self.visit(stmt)
  File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/misc.py", line 155, in visit
    return visitor(node)
  File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/codegen.py", line 179, in visit_Expr
    self.generic_visit(node)
  File "/usr/lib/python3.4/ast.py", line 255, in generic_visit
    self.visit(value)
  File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/misc.py", line 155, in visit
    return visitor(node)
  File "/home/bilbo/.virtualenvs/hy34/lib/python3.4/site-packages/astor-0.4_dev-py3.4.egg/astor/misc.py", line 149, in abort_visit
    raise AttributeError(msg % node.__class__.__name__)
AttributeError: No defined handler for node of type NameConstant

I am facing a subtle problem when a big float literal is converted to infinity by Python. This is illustrated in the following snippet (tested with astor 0.5 on Python 3.4):

print (astor.to_source(ast.parse('a = 1e400')))
# --> 'a = inf'
print (astor.to_source(ast.parse('a = -1e400')))
# --> 'a = (- inf)'

These round-trip calls are supposed to produce equivalent Python code that is syntactically valid. This is not the case here since inf is not a builtin object of Python. Actually, the bug occurs because the infinity value is already in the ast node; I assume that astor.to_source calls repr on the value carried by this node, which indeed returns the string 'inf'. A satisfying solution for me could be this:

print (astor.to_source(ast.parse('a = 1e400')))
# --> "a = float('inf')"
print (astor.to_source(ast.parse('a = -1e400')))
# --> "a = - float('inf')"

In[15]: astor.to_source(ast.parse('1+2'))
Out[15]: '(1 + 2)'

It looks like astor is not ready for some of the newer Python features.

Python 3.6.1

>>> import astor, ast
>>> astor.__version__
'0.5'
>>> compile("{1: 1, **{2: 2}}", "<str>", "exec", ast.PyCF_ONLY_AST)
<_ast.Module object at 0x00000180BB3F1F28>
>>> astor.codegen.to_source(_)
'{1: 1, None: {2: 2, }, }'

The stars are None: .

Currently, the only command line supported is python -m astor.rtrip [readonly] [<source file or directory>]

Locally, I've enhanced my own copy of rtrip with a few more options, but the way I did that was, uhhhh, non-standard.

When I started this project, I had some Python interpreter where -m <module> ate any subsequent flags on the command line, e.g. -r didn't work. I don't know what that was, haven't found it again, can't recreate it, don't care.

I also have to confess that I don't know which command line argument parser is winning the hearts and minds of astor-lovers.

So, I have some functionality I'd like to add in a branch, but I haven't created a PR because I assume it's just really wrong code.

I don't know if we want to add a __main__.py and be able to support python -m rtrip ... as well as other options, or whether we just want to do a better job of doing the astor.rtrip command line.

If anybody could help, that'd be awesome.

We're missing a proper testing suite.