Comments (8)
dsagal
commented on May 28, 2024
The xonsh parser produces its own AST. Looking at the source code, it seems to use enough Python that its AST is likely an extension of Python's AST, with extra tokens and nodes. You'd need to fork asttokens and make it aware of these addititions, for it to be able to parse xonsh AST. It's hard to say how much work that would be. It might be easy for someone familiar with xonsh internals, but hard for everyone else :)
from asttokens.
anki-code
commented on May 28, 2024
Got it! Thanks!
from asttokens.
alexmojaki
commented on May 28, 2024
Just to add to what @dsagal said, asttokens has to specially handle many kinds of nodes whose tokens are not correctly inferred by the initial generic algorithm. For example in the traceback it says it was trying to specially handle an attribute node and failed because it couldn't find a dot. Does xonsh have a different concept of Attribute in the AST? Either way, you would probably need to add special cases for some (probably not all) of the additional nodes found in xonsh.
from asttokens.
scopatz
commented on May 28, 2024
FWIW xonsh produces a plain-old Python AST with no additional nodes.
from asttokens.
anki-code
commented on May 28, 2024
I tried to walk the tree:
# To run this code just do `pip install xonsh`, run `xonsh` and copy-paste this code
import ast
def str_node(node):
if isinstance(node, ast.AST):
fields = [(name, str_node(val)) for name, val in ast.iter_fields(node) if name not in ('left', 'right')]
rv = '%s(%s' % (node.__class__.__name__, ', '.join('%s=%s' % field for field in fields))
return rv + ')'
else:
return repr(node)
def ast_visit(node, level=0):
print(' ' * level + str_node(node))
for field, value in ast.iter_fields(node):
if isinstance(value, list):
for item in value:
if isinstance(item, ast.AST):
ast_visit(item, level=level+1)
elif isinstance(value, ast.AST):
ast_visit(value, level=level+1)
cmd = 'echo @("hello") | head'
ast_visit(__xonsh__.execer.parse(cmd, ctx=__xonsh__.ctx))And got:
Expression(body=Call(func=Attribute(value=Name(id='__xonsh__', ctx=Load()), attr='subproc_captured_hiddenobject', ctx=Load()), args=[<_ast.BinOp object at 0x7f374f478460>, <_ast.Constant object at 0x7f374f584850>, <_ast.List object at 0x7f374f478610>], keywords=[]))
Call(func=Attribute(value=Name(id='__xonsh__', ctx=Load()), attr='subproc_captured_hiddenobject', ctx=Load()), args=[<_ast.BinOp object at 0x7f374f478460>, <_ast.Constant object at 0x7f374f584850>, <_ast.List object at 0x7f374f478610>], keywords=[])
Attribute(value=Name(id='__xonsh__', ctx=Load()), attr='subproc_captured_hiddenobject', ctx=Load())
Name(id='__xonsh__', ctx=Load())
Load()
Load()
BinOp(op=Add())
List(elts=[<_ast.Call object at 0x7f374f9f2a90>], ctx=Load())
Call(func=Attribute(value=Name(id='__xonsh__', ctx=Load()), attr='expand_path', ctx=Load()), args=[<_ast.Constant object at 0x7f374f9f23a0>], keywords=[])
Attribute(value=Name(id='__xonsh__', ctx=Load()), attr='expand_path', ctx=Load())
Name(id='__xonsh__', ctx=Load())
Load()
Load()
Constant(value='echo')
Load()
Add()
Call(func=Attribute(value=Name(id='__xonsh__', ctx=Load()), attr='list_of_strs_or_callables', ctx=Load()), args=[<_ast.Constant object at 0x7f374f478d60>], keywords=[])
Attribute(value=Name(id='__xonsh__', ctx=Load()), attr='list_of_strs_or_callables', ctx=Load())
Name(id='__xonsh__', ctx=Load())
Load()
Load()
Constant(value='hello')
Constant(value='|')
List(elts=[<_ast.Call object at 0x7f374f478c70>], ctx=Load())
Call(func=Attribute(value=Name(id='__xonsh__', ctx=Load()), attr='expand_path', ctx=Load()), args=[<_ast.Constant object at 0x7f374f478b20>], keywords=[])
Attribute(value=Name(id='__xonsh__', ctx=Load()), attr='expand_path', ctx=Load())
Name(id='__xonsh__', ctx=Load())
Load()
Load()
Constant(value='head')
Load()
Here I see no any fields with source position information. It means there is no data about it or the place where I'm searching is wrong?
from asttokens.
anki-code
commented on May 28, 2024
Oh, I see, we can just avoid trying to specially handle an attribute node. I'm working on PR.
from asttokens.
alexmojaki
commented on May 28, 2024
Here I see no any fields with source position information. It means there is no data about it or the place where I'm searching is wrong?
It's not present in ast.iter_fields(node), but the attributes are there. You can find them in the __dict__. Or you can use ast.dump(node, include_attributes=True, indent=2), but you will need Python 3.9 for indent=2.
from asttokens.
anki-code
commented on May 28, 2024
Thank you for you help and fast responses! We discussed the questions in the PR and I have info to continue thinking.
from asttokens.
Related Issues (20)
- DeprecationWarning: The 'astroid.node_classes' module is deprecated HOT 5
- ASTText astroid support HOT 1
- IndexError in asttokens.ASTTokens HOT 2
- Timelines around support for legacy Python HOT 2
- Replace deprecated imp module with importlib HOT 2
- Get node from token? HOT 2
- Official support for 3.9 and maybe 3.10
- 2.0.5 tag missing HOT 2
- Test failure with py3.8 and astroid-2.6+: tests/test_astroid.py::TestAstroid::test_slices - AssertionError: Items in the first set but not the second: HOT 3
- Comma misannotation bug HOT 2
- Fix travis build
- Test new syntax in 3.10 HOT 2
- 2.0.5: pytest is failing in few `astroid` units HOT 15
- `asttokens` 2.0.6 has undeclared dependency on `astroid` HOT 8
- Followup to astroid import fix
- mypy reports `error: Name "asttokens.ASTTokens" is not defined` HOT 4
- Support for constructing around a list of existing tokens HOT 6
- Docs outdated HOT 3
- Consider a clearer optional-functionality API HOT 15
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from asttokens.