Giter VIP home page Giter VIP logo

gast's Introduction

GAST, daou naer!

A generic AST to represent Python2 and Python3's Abstract Syntax Tree(AST).

GAST provides a compatibility layer between the AST of various Python versions, as produced by ast.parse from the standard ast module.

Basic Usage

API

gast provides the same API as the ast module. All functions and classes from the ast module are also available in the gast module, and work accordingly on the gast tree.

Three notable exceptions:

  1. gast.parse directly produces a GAST node. It's equivalent to running

    gast.ast_to_gast on the output of ast.parse.

  2. gast.dump dumps the gast common representation, not the original

    one.

  3. gast.gast_to_ast and gast.ast_to_gast can be used to convert

    from one ast to the other, back and forth.

Version Compatibility

GAST is tested using tox and Travis on the following Python versions:

  • 2.7
  • 3.4
  • 3.5
  • 3.6
  • 3.7
  • 3.8

AST Changes

Python3

The AST used by GAST is the same as the one used in Python3.8, with the notable exception of ast.arg being replaced by an ast.Name with an ast.Param context.

For minor version before 3.8, please note that Ellipsis, Num, Str, Bytes and NamedConstant are represented as Constant.

Python2

To cope with Python3 features, several nodes from the Python2 AST are extended with some new attributes/children, or represented by different nodes

  • ModuleDef nodes have a type_ignores attribute.
  • FunctionDef nodes have a returns attribute and a type_comment attribute.
  • ClassDef nodes have a keywords attribute.
  • With's context_expr and optional_vars fields are hold in a withitem object.
  • For nodes have a type_comment attribute.
  • Raise's type, inst and tback fields are hold in a single exc field, using the transformation raise E, V, T => raise E(V).with_traceback(T).
  • TryExcept and TryFinally nodes are merged in the Try node.
  • arguments nodes have a kwonlyargs and kw_defaults attributes.
  • Call nodes loose their starargs attribute, replaced by an argument wrapped in a Starred node. They also loose their kwargs attribute, wrapped in a keyword node with the identifier set to None, as done in Python3.
  • comprehension nodes have an async attribute (that is always set to 0).
  • Ellipsis, Num and Str nodes are represented as Constant.
  • Subscript which don't have any Slice node as slice attribute (and Ellipsis are not Slice nodes) no longer hold an ExtSlice but an Index(Tuple(...)) instead.

Pit Falls

  • In Python3, None, True and False are parsed as Constant while they are parsed as regular Name in Python2.

ASDL

-- ASDL's six builtin types are identifier, int, string, bytes, object, singleton

module Python
{
    mod = Module(stmt* body, type_ignore *type_ignores)
        | Interactive(stmt* body)
        | Expression(expr body)
        | FunctionType(expr* argtypes, expr returns)

        -- not really an actual node but useful in Jython's typesystem.
        | Suite(stmt* body)

    stmt = FunctionDef(identifier name, arguments args,
                       stmt* body, expr* decorator_list, expr? returns,
                       string? type_comment)
          | AsyncFunctionDef(identifier name, arguments args,
                             stmt* body, expr* decorator_list, expr? returns,
                             string? type_comment)

          | ClassDef(identifier name,
             expr* bases,
             keyword* keywords,
             stmt* body,
             expr* decorator_list)
          | Return(expr? value)

          | Delete(expr* targets)
          | Assign(expr* targets, expr value, string? type_comment)
          | AugAssign(expr target, operator op, expr value)
          -- 'simple' indicates that we annotate simple name without parens
          | AnnAssign(expr target, expr annotation, expr? value, int simple)

          -- not sure if bool is allowed, can always use int
          | Print(expr? dest, expr* values, bool nl)

          -- use 'orelse' because else is a keyword in target languages
          | For(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment)
          | AsyncFor(expr target, expr iter, stmt* body, stmt* orelse, string? type_comment)
          | While(expr test, stmt* body, stmt* orelse)
          | If(expr test, stmt* body, stmt* orelse)
          | With(withitem* items, stmt* body, string? type_comment)
          | AsyncWith(withitem* items, stmt* body, string? type_comment)

          | Raise(expr? exc, expr? cause)
          | Try(stmt* body, excepthandler* handlers, stmt* orelse, stmt* finalbody)
          | Assert(expr test, expr? msg)

          | Import(alias* names)
          | ImportFrom(identifier? module, alias* names, int? level)

          -- Doesn't capture requirement that locals must be
          -- defined if globals is
          -- still supports use as a function!
          | Exec(expr body, expr? globals, expr? locals)

          | Global(identifier* names)
          | Nonlocal(identifier* names)
          | Expr(expr value)
          | Pass | Break | Continue

          -- XXX Jython will be different
          -- col_offset is the byte offset in the utf8 string the parser uses
          attributes (int lineno, int col_offset)

          -- BoolOp() can use left & right?
    expr = BoolOp(boolop op, expr* values)
         | BinOp(expr left, operator op, expr right)
         | UnaryOp(unaryop op, expr operand)
         | Lambda(arguments args, expr body)
         | IfExp(expr test, expr body, expr orelse)
         | Dict(expr* keys, expr* values)
         | Set(expr* elts)
         | ListComp(expr elt, comprehension* generators)
         | SetComp(expr elt, comprehension* generators)
         | DictComp(expr key, expr value, comprehension* generators)
         | GeneratorExp(expr elt, comprehension* generators)
         -- the grammar constrains where yield expressions can occur
         | Await(expr value)
         | Yield(expr? value)
         | YieldFrom(expr value)
         -- need sequences for compare to distinguish between
         -- x < 4 < 3 and (x < 4) < 3
         | Compare(expr left, cmpop* ops, expr* comparators)
         | Call(expr func, expr* args, keyword* keywords)
         | Repr(expr value)
         | FormattedValue(expr value, int? conversion, expr? format_spec)
         | JoinedStr(expr* values)
         | Constant(constant value, string? kind)

         -- the following expression can appear in assignment context
         | Attribute(expr value, identifier attr, expr_context ctx)
         | Subscript(expr value, slice slice, expr_context ctx)
         | Starred(expr value, expr_context ctx)
         | Name(identifier id, expr_context ctx, expr? annotation,
                string? type_comment)
         | List(expr* elts, expr_context ctx)
         | Tuple(expr* elts, expr_context ctx)

          -- col_offset is the byte offset in the utf8 string the parser uses
          attributes (int lineno, int col_offset)

    expr_context = Load | Store | Del | AugLoad | AugStore | Param

    slice = Slice(expr? lower, expr? upper, expr? step)
          | ExtSlice(slice* dims)
          | Index(expr value)

    boolop = And | Or

    operator = Add | Sub | Mult | MatMult | Div | Mod | Pow | LShift
                 | RShift | BitOr | BitXor | BitAnd | FloorDiv

    unaryop = Invert | Not | UAdd | USub

    cmpop = Eq | NotEq | Lt | LtE | Gt | GtE | Is | IsNot | In | NotIn

    comprehension = (expr target, expr iter, expr* ifs, int is_async)

    excepthandler = ExceptHandler(expr? type, expr? name, stmt* body)
                    attributes (int lineno, int col_offset, int? end_lineno, int? end_col_offset)

    arguments = (expr* args, expr* posonlyargs, expr? vararg, expr* kwonlyargs,
                 expr* kw_defaults, expr? kwarg, expr* defaults)

    -- keyword arguments supplied to call (NULL identifier for **kwargs)
    keyword = (identifier? arg, expr value)

    -- import name with optional 'as' alias.
    alias = (identifier name, identifier? asname)

    withitem = (expr context_expr, expr? optional_vars)

    type_ignore = TypeIgnore(int lineno, string tag)
}

gast's People

Contributors

curtlh avatar dbieber avatar ghisvail avatar hugovk avatar jacklabarba avatar jendrikseipp avatar jfinkels avatar serge-sans-paille avatar

Stargazers

 avatar  avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.