The dst
package enables manipulation of a Go syntax tree with high fidelity. Decorations (e.g.
comments and line spacing) remain attached to the correct nodes as the tree is modified.
The go/ast
package wasn't created with source manipulation as an intended use-case. Comments are
stored by their byte offset instead of attached to nodes. Because of this, re-arranging nodes breaks
the output. See this golang issue for more information.
Consider this example where we want to reverse the order of the two statements. As you can see the comments don't remain attached to the correct nodes:
code := `package a
func main(){
var a int // foo
var b string // bar
}
`
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "a.go", code, parser.ParseComments)
if err != nil {
panic(err)
}
list := f.Decls[0].(*ast.FuncDecl).Body.List
list[0], list[1] = list[1], list[0]
if err := format.Node(os.Stdout, fset, f); err != nil {
panic(err)
}
//Output:
//package a
//
//func main() {
// // foo
// var b string
// var a int
// // bar
//}
Here's the same example using dst
:
code := `package a
func main(){
var a int // foo
var b string // bar
}
`
f, err := decorator.Parse(code)
if err != nil {
panic(err)
}
list := f.Decls[0].(*dst.FuncDecl).Body.List
list[0], list[1] = list[1], list[0]
if err := decorator.Print(f); err != nil {
panic(err)
}
//Output:
//package a
//
//func main() {
// var b string // bar
// var a int // foo
//}
Parsing a source file to dst
and printing the results after modification can be accomplished with
several Parse
and Print
convenience functions in the decorator
package.
For more fine-grained control you can use Decorator
to convert from ast
to dst
, and Restorer
to convert back again. See the go/types
section below for a demonstration.
Comments are added at decoration attachment points. See decorations-types-generated.go for a full list of these points, along with demonstration code of where they are rendered in the output.
The decoration attachment points have convenience functions Append
, Prepend
, Replace
, Clear
and All
to accomplish common tasks. Use the full text of your comment including the //
or /**/
markers. When adding a line comment, a newline is automatically rendered.
code := `package main
func main() {
println("Hello World!")
}`
f, err := decorator.Parse(code)
if err != nil {
panic(err)
}
call := f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr)
call.Decs.Start.Append("// you can add comments at the start...")
call.Decs.Fun.Append("/* ...in the middle... */")
call.Decs.End.Append("// or at the end.")
if err := decorator.Print(f); err != nil {
panic(err)
}
//Output:
//package main
//
//func main() {
// // you can add comments at the start...
// println /* ...in the middle... */ ("Hello World!") // or at the end.
//}
The Space
property marks the node as having a line space (new line or empty line) before the node.
These spaces are rendered before any decorations attached to the Start
decoration point. The After
property is similar but rendered after the node (and after any End
decorations).
code := `package main
func main() {
println(a, b, c)
}`
f, err := decorator.Parse(code)
if err != nil {
panic(err)
}
call := f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr)
call.Decs.Space = dst.EmptyLine
call.Decs.After = dst.EmptyLine
for _, v := range call.Args {
v := v.(*dst.Ident)
v.Decs.Space = dst.NewLine
v.Decs.After = dst.NewLine
}
if err := decorator.Print(f); err != nil {
panic(err)
}
//Output:
//package main
//
//func main() {
//
// println(
// a,
// b,
// c,
// )
//
//}
The common decoration properties (Start
, End
, Space
and After
) occur on all nodes, and can be
accessed with the Decorations()
method on the Node
interface:
code := `package main
func main() {
var i int
i++
println(i)
}`
f, err := decorator.Parse(code)
if err != nil {
panic(err)
}
list := f.Decls[0].(*dst.FuncDecl).Body.List
list[0].Decorations().Space = dst.EmptyLine
list[0].Decorations().End.Append("// the Decorations method allows access to the common")
list[1].Decorations().End.Append("// decoration properties (Space, Start, End and After)")
list[2].Decorations().End.Append("// for all nodes.")
list[2].Decorations().After = dst.EmptyLine
if err := decorator.Print(f); err != nil {
panic(err)
}
//Output:
//package main
//
//func main() {
//
// var i int // the Decorations method allows access to the common
// i++ // decoration properties (Space, Start, End and After)
// println(i) // for all nodes.
//
//}
The Space
and After
properties cover the majority of cases, but occasionally a newline needs to
be rendered inside a node. Simply add a \n
decoration to accomplish this.
The dstutil package is a fork of golang.org/x/tools/go/ast/astutil
,
and provides the Apply
function with similar semantics.
Adapting the go/types
package to use dst
as input is non-trivial because go/types
uses
position information in several places. A work-around is to convert from ast
to dst
using
Decorator. After conversion, this
exposes DstNodes
and AstNodes
which map between ast.Node
and dst.Node
. This way, the
go/types
package can be used:
code := `package main
func main() {
var i int
i++
println(i)
}`
// Parse the code to AST
fset := token.NewFileSet()
astFile, err := parser.ParseFile(fset, "a.go", code, parser.ParseComments)
if err != nil {
panic(err)
}
// Invoke the type checker using AST as input
typesInfo := types.Info{
Defs: make(map[*ast.Ident]types.Object),
Uses: make(map[*ast.Ident]types.Object),
}
conf := &types.Config{}
if _, err := conf.Check("a", fset, []*ast.File{astFile}, &typesInfo); err != nil {
panic(err)
}
// Decorate the *ast.File to give us a *dst.File
dec := decorator.New()
f := dec.Decorate(fset, astFile).(*dst.File)
// Find the *dst.Ident for the definition of "i"
dstDef := f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.DeclStmt).Decl.(*dst.GenDecl).Specs[0].(*dst.ValueSpec).Names[0]
// Find the *ast.Ident using the AstNodes mapping
astDef := dec.AstNodes[dstDef].(*ast.Ident)
// Find the types.Object corresponding to "i"
obj := typesInfo.Defs[astDef]
// Find all the uses of that object
var astUses []*ast.Ident
for id, ob := range typesInfo.Uses {
if ob != obj {
continue
}
astUses = append(astUses, id)
}
// Find each *dst.Ident in the DstNodes mapping
var dstUses []*dst.Ident
for _, id := range astUses {
dstUses = append(dstUses, dec.DstNodes[id].(*dst.Ident))
}
// Change the name of the original definition and all uses
dstDef.Name = "foo"
for _, id := range dstUses {
id.Name = "foo"
}
// Print the DST
if err := decorator.Print(f); err != nil {
panic(err)
}
//Output:
//package main
//
//func main() {
// var foo int
// foo++
// println(foo)
//}
If you would like to help create a fully dst
compatible version of go/types
, feel free to
continue my work in the types branch.
This is an experimental package under development, but the API is not expected to change much going forward. Please try it out and give feedback.
Feel free to create an issue or chat in the #dst Gophers Slack channel.