I tried to use for-else, only to discover it does not exist (syntax error at 'else': expected expression).

for root in roots:
    if src.path.startswith(root):
        modules.append(create_module(name = src.path[len(root):], file = src))
        break
else:
    fail("Module roots does not contain %s" % src.path, "srcs")

This is surprising to someone familiar with Python control structures.

It seems skylark has landed on it's rename to starlark. Is there a plan for renaming this package?

According to the specification and the implementation, string.capitalize modifies the first letter of every word. Therefore "aBc dEf".capitalize() returns "ABc DEf".

This is a bug:

• This is inconsistent with both Bazel and Python, because they return "Abc def".
• string.capitalize seems redundant with behave as string.title.

From looking at eval.go, it looks like the parsing of a script could be done once and then re-evaluated with different globals multiple times, saving the overhead of the parsing step. Is that something you'd be interested in supporting? I know skylark is used just for configuration in bazel, and thus is probably not rerun unless the script changes, but I would like to be able to use this for user-created logic in Go programs, such as adding custom functions to a template parser (like Hugo). In those cases, the same skylark code would be run over and over with different globals. It would be nice if this were supported in the skylark package itself.

UnpackPositionalArg seems to not work if I have a custom type.

What should I do? Make my own UnpackArgs with my custom type(s)?

Use of reflect? Or some kind of Unpacker struct that lets me register the extra types?

I'm working on time and duration. I'd like to be able to do something like

duration = time1 - time2

however, it looks like operators are hard coded right now. Could that be defined as an interface to implement, like Comparable?

Ideally I'd like to be able to do

time2 = time1 + duration

So unlike CompareSameType, it would not require the two to be the same type. The function would need to figure out the type and behave accordingly.

Currently the hash table used by dict and set uses slot = hash(x) % len(table) to choose a bucket. This is vulnerable to hash flooding, a denial of service attack in in which the adversary chooses values of x that result in the same slot, causing the table to degenerate to a single bucket and server performance to suffer.

To mitigate this, we'll need to change the signature of Value.Hash to accept a seed value, and change implementations to incorporate this seed value into their result in an order-dependent manner, such as multiplication but not simply XOR. Each table will need to provide a random seed value, created during table initialization and constant for the life of each table, to hash(x, seed). It is not sufficient to merely incorporate the seed value after computing hash(x).

This will not change the semantics of the hash table observed by the Skylark program; in particular, iteration will still be in insertion order.

for example: 2 ** 2 produces an error.

Currently the scanner refuses to parse integer literals that can't be represented as int64:

$ skylark 
Welcome to Skylark (github.com/google/skylark)
>>> 1234567890123456789
1234567890123456789
>>> 12345678901234567890
... 
<stdin>:1:1: invalid int literal

The solution is to change the syntax.Literal.Value to allow string, int64, or *big.Int, and change the interpreter accordingly.

Also, the comment next to the Value field is wrong: it says int, not int64.

$ skylark
>>> "{{ {0} }}".format("hello")
"{ hello }}"

The correct output is "{ hello }".

This issue will be used to accumulate a list of proposed breaking API changes to be implemented as a batch (to minimize disruption) before we freeze the API, which may happen when the Bazel team decides how to rename Skylark.

Item: currently, both Function.Call and Builtin.Call do new(Frame), thread.push(frame), call, thread.pop(frame). User-defined implementations of Callable should also do the same for calls of such functions to appear in stack backtraces. Rather than require each implementation to do it, we should do it in the skylark.Call wrapper, and disallow direct calls to the Callable.Call interface method by renaming it CallInternal or CallImpl.

The parser should have an option to capture the comments and attach them to AST nodes (https://talks.go-zh.org/2015/gofmt-en.slide#31). For convergence, we probably want to do like in Buildifier (https://github.com/bazelbuild/buildtools/blob/master/build/syntax.go#L59).

Comments are useful for a wide range of applications (linter, formatting, refactoring...).

The built-in int function of Skylark-in-Java (and Pythons 2 and 3) supports binary using the 0b prefix:

>>> int("0b1010", 0)
10
>>> int("1010", 2)
10

Skylark-in-Go should do also.

Pythons 2 and 3 also allow binary integer literals, though Skylark-in-Java does not:

>>> 0b1010
10

For consistency, Skylark probably should too.

https://godoc.org/github.com/google/skylark/resolve#File

resolve.File takes an arg isBuiltIn.... I thought this was supposed to return true both for builtins in skylark.Universe, and for ones I passed in as globals, since the ones I pass in are created are of type skylark.BuiltIn. However, when I do that, I get nil pointer panics (will create another bug for that). I think there's a couple problems here:

1. Calling functions you create and pass in as globals "builtins" is very confusing. They're not "built in" per se, even if they're also not defined in the script itself.

2. resolve.File seems to have a different definition of what a builtIn is, which seems to want only things in skylark.Universe. And if that's so, why do I need to pass in an argument for that at all?

.bzl files suffer from the same type safety problems as as untyped Python. It is much harder for me to trace through .bzl coded compared to Go.

Is there any work to support static type declarations? Copying Python's type semantics seems most natural since the syntax of the languages is so close.

Since Skylark has been renamed to Starlark, it would be useful to mention this in the README.

Example 1:

def f():
    print(len)
    len = 2
    print(len)

f()

Example 2:

print(len)
len = 2
print(len)

The first example fails, because local variable len referenced before assignment. The second example should fail for the same reason.

From the spec: "If name is bound anywhere within a block, all uses of the name within the block are treated as references to that binding, even uses that appear before the binding."

Are there any examples of parsing BUILD files using the skylark parser? For https://github.com/bazelbuild/bazel-watcher it would be great if it were possible to parse the BUILD files without taking the Bazel lock so we could, in parallel, build and compute source files to watch. Do you know of any code that is doing this already?

$ skylark
>>> x ! 0
... 
<stdin>:1:4: unexpected input character '!'

The position of the ! in the error message is off by one.

I have another experimental branch, based on the branch I linked in #124, in which I implemented a simple POC for Python-style try/except syntax. It's not yet complete, but the basics seem to be working. The feature branch also includes a ton of other changes since it was branched from another of my feature branches, but the gist of the changes needed to implement try/except are in the following commit: wdamron@d3c3a67

For reference, here's the try/except branch: https://github.com/wdamron/skylark/tree/try-except

Like the other issue I opened (#124), this isn't at all a bug and is probably out of scope for this project, but I figured I'd open the issue in case it could be of interest.

I really like the name skylark. It's evocative, makes for a nice file extension - .sky... opens up a lot of unused namespaces in the go community.

According to the specification:

Range values are comparable: two range values compare equal if they denote the same sequence of integers, even if they were created using different parameters.

The interpreter contains a bug: range(0, 5, 10) == range(0, 5, 11) returns False.
Bazel, Python2, and Python3 return True (Bazel's implementation is not necessarily the best, as it iterates over the values).

I've been experimenting with some changes to the interpreter: I added the ability for threads to be suspended and fully serialized then later deserialized and resumed (potentially elsewhere). This doesn't work if there are builtin Callable values within the call-stack, as builtin functions are inherently not resumable. An additional Resumable interface would be need to be implemented to support this, which I haven't gotten around to.

The majority of the my changes are in the new Encoder/Decoder types, and I split out the common Value types into their own files (from values.go).

The absence of these features isn't at all a "bug", and is arguably way out of scope for this project, so a pull request didn't seem appropriate, but I'm including a link to my experimental branch in case these changes would be of any interest:

github.com/wdamron/skylark/tree/reentrant

@alandonovan Feel free to close this.

Apparently, this repo is deprecated and moved to https://github.com/google/starlark-go, please archive it to make it clear.

Docs for String says:

A String is an immutable sequence of bytes. Strings are iterable; iteration over a string yields each of its 1-byte substrings in order.

But String doesn't implement Iterable's Iterate method.

Is this intentional?

String could implement Iterable, in theory, by returning an iterator which passes each one-byte substring into Next.

This project is great! I'm just in the beginning stages of using it for a custom configuration language. Here is an example builtin callback:

# test.sky
local_workspace(
    name = "foo",
    root = ".",
    files = ["a.txt", "b.txt"],
)

func (i *Interpreter) handleLocalWorkspace(thread *skylark.Thread, _ *skylark.Builtin, args skylark.Tuple, kwargs []skylark.Tuple) (skylark.Value, error) {
	rule := &LocalWorkspaceRule{}
	files := skylark.NewList([]skylark.Value{})
	
	if err := skylark.UnpackArgs("local_workspace", args, kwargs,
		"name", &rule.Name,
		"root?", &rule.Root,
		"files?", &files,
	); err != nil {
		return nil, err
	}

	if rule.Name == "" {
		return nil, fmt.Errorf("local_workspace attribute 'name' is required")
	}

	iter := files.Iterate()
	defer iter.Done()
	var value skylark.Value
	for iter.Next(&value) {
		log.Printf("filename: %v\n", value)
		if value.Type() == "string" {
			rule.Files = append(rule.Files, value.String())
		} else {
			return nil, fmt.Errorf("`files` attribute element must be a string, not %s (%v)", value.Type(), value.String())
		}
	}

	i.LocalWorkspaceRule = append(i.LocalWorkspaceRule, rule)
	return skylark.None, nil
}

Here I've defined a "rule" called local_workspace. This works!

However, I immediately find myself wondering if there is an existing library or plans to support bazel-like custom rule definitions, where I don't have to do all the validation myself. Basically, I'd like to be able to build up custom rule definitions that have string_list attributes (in Go), and have the rule automap the args to some struct type or protobuf type.

This of course is not terribly hard to write, but it seems like something that may already be in the works so I thought I'd ask.

The Go implementation dynamically rejects all attempts to update a list during iteration, except list[i] = ..., which it considers a non-structural mutation:

$ cat ~/a.sky
def iterator1():
    list = [0, 1, 2]
    for x in list:
        list[x] = 2 * x
    return list

print(iterator1(), [0, 2, 4])
$ skylark a.sky
[0, 2, 4] [0, 2, 4]

The Java and Rust implementations reject even this.

The Go semantics came out of a discussion with the Bazel team that was never concluded. The task of this issue is to clarify the semantics for mutation during iteration. See Google Issue b/37964285 for context.

@laurentlb

tldr; no obvious way to exit repl and ctrl+c does not work.

The AST is relatively faithful to the input file and contains a lot of information (locations, raw data). One notable exception is due to an optimization in the parser: "a" + "b" is replaced with "ab".

Is there a way to preserve this information?
Maybe optimizations on the AST could be done in a separate pass (in case we want to do other things like this)?

A smallish example:

package main

import (
	"fmt"

	"github.com/google/skylark/syntax"
)

func test1(filename string, src []byte) error {
	_, err := syntax.Parse(filename, src)
	return err
}

func test2(filename string, src interface{}) error {
	_, err := syntax.Parse(filename, src)
	return err
}

func main() {
	fmt.Printf("%s\n", test1("test.sky", nil))
	fmt.Printf("%s\n", test2("test.sky", nil))
}

which prints

%!s(<nil>)
reading test.sky: open test.sky: no such file or directory

This is a little surprising; you'd expect both to do the same thing, but if src is passed as a nil []byte instead of an untyped nil, it's still used. From reading the docs, I assume that's not what's intended.

Happy to send a little PR for this if you agree.

Consider the following code:

$ skylark
Welcome to Skylark (github.com/google/skylark)
>>> x = ["a"]
>>> [f for f in x if x == "a" if x == "b"]
... 
<stdin>:1:27: conditional expression without else clause

According to the Skylark spec, this should be valid. The skylark spec defines list comprehensions like this:

ListComp = '[' Test {CompClause} ']'.
CompClause = 'for' LoopVariables 'in' Test
           | 'if' Test .

So the second if clause should be parsed as a CompClause. Instead, it appears to be trying to parse the second if clause as a IfExpr

Arguably, this is a bug in the grammar. The Python3 grammar has a workaround for this issue, defining list comprehensions like this:

comprehension ::=  expression comp_for
comp_for      ::=  "for" target_list "in" or_test [comp_iter]
comp_iter     ::=  comp_for | comp_if
comp_if       ::=  "if" expression_nocond [comp_iter]

I see this from time to time:

=== RUN ExampleLoadParallelCycle
--- FAIL: ExampleLoadParallelCycle (0.00s)
got:
cannot load a.sky: cannot load c.sky: cannot load b.sky: cycle in load graph
cannot load b.sky: cycle in load graph
want:
cannot load a.sky: cannot load c.sky: cycle in load graph
cannot load b.sky: cannot load a.sky: cannot load c.sky: cycle in load graph
FAIL
FAIL github.com/google/skylark 0.054s

Referring to: https://github.com/google/skylark/blob/master/resolve/resolve.go#L273

Shouldn't the debug string returned in the b.comp != nil case include the comprehension span (currently function span)?

I hit this when I was wrestling with #9.

If skylark.Call is passed a nil function, it will panic here:

(this was the result of a user error, but I would imagine it would be good not to panic in this situation regardless)

It feels very weird to have a Skylark implementation, that I cannot build with Bazel.

In skylark, repeated assignments to global variables are supposed to be a compile time error. E.g. a="foo"; a="bar" is an error -- "cannot reassign global a declared at [...]".

However, if I define a global in my go code that is calling skylark.Exec(...), and then assign to that in my skylark script, there's no error. This seems inconsistent and surprising.

It seems like the globals I provide to Exec are treated roughly as if they're in the universe block in that assignments can shadow them... but they're also not treated like the universe block because they'll still be in the globals StringDict after the Exec returns.

I'm not sure if this is a bug or just a documentation request. :)

(In case it's relevant, here's why I'm asking: I'd like to be providing some more builtin functions to use without load statements in my skylark scripts... and I don't want them to appear in the globals StringDict result of the module's evaluation. Is putting my additional builtins into reach by putting them in the globals when I start Exec going about this the wrong way, and I should be using a different approach? Thanks for any advice.)

$ cat a.sky

def b(*args, **kwargs):
	print(args, kwargs)

def a(*args, **kwargs):
	b(*args, k = "foo", **kwargs)

a(1, 2, x=3)

Both Python and Skylark-in-Java print this result:

((1, 2), {'x': 3, 'k': 'foo'})

But the Go implementation crashes in the compiler:

$ skylark a.sky 
panic: interface conversion: syntax.Expr is *syntax.UnaryExpr, not *syntax.BinaryExpr

goroutine 1 [running]:
google3/third_party/golang/skylark/internal/compile/compile.(*fcomp).args(0xc0000ef7b0, 0xc00009a1e0, 0xc000069900)
	third_party/golang/skylark/internal/compile/compile.go:1512 +0x5a6
google3/third_party/golang/skylark/internal/compile/compile.(*fcomp).call(0xc0000ef7b0, 0xc00009a1e0)
	third_party/golang/skylark/internal/compile/compile.go:1470 +0x5e
google3/third_party/golang/skylark/internal/compile/compile.(*fcomp).expr(0xc0000ef7b0, 0x5edc00, 0xc00009a1e0)
	third_party/golang/skylark/internal/compile/compile.go:1270 +0x26c
google3/third_party/golang/skylark/internal/compile/compile.(*fcomp).stmt(0xc0000ef7b0, 0x5eddc0, 0xc00000c960)
	third_party/golang/skylark/internal/compile/compile.go:910 +0x102a
google3/third_party/golang/skylark/internal/compile/compile.(*fcomp).stmts(0xc0000ef7b0, 0xc000057530, 0x1, 0x1)
	third_party/golang/skylark/internal/compile/compile.go:899 +0x68
google3/third_party/golang/skylark/internal/compile/compile.(*pcomp).function(0xc00000ca20, 0x700161, 0x1, 0xc0000574b0, 0x100000005, 0xc000057530, 0x1, 0x1, 0xc0000575b0, 0x2, ...)
	third_party/golang/skylark/internal/compile/compile.go:440 +0x292
google3/third_party/golang/skylark/internal/compile/compile.(*fcomp).function(0xc0000efbc8, 0xc0000574b0, 0x100000005, 0x700161, 0x1, 0xc0000b2160)
	third_party/golang/skylark/internal/compile/compile.go:1627 +0x201
google3/third_party/golang/skylark/internal/compile/compile.(*fcomp).stmt(0xc0000efbc8, 0x5edcc0, 0xc0000b2140)
	third_party/golang/skylark/internal/compile/compile.go:1013 +0xf7d
google3/third_party/golang/skylark/internal/compile/compile.(*fcomp).stmts(0xc0000efbc8, 0xc000069c80, 0x3, 0x4)
	third_party/golang/skylark/internal/compile/compile.go:899 +0x68
google3/third_party/golang/skylark/internal/compile/compile.(*pcomp).function(0xc00000ca20, 0x5c1ca2, 0xa, 0xc0000574b0, 0x100000002, 0xc000069c80, 0x3, 0x4, 0x0, 0x0, ...)
	third_party/golang/skylark/internal/compile/compile.go:440 +0x292
google3/third_party/golang/skylark/internal/compile/compile.File(0xc000069c80, 0x3, 0x4, 0x0, 0x0, 0x0, 0xc000057580, 0x2, 0x2, 0x593640)
...
`

OS Platform and Distribution

• Windows 10 professional
• Docker image based on Ubuntu:16.04.4 LTS (xenial)
• Docker 18.03.0-ce-win59

Go install procedure

curl https://dl.google.com/go/go1.10.1.linux-amd64.tar.gz -o /go1.10.1.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.10.1.linux-amd64.tar.gz && rm go1.10.1.linux-amd64.tar.gz
export PATH = $PATH:/usr/local/go/bin

Exact command to reproduce

go get github.com/google/skylark
go build github.com/google/skylark/cmd/skylark

triggers the following

Error message

root/go/src/github.com/google/skylark/repl/repl.go:33:2: cannot find package "github.com/chzyer/readline" in any of:
/usr/local/go/src/github.com/chzyer/readline (from $GOROOT)
/root/go/src/github.com/chzyer/readline (from $GOPATH)

in real Python, user-defined classes can implement the slice operator (x[i:j])

in Skylark, the slice operator can only be used on String, List, and Tuple
https://github.com/google/skylark/blob/master/eval.go#L883

would you accept a PR to make this an interface, like other Skylark operators (HasAttrs, etc)?