Giter VIP home page Giter VIP logo

go-jsonnet's Issues

Tests metadata

Annotate individual end to end tests with things like:

  • Whether it should succeed, end with runtime error or end with a static error.
  • extvars and tlas
  • error formatting settings (in case error is expected)
  • native functions
  • import callbacks
  • Maximum stack
  • Maximum number of printed frames on stack trace
  • etc.

This is needed to properly test some features and it would be very helpful in unifying the Go and C++ test suites.

Full commandline support

It would be nice to use go version as a commandline drop-in replacement for C++ version.

We can avoid implementing it in Go for now if we implement libjsonnet #46 and use that from C++ commandline. The advantage is full compatibility, more tests of go-jsonnet (currently libjsonnet tests are very minimal, interesting tests happen through invoking jsonnet command) and we get it almost for free once we have #46.

Add comments to AST (and parser/lexer)

If I were writing a jsonnet formatter, I'd like to have access to comments as AST nodes so I can include them when I'm rewriting the source. It would also be helpful if using the AST to generate jsonnet as a conversion from another source.

Will go-jsonnet supersede the c++ version?

Hi jsonnet team,

As golang is now the preferred language for ops, will go-jsonnet become the de facto implementation for jsonnet?

IMHO, the go version is more likely to be adopted by the community as a library. Also from code readability perspective, golang version is more likely to get merge-able PRs.

libjsonnet compatiblity

We want to be able to use go version as a drop-in replacement for C++ version (for evaluation only, no formatting). This will probably require some changes to happen first on C++ side, e.g. separation of evaluation and formatting.

It's a part of the larger API discussion.

Dead ?

Just asking because i am using this with kubernetes and trying to avoid any c++ dependencies.

Pass through JSON

I've set myself an arbitrary objective today to allow passing through JSON, i.e. add support for desugaring, checking, executing, manifesting basic objects and arrays containing other objects / arrays / primitives.

Make super a value?

Super can be a perfectly good value. It's not an object, though. It would be just a selfBinding - it's indexable, but + doesn't make sense on it.

We could desugar in super etc. and avoid treating it specially during evaluation.

RUNTIME ERROR: Couldn't manifest function in JSON output.

I found this in code:

case *valueFunction:
		return nil, makeRuntimeError("Couldn't manifest function in JSON output.", i.getCurrentStackTrace(trace))

and my input is something like this:

local a = import "../../../applib/a.libsonnet";
local b = import "../../../applib/b.libsonnet";

function(config={})
  local overall_config = default_config + config;
  (if a.match(overall_config) then
    {
      ...
    }
  else
    {
      ...
    }) + {
    ...
  }

And i get this:

RUNTIME ERROR: Couldn't manifest function in JSON output.
	During manifestation

How can i resolve this?
Thanks!

Execution performance ideas

First we need to have some benchmarking in place, and some reasonable "corpus" of jsonnet to test on. We can start by using current test suite, but dedicated benchmarks make more sense.

Once we have that, we can try to find bottlenecks and potentially optimize. Instead of having them scattered throughout the code, I'll dump some of these ideas here:

  • caching of object locals (they're desugared to locals in each field now)
  • caching of object fields (controversial and important, we need a discussion first)
  • memory usage of tail recursion
  • making sure that we don't keep objects for too long
  • generate unique numbers for variables statically and use that instead of names
  • faster implementation of std.join, uniq, setInter and other array building builtins
  • native implementation of std.sort, std.parseInt and other functions that the user may expect to be optimized
  • interning of booleans for faster comparisons (by using the pointer directly and never actually dereferencing) and lower memusage.
  • avoid desugaring operators, call builtins directly instead (fewer things to check, no need to do std lookup and verifying that it's actually a function etc.).
  • fast & easy way for users to build arrays- perhaps a more sophisticated representation of an array would help, so that + behaves reasonably and/or more native higher-order functions. This stuff often applies to strings, too.
  • faster indexing of objects with deep inheritance chains
  • reduce overhead of function call argument checking (currently we check for a lot of special cases, because of optional arguments and that requires building maps and multiple lookups for each argument - maybe it would help to skip these checks when there are only positional arguments)
  • check performance of environment capturing (and reuse the environment when possible)
  • interning of identifiers
  • try other representations of cachedThunk

Before we implement any of these we should have a benchmark that proves that it actually helps.

Prevent changing meaning of std in desugared expressions

Some constructs are desugared to functions from std. The problem is that if the user wraps them in their own definition of std they can change the meaning of core language constructs (probably just break them) which will be extremely confusing.

We can have special alias for std, unavailable for the user, for example $td.

Consider exposing interface for repeated execution with varying environment

At present, it appears that go-jsonnet's public interface starts with jsonnet.MakeVM, against which one can call (*VM).ExtVar, (*VM).ExtCode, and (*VM).Importer to prepare an environment, and eventually call (*VM).EvaluateSnippet against a sequence of Jsonnet expressions.

That call to (*VM).EvaluateSnippet eventually calls on the internal snippetToAST function, which yields a parsed ast.Node ready for later use in the evaluate function. In there, we find some more code-to-AST translation for environment entries that are Jsonnet code.

Is it viable to cleave a new break into this interface, so that it's possible to parse some Jsonnet—together with its import callbacks, and perhaps code entries from the environment—and yield a prepared object that could then be used for repeated evaluation against different TLAs?

The use case I have in mind is a server that would take a Jsonnet function that expects to receive some JSON as a TLA and evaluate the same Jsonnet function repeatedly and frequently, just with its input changing. It's unfortunate to have to restart this procedure each time from raw text when the code involved won't change between evaluations, throwing away the parsed form of the input text.

I concede that I have no measurements to critique the cost of the current interface compared to the separate prepare-once/evaluate-repeatedly approach sketched above. Has this project considered and rejected this approach before? If not, I'm willing to help pursue it.

A few ideas for an interface:

  • (*vm).PrepareSnippet(filename string, snippet string) (*Program, error)
  • (*Program).Eval(vs map[string]string) (string, error)
  • (*Program).EvalTo(w io.Writer, vs map[string]string) error

import support

Hi,

Thanks for open sourcing go-jsonnet library. Do you have any plan implementing the 'import' function in jsonnet language any time soon?

Thanks,
Jim

Missing builtins

Still there is a bunch of builtins that are available in C++ version and not here.

Current status:
builtins["makeArray"] = &Interpreter::builtinMakeArray; DONE
builtins["pow"] = &Interpreter::builtinPow; DONE
builtins["floor"] = &Interpreter::builtinFloor; DONE
builtins["ceil"] = &Interpreter::builtinCeil; DONE
builtins["sqrt"] = &Interpreter::builtinSqrt; DONE
builtins["sin"] = &Interpreter::builtinSin; DONE
builtins["cos"] = &Interpreter::builtinCos; DONE
builtins["tan"] = &Interpreter::builtinTan; DONE
builtins["asin"] = &Interpreter::builtinAsin; DONE
builtins["acos"] = &Interpreter::builtinAcos; DONE
builtins["atan"] = &Interpreter::builtinAtan; DONE
builtins["type"] = &Interpreter::builtinType; DONE
builtins["filter"] = &Interpreter::builtinFilter; DONE
builtins["objectHasEx"] = &Interpreter::builtinObjectHasEx; DONE
builtins["length"] = &Interpreter::builtinLength; DONE
builtins["objectFieldsEx"] = &Interpreter::builtinObjectFieldsEx; DONE
builtins["codepoint"] = &Interpreter::builtinCodepoint; DONE
builtins["char"] = &Interpreter::builtinChar; DONE
builtins["log"] = &Interpreter::builtinLog; DONE
builtins["exp"] = &Interpreter::builtinExp; DONE
builtins["mantissa"] = &Interpreter::builtinMantissa; DONE
builtins["exponent"] = &Interpreter::builtinExponent; DONE
builtins["modulo"] = &Interpreter::builtinModulo; DONE
builtins["extVar"] = &Interpreter::builtinExtVar; DONE
builtins["primitiveEquals"] = &Interpreter::builtinPrimitiveEquals; DONE
builtins["native"] = &Interpreter::builtinNative; TODO: see #44
builtins["md5"] = &Interpreter::builtinMd5; DONE

tailstrict does not prevent blowing the stack

$ go-jsonnet --max-stack 7 -e 'local f(n, c=0) = if n == 0 then c else f(n - 1, c + n) tailstrict; f(100)'
RUNTIME ERROR: Max stack frames exceeded.
        <std>:979:29-30 thunk from <thunk <ta> from <function <anonymous>>>
        <builtin>       builtin function <type>
        <std>:981:33-35 thunk from <function <anonymous>>
        <builtin>       builtin function <primitiveEquals>
        <std>:981:12-40 function <anonymous>

        <cmdline>:1:69-75       $
        During evaluation

When fixed, remove special go golden for test_cmd case max_trace4

Pre-parsed stdlib

@mqliang wrote in #89:

It would be very interesting if we can dump ast as go source files, then we can dump ast of stdlib as go source code, and embed it into vm, thus avoid parsing stdlib every time. Any idea?

I think we should look into it. It sound quite easy.

Object field caching (discussion needed)

Accessing a field multiple times results in multiple evaluations. This can be very easily avoided and innocent looking code may be in fact exponential.

I know there are some concerns about memory usage.

Cannot return RuntimeError from native function

While the RuntimeError type is exported, there does not seem to be an exported way to construct a RuntimeError or an exported way to populate the StackTrace ([]TraceFrame) from.

It would be useful to be able to have an exported variant of the get* functions in evaluator.go that return Go types (e.g. string instead of valueString) and a RuntimeError. However, just being able to get the stacktrace and exporting makeRuntimeError would be sufficient.

Returning other error types from a native function results in an "INTERNAL ERROR" error message, requesting that the user file a bug. The lack of a stack trace in the error message can make it difficult to know what caused the error.

Parameter names of builtins

The parameter names are visible to the user - they can use named arguments instead of positional for every function, including those in stdlib and builtins. Generally the names of parameters should be considered a part of the interface.

So we need to make sure they are right (consistent with http://jsonnet.org/docs/stdlib.html) and it should be verified by tests.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.