cuelang / cue Goto Github PK

Allow diffing between (snapshots of) previous version and current version.

Not sure if this is just not implemented yet, but the examples of breaking reference cycles don't work as of current master:

// CUE knows how to resolve the following:
x: 200
x: y + 100
y: x - 100

constraint violated: _|_(200:delayed constraint (200 & (<0>.y + 100)) violated)

https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/stringraw.md

Note the markdown uses "```json" causing the red rendering

ie this...

$ cue eval stringraw.cue

{
    msg1: "The sequence \"\\U0001F604\" renders as 😄."
    msg2: """
        A regular expression can conveniently be written as:
        
            #\"\\d{3}\"#
        
        This construct works for bytes, strings and their
        multi-line variants.
        """
}

should be ...

$ cue eval stringraw.cue

{
    msg1: "the sequence \"\\U0001F604\" renders as 😄."
    msg2: """
        A regular expression can conveniently be written as:
        
            #\"\\d{3}\"#
        
        This construct works for bytes, strings and their
        multi-line variants.
        """
}

Similarly, https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/bytes.md exhibits similar (missing quotes around cue export rendered json due to missing quotes in md)

ref also #48

a: [1] + [2]

gives

unsupported op +(list, list)

at current master (7d0797b).

This likely means that the cli docs should be generated from the tool code.

The output from eval is a mix of JSON and (invalid) CUE and indents weirdly. It was quickly written before package format was written.

Replace the output with using package format.

Also, it would be useful to see partially evaluated configurations.

How is CUE different from jsonnet, both coming from Google, and the latter being largely promoted especially in the context of Kubernetes?

Atm. cue has no builtin's for encoding and decoding strings with base64.
base64 encoded values are used by kubernetes secrets.

I guess a good starting point is the encoding/hex package. I also would like to implement the missing functionality.

Or really any programming language, but Go is a good first target.

• Generate Go structs (with comments) from CUE definitions
• Generate validation code given the CUE constraints defined on these definitions (well, its the same thing in CUE, but for Go it needs to be split out)

In #33, @mpvl does IMHO a very good job at highlighting the core difference between cue and jsonnet.

I like cue very much; it really has the potential to tame the complexity of configs.

Its generative aspect works well for templating when the templates have been explicitly designed for extension (and arguably forcing the author to be explicit is instrumental for the success of taming the complexity in the first place)

One of my main annoyances with Helm, is that chart authors are effectively forced to expose every possible field of every k8s object as values in the values.yaml template, because sooner or later some consumer of that chart (usually from outside the administrative domain of the chart author) will have a reasonable use case for the override.

In jsonnet, you can just take an upstream yaml or json object, parse it and apply overrides with jsonnet.

For cue to be useful in this scenario, would we need a function that converts a cue object tree into an object tree that replaces every field value x with *x | _.

Then we could do something like (I know that "Defaultify" is a horrible name, cannot currently think of a better name right now):

import (
  "encoding/yaml"
  "override"
)

res: override.Defaultify(yaml.Unmarshal("""
     foo:
       bar:
         a: 1
         b: 2
  """
)) & {
  foo bar b: 10
}

once an external value is lifted into cue value model, it can be further refined applying type and value limitations that make sense to the user.

Perhaps I'm missing something. Is there already a way to achieve this today?

(I saw there is an interesting approach for importing external legacy configs based on source level imports, but I think that solves a different use case)

Analogous to the Go's go generate.

CUE's is designed with code generation in mind and it is to be expected that many CUE files will be generated. A cue:generate directive would allow annotating the methods used to generate the files in a way familiar to Go programmers.

Excellent effort, would be a great replacement for our inhouse python based system. However we are missing an important feature, documenting fields which are accessible from api/programming which helps auto generating documentation, provides schema comments when autogenerating schemas for third party systems etc.

Please take a look at https://github.com/jpillora/opts/tree/subcommands (work in progress). I think cue is a fascinating project and happy to contribute (I have started a similar project using protobufs as starting point).

Thought helping move from Cobra to Opts might be a good first start.

Related to #25, #9

One of this nice things about the work in progress on Opts is basic bash, zsh and fish command completion is free and highly customisable for more complex requirements.

The behavior of struct unification specified as

The successful unification of structs a and b is a new struct c which has all fields of both a and b

Cannot define an upper bound for structs. Specifically, CUE cannot help us verify that the resulting expression is limited to allowed fields.

For example, the unification

base: {
  onlyViableField: string | *"myDefault"
}

{
  base & {onlyViableFeild: "nonDefaultValue"}
}

yields

{
  onlyViableField: "myDefault"
  onlyViableFeild: "nonDefaultValue"
}

In kubernetes, the existence of extraneous fields causes failure when kubectl applys the resulting YAML, whereas in other systems the expression above could use silently use the default value erroneously.

When comparing CUE to a typed configuration language (like dhall), this is a qualitative difference in capabilities, since the extraneous field would be found by explicitly typing the result.

Extraneous fields can be hard to find - especially in large, opaque, auto-generated config schemes.

The tutorial page on Bottom makes it seem like the user can see partial output like this when errors occur:

a:    _|_
l:    _|_
list: [0, 1, 2]
val:  _|_

But instead, the tool outputs messages like this:

cannot unify numbers 2 and 3:
    [...]
cannot unify numbers 4 and 5:
    [...]
index 3 out of bounds:
    [...]

From #13 it sounds like the tool just doesn't support this yet. In the meantime, perhaps the docs should be edited to clarify that the output shown is theoretical, so users know they didn't do something wrong.

Using types.cue from https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/types.md errors on running cue eval.

$ cue eval -i types.cue 
--- .
unsupported op &((float)*, int):
    ./types.cue:2:8
    ./types.cue:7:10
    
// /home/ghchinoy/dev/cueexperments
{
    point: {
        x:  float
        y:  float
    }
    xaxis: {
        x:  _|_ // unsupported op &((float)*, int)
        y:  float
    }
    yaxis: {
        x:  float
        y:  _|_ // unsupported op &((float)*, int)
    }
    origin: {
        x:  _|_ // unsupported op &((float)*, int)
        y:  _|_ // unsupported op &((float)*, int)
    }
}```

Thanks for the nice walkthrough, that shows some of the great power of Cue :-)
Parts of it refers to previous versions of Cue:

cue eval ./... > snapshot2 doesn't fail as indicated, since now eval can output non-atomic string

Under "1 is the default", 1 is not marked as default.

I am not sure if this is pilot error, or a recent commit changed something, but I can't see any objects being listed or dumped.

cd cuelang/cue/doc/tutorial/kubernetes/manual/services
cue cmd dump ./...
cue cmd dump
cue cmd ls 
cue cmd ls ./...

None of the above commands produce any output. cmd ls|dump works if I specify a path to a directory. For example - this works:

cue ls frontend/waiter 
Service         frontend        waiter
Deployment      frontend        waiter

Is this expected behavior?

I'm not sure if this is just not implemented yet, but filing an issue to track either way.

At current master, the examples for string interpolation in field names don't seem to work. Any field with an interpolation in the name is silently dropped from the output.

There is currently no way to reference a top-level field with a non-standard character:

"foo-bar": "can't touch this"

a: // cannot reference "foo-bar"

One solution is to introduce $ to refer to the root object. This is possible, but such references often result in poor configuration file design. It is also one other thing to learn for a user.

An alternative approach is to allow a syntax for identifiers with non-standard characters. Swift and GCL allow back quotes for this purpose:

"foo-bar": "can touch this"

a: `foo-bar`

Main drawback: now we have three ways of referring to identifiers. Allowing back quotes also on the left hand side might mitigate this issue. The string approach may be considered a relic of JSON and hardly used unless needed for interpolation.

`foo-bar`: "can touch this"

a: `foo-bar`

"dynamic\(interpolated)": 42

cue fmt could be used to canonicalize LHS identifiers according to this rule.

This is a tracking Issue for any OpenAPI changes until most of it is completed. This includes:

• Convert CUE to OpenAPI
• Convert OpenAPI to CUE

CL https://cue-review.googlesource.com/c/cue/+/1660 introduced an alternative semantics for default values that allow also default values to be fully associative, commutative, idempotent, and distributive.

This still needs to be implemented. The implementation should go hand in hand with tidying up and clarifying the semantics of references and the use of defaults in arithmetic and the combination thereof.

The current output points folks to http://cuelang.org/pkg/tool for information on available tasks. However that URL doesn't have any content.

$ curl -i http://cuelang.org/pkg/tool
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
X-Cloud-Trace-Context: 06cb78cc27dfe93e8cc9dd3bde03b522;o=1
Date: Mon, 22 Apr 2019 10:32:57 GMT
Server: Google Frontend
Content-Length: 19

404 page not found

When converting an overlay system to use CUE, "parent" template will often hard code defaults, about which CUE will complain if any instances unify a struct overriding such defaults. As this is likely a common case, it makes sense to add refactor support for this.

The relax command can rewrite parent structs to relax constraints. It is a simple variant of anti-unification as proposed in #7.

In a typical workflow, a user would run relax and then manually verify in the vc diffs which relaxations to keep and which reveal bugs.

Swift raw strings seem to be the nicest design for raw strings out there.

Benefits:

• just a generalization of normal strings, including multi-line
• allow string interpolation within raw strings
• frees up back quotes for, for instance, identifier literals (with non-standard characters).
• extends to both string and byte literals
• can still use escape sequences within the raw strings
• but typically does't require escaping any.

#"This is \ a raw string"#
#"Raw string with interpolation \#(interpolated)."#
##"Raw string without interpolation \#(interpolated)."##

etc.

This is a tracking issue related to extracting CUE from Go code. The functionality in this issue focusses on use cases where Go is used as the source of truth. Issue #6 focusses on generating Go assuming that CUE is used as the source of truth.

injection:
Allow annotation of Go types with additional CUE constraints and provide a simple API to validate and complete Go values.

extraction:
The primary extraction method looks at struct definitions in Go code, determines how these would be interpreted by the encoding/json package, and then generates CUE definitions for such structs.
In addition, we can extract CUE annotations used in Go code for validation to augment such definitions.

Since CUE is a superset of JSON, I understand that you can paste valid JSON into a .cue file and it will be guaranteed valid CUE as well.

My question is, is it guaranteed that valid JSON pasted into a .cue file will never accidentally invoke/trigger any features of CUE that are not in JSON?

For example, I'm guessing that string interpolation can't be triggered accidentally by valid JSON, since (I presume) the escape sequence \( is invalid JSON. Is CUE designed in such a way that all such features are systematically impossible to trigger from valid JSON?

In other words, is all valid JSON guaranteed to produce exactly the same output when treated as a CUE program? I haven't seen mention of this in the docs.

Allow constraining string values with regular expressions:

a: re("[a-z][a-z][a-z]")
a: "foo"

The syntax is yet to be determined. But see #1.

The reason for this is that cue evaluates list types too aggressively. This does not change the result of a usual CUE evaluation, but it does disallow introspection of intermediate results as is needed by the openapi package.

There are several ways to handle this, but most ideally and most consistently, cue could treat list item types analoguously to struct templates.

Alternatives are to track more origin information in result nodes.

Referential cycles are currently implemented. Structural cycles are not, however. So CUE currently allows the definition of infinite structures like:

{ f: { a: f } }

As defined by the spec, infinite structures are not allowed. Infinite structures should be detected and disallowed using an occurs check, as is commonly done in graph unification implementations.

There are some design decisions to be made though. A careful choice of semantics in the occurs check could make CUE not Turing complete and even guarantee termination. This weakens the power of CUE, but may be a desirable property if CUE is to be used for things like smart contracts, where the ability to determine termination would be a major win.

For an example of Turing completeness in CUE see the test called "reentrance" in resolve_test.go, which implements Fibonacci in CUE. The specifics of the semantics of the occurs check will determine whether these shenanigans would be allowed or not.

Depending on the level of restrictions, it should be investigated whether the occurs check can be implemented cheaply using Tomabechi-style cycle detection.

kubectl allows patching using a "strategic merge". In short, it allows lists to be treated like maps so that the right elements can be merged.

In general, unification of lists is very tedious for automated configuration. A possible solution is for users to define mappings manually. Although CUE can handle reconstructing and merging unknown API, it is still tedious. Ideally users would just use the native API of whatever system they work on.

Also, ideally, mappings should be able to be applied with the same separation of concerns as is possible with normal objects.

One possible approach would be to extend the emit mechanisms for inner objects:

myList: [ ...{ name: string } ]

// before evaluation, the list is converted to a map like this
myList: {
    <- "\(x.name)" : x for x in $ // $ means self or current object
    // after evaluation, it is converted back to this format.
    -> [ x for x in $ ]
}

The top-level emit is then -> Expr where the -> can be elided as the top-level value is always a map.

This needs a lot more thought, but having some kind of mechanism like this can be important.

There may also be overlap with a design for field attributes.

I have cue file test.cue:

address: string

And when I call this, I got an error:

$ cue export test.cue 
json: error calling MarshalJSON for type cue.Value: json: error calling MarshalJSON for type cue.Value: cannot convert incomplete value "string" to JSON

For instance:

a: >5         // This field must be greater than 5
b: !=null     // This field must not be null

c: >3.0 & <10  // (3.0, 10.0) open ended range on both sides

Supported operators:

< > >= <= != ==

These would be defined on the same types as their binary equivalents. Semantically such expressions would define disjunctions of possible values, analogous to ranges.

Introducing these would solve various limitations that are currently present. Such as defining open ended ranges for floats or requiring that a value should not be of a specific type.

This is a tracking issue for the implementation of the commented out sections in the spec for conversions.

For instance:

• convert between the integer types
• between byte lists and bytes
• etc

But also to convert a struct to a specific struct type. The latter is quite handy for various purposes.

With this code

x: {a:1}|{a:2}
y: x & {a:3}

I get the following not so helpful message:

empty disjunction: &{{0xc0001fe600} <nil> <nil> [] [{4 0xc00007ee00 <nil> <nil>}] 0xc0001eed20}:
    /...<filename>:2:4
    
terminating because of errors

Implement cue hoist to extract templates from concrete data definitions. One could see this as cue trim in the opposite direction.

This can be implemented using the known anti-unification algorithm. It probably makes sense, though, to disregard disjunctions or to limit the number of elements.

Tracking issue for adding protobuf support to CUE:

• import command reads text and binary proto instance
• import command reads proto definitions
• export command supports the converse
• core libs

I was a little surprised to see the '.' here:
https://cue.googlesource.com/cue/+/HEAD/doc/tutorial/basics/rangedef.md

int16 >=-32_768 & <=.32_767

As of current master, the following CUE program doesn't evaluate as expected:

import "encoding/json"

input: "{\"one\": 1, \"two\": 2, \"three\": 3}"
result: json.Unmarshal(input)

Expected output:

{
    "input": "{\"one\": 1, \"two\": 2, \"three\": 3}",
    "result": {
        "one": 1,
        "two": 2,
        "three": 3
    }
}

Actual output (cue export):

{
    "input": "{\"one\": 1, \"two\": 2, \"three\": 3}",
    "result": {
        "Valid": 1,
        "Compact": 2,
        "Indent": 3
    }
}

Notice the field names in the unmarshaled result have been replaced with the names of methods in pkg/encoding/json.

Current CUE supports the more old-school 0777 notation. 0o, however, is more legible, has better properties regarding scanning, and seems to be the standard languages are converging on.

The CUE-YAML importer would need to be modified to convert octal numbers accordingly.

The old format should be dropped.

cue seems to detect what the spec describes as reference cycles. But for structural cycles, it runs into a stack overflow:

goroutine 1 [running]:
runtime.heapBitsSetType(0xc04ab64070, 0x8, 0x8, 0xc5bcc0)
        /usr/lib/google-golang/src/runtime/mbitmap.go:938 +0xa55 fp=0xc028000350 sp=0xc028000348 pc=0x52cea5
runtime.mallocgc(0x8, 0xc5bcc0, 0x1, 0x0)
        /usr/lib/google-golang/src/runtime/malloc.go:969 +0x50e fp=0xc0280003f0 sp=0xc028000350 pc=0x522c3e
runtime.newobject(0xc5bcc0, 0x0)
        /usr/lib/google-golang/src/runtime/malloc.go:1068 +0x38 fp=0xc028000420 sp=0xc0280003f0 pc=0x523298
cuelang.org/go/cue.(*structLit).expandFields(0xc00025c000, 0xc01907de60, 0xc01907de60)
        /usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/value.go:646 +0x42 fp=0xc028000658 sp=0xc028000420 pc=0x7d88c2
cuelang.org/go/cue.Value.structValOpts(0xc000240b90, 0xc04ab63a00, 0xc01907de60, 0x101000000, 0xc04ab63a00, 0xc00025c000, 0xd98c20, 0xc00025c000, 0xd96780)
        /usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/types.go:813 +0xcd fp=0xc028000788 sp=0xc028000658 pc=0x7d239d
cuelang.org/go/cue.Value.structVal(...)
        /usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/types.go:799
cuelang.org/go/cue.Value.Walk(0xc000240b90, 0xc04ab63a00, 0xc047fff870, 0x0)
        /usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/types.go:1126 +0xe9 fp=0xc028000890 sp=0xc028000788 pc=0x7d4649
cuelang.org/go/cue.Value.Walk(0xc000240b90, 0xc04ab639c0, 0xc047fff870, 0x0)
        /usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/types.go:1129 +0x161 fp=0xc028000998 sp=0xc028000890 pc=0x7d46c1
cuelang.org/go/cue.Value.Walk(0xc000240b90, 0xc04ab63980, 0xc047fff870, 0x0)
        /usr/local/google/home/freinartz/repos/src/cuelang.org/go/cue/types.go:1129 +0x161 fp=0xc028000aa0 sp=0xc028000998 pc=0x7d46c1
cuelang.org/go/cue.Value.Walk(0xc000240b90, 0xc04ab63940, 0xc047fff870, 0x0)
...

This is a tracking issue for improved error support

• more location information
• avoid obscure words like unify
• expose more structured information in the API
• better localization support

This input:

outer middle inner: 3

produces this with CUE

{
   "outer": {
      "middle": {
         "inner": 3
      }
   }
}

and this with YAML:

{
   "outer middle inner": 3
}

this seems like too much sugar to be worth it

https://github.com/cuelang/cue/blob/master/doc/tutorial/basics/fold.md

a <Name>: foo
a b: {}

gives the following on eval:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0x7dac0e]

goroutine 1 [running]:
cuelang.org/go/cmd/cue/cmd.Execute.func1()
/usr/local/google/home/bde/go/src/cuelang.org/go/cmd/cue/cmd/root.go:86 +0x10a
panic(0xc00980, 0x12eb230)
/usr/lib/google-golang/src/runtime/panic.go:522 +0x1b5
cuelang.org/go/cue.(*structValue).Len(...)
/usr/local/google/home/bde/go/src/cuelang.org/go/cue/types.go:83
cuelang.org/go/cue.Value.Walk(0xc000242d70, 0xc000242e60, 0xc000219870, 0x0)
/usr/local/google/home/bde/go/src/cuelang.org/go/cue/types.go:1327 +0x17e
cuelang.org/go/cue.Value.Walk(0xc000242d70, 0xc000242e10, 0xc000219870, 0x0)
/usr/local/google/home/bde/go/src/cuelang.org/go/cue/types.go:1329 +0x16d
cuelang.org/go/cue.Value.Walk(0xc000242d70, 0xc000242dc0, 0xc000219870, 0x0)
/usr/local/google/home/bde/go/src/cuelang.org/go/cue/types.go:1329 +0x16d
cuelang.org/go/cue.Value.Validate(0xc000242d70, 0xc000242dc0, 0x0, 0x0, 0x0, 0x1, 0x1)
/usr/local/google/home/bde/go/src/cuelang.org/go/cue/types.go:1274 +0x106
cuelang.org/go/cmd/cue/cmd.buildInstances(0x12f5080, 0xc000010c80, 0x1, 0x1, 0xc000010c80, 0x1, 0x1)
/usr/local/google/home/bde/go/src/cuelang.org/go/cmd/cue/cmd/common.go:102 +0x110
cuelang.org/go/cmd/cue/cmd.buildFromArgs(0x12f5080, 0xc00023e950, 0x1, 0x1, 0x8a7560, 0xbee4e0, 0x0)
/usr/local/google/home/bde/go/src/cuelang.org/go/cmd/cue/cmd/common.go:72 +0x7e
cuelang.org/go/cmd/cue/cmd.glob..func2(0x12f5080, 0xc00023e950, 0x1, 0x1, 0x0, 0x0)
/usr/local/google/home/bde/go/src/cuelang.org/go/cmd/cue/cmd/eval.go:50 +0x6a
github.com/spf13/cobra.(*Command).execute(0x12f5080, 0xc00023e920, 0x1, 0x1, 0x12f5080, 0xc00023e920)
/usr/local/google/home/bde/go/src/github.com/spf13/cobra/command.go:762 +0x465
github.com/spf13/cobra.(*Command).ExecuteC(0x12f4960, 0xc000219da0, 0x7ffd2f1f1fba, 0x4)
/usr/local/google/home/bde/go/src/github.com/spf13/cobra/command.go:850 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
/usr/local/google/home/bde/go/src/github.com/spf13/cobra/command.go:800
cuelang.org/go/cmd/cue/cmd.Execute()
/usr/local/google/home/bde/go/src/cuelang.org/go/cmd/cue/cmd/root.go:129 +0xd9
main.main()

While running:
go get -u cuelang.org/go/cmd/cue

I get:
cmd/cue/cmd/vet.go:66:7: undefined: fVerbose

https://github.com/cuelang/cue/blob/master/cmd/cue/cmd/vet.go#L66

• keeping the links in sync is kind of tedious.
• would be nice to have some proper formatting
• use in-browser evaluation of example (using the GopherJS port)

To display the current build and version.

I can't seem to get import of a local package to work. An example would be great. Maybe that could be folded into kubernetes/manual?

I was experimenting with adding a set of validation commands to my cue configuration, in _tools.cue files. The commands they exec out to will return non-zero exit codes when those commands fail, but running them via the cue <cmd> runner eats those exit codes and always returns 0. Checking the code this doesn't appear to be something I can add to the print task.

Is there a way of achieving this at present? Supporting passing through the exit codes would make integration of cue commands into pipelines easier.