geofffranks / spruce Goto Github PK

A BOSH template merge tool

License: MIT License

Go 99.42% Shell 0.38% Makefile 0.06% Dockerfile 0.14%

spruce's Introduction

          *          .---. ,---.  ,---.  .-. .-.  ,--,  ,---.         *
         /.\        ( .-._)| .-.\ | .-.\ | | | |.' .')  | .-'        /.\
        /..'\      (_) \   | |-' )| `-'/ | | | ||  |(_) | `-.       /..'\
        /'.'\      _  \ \  | |--' |   (  | | | |\  \    | .-'       /'.'\
       /.''.'\    ( `-'  ) | |    | |\ \ | `-')| \  `-. |  `--.    /.''.'\
       /.'.'.\     `----'  /(     |_| \)\`---(_)  \____\/( __.'    /.'.'.\
"'""""/'.''.'.\""'"'""""""(__)""""""""(__)"""""""""""""(__)""'""""/'.''.'.\""'"'"
      ^^^[_]^^^                                                   ^^^[_]^^^

( We'll be in #spruce)

Introducing Spruce

spruce is a general purpose YAML & JSON merging tool.

It is designed to be an intuitive utility for merging YAML/JSON templates together to generate complicated YAML/JSON config files in a repeatable fashion. It can be used to stitch together some generic/top level definitions for the config and pull in overrides for site-specific configurations to DRY your configs up as much as possible.

How do I get started?

spruce is available via Homebrew, just brew tap starkandwayne/cf; brew install spruce

Alternatively, you can download a prebuilt binaries for 64-bit Linux, or Mac OS X

How do I compile from source?

Install Go, e.g. on Ubuntu sudo snap install --classic go
Fetch sources via go get github.com/geofffranks/spruce
Change current directory to the source root cd ~/go/src/github.com/geofffranks/spruce/
Compile and execute tests make all

A Quick Example

# Let's build the first yaml file we will merge
$ cat <<EOF first.yml
some_data: this will be overwritten later
a_random_map:
  key1: some data
heres_an_array:
- first element
EOF

# and now build the second yaml file to merge on top of it
$ cat <<EOF second.yml
some_data: 42
a_random_map:
  key2: adding more data
heres_an_array:
- (( prepend ))
- zeroth element
more_data: 84

# what happens when we spruce merge?
$ spruce merge first.yml second.yml
a_random_map:
  key1: some data
  key2: adding more data
heres_an_array:
- zeroth element
- first element
more_data: 84
some_data: 42

The data in second.yml is overlayed on top of the data in first.yml. Check out the merge semantics and array merging for more info on how that was done. Or, check out [this example on play.spruce.cf][play.spruce-example]

Documentation

What else can Spruce do for you?

spruce doesn't just stop at merging datastructures together. It also has the following helpful subcommands:

spruce diff - Allows you to get a useful diff of two YAML files, to see where they differ semantically. This is more than a simple diff tool, as it examines the functional differences, rather than just textual (e.g. key-ordering differences would be ignored)

spruce json - Allows you to convert a YAML document into JSON, for consumption by something that requires a JSON input. spruce merge will handle both YAML + JSON documents, but produce only YAML output.

spruce vaultinfo - Takes a list of files that would be merged together, and analyzes what paths in Vault would be looked up. Useful for determining explicitly what access an automated process might need to Vault to obtain the right credentials, and nothing more. Also useful if you need to audit what credentials your configs are retrieving for a system..

License

Licensed under the MIT License

spruce's People

Contributors

Stargazers

Watchers

spruce's Issues

Problem with exclamation marks in YML

Hi everyone, I was trying to use spruce in the following way:

bosh download manifest cf.yml 
spruce merge cf.yml mytemplate.yml

And I found that spruce apparently has issues with ! , and the manifest deployed by Bosh contains them:

properties: 
    route_registrar:
      routes:
      - name: uaa
        port: 8080
        tags:
          component: uaa
        uris:
        - uaa.fabric.ng.bluemix.net
        - ! '*.uaa.fabric.ng.bluemix.net'

The error that I get is:

tmp/cf.yml: unmarshal []byte to yaml failed: yaml: line 525: did not find URI escaped octet

Any idea of what I could do? Thanks in advance

PostProcessor overhaul

Rework the postprocessing logic to be a command-dispatch type processor, with proper recursion support, to fix weird edge cases + quasi-recursion bugs.

Post Processing of Merged data

Add hooks/support for post processing of the final merged data to handle things like:

static IP calculations
resource pool size auto-calculation
Anything else required

Multiple ranges of static IPs causes interesting failures

If multiple static_ip ranges are specified in a subnet, the pool of addresses calculated is incorrect. Instead of being the first range plus second range and so on, it is the Nth range repeated N times.

http://play.spruce.cf/#bb8ea008d705965abee8

Support an inject mechanism

spiff's custom yaml parser allowed for <<: (( thing )) to pull in attributes from thing, and place them in the parent. Since spruce is using goyaml, and <<: is actually a YAML construct when used with YAML anchors, we can't use this with spruce.

Implement an (( inject stuff )) operator so this is once again possible.

Concat'in a concat bug

when trying to use a value that you calculated by concat can't be used when merged sometimes. Example below is how to replicate issue

$ spruce merge test.yml test2.yml
properties:
  blah: foo.xip.iobar
  domain: foo.xip.io



$ spruce merge test2.yml test.yml
properties:
  blah: (( concat "foo" ".xip.io" ))bar
  domain: foo.xip.io


$ cat test2.yml
properties:
  blah: (( concat properties.domain "bar" ))

$ cat test.yml
properties:
  domain: (( concat "foo" ".xip.io" ))

Better Debugging for post-processing

Show yaml dumps in-between post-processing steps to aid in debugging.

Make VAULT_TOKEN optional (read from ~) in (( vault ... )) operator

We added the requirement on VAULT_TOKEN for CI deployments via concourse, so that we could interact directly via HTTP API and forego the Vault CLI.

This is causing no end of confusion when trying to run make withcreds, since the Vault has been authed to, but the token is in ~/.vault-token

If VAULT_TOKEN is not set in the environment, we should attempt to read it from ~/.vault-token

Better debugging about the VAULT not being used would also be nice.

Concat with non-overridden param does not throw an error

For example:

http://play.spruce.cf/#a811a3dc3e2f70141779

Infinity loop when assign static IPs

In commit id 6f5a63a, Infinity loop occurs when reserve static IPs more then 256.

Following is an example.

$ cat template.yml
networks:
- name: cf1
  subnets:
  - static:
    - 10.0.0.2 - 10.0.1.254
jobs:
- name: foo
  templates:
  - name: bar
  instances: 1
  networks:
  - name: cf1
    static_ips: (( static_ips(0) ))

$ spruce -D merge template.yml
DEBUG> Processing file 'template.yml'
DEBUG> $.networks: not found upstream, adding it
DEBUG> $.jobs: not found upstream, adding it
DEBUG> parsing `(( static_ips(0) ))': looks like a (( static_ips ... )) operator
DEBUG>  arguments:
DEBUG>   #0: parsed as unquoted integer literal '0'
DEBUG>
DEBUG> patching up YAML by evaluating outstanding operators
DEBUG>
DEBUG> running (( static_ips ... )) operation at $.jobs.0.networks.0.static_ips
DEBUG>   determining what job context (( static_ips ... )) was called in
DEBUG>   got it.  $.jobs.0
DEBUG>
DEBUG>   extracting job name from $.jobs.0.name
DEBUG>   got it.  job is foo
DEBUG>
DEBUG>   determining how many instances of job foo there are
DEBUG>   got it.  there are 1 instances of foo
DEBUG>
DEBUG>   checking to see if the caller asked for enough static_ips to provision all job instances (need at least 1)
DEBUG>   looks good.  asking for 1 IPs for a job with 1 instances
DEBUG>
DEBUG>   determining the pool of static IPs from which to provision

# gone away!

[Feature]: Operation to read from environment variables

It would be great to have an operation to read environment variables, so that you can override some values easy.

Something like:

server:
  aws_secret: (( env(AWS_SECRET) ))
  url; (( env(SERVER_URL) || 'localhost' ))

What do you think?

Better Array merging syntax + Support

Replacing current behaviors of merging:

(( append ))
(( prepend ))
New:
(( inline )) - this one merges objects index-by-index until it runs out, and then appends the remainder, if any.

Use starkandwayne Dockerfiles for concourse

And remove the custom ci_image from the pipeline.

The concourse-go image, tag 1.5, should be suitable, it just needs to be amended to pre-generate the goxc toolchain for the desired architectures (probably all of them). I'll put a separate ticket in over there.

[Feature] show what file each field came from

Move `(( inject ... ))` Into an Interim Phase

Resolution of (( inject ... )) calls needs to move to its own interim phase, in between doc-merge and oper-eval. We may need to do DFA here too, to make sure that we are sub-injecting properly.

This is a prereq for the || syntax in operand evaluation, to simplify DFA there (in oper-eval)

Inject not merging properly

Inject is not merging arrays properly when injecting. It replaces rather than merges when that option is available. The yaml at http://play.spruce.cf/#bdeb1d30d979361217a5 should result in the below output, but does not (the my_job and my_other_job elements are missing from templates).


---
jobs:
  api_node:
    properties:
      foo: bar
      this: that
    templates:
    - name: my_job
      release: my_release
    - name: my_other_job
      release: my_other_release
    - name: my_superspecial_job
      release: my_superspecial_release

Make arrays merge by default

Everything else merges by default, make arrays work like this by default:

Try to merge maps by name
If not all maps, or not all maps have name key, error indicating no specified array merge plan

require explicit override with null

stub1.yml:

baz:
  foo:
    bar: (( param "I need a bar" ))

stub2.yml:

baz:
  foo:

$ spruce merge stub1.yml stub2.yml
baz:
  foo: null

I expected the missing param bar to cause an error.

I think if I wanted the result to be what I pasted above I should have to have the following in stub2.yml

baz:
  foo: ~

What do you think?

Add "releases" subcommand

It will ensure the currently targeted bosh target (or provide a bosh target via a CLI flag?) has the releases specified in a given manifest uploaded to it

Release binaries structure

Hey,
I'm setting up a jumpbox provisioner script. Is there an automatic way to download the 'latest' version of spiff without installing go?

Haydon

non-helpful error message on missing param

1 error(s) detected:
 - $.jobs.cluster_monitor.properties.nats-to-syslog.nats.password: BOSH NATS password

for (( param "BOSH NATS password" ))

Luckily I knew what was wrong because I was testing my stubs but a slightly more useful error message would be immensely helpful for users who don't actually know what's in the stubs.

Version Output

Add a -v/--version flag to spruce to output the current version

Unable to concat int

When merging the files below I get interface conversion: interface is int, not string

releases:
- name: bosh
  url: file:///tmp/bosh-release.tgz
  version: 5

releases:
- name: bosh
  url: (( concat "http://bosh.io/d/github.com/cloudfoundry/bosh?v=" releases.bosh.version ))

Convert spruce to sub-command based app

Move merging behavior into a sub-command called "merge", opening up the way for more subcommands

panic: runtime error: invalid memory address or nil pointer dereference

Yay for useful error messages

This occurs when you set a field like this

consul: ~

This is a null and when you reference it with grab (( grab consul ))... you get the "panic: runtime error: invalid memory address or nil pointer dereference"

Add travis CI support

Add a 'defer' mechanism to force keys to be specified

Add a (( defer <message> )) to allow people to force a property to be defined by a later file in the merge.

Should post-process after all static_ips, grab, and pruning, and cause errors if any properties were not defined.

Data flow issue with inject

http://play.spruce.cf/#6c669324bf3b7d5a96ed

Injection of named blocks that also contain injections do not inject the original injection

@jhunt indicated this is an aliasing bug of x.0 vs x.name

Crash when running first example

When running the first example current spruce master crashes. It does not crash when running v0.12.0.

jonty@towel:~/code/go/src/github.com/geofffranks/spruce (master)$ git rev-parse HEAD
293a7e78d6e370731706ceb951320a468d34b810

jonty@towel:~/code/go/src/github.com/geofffranks/spruce (master)$ spruce merge assets/examples/example.yml assets/examples/example2.yml
panic: interface conversion: interface is int, not string

goroutine 1 [running]:
main.(*Evaluator).DataFlow.func2(0x20b4a0, 0xc82006b290)
    /Users/jonty/code/go/src/github.com/geofffranks/spruce/evaluator.go:61 +0x5b4
main.(*Evaluator).DataFlow.func1(0x20b4a0, 0xc82006b290)
    /Users/jonty/code/go/src/github.com/geofffranks/spruce/evaluator.go:53 +0xa49
main.(*Evaluator).DataFlow.func2(0x20b4a0, 0xc82006af60)
    /Users/jonty/code/go/src/github.com/geofffranks/spruce/evaluator.go:62 +0x64b
main.(*Evaluator).DataFlow(0xc82008de00, 0x0, 0x0)
    /Users/jonty/code/go/src/github.com/geofffranks/spruce/evaluator.go:75 +0x258
main.(*Evaluator).Run(0xc82008de00, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/jonty/code/go/src/github.com/geofffranks/spruce/evaluator.go:376 +0xbf
main.main()
    /Users/jonty/code/go/src/github.com/geofffranks/spruce/main.go:133 +0x634

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/Cellar/go/1.5.1/libexec/src/runtime/asm_amd64.s:1696 +0x1

Handle Vault redirects better

If you set the VAULT_ADDR environment variable to a non-leader Vault in a clustered Vault setup (tested on Consul), Spruce is unable to properly follow the 307 Temporary Redirect, yielding "missing client token" errors responses (400 Bad Request) from the actual leader node.

Exchange is:

  C: GET /v1/secret/whatever
     Host: <non-leader>
     X-Vault-Token: <TOKEN>
  S: HTTP 307 Temporary Redirect
     Location: <real leader>
  C: GET /v1/secret/whatever
     Host: <real leader>
     (no X-Vault-Token header)

We should either specify a Redirect Policy to net/http that will re-submit with headers intact, or forgo automatic redirection and handle it ourselves.

Static ip should return empty array if instance count is zero

In the usecase of specifying static ips but the instance count is 0, this should return empty array via grab/concat and also return only number of static ips as they are instance counts

Can not grab static_ips correctly

The spruce can not grab static_ips correctoly in 99de7eb.

Following manifest can grab correctly.

$ cat manifest.yml
networks:
  - name: test-network
    subnets:
      - static: [ 10.0.0.5 - 10.0.0.10 ]
jobs:
  - name: job1
    instances: 1
    networks:
      - name: test-network
        static_ips: (( static_ips(0) ))

z: (( grab jobs.job1.networks.test-network.static_ips ))
$ ./spruce merge manifest.yml
jobs:
- instances: 1
  name: job1
  networks:
  - name: test-network
    static_ips:
    - 10.0.0.5
networks:
- name: test-network
  subnets:
  - static:
    - 10.0.0.5 - 10.0.0.10
z:
- 10.0.0.5

But next manifest can not grab correctly.

$ cat manifest.yml
networks:
  - name: test-network
    subnets:
      - static: [ 10.0.0.5 - 10.0.0.10 ]
jobs:
  - name: job1
    instances: 1
    networks:
      - name: test-network
        static_ips: (( static_ips(0) ))

a: (( grab jobs.job1.networks.test-network.static_ips ))
$ ./spruce merge manifest.yml
a: (( static_ips(0) ))
jobs:
- instances: 1
  name: job1
  networks:
  - name: test-network
    static_ips:
    - 10.0.0.5
networks:
- name: test-network
  subnets:
  - static:
    - 10.0.0.5 - 10.0.0.10

It seem to depend in order of key in yaml.

static_ips doesn't work when you merge static section of subnet from other place

Hey, everyone.

I have the following error. Here are template files:

jobs.yml

jobs:
- name: api_z1
  instances: 3
  networks:
  - name: net1
    static_ips: (( static_ips(0, 1, 2) ))

networks.yml

networks:
- name: net1
  subnets:
  - static: (( network_settings.static ))

settings.yml

network_settings:
  static:
  - 10.10.2.65 - 10.10.2.70

When I run spruce merge jobs.yml network.yml settings.yml I've got the following error message:

1 error(s) detected:
 - $.jobs.api_z1.networks.net1.static_ips: `$.networks.net1.subnets.[0].static` is not an array

concat fails when square brackets contains colon

When concat is used within square brackets, and the the concat string contains a colon, spruce merge fails.
For eg.,
template.yml

---
top: {url: (( concat "https://" some.variable  ".com" ))}

properties.yml

---
some:
  variable: example

spruce merge properties.yml template.yml results in the following error
template.yml: unmarshal []byte to yaml failed: yaml: line 1: found unexpected ':'

Spruce is incorrectly producing an error

Recently encountered the following:

$ make refresh manifest
Refreshing global data for sandbox
Updated global data
Refreshing site data for sandbox
Updated site data
Building asv / sandbox
Working in /home/quinn/deployments/cloud-foundry-deployments/sandbox
1 error(s) detected:
 - $.jobs.3.templates.0: (( append )) operator not defined


make: *** [manifest] Error 2

$ cd ../

$ grep -R "append" *
sandbox/.site/jobs.yml:  - (( append ))
site/jobs.yml:  - (( append ))

$ less site/jobs.yml

$ spruce -v
spruce - Version 0.12.0 (master)

The lines in question are:

 72   templates:
 73   - (( append ))
 74   - name: shield-agent
 75     release: shield

Tests fail due to pesky punctuation

When I run go tests (latest everything), I get:

x....
Failures:

  * /Users/jhunt/go/src/github.com/filefrog/spruce/main_test.go
  Line 19:
  Expected      'unmarshal []byte to yaml failed'
  to start with 'unmarshal []byte to yaml failed:'
  (but it didn't)!


10 total assertions

--- FAIL: TestParseYAML (0.00s)

[Bug] static_ips operator hangs with bad ranges

If a static range ends before it starts, the static_ips operator loops for quite some time (if not infinitely).

This should be fixed.

Example:

networks:
- name: default
  subnets:
  - cloud_properties:
      random: stuff
    range: 10.244.10.0/24
    reserved: 10.244.10.5 - 10.244.10.254
    static: 10.244.10.2 - 10.244.1.3

Add "stemcells" subcommand

It will ensure the currently targeted bosh target (or provide a bosh target via a CLI flag?) has the stemcells specified in a given manifest uploaded to it

Add "diff" subcommand

Add a diff subcommand for diffing two yaml manifests, and getting easily understood diff output

Accumulate all errors before exiting

Find all of the errors for a post-process step before bailing out, to provide a more useful user experience, rather than just latest error, bail out, fix, next error, bail out, fix iterations.

Concat string to array elements

We often have the need to append a certain string to all elements of an array. The primary use case is adding a port number to a list of IPs:

ips:
- 1.1.1.1
- 1.1.1.2
port: 8080
addr: (( ip ":" port ))

should produce

ips:
- 1.1.1.1
- 1.1.1.2
port: 8080
addr:
- 1.1.1.1:8080
- 1.1.1.2:8080

Can spruce support complex replacement?

Hi,

We have a yaml file likes the following, want to check with you if it works with spruce? We hope spruce can grab repositories.paas.path and repositories.paas.version correctly.

name: Charlie
version: 42

repositories:

name: paas
path: paas-app-artifacts
version: "23"
name: iaas
path: iaas-app-artifacts
version: 42
artifact: "(( grab repositories.paas.path))/((grab repositories.paas.version))/meta.yml"

Is spruce obsolete now that bosh links have landed?

http://bosh.io/docs/links.html cc @drnic

Add your spruce playground to fiddles.io

You should add your spruce fiddler to https://fiddles.io/

Spruce should error on `(( merge ))` operator in non-list context

Ran into a case converting spiff templates to spruce, where a few stray (( merge )) calls leftover from the old templates caused some unexpected (but not unexplained) behavior.

See http://play.spruce.cf/#4e03238d5bcc00f8e19c

It would be nice if, in the above example, spruce would error on the non-list context application of (( merge ))

Add debugging support

Make it easy to figure out what merges/overrides are happening, from what files, when, to make it easier to track down issues.

Also add debugging for stemcells/releases subcommands to show what steps are going on there

Support concat of strings

eg. (( grab foo.bar ":qux" )) would result in "foobar:qux" if foo.bar is "foobar"

Make cf-release work with spruce

I'm exploring whether it's possible to make cf-release work with spruce instead of spiff. For the most part, it's meant I can delete a bunch of some_prop: null all over the place, which is awesome. I'm trying to validate that this works by running the manifest generation tests in cf-release. So far, I've been able to get it working for 1 out of the 4 infrastructures (namely vsphere):

https://github.com/Amit-PivotalLabs/cf-release/tree/wip-spiff-to-spruce

Here's output from the test run:

$ bundle exec rspec spec
6 error(s) detected:
 - $.jobs.api_worker_z1.properties.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects
 - $.jobs.api_worker_z2.properties.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects
 - $.jobs.api_z1.properties.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects
 - $.jobs.api_z2.properties.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects
 - $.meta.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects
 - $.properties.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects


F6 error(s) detected:
 - $.jobs.api_worker_z1.properties.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects
 - $.jobs.api_worker_z2.properties.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects
 - $.jobs.api_z1.properties.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects
 - $.jobs.api_z2.properties.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects
 - $.meta.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects
 - $.properties.nfs_server.address: Unable to resolve `jobs.nfs_z1.networks.cf1.static_ips.[0]`: `jobs.nfs_z1.networks.cf1.static_ips` has no sub-objects


F7 error(s) detected:
 - $.jobs.api_worker_z1.properties.nfs_server.allow_from_entries.[1]: Unable to resolve `networks.cf2.subnets.[0].range`: `networks.cf2` could not be found in the YAML datastructure
 - $.jobs.api_worker_z2.properties.nfs_server.allow_from_entries.[1]: Unable to resolve `networks.cf2.subnets.[0].range`: `networks.cf2` could not be found in the YAML datastructure
 - $.jobs.api_z1.properties.nfs_server.allow_from_entries.[1]: Unable to resolve `networks.cf2.subnets.[0].range`: `networks.cf2` could not be found in the YAML datastructure
 - $.jobs.api_z2.properties.nfs_server.allow_from_entries.[1]: Unable to resolve `networks.cf2.subnets.[0].range`: `networks.cf2` could not be found in the YAML datastructure
 - $.meta.nfs_client_ranges.[1]: Unable to resolve `networks.cf2.subnets.[0].range`: `networks.cf2` could not be found in the YAML datastructure
 - $.meta.nfs_server.allow_from_entries.[1]: Unable to resolve `networks.cf2.subnets.[0].range`: `networks.cf2` could not be found in the YAML datastructure
 - $.properties.nfs_server.allow_from_entries.[1]: Unable to resolve `networks.cf2.subnets.[0].range`: `networks.cf2` could not be found in the YAML datastructure


F.

Failures:

  1) Manifest Generation aws behaves like generating manifests builds the correct manifest for aws
     Failure/Error: expect($?.exitstatus).to eq(0)

       expected: 0
            got: 2

       (compared using ==)
     Shared Example Group: "generating manifests" called from ./spec/manifest_generation_spec.rb:24
     # ./spec/manifest_generation_spec.rb:12:in `block (3 levels) in <top (required)>'

  2) Manifest Generation warden behaves like generating manifests builds the correct manifest for warden
     Failure/Error: expect($?.exitstatus).to eq(0)

       expected: 0
            got: 2

       (compared using ==)
     Shared Example Group: "generating manifests" called from ./spec/manifest_generation_spec.rb:28
     # ./spec/manifest_generation_spec.rb:12:in `block (3 levels) in <top (required)>'

  3) Manifest Generation openstack behaves like generating manifests builds the correct manifest for openstack
     Failure/Error: expect($?.exitstatus).to eq(0)

       expected: 0
            got: 2

       (compared using ==)
     Shared Example Group: "generating manifests" called from ./spec/manifest_generation_spec.rb:32
     # ./spec/manifest_generation_spec.rb:12:in `block (3 levels) in <top (required)>'

Finished in 1.35 seconds (files took 0.14466 seconds to load)
4 examples, 3 failures

Failed examples:

rspec ./spec/manifest_generation_spec.rb[1:1:1:1] # Manifest Generation aws behaves like generating manifests builds the correct manifest for aws
rspec ./spec/manifest_generation_spec.rb[1:2:1:1] # Manifest Generation warden behaves like generating manifests builds the correct manifest for warden
rspec ./spec/manifest_generation_spec.rb[1:3:1:1] # Manifest Generation openstack behaves like generating manifests builds the correct manifest for openstack

The failures stem from the following facts:

for AWS and warden, we don't use NFS, so we obviously don't assign static IPs to the NFS job. With spiff we can do (( jobs.nfs_z1.networks.cf1.static_ips.[0] || nil )); spruce doesn't support this feature so we have to remove the || nil and end up with ``jobs.nfs_z1.networks.cf1.static_ips has no sub-objects.
for openstack, we only have one network, so there is no cf2 network. Again, with spiff we can do (( networks.cf2.subnets.[0].range || nil )) but with spruce the best we can hope for is to remove the || nil, giving ``networks.cf2 could not be found in the YAML datastructure.

Any thoughts for good ideas on how to proceed?

Compiled releases are dynamically linked

The releases published in this repo are dynamically linked:

$ file spruce
spruce: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), not stripped

This can be a problem in scenarios like docker alpine. For instance, this Dockerfile, if you build it:

cat <<EOF > Dockerfile
FROM alpine:3.3

ENV SPRUCE_VERSION 0.13.0

RUN apk add --update wget ca-certificates \
  && wget https://github.com/geofffranks/spruce/releases/download/v${SPRUCE_VERSION}/spruce_${SPRUCE_VERSION}_linux_amd64.tar.gz \
  && tar -xvzf spruce_${SPRUCE_VERSION}_linux_amd64.tar.gz \
  && mv spruce_${SPRUCE_VERSION}_linux_amd64/spruce /usr/local/bin/ \
  && rm -rf spruce_${SPRUCE_VERSION}_linux_amd64* \
  && apk del wget ca-certificates \
  && rm -rf /var/cache/apk/*
EOF
docker build -t alpine-spruce .
docker run -i -t alpine-spruce ldd /usr/local/bin/spruce

It fails:

$ docker run -i -t alpine-spruce ldd /usr/local/bin/spruce
    /lib64/ld-linux-x86-64.so.2 (0x557b2cfc0000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x557b2cfc0000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x557b2cfc0000)
Error relocating /usr/local/bin/spruce: __vfprintf_chk: symbol not found

In this thread they discuss the topic and it seems that the magic solution is compile the binary with:

go build -tags netgo -installsuffix netgo .

Which indeed generates a static file:

$ file spruce 
spruce: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), statically linked, not stripped

Inject doesn't append to array

Inject doesn't append to array instead replace values

File:

meta:
  jobs:
    consul:
      templates:
      - foo
      - bar
      foo: fun

jobs:
- name: consul
  templates:
  - qux
  inject: (( inject meta.jobs.consul ))
  quack: boo

Result:

jobs:
- foo: fun
  name: consul
  quack: boo
  templates:
  - qux
  - bar
meta:
  jobs:
    consul:
      foo: fun
      templates:
      - foo
      - bar

Desired Result:

jobs:
- foo: fun
  name: consul
  quack: boo
  templates:
  - foo
  - bar
  - qux
meta:
  jobs:
    consul:
      foo: fun
      templates:
      - foo
      - bar