cloudfoundry-attic / bosh-notes Goto Github PK

On the the bosh-bootloader team, we've had some issues with string interpolation in our pipeline of Terraform -> Bash -> YAML/Bosh manifest. Our tool essentially maintains a script like this:

#!/bin/sh
bosh create-env \
  ${BBL_STATE_DIR}/bosh-deployment/bosh.yml \
  --state  ${BBL_STATE_DIR}/vars/bosh-state.json \
  --vars-store  ${BBL_STATE_DIR}/vars/director-vars-store.yml \
  --vars-file  ${BBL_STATE_DIR}/vars/director-vars-file.yml \
  -o  ${BBL_STATE_DIR}/bosh-deployment/gcp/cpi.yml \
  -o  ${BBL_STATE_DIR}/bosh-deployment/jumpbox-user.yml \
  -o  ${BBL_STATE_DIR}/bosh-deployment/uaa.yml \
  -o  ${BBL_STATE_DIR}/bosh-deployment/credhub.yml \
  -o  ${BBL_STATE_DIR}/bbl-ops-files/gcp/bosh-director-ephemeral-ip-ops.yml \
  --var-file  gcp_credentials_json="${BBL_GCP_SERVICE_ACCOUNT_KEY_PATH}" \
  -v  project_id="${BBL_GCP_PROJECT_ID}" \
  -v zone="${BBL_GCP_ZONE}" 

This script stores the "state" of the director in a sense. What ops files it was made with, in particular.

The issues have motivated us to explore replacing that script with a YAML file, or something. This would make the invocation of create-env so trivial that we wouldn't bother documenting it as heavily anymore:

bosh --ini-config bosh-arg-config.yml create-env

The go-flags library already supports this and calls it "INI files":

[create-env]

; comments

o=ops.yml

v=variable=value

[delete-env]

o=ops.yml

But that could just as easily be YAML, because this is Bosh after all:

---

create-env:

  o: "ops.yml"

  v: variable="value"

delete-env:

  o: "ops.yml"

Given this file above, you could also supply it to delete-env and have an effect:

bosh --ini-config bosh-arg-config.yml delete-env

@cdutra is already comfortable guiding our team on implementing this, and we could PR it, and @freddesbiens from the Bosh team has shown at least some interest in this direction so maybe we could get it merged as outline above, too.

However, we're interested in feedback to be better:

1. Help us come up with a better name for this.
2. Help us think of other usecases, like documenting a full deployment example more succinctly in cf-deployment. @heyjcollins
3. Help us think of how this format could be extended with more metadata, like maybe this could assist users in determining what ops files work with what other opsfiles, or maybe you could use these to prompt the user for input and make decisions kind of like outline in the Bosh bundles proposal

Description in this repo and on bosh.io/docs only give examples for OpenStack. How would one create a cpi-config for Garden CPI?

I was thinking more about addons...

It seems like addons will often be maintained by independent teams (i.e. logging/monitoring, antivirus on-site). It seems weird to have a single file where everyone will be touching things. Rather than a single file of many addons that many people have to work in, perhaps they could be individually maintained. It seems like it'd be much more readable, concise, and easier to maintain.

$ bosh update addon /dev/stdin << EOF
name: ipsec
releases:
  - { name: ipsec, version: 27 }
templates:
  - { name: ipsec_agent, release: ipspec }
  - { name: ipsec_openswan, release: ipsec }
properties:
  ipsec:
    shared_key: UYasbvh872f
EOF

Internally, they could be versioned such that deployments point to which addons they're currently using. This seems like it'd be easier than deployments pointing to a single entry of all addons which may not actually be relevant to the deployment (like cloud-config)...

$ psql 'select * from deployments_addons;'
{ "deployment": "always_updated",  "addon_version": "3", "addon_version.addon.name": "ipsec" }
{ "deployment": "outdated_deploy", "addon_version": "1", "addon_version.addon.name": "ipsec" }

$ bosh addons
[ { "name": "ipsec",
    "versions": [ "1", "2", "3" ] },
  { "name": "logsearch-shipper",
    "versions": [ "4", "5", "6" ] } ]

$ bosh deployment addons "outdated_deploy"
[ { "name": "ipsec",
    "version": "1",
    "latest_version": "3" } ]

Rather than versions, it might be worthwhile to consider using timestamps. Versions might imply correlations to the addon's .releases[].version and cause confusion. For example, the ipsec addon is using release version 27, but the BOSH director might consider the addon config version to be 3 causing users to become worried it's out of date.

On the releases[].version discussion of latest, I'd personally rather see it be explicit where I as the addon operator need to state the release version I expect deployments to use.

Personally, I'd like to see addons not be installed by default, but rather let deployment operators elect which addons/features are going to be transforming their deployments. Maybe that's where tags.excluded_on is useful? Seems odd to make addon operators responsible for knowing which deployments are going to be deployed if those addons are more service-oriented. As mentioned in the file, some addons might need network security group customizations and on some CPIs that'll be affecting how I plan those out and I need to be aware of that. Also, some addons may just be for convenience than having to retype a bunch of properties or global templates but aren't necessarily appropriate for everything I deploy. For that, I think it'd be nice to have a deployment manifest addons property which allows me to elect (or possibly ignore) addons...

name: my_deployment
director_uuid: 423a77e8-1aca-45c6-a5d0-d70cf6417ce6
addons:
  - ramdisk             # use the ramdisk addon which might not be installed by default
  - !logsearch-shipper  # do not use the addon which might be installed by default
  - ipsec?              # if there's an ipsec addon, use it; otherwise ignore
templates:
  ...snip...

Other crazy idea... what if deployments could expose an addon since it seems like some addons might be closely bound to a deployment's services. For example, a deployment manifest managing a logsearch cluster might expose an addon for shipper which will ship logs into the deployment. Other deployments could opt into that by referencing the logsearch.shipper addon it exposed. Haven't really thought that one through though.

Anyways... just some ideas which seemed easier to consider with examples.

For debugging it would be useful to show recent tasks filtered by deployment.

Ie: Today i know that i have a problem that occurred with logging. This problem is probably in the deployment cf-cfapps-io2 However, at least as far as i know i cannot looks at recent tasks that have occurred specifically to this deployment.

alternatively, if in the recent tasks we had another field shown for deployment we would be able to parse this on our own using AWK or other UNIX tools. Screenie below to illustrate what i am talking about here

Thx for consideration =D

Typically DNS for BOSH deployed software is seen as a pre-install concern, but I would like to propose that it be something that can be implemented in each CPI. CFAR and CFCR both have some implicit requirements for configuring external DNS to meet their conventions (CFAR has always had this need, CFCR has just added this pre-requisite).

AWS, GCP, Azure, and Openstack all have some form of DNSaaS, so this is nearly a universal Cloud resource. Even for users of vSphere or in regions which do not support DNS (China, GovCloud), there could be significant benefit just to know explicitly what are the requirements around DNS in the sample deployment manifest for a specific SemVer of a BOSH release rather than correlate documentation and software to infer this information.

If this were available to be configured via manifests and cloud configs, then we also could begin to implement BOSH releases which are much more dependent on runtime modification of DNS records. For example, Let's Encrypt's wildcard certs only work with DNS based validation, where a DNS entry must be made containing a challenge response. This challenge must be renewed every three weeks, so doing this during bootstrapping is of limited benefit.

Some counterpoints / risks I see:

• Roles/Permissions/IAM: How would we enable an operator to say "these users/deployments can modify DNS rules on this CPI but these users/deployments cannot"? Perhaps the operator would have to install one cpi with a role that doesn't allow modification of DNS, and another version of the cpi with roles that can modify DNS.

Hello there,

We found a situation where we wanted to add a feature to an errand, but without changing it's job code, for compatibility reasons. The best solution we could think of would be adding this feature as a separate job/template to the errand, but we found out that bosh didn't like that very much.
It would be pretty cool if bosh supported this, or if there's some workaround to achieve similar results we would love to hear!

Many thanks!

I think that templates for jobs would be easier to create if bosh-template library had something like "partial templates" helpers.

For instance you can look at director.yml.erb.erb. It has a large block for each IaaS it supports. If we could use partials, we are able to extract this IaaS specific settings to distinct templates. The resulting code will be much cleaner, something like this:

<% if_p('aws') do %>
  <%= render 'cpi/aws' %>
<% end %>
<% if_p('openstack') do %>
  <%= render 'cpi/openstack' %>
<% end %>
...

What do you think?

https://github.com/cloudfoundry/bosh-notes/blob/master/availability-zones.md

In our deployment we have a vSphere datacenter to separate AZs. So, this is a reminder that if you fix AZ support for AWS and not for our use case do what you can to help ensure cf-release continues to provide a way to uniquely identify a job/index.

Currently the CC for example has template like this:
https://github.com/cloudfoundry/cloud_controller_ng/blob/5257a8af6990e71cd1e34ae8978dfe4773b32826/bosh-templates/cloud_controller_worker_ctl.erb#L48

If AWS AZ is fixed I would be afraid that runtime developers would begin assuming that a CF job will be uniquely identifiable via index only:

exec chpst -u vcap:vcap bundle exec rake <%= task_name %>[cc_global_worker.<%= spec.index %>.$INDEX]

Config like that will break those of us that still have to spread CF installations across several deployments.

BUCC is really cool. It has one thing in particular that is a super power I'd like to give to all things BOSH: after you install Concourse, you are configured to use Concourse.

Here's what it is normally like:

1. run this command to install concourse
2. now interrogate credhub to figure out what secrets/infrastructure were generated
3. use the results of that interrogation to log in to concourse and fly set-pipeline

What if it were like this:

1. run this command to install concourse:bosh deploy -d concourse -v target-name=myconcourse
2. fly -t myconcourse set-pipeline --pipeline my-pipeline --config pipeline.yml
3. There is no step three.

This same concept applies to CFAR and the CF cli, and to CFCR and the kubectl cli. This would be something to run locally after the whole release is resolved, possibly after the whole manifest has successfully deployed.

Risks/Challenges/Thoughts:

• well, for starters, this is probably somewhat operating system specific. The process by which I target concourse on Windows is probably different than on Linux.
• This is so convenient that I will want to re-enter it, so I may want to run just the local-post-deploy hooks in a release.
• This almost necessarily involves secrets, and a lot of baggage comes along with that: what if I don't want this written to disk? What if it is rotated while I'm trying to use it?
• In addition to implicit targeting One MVP iteration of this would be to simply output to screen the whole host of variables you need to log into your concourse:

Thanks for installing concourse! Run this to log into your concourse:
fly -t myconcourse --api https://54.172.12.7 --password qwertyuiop --user admin

Initial proposal: availability-zones.md

Currently it is hard to build stemcells, especially if you want to build them to a new clouds.

There are several reasons for it:

• you have to patch bosh, I had to patch bosh-release, bosh-stemcell, bosh-dev projects except of stemcell_builder.
• it takes a lot of time to check your changes had necessary effects
• you can’t build “light” stemcells for new infrastructures easily using current BOSH stemcell builder.

I suppose that live would be much more better after BOSH will go from compiling MicroBOSH into stemcell, but still there are some possible ways to improve it.

Here are some suggestions to improve stemcell builder:

• Run stemcell builder locally. I had customers that didn’t wanted to use any external infrastructure (like AWS) to build stemcell, this meant they wanted to run them into their IaaS. Vagrant box didn’t played well in such situations because of absence of nested virtualization.
• Build “light” stemcells for external clouds. Some clouds doesn’t have “custom images” as a feature. And the only one option there is to build instance templates from existing VMs. For this kind of solutions it is necessary to run stemcell builder on the cloud instence and build stemcell into attached disks or a system disk.
• Rollback of state of file system after failed stages. Currently you spend a lot of time rebuilding stemcell after small changes that leaded to errors. I suggest to use something like aufs to keep state of file system to rollback after stage failed. This will speed up development process.
• Declarative way to define stages. That's ok to have such DSL as BOSH currently has, but it would be much more easier for people not to mess with ruby code in stage_collection.rb file, but to pass some .yml manifest to the builder.

Also I have a strange feeling that it would be better to rewrite stemcell builder in go lang :)

What do you think ?

Initial proposal: https://github.com/cloudfoundry/bosh-notes/blob/master/cloud-config.md

Initial proposal: https://github.com/cloudfoundry/bosh-notes/blob/master/deployment-naming.md

Hi. I found bosh's plan to remove registry.
https://github.com/cloudfoundry/bosh-notes/blob/a34e33e95bda0712aa24ba03c1e49e4a1f147be1/cpi-api-v2.md
https://github.com/cloudfoundry/bosh-notes/blob/master/registry-removal.md

But I did not know the reason for this? Do we have any related discussions?

Thank you.

It would be great to have bosh job auto-scaling functionality either built into BOSH core or available as an add-on service.

Two example popular use cases which come to mind:

• Cell Autoscaling
  Triggered either by avgerage free memory across the Cells or maybe by looking at some more complicated metrics from CF (CC or BBS) to tell that jobs destined for a specific job/isolation segment can not be placed.

• CF goRouter Autoscaling
  Triggered by CPU % of the routers or something more complicated like requests received per second

Hey, all.

A lot of errors can occur because of a "human factor". I mean that errors in BOSH manifests can be caused by misspelling or misunderstanding of BOSH release properties. Even if BOSH can check some type of errors in manifest properties, often it is not enough.

I am pretty sure that teams that are implementing BOSH release are the best people who can put reasonable validations to properties fields. That's why I'm sure that it will be a good thing to provide a way to add validations to BOSH release properties during implementation. At this moment job spec (example) contains only following fields: description, default and the name of the property. I suggest adding type and validations fields to this sections. The easiest way to validate properties will be checking their types and validating across regexp.

This feature will allow to aware user to set database schema to postgresql, not postgres and will save user's time to find out what's wrong.

What do you think about this feature?

I feel strong demand in writing tests while I develop BOSH releases.

There are lots of positive sides of it. First of all it could speed up development process, when developer will not be needed to deploy bosh-release to separate test infrastructure. Another positive side of tested releases would be possibility to check that PRs work (for instance you'll be 100% sure that merging some PR from side developer to admin-iu-boshrelease will not break any features from acceptance tests).

I guess we can run this tests inside of containers on developers host, possibly we can use Docker there. What are your thoughts on this?

TDD+BOSH FTW!

Hello,

I would like to know if the feature errand v2 (https://github.com/cloudfoundry/bosh-notes/blob/master/errands-v2.md) is fully available.
I have install BOSH 263.1.0 as a pre-requisite. It is OK with the manifest file.
But when I launch the command "bosh -d myDeployment errands", my errand doesn't appear in stdout.
It is not possible to trigger it. Maybe the
Thanks in advance and regards,

Initial proposal: https://github.com/cloudfoundry/bosh-notes/blob/master/uaa.md