cloudfoundry-attic / bosh-notes Goto Github PK
View Code? Open in Web Editor NEWCollection of proposals for BOSH
License: Apache License 2.0
Collection of proposals for BOSH
License: Apache License 2.0
On the the bosh-bootloader team, we've had some issues with string interpolation in our pipeline of Terraform -> Bash -> YAML/Bosh manifest. Our tool essentially maintains a script like this:
#!/bin/sh
bosh create-env \
${BBL_STATE_DIR}/bosh-deployment/bosh.yml \
--state ${BBL_STATE_DIR}/vars/bosh-state.json \
--vars-store ${BBL_STATE_DIR}/vars/director-vars-store.yml \
--vars-file ${BBL_STATE_DIR}/vars/director-vars-file.yml \
-o ${BBL_STATE_DIR}/bosh-deployment/gcp/cpi.yml \
-o ${BBL_STATE_DIR}/bosh-deployment/jumpbox-user.yml \
-o ${BBL_STATE_DIR}/bosh-deployment/uaa.yml \
-o ${BBL_STATE_DIR}/bosh-deployment/credhub.yml \
-o ${BBL_STATE_DIR}/bbl-ops-files/gcp/bosh-director-ephemeral-ip-ops.yml \
--var-file gcp_credentials_json="${BBL_GCP_SERVICE_ACCOUNT_KEY_PATH}" \
-v project_id="${BBL_GCP_PROJECT_ID}" \
-v zone="${BBL_GCP_ZONE}"
This script stores the "state" of the director in a sense. What ops files it was made with, in particular.
The issues have motivated us to explore replacing that script with a YAML file, or something. This would make the invocation of create-env
so trivial that we wouldn't bother documenting it as heavily anymore:
bosh --ini-config bosh-arg-config.yml create-env
The go-flags library already supports this and calls it "INI files":
[create-env]
; comments
o=ops.yml
v=variable=value
[delete-env]
o=ops.yml
But that could just as easily be YAML, because this is Bosh after all:
---
create-env:
o: "ops.yml"
v: variable="value"
delete-env:
o: "ops.yml"
Given this file above, you could also supply it to delete-env
and have an effect:
bosh --ini-config bosh-arg-config.yml delete-env
@cdutra is already comfortable guiding our team on implementing this, and we could PR it, and @freddesbiens from the Bosh team has shown at least some interest in this direction so maybe we could get it merged as outline above, too.
However, we're interested in feedback to be better:
Description in this repo and on bosh.io/docs only give examples for OpenStack. How would one create a cpi-config for Garden CPI?
I was thinking more about addons...
It seems like addons will often be maintained by independent teams (i.e. logging/monitoring, antivirus on-site). It seems weird to have a single file where everyone will be touching things. Rather than a single file of many addons that many people have to work in, perhaps they could be individually maintained. It seems like it'd be much more readable, concise, and easier to maintain.
$ bosh update addon /dev/stdin << EOF
name: ipsec
releases:
- { name: ipsec, version: 27 }
templates:
- { name: ipsec_agent, release: ipspec }
- { name: ipsec_openswan, release: ipsec }
properties:
ipsec:
shared_key: UYasbvh872f
EOF
Internally, they could be versioned such that deployments point to which addons they're currently using. This seems like it'd be easier than deployments pointing to a single entry of all addons which may not actually be relevant to the deployment (like cloud-config)...
$ psql 'select * from deployments_addons;'
{ "deployment": "always_updated", "addon_version": "3", "addon_version.addon.name": "ipsec" }
{ "deployment": "outdated_deploy", "addon_version": "1", "addon_version.addon.name": "ipsec" }
$ bosh addons
[ { "name": "ipsec",
"versions": [ "1", "2", "3" ] },
{ "name": "logsearch-shipper",
"versions": [ "4", "5", "6" ] } ]
$ bosh deployment addons "outdated_deploy"
[ { "name": "ipsec",
"version": "1",
"latest_version": "3" } ]
Rather than versions, it might be worthwhile to consider using timestamps. Versions might imply correlations to the addon's .releases[].version
and cause confusion. For example, the ipsec
addon is using release version 27
, but the BOSH director might consider the addon config version to be 3
causing users to become worried it's out of date.
On the releases[].version
discussion of latest
, I'd personally rather see it be explicit where I as the addon operator need to state the release version I expect deployments to use.
Personally, I'd like to see addons
not be installed by default, but rather let deployment operators elect which addons/features are going to be transforming their deployments. Maybe that's where tags.excluded_on
is useful? Seems odd to make addon operators responsible for knowing which deployments are going to be deployed if those addons are more service-oriented. As mentioned in the file, some addons might need network security group customizations and on some CPIs that'll be affecting how I plan those out and I need to be aware of that. Also, some addons may just be for convenience than having to retype a bunch of properties or global templates but aren't necessarily appropriate for everything I deploy. For that, I think it'd be nice to have a deployment manifest addons
property which allows me to elect (or possibly ignore) addons...
name: my_deployment
director_uuid: 423a77e8-1aca-45c6-a5d0-d70cf6417ce6
addons:
- ramdisk # use the ramdisk addon which might not be installed by default
- !logsearch-shipper # do not use the addon which might be installed by default
- ipsec? # if there's an ipsec addon, use it; otherwise ignore
templates:
...snip...
Other crazy idea... what if deployments could expose an addon since it seems like some addons might be closely bound to a deployment's services. For example, a deployment manifest managing a logsearch
cluster might expose an addon for shipper
which will ship logs into the deployment. Other deployments could opt into that by referencing the logsearch.shipper
addon it exposed. Haven't really thought that one through though.
Anyways... just some ideas which seemed easier to consider with examples.
For debugging it would be useful to show recent tasks filtered by deployment.
Ie: Today i know that i have a problem that occurred with logging. This problem is probably in the deployment cf-cfapps-io2
However, at least as far as i know i cannot looks at recent tasks that have occurred specifically to this deployment.
alternatively, if in the recent tasks we had another field shown for deployment we would be able to parse this on our own using AWK or other UNIX tools. Screenie below to illustrate what i am talking about here
Thx for consideration =D
Typically DNS for BOSH deployed software is seen as a pre-install concern, but I would like to propose that it be something that can be implemented in each CPI. CFAR and CFCR both have some implicit requirements for configuring external DNS to meet their conventions (CFAR has always had this need, CFCR has just added this pre-requisite).
AWS, GCP, Azure, and Openstack all have some form of DNSaaS, so this is nearly a universal Cloud resource. Even for users of vSphere or in regions which do not support DNS (China, GovCloud), there could be significant benefit just to know explicitly what are the requirements around DNS in the sample deployment manifest for a specific SemVer of a BOSH release rather than correlate documentation and software to infer this information.
If this were available to be configured via manifests and cloud configs, then we also could begin to implement BOSH releases which are much more dependent on runtime modification of DNS records. For example, Let's Encrypt's wildcard certs only work with DNS based validation, where a DNS entry must be made containing a challenge response. This challenge must be renewed every three weeks, so doing this during bootstrapping is of limited benefit.
Some counterpoints / risks I see:
Hello there,
We found a situation where we wanted to add a feature to an errand, but without changing it's job code, for compatibility reasons. The best solution we could think of would be adding this feature as a separate job/template to the errand, but we found out that bosh didn't like that very much.
It would be pretty cool if bosh supported this, or if there's some workaround to achieve similar results we would love to hear!
Many thanks!
I think that templates for jobs would be easier to create if bosh-template library had something like "partial templates" helpers.
For instance you can look at director.yml.erb.erb. It has a large block for each IaaS it supports. If we could use partials, we are able to extract this IaaS specific settings to distinct templates. The resulting code will be much cleaner, something like this:
<% if_p('aws') do %>
<%= render 'cpi/aws' %>
<% end %>
<% if_p('openstack') do %>
<%= render 'cpi/openstack' %>
<% end %>
...
What do you think?
https://github.com/cloudfoundry/bosh-notes/blob/master/availability-zones.md
In our deployment we have a vSphere datacenter to separate AZs. So, this is a reminder that if you fix AZ support for AWS and not for our use case do what you can to help ensure cf-release continues to provide a way to uniquely identify a job/index.
Currently the CC for example has template like this:
https://github.com/cloudfoundry/cloud_controller_ng/blob/5257a8af6990e71cd1e34ae8978dfe4773b32826/bosh-templates/cloud_controller_worker_ctl.erb#L48
If AWS AZ is fixed I would be afraid that runtime developers would begin assuming that a CF job will be uniquely identifiable via index only:
exec chpst -u vcap:vcap bundle exec rake <%= task_name %>[cc_global_worker.<%= spec.index %>.$INDEX]
Config like that will break those of us that still have to spread CF installations across several deployments.
BUCC is really cool. It has one thing in particular that is a super power I'd like to give to all things BOSH: after you install Concourse, you are configured to use Concourse.
Here's what it is normally like:
fly set-pipeline
What if it were like this:
bosh deploy -d concourse -v target-name=myconcourse
fly -t myconcourse set-pipeline --pipeline my-pipeline --config pipeline.yml
This same concept applies to CFAR and the CF cli, and to CFCR and the kubectl cli. This would be something to run locally after the whole release is resolved, possibly after the whole manifest has successfully deployed.
Risks/Challenges/Thoughts:
Thanks for installing concourse! Run this to log into your concourse:
fly -t myconcourse --api https://54.172.12.7 --password qwertyuiop --user admin
Initial proposal: availability-zones.md
Currently it is hard to build stemcells, especially if you want to build them to a new clouds.
There are several reasons for it:
I suppose that live would be much more better after BOSH will go from compiling MicroBOSH into stemcell, but still there are some possible ways to improve it.
Here are some suggestions to improve stemcell builder:
Also I have a strange feeling that it would be better to rewrite stemcell builder in go lang :)
What do you think ?
Initial proposal: https://github.com/cloudfoundry/bosh-notes/blob/master/cloud-config.md
Hi. I found bosh's plan to remove registry.
https://github.com/cloudfoundry/bosh-notes/blob/a34e33e95bda0712aa24ba03c1e49e4a1f147be1/cpi-api-v2.md
https://github.com/cloudfoundry/bosh-notes/blob/master/registry-removal.md
But I did not know the reason for this? Do we have any related discussions?
Thank you.
It would be great to have bosh job auto-scaling functionality either built into BOSH core or available as an add-on service.
Two example popular use cases which come to mind:
Cell Autoscaling
Triggered either by avgerage free memory across the Cells or maybe by looking at some more complicated metrics from CF (CC or BBS) to tell that jobs destined for a specific job/isolation segment can not be placed.
CF goRouter Autoscaling
Triggered by CPU % of the routers or something more complicated like requests received per second
Hey, all.
A lot of errors can occur because of a "human factor". I mean that errors in BOSH manifests can be caused by misspelling or misunderstanding of BOSH release properties. Even if BOSH can check some type of errors in manifest properties, often it is not enough.
I am pretty sure that teams that are implementing BOSH release are the best people who can put reasonable validations to properties fields. That's why I'm sure that it will be a good thing to provide a way to add validations to BOSH release properties during implementation. At this moment job spec (example) contains only following fields: description
, default
and the name of the property. I suggest adding type
and validations
fields to this sections. The easiest way to validate properties will be checking their types and validating across regexp.
This feature will allow to aware user to set database schema to postgresql
, not postgres
and will save user's time to find out what's wrong.
What do you think about this feature?
I feel strong demand in writing tests while I develop BOSH releases.
There are lots of positive sides of it. First of all it could speed up development process, when developer will not be needed to deploy bosh-release to separate test infrastructure. Another positive side of tested releases would be possibility to check that PRs work (for instance you'll be 100% sure that merging some PR from side developer to admin-iu-boshrelease will not break any features from acceptance tests).
I guess we can run this tests inside of containers on developers host, possibly we can use Docker there. What are your thoughts on this?
TDD+BOSH FTW!
Hello,
I would like to know if the feature errand v2 (https://github.com/cloudfoundry/bosh-notes/blob/master/errands-v2.md) is fully available.
I have install BOSH 263.1.0 as a pre-requisite. It is OK with the manifest file.
But when I launch the command "bosh -d myDeployment errands", my errand doesn't appear in stdout.
It is not possible to trigger it. Maybe the
Thanks in advance and regards,
Initial proposal: https://github.com/cloudfoundry/bosh-notes/blob/master/uaa.md
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.