In-place Agent Upgrades about bosh-windows-stemcell-builder HOT 11 CLOSED

@sunjayBhatia a full clone is fine, thanks!

from bosh-windows-stemcell-builder.

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/143613051

The labels on this github issue will be updated when the story is started.

from bosh-windows-stemcell-builder.

@mxplusb Can you clarify why the stemcell creation process does not always start from the latest version of the "golden master" that has been updated? How come the snapshot image you are starting from already had the stemcell build process applied to it? Stemcells are generally supposed to be as minimal and reproducible as possible, it is a little unorthodox to start with some leftover state of a previous stemcell build.

For example, in our pipelines, we start from a base image and just install Windows updates which is our version of the "golden master". We then take that updated image and apply the whole stemcell build process (agent install, firewall configuration, lgpo, etc.).

Since we will already have issues with users starting from "golden master" states that we cannot look at and we cannot know for sure what all has been done to these images, we would recommend starting from a clean as possible state to prevent as much difference between our "reference" stemcells and theirs.

from bosh-windows-stemcell-builder.

from bosh-windows-stemcell-builder.

@mxplusb To clarify, we don't start from a scratch image every time in our pipeline, we have an image that we keep updating and snapshot after each round of updates. We then use this latest image to install the agent on and turn into a stemcell. If you're able to use the same pattern, that would be optimal.

Can we close this out given this info? Thanks for your feedback as well!

from bosh-windows-stemcell-builder.

Closing this issue as it seems to be solved.

from bosh-windows-stemcell-builder.

@sunjayBhatia this definitely isn't solved at all, not sure why you closed it. You have to give people time to respond. I was on-site with a Cloud Foundry customer and there were other, more important priorities that didn't leave me enough time to respond in 1 business day.

To follow through, we can't really use the same pattern. The golden image is an always-on server so it can maintain parity with patches and security updates from their internal systems. So installing the agent in-place is a requirement, especially as the process will be automated. If the solution is to just blow away all previous Bosh Agent configs, that's acceptable, but some documentation would need to be provided around where the configs so they can be purged.

/cc @eamonryan

from bosh-windows-stemcell-builder.

So @sunjayBhatia if I have this right - you snapshot after each round of updates, then install the BOSH agent and then you export the image to OVA and revert the snapshot on the original VM afterward to a state with no BOSH agent installed?

I could see a few issues with this:

1. You will continue to add more and more snapshots which is known to eventually cause problems in vSphere once the chain gets too long, unless you periodically commit them to disk, this adds overhead on the IaaS side and pretty IaaS specific overhead

2. If there is a plan to eventually support OpenStack the process will be different again as a result, since if there was just an agent reinstall process it could apply to all IaaS platforms

3. Snapshots, while a feature that has been around for a long time in vSphere, are not perfect and historically cause issues with broken snapshot chains, missing headers or unexpected disk expansion since each new delta disk can grow as large as the original base disk if some process executes that changes the bits so there is possibility for other failure here

All of this could be avoided and simplified at the Guest OS level with either an uninstall or reinstall script for the BOSH agent and any other components our scripts insert.

from bosh-windows-stemcell-builder.

@mxplusb @eamonryan

To follow through, we can't really use the same pattern. The golden image is an always-on server so it can maintain parity with patches and security updates from their internal systems. So installing the agent in-place is a requirement, especially as the process will be automated. If the solution is to just blow away all previous Bosh Agent configs, that's acceptable, but some documentation would need to be provided around where the configs so they can be purged.

Using an "always on" image is totally ok as long as it does not have the agent installed. The pattern would be to have an always on server that gets the patches as you specified, then clone it and go through the stemcell creation process. Windows updates often require restarts and the BOSH agent is a service that starts the next time the VM it starts on is turned on. You will have your Event Log and BOSH logs polluted by error messages.

if I have this right - you snapshot after each round of updates, then install the BOSH agent and then you export the image to OVA and revert the snapshot on the original VM afterward to a state with no BOSH agent installed?

1. You will continue to add more and more snapshots which is known to eventually cause problems in vSphere once the chain gets too long, unless you periodically commit them to disk, this adds overhead on the IaaS side and pretty IaaS specific overhead

We do not "snapshot" we full clone the source VM (apologies for using the wrong term earlier) after updates. There is no snapshotting and reverting of the VM we update, we just clone the updated VM and install the BOSH Agent on it. That becomes the stemcell.

We don't support reinstalling the agent and it's various dependencies, for fear that things like LGPO may lead to conflicts. We do not test doing an agent reinstall and make no guarantees it will work with edge cases like this. It sort of goes against the philosophy BOSH has regarding stemcells, they are a minimal OS image with a small amount of software and hardening added.

from bosh-windows-stemcell-builder.

@sunjayBhatia a full clone could be okay, let me follow up on Monday with the customer, and I'll try to get back to you early next week. 😄

from bosh-windows-stemcell-builder.

@mxplusb any word on this?

from bosh-windows-stemcell-builder.