Currently, the variable COREOS_VERSION must be set in at least two places, and possibly a third:

epoxy-images$ grep -r COREOS_VERSION= .
./cloudbuild.yaml:   - 'COREOS_VERSION=2079.6.0'
./.travis.yml:- COREOS_VERSION="2079.6.0"
./manual/manually_generate_images.sh:COREOS_VERSION="2079.6.0"

This means that every time we decided to update the CoreOS version, the person doing it has to remember to update it in three places, which is not ideal. We should figure out a way that will allow us to defined it in a single location.

Now that index2ip supports IPv6 m-lab/index2ip#10 we should configure the IPv6 host network.

• NOTE: this should be done after epoxy_client completes until the epoxy server has native support for IPv6.

https://github.com/m-lab/epoxy-images/blob/dev/configs/stage3_coreos/resources/generate_network_config.sh

Note, this change should also update to use the epoxy.ipv4= and epoxy.ipv6= kernel parameters.

Currently, epoxy-images deployment overwrites old versions with the new version. This makes rollback impossible without a rebuild. Instead, the deployment step should make two copies: 1) to latest, 2) to the current version.

Boot scripts should reference 'latest' and the explicit version directory would serve as a backup.

The stage1-template.ipxe script and however we generate stage1 images for legacy hardware, we should encode some known hardware information into the kargs so that they are available to epoxy during boot.

In particular, this could be helpful for updating the mlx rom. For example, if the kargs include:

epoxy.model=Mellanox epoxy.device=ConnectX3

Then the process that downloads and flashes the rom could construct a URL to locate the latest image, e.g.

https://storage.googleapis.com/epoxy-mlab-staging/roms/{{.kargs.epoxy.model}}/{{.kargs.epoxy.device}}/{{.kargs.epoxy.hostname}}.mrom

We recently discovered IPv6 was down for a good number of sites, and we were never aware of it. Most of the issues are likely legitimate IPv6 issues with the transit providers. However, for one site, nbo01, something else is going on. IPv6 works to the host machine, but not to containers. Interestingly, rebooting a machine will cause IPv6 to work for about 30m, then go down again. Further investigation revealed that the usual default route on nbo01 containers was getting removed, causing IPv6 to break. Digging into that fact a bit revealed that this is likely an issue with SLACC and RA (Router Advertisements). There is a kernel setting, net.ipv6.conf.<iface>.accept_ra, which instructs the network stack to either accept or ignore RAs. It turns out that this setting is set to 0 in the default network namespace, but to 1 in a container namespace. I don't know why this has thus far only affected nbo01, seemingly, but it is likely because most upstream providers have SLAAC disabled on the interface that connects the M-Lab equipment.

This unit is failing with errors like:

api-platform-cluster-us-west2-c reboot-api-node.sh[2973302]: {"level":"warn","ts":"2023-11-29T15:00:07.066Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0003c0700/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection closed before server preface received"}

api-platform-cluster-us-west2-c reboot-api-node.sh[2973302]: Error: context deadline exceeded

After some investigation, I found that etcdctl was not getting the proper configurations from its environment. The necessary ENV variables are set in /root/.bashrc, and the unit script sources /root/.profile, which itself should source /root/.bashrc. However, .bashrc's first line checks if the script is running interactively, and if not exits.

This used to work so something must have changed in etcd when we upgraded to a newer version of k8s. The underlying errors are TLS related. Maybe etcd because more strict about TLS in some way.

What is confusing here is the create-control-plane.sh script should be putting the necessary ENV variables into both /root/.profile and /root/.bashrc, yet for some reason the ENV variables are no present in .profile:

https://github.com/m-lab/epoxy-images/blob/main/configs/virtual_ubuntu/opt/mlab/bin/create-control-plane.sh#L292

I need to figure out why create-control-plane.sh is not getting the variables into .profile.

We have recently discovered that when configuring a new site the nodes do not have access to the Internet until the full switch configuration is applied. This might happen some time after remote hands have set up boot sequence and powered on the server, due to timezone differences / after-the-fact communication from remote hands / Ops team not immediately available.

Currently, the lack of Internet connection means that boot fails even if the connection becomes available later (e.g. when we apply the full switch configuration). Implementing some kind of retry logic would simplify the new site deployment significantly.

Tasks:

• #28

We currently apply both a basic DRAC configuration in stage1 and a full one during stage2. To apply these configurations we use ipmitool, which awaits for confirmation from DRAC after sending each command. For unknown reasons, on R640s some of the commands can take a long time to be confirmed and ipmitool times out.

Both the stage1 and the stage2 scripts should be modified to tolerate these transient failures and keep retrying each command for a few times (e.g. 10).

We have an automated way of updating stage1 images for the R630s booting from Mellanox NICs (mlx_update stage3 images), but we have no automated way of updating stage1 boot images on machines that boot from NICs. This needs to be addressed.

Today, updaterom.sh performs the logic of parsing /proc/cmdline, finding the epoxy.mrom prefix URL, discovering the device model, and building a full URL to download a new mrom image before burning it.

The URL discovery and construction logic should be executed by epoxy_client. And epoxy_client action should be able to specify a URL variable as:

{{kargs `epoxy.mrom_url`}}/{{.vars.device}}/{{.vars.hostname}}.mrom

Then download that image, and pass the local file to a modified updaterom.sh script that accepts a rom file as parameter.

With this support updaterom.sh can be replaced with flashrom.sh.

The debootstrap utility can only source files from a single apt repository. In our case, when generating machine images we point it to the point release repository. What this means is that the images built by epoxy-images will only ever contain the packages at the versions they had at the time of the point release. For LTS releases of Ubuntu, this can mean we are running very old versions of packages, with known bugs which have been resolved in updates versions.

The OS is configured to automatically install security updates, but is not configured to install regular package updates. This is how we want it, but we need a mechanism to get updated packages from our base OS release onto the platform.

• To set up pacing correctly on the host
• Seems like downloading from siteinfo is the right approach
• May not be relevant until PAUSE is disabled

Tasks:

• #22

Tasks:

• #23

For a long time we have talked about the possibility of regularly rebooting the entire fleet. In the past year we launched a new reboot management system called Kured. Kured runs as a pod on all machines (a DaemonSet), and watches for the existence of a configurable "sentinel" file. When that file exists, the service will queue the machine for a reboot.

Because Kured operates at the local filesystem level, this opens the possibility that we could write a simple systemd service (with associated timer unit) which could regularly (daily?) compare the uptime of the machine to some configurable maximum value (60d?), and if the uptime is greater than that value to write out the sentinel file, triggering a reboot.

Consideration: automatically rebooting a node will automatically pull in any new changes in the underlying ePoxy stage3 boot images. For example, if we are in the process of updating to a new kubernetes version and merge those changes to the master branch, or tag the repository, before the API cluster is updated, then nodes will fail to join the cluster. This is just the first example that occurs to me. It means that we will need to be somewhat deliberate about the changes that get merged into master of this repo, and especially considerate of tagging the repo, knowing that those changes will get automatically applied to all nodes somewhere between 0 and, say, 60d.

In a discussion with @stephen-soltesz today, we realized that deployment of stage3 images into production is currently condensed into a single, non-versioned step, but should probably be two.

Currently, when the epoxy-images repository is tagged, new stage1 and stage3 images are built and published to static, non-versioned locations in GCS. This means that the URL for images never changes, while the underlying file that backs the URL does. This can lead to confusion about what is actually being deployed and booting at any given time.

We should probably implement versioned stage3 image URLs, such that tagging epoxy-images builds versioned images (and URLs), and then a second step would be required to cause them to be deployed. That second step would be to somehow modify the GCD entity for every node to point to the new versioned URL.

One of the root causes of the pusher/TSO outage was due to epoxy-images creating new boot images that included later versions of k8s than the k8s master supported.

We should enforce that epoxy-images are only updated after a k8s master version is updated to support the client.

Currently, Ubuntu platform nodes are not automatically updated in any way, unless the stage3 image is recreated with updated packages. We should be automatically applying security fixes.

This PR dovetails with moving to automatic node reboots with Kured, since Kured, by default, looks for the Ubuntu flag that an update requires a reboot.

With modifications to install them in /usr/bin instead of /opt/bin

Today, on the legacy PLC platform, if a node fails to boot fully to PLC for some reason the node will fall-back to "SafeBoot" mode, allowing an operator or PLC admin to login to inspect the issue. However, our ePoxy stage1 images don't have this facility, though it does appear that it occurred to @stephen-soltesz when developing ePoxy.

One use case for this feature is if ePoxy boot fails for any reason, an operator can fix the issue with ePoxy, login to the machine and reboot it. Generally the reboot could be done via the DRAC, but at new sites the DRAC may not yet have been configured. This is probably another good use case for configuring DRAC basics during stage1 boot.

Now that ROMs can target epoxy server in different GCP projects, this variable should be accessible during runtime so epoxy_client can automatically fetch images from the correct project buckets.

In particular, the later stage actions should refer to URLs that reference the GCP project.

"files" : {
     "vmlinuz" : {
        "url" : "https://storage.googleapis.com/epoxy-{{get_karg `epoxy.project`}}/stage3_coreos/coreos_production_pxe.vmlinuz"
     },
     "initram" : {
        "url" : "https://storage.googleapis.com/epoxy-{{get_karg `epoxy.project`}}/stage3_coreos/coreos_custom_pxe_image.cpio.gz"
     }
 },

Tasks:

• #25

We are currently setting the TCP pacing maxrate to the max rate of the uplink. This may be fine, but once ethernet flow control (PAUSE frames) is turned off on all ports, we'll want to keep a close eye on discards at sites. It's possible that we may want to reduce the maxrate for pacing to something less than the uplink's capacity if we see discards.

Right now epoxy-images build everything on commits and tags.

Instead, we want to have differential builds for certain changes that can occur much faster.

• build only generic images (e.g. stage3*)
• build only stage1 images (e.g. roms, isos, usbs, etc)
• build only new stage1 images (it's redundant to rebuild images that haven't changed)

Tasks:

• #30

Recently a rollout of the host DS stalled for multiple days b/c several machines had 2 containers hung during shutdown. Currently, fix-hung-shim.sh expects only a single shim process to hang. It needs to detect multiple hung shim processes.

For example:

$ docker ps | grep host
06ee5a8d016e        e4c310f63d8a    "/traceroute-caller …"   13 days ago         Up 13 days                              k8s_traceroute_host-cpzdj_default_dd0c6aab-3dbb-44aa-8ea1-3dd6fd2bf299_0
e38f71d2de57        c478ac8aa02b    "/bin/tcp-info -prom…"   13 days ago         Up 13 days                              k8s_tcpinfo_host-cpzdj_default_dd0c6aab-3dbb-44aa-8ea1-3dd6fd2bf299_0

Up to now all ROM update processes have relied on downloading a suitable ROM image over the network. In some cases, this may not be possible, such as if the NIC is non-operational due to a bad firmware update.

To support cases like this we need an all-in-one ROM update image that bundles the suitable ROMs in the ISO.

https://coreos.com/ignition/docs/latest/

On multiple occassions now, the epoxy_client receives a context deadline exceeded error during download. When it happens, this interrupts the download and causes the client to fail.

This should really almost never happen, or only after many failures.

This file needs to be modified to work with v2 names:

https://github.com/m-lab/epoxy-images/blob/master/configs/stage3_coreos/resources/cloud-config.yml#L225

NOTE: This is already fixed for Ubuntu.

Stage2 uses epoxy-client to boot the third stage.

The stage3 coreos and stage3 mlxupdate images should also include the epoxy client so that they can execute any final nextboot commands and report success (or failure).

https://github.com/m-lab/epoxy-images/blob/master/Dockerfile#L17

It should use a pinned version.

The latest linux-source versions 4.4.0-109.132 and 4.4.0-108.131 do not build.

The cause appears to be fixed according to the ChangeLog in 4.4.111

However, that version is not yet available in xenial packages. So, temporarily, we will pin the linux-source-4.4.0 version to 4.4.0-104.127

Currently, the setup_stage1.sh script uses mlabconfig.py to get a list of nodes and their IP addresses. mlabconfig.py uses the old PLC-based file sites.py, and we want to move away from using sites.py and to start using siteinfo, or some API which is backed by siteinfo.

At least this one location will need updating, and perhaps more (?):

https://github.com/m-lab/epoxy-images/blob/master/setup_stage1_isos.sh#L22

R640's should be configured to use the "performance" CPU frequency governor rather than the default "powersave".

After manually setting the "performance" governor on mlab3-hkg01 using:

sudo apt-get install cpufrequtils
for i in `seq 0 31` ; do sudo cpufreq-set --cpu $i --governor performance ; done

Shortly after the very high 95th% polling interval from tcpinfo became much closer to the median.

And, the overall pod CPU usage and load dropped, without an apparent increase in CPU temperature.

Currently DRAC configuration is done manually after the nodes are up and running CoreOS. However, if anything goes wrong during the boot process we have no way of accessing the node and debugging the issue, unless we involve remote hands.

We should configure DRACs as soon as possible, which probably means during stage1.