Mender delta updates fail to work due to systemd-machine-id-setup service about tegra-demo-distro HOT 3 CLOSED

The way I've dealt with this in my projects is to use systemd's volatile root feature, setting systemd.volatile=overlay on the kernel command line, documented here, then handling things like host name and timezone settings with additional programs/scripts to stash the settings in a persistent location elsewhere and restore them at boot time.

Getting this to work right with Yocto builds involves a bit of additional work, essentially turning off the volatile-binds stuff (/var/volatile is unnecessary in this setup) and adjusting files/fs-perms.txt. I have a subset of the changes in my test distro.

There are some issues with overlayfs support in the 4.9 kernel which can cause some issues, so I also wound up importing the back-port of the 4.19 overlayfs support to 4.9 that one of the overlayfs developers maintains to address those.

from tegra-demo-distro.

Thanks @vinotion!

This relates to #198 as well as OE4T/meta-mender-community#8 (comment) and OE4T/meta-tegra#527. It's also related to systemd/systemd#14131 Cross referencing to setup links to these.

At the beginning of the boot process, systemd-machine-id-setup will change the machine ID in /etc/machine-id.

The change at https://github.com/OE4T/meta-tegra/pull/527/files#diff-6952bcad754469ed729bf94101a17d36ff760011bb19601d5e3b0f50d74a546f (see https://github.com/OE4T/meta-tegra/pull/527/files#diff-6952bcad754469ed729bf94101a17d36ff760011bb19601d5e3b0f50d74a546f for instance) puts the systemd.machine_id on the command line. See this example from my booting xavier-nx-devkit-emmc image running kirkstone at f8b9c37 with source setup-env --machine jetson-xavier-nx-devkit-emmc --distro tegrademo-mender and bitbake demo-image-base

root@jetson-xavier-nx-devkit-emmc:~# cat /proc/cmdline
console=ttyTCU0,115200 console=tty0 fbcon=map:0 video=tegrafb earlycon=tegra_comb_uart,mmio32,0x0c168000 gpt rootfs.slot_suffix= usbcore.old_scheme_first=1 tegraid=19.1.2.0.0 maxcpus=6 boot.slot_suffix=_b boot.ratchetvalues=0.4.2 vpr_resize sdhci_tegra.en_boot_part_access=1 systemd.machine_id=4e2dca41797e4f92af897d88f06d9409

However, systemd still writes /etc/machine-id at boot, even when specified on the command line. This detail is not included in https://www.freedesktop.org/software/systemd/man/machine-id.html, however you can see this logic at https://github.com/systemd/systemd/blob/b33c2757d84d4f14f6c31da1c79dc343c43682e2/src/shared/machine-id-setup.c#L135-L161 and the file is populated in etc with the same content.

root@jetson-xavier-nx-devkit-emmc:~# cat /etc/machine-id
4e2dca41797e4f92af897d88f06d9409

This thread provides some systemd philosophy regarding read only rootfs: https://lists.freedesktop.org/archives/systemd-devel/2021-February/046149.html. The summary is they expect /etc to be writeable and don't appear to be contemplating delta updates, at least in that thread. The thread at systemd/systemd#14131 is also interesting, and also doesn't appear to contemplate delta updates.

The part I don't understand yet is why more platforms using mender delta update don't have this problem and where we are different in this regard. My guess is that the comment at #198 (reply in thread) is related and there's something different about cboot builds which means we don't get the ro kernel argument with IMAGE_FEATURES including read-only-rootfs, probably relates to https://github.com/OE4T/tegra-demo-distro/blob/402f67f3e9ad7bd4c92f4688d75e1a0fbe97224a/layers/meta-tegrademo/classes/rootfs-postcommands-overrides.bbclass in a way I don't understand yet.

from tegra-demo-distro.

Ugh, apologies for not getting back to this! The suggested links to discussions and solutions are great. Thanks @dwalkes @madisongh 👍🏻

We ended up adding some systemd scripts to deal with things like setting the host name from the mounted overlay settings etc. Not the prettiest solution, but simple and very reliable. Perhaps in a future incarnation of our Yocto project we will investigate a more thorough approach.

from tegra-demo-distro.