Fail on boot due to `zfs` mount point being non-empty. about disko HOT 7 CLOSED

Yes. You can explore the result like this:

╭─[~/git/infra]─[ci]
nix-shell ╰─ % nix repl
Welcome to Nix 2.12.0. Type :? for help.

nix-repl> :lf github:nix-community/infra
Added 19 variables.

nix-repl> outputs.nixosConfigurations."build04.nix-community.org"
{ _module = { ... }; config = { ... }; extendModules = «lambda @ /nix/store/pm0s0wqyzr0vjpfcs423ckmy0222vslb-source/lib/modules.nix:329:23»; extraArgs = { ... }; options = { ... }; pkgs = { ... }; type = { ... }; }

nix-repl> outputs.nixosConfigurations."build04.nix-community.org".config.fileSystems
{ "/" = { ... }; "/boot" = { ... }; "/home" = { ... }; }

nix-repl> outputs.nixosConfigurations."build04.nix-community.org".config.fileSystems."/home"
{ autoFormat = false; autoResize = false; depends = [ ... ]; device = "zroot/root/home"; encrypted = { ... }; formatOptions = ""; fsType = "zfs"; label = null; mountPoint = "/home"; neededForBoot = false; noCheck = false; options = [ ... ]; }

from disko.

I usually run zpool export -fa after I installed the system before rebooting, so the pool can be cleanly imported after the reboot. It would be nice if disko could do this for you. But I'm unsure how it could do that :D

please report if this fixes the problem, makes it worse or has no impact on it.

apart from that I cannot spot any misconfiguration.

from disko.

Thank you for the quick reply!

I usually run zpool export -fa after I installed the system before rebooting, so the pool can be cleanly imported after the reboot. It would be nice if disko could do this for you. But I'm unsure how it could do that :D

please report if this fixes the problem, makes it worse or has no impact on it.

I tried this solution without reinstalling on my current system. So I just booted to an installation medium, mounted the zfs drives with disko mount and ran zfs export -fa. After some unmounting issues this ran successfully. But no change in the bug I have described here. As I understand you probably suggest doing a whole clean reinstall so this was just a hail mary on my part as I do not have an understanding about zfs yet. Might do one later but do not have the time atm :/

However, I did find a crude (and potentially damaging later?) method to remove the non-empty error by adding

{
  systemd.services."zfs-mount".serviceConfig.ExecStartPre =
    "${pkgs.zfs}/sbin/zfs mount -O -a";
]

to my configuration. The -O i.e. overlay option is the new thing to the previous ExecStart command of the zfs-mount service. It allows mounting on directories that are non-empty (i.e. it is potentially very dangerous) as I understand it. I could not override the ExecStart command (i.e. zfs-mount".serviceConfig.ExecStart = lib.mkAfter "-O";) as there is some weirdness in how it is actually added to the service config. So now it just runs the mount commands twice, but because the ExecStartPre command succeeds in mounting the ExecStart part probably does nothing (?).

from disko.

Aaaaand an update: the bug has not disappeared, rather it appears sporadically/randomly. My guess, which probably is no big revelation, is that some systemd service generates the empty user directory and there is a race condition between the zfs-mount and that other service.

from disko.

Seems like this is better discussed in nixpkgs. Disko only generates the implementation but is not responsible for how things are mounted in the end.

from disko.

Seems like this is better discussed in nixpkgs. Disko only generates the implementation but is not responsible for how things are mounted in the end.

I will post an issue on nixpkgs. The configuration generated by disko is (mainly) in the fileSystems attribute of the final config as I understand? I suppose that is the relevant part for creating an issue in nixpkgs.

from disko.

Thank you, I opted to not using a dataset for the /home/ to still continue using zfs-mount automount functionality. Switching to
legacy seems to be the only way to avoid the race condition.

Can the automount functionality and using legacy mountpoints be mixed freely? I.e. using legacy for /home.

This issue can be considered "fixed" for me as I now know what to avoid what the expected errors might be. You can close it if
deemed suitable or keep it open for future reference. I suppose this is a quite a major caveat when using the automount
functionality of zfs.

Using a legacy mountpoint for the /home dataset seems to be the only solution to avoid this race condition. You can check the discussion at NixOS/nixpkgs#212762 where I pasted the above quote if you want to check the short discussion.

Beyond that this issue can be considered solved for me.

from disko.