Hangs at `copying closure...` about nixops HOT 13 OPEN

Thanks for the workaround! A slightly less reproducable but easier to remember nix-shell command:
$ nix-shell -p nixVersions.nix_2_14

from nixops.

@lelit you are a life-saver.

$ nix shell github:nixos/nixpkgs/8ad5e8132c5dcf977e308e7bf5517cc6cc0bf7d8#nix -c nixops deploy

working like a charm. Is this a documented issue somewhere else? If not I'll start a bisect -- I'm not even sure if the issue would belong here, at nixpkgs/nix, or elsewhere.

from nixops.

NixOS/nix#8329 was fixed, but only for cases where the master could be found without NIX_SSHOPTS. The remaining issue is NixOS/nix#8480, and my proposed fix is in NixOS/nix#8506.

from nixops.

I had the same problem and after some investigation I determined that it was due to using nix 2.15, so the quickest workaround has been something like the following

let
  # As of 2023-04-19, nix 2.15 breaks nixops: it becomes stalled at "copying closure" step
  # Use latest version known to be working, 2.14.1
  currentMasterPkgs = import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/8ad5e8132c5dcf977e308e7bf5517cc6cc0bf7d8.tar.gz";
  }) {};
  nix214 = currentMasterPkgs.nixVersions.nix_2_14;
in mkShell {
  buildInputs = [
    nix214
    pkgs.nixops
  ];
}

from nixops.

I looked around but did not find anything. I identified latest nix as the culprit by sheer luck, knowing that the previous deploy of a week before worked as usual, and the only difference on my work machine has been a monthly update of its NixOS.

from nixops.

So glad to have finally found this - is there an issue opened against NixOs/nix for this?

from nixops.

I tried to make a script to bisect nixpkgs but couldn't find the culprit commit. Maybe I'm bisecting the wrong codebase.

from nixops.

Just investigating this a little...

nixops is running

export NIX_SSHOPTS="-o Port=22 -o StrictHostKeyChecking=accept-new -i /tmp/nixops-tmpnsxeilim/id_nixops-*** -o ControlPath=/tmp/nixops-ssh-tmplqvmzvpi/master-socket"
nix-copy-closure --to *** /nix/store/5d1604l152flbxdzashz59165bqqc9hr-nixos-system-*** --use-substitutes

The socket doesn't actually exist. It doesn't hang without the -o ControlPath. I'll bisect nix, but it looks like it might be partially the fault of nixops.

from nixops.

NixOS/nix@5291a82

Is the first bad commit on nix.

from nixops.

That's what I was finding: if you run the command without -oControlPath, it works great. With, it hangs, and because nix-copy-closure does one request to figure out what to copy and then starts doing copies, I think it's guaranteed to trigger the bug here.

from nixops.

NixOS/nix@5291a82

Is the first bad commit on nix.

That checks out with NixOS/nix#8329. I did find that downgrading my NixOps shell.nix to Nix 2.13 fixed the issue.

from nixops.