explain cache misses about nix HOT 6 CLOSED

@KiaraGrouwstra what you write indicates to me that the mechanism you desire may indeed be more suitable to be implemented on top of Nix rather than as part of it, for example as part of a (still hypothetical) Nixpkgs CLI.

I have a hard time imagining how caching derivations would help, because all that gives you are more opaque store paths. What we'd really need for actionable suggestions is information about Nixpkgs recipe parameters, and we simply don't have them at the Nix layer. The derivation's name attribute is a clumsy proxy for actual metadata.

Another thing that could get us closer to your vision may be a generic cache mechanism where one adds an attribute set to the argument to derivation, which is serialised (only) into the cache. Then consumers could figure out what to do with that. But the implications of this would need to be considered properly.

from nix.

Triaged in Nix team meeting:

• @edolstra: there is no way we can implement that, because the evaluation model does not allow that

  maybe package foo with this version x of dependency bar is not in the cache, but using version y it might have been.

  Nix cannot answer that question. It doesn't know what's in the cache, and cannot map that back to arguments or attributes Nixpkgs

• @fricklerhandwerk if anything, this could only (in principle) addressed at the Nixpkgs layer, although even then it's not clear how exactly that would work without combinatorial explosion

• decision: closing it since we don't know how to implement it. feel free to re-open with a concrete proposal.

from nix.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-05-15-nix-team-meeting-minutes-146/45491/1

from nix.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/overlaying-packages-using-cuda-to-use-zluda/45374/5

from nix.

i thought about this a bit more. @fricklerhandwerk

i think caches currently not having the info needed for this means we could address this for the future by allowing caches to start storing not only a hash's build results, but also the (instantiated) derivation that produced that result, as we do in /nix/store.
while this could yield a trade-off with privacy (-> make this opt-in), i think that should take care of the info needed to answer such queries for future cache pushes. (having this info in the cache could further bring verifiability to build results.)

i think the concern around combinatorial explosion could be alleviated by:

• making this opt-in to the cache
• making this opt-in to the pushing user
• potentially limiting this further from derivations to packages specifically (i.e. has pname) which i think is the 20/80
• the ability to then group/filter cache hits by e.g. such a pname if not optionally also version string/rev - while a limitation, potentially again a 20/80
• requiring a manual action on the part of the user to request a miss explanation for a given derivation (/package)
• the number of cached versions of a package in practice already being limited by the cache actually having to store all their cached results
• potentially offloading work regarding the explanation to a client running at the requesting user, limiting the cache's work on such a request to sending instantiated derivations with cached build results for a given pname (+ version)
• summarizing likely reasons by presenting the user with a ranking of tallied key paths that differ between the requested vs. cached derivations. i.e., 3 cached derivations used a different value for foo.bar, 2 additionally had a value for key baz that is missing in the requested derivation. if the user picks such an acceptable difference, they could then be presented with top similar cached derivations (with similarity measured in e.g. number of key paths with differing values).

i hope that i properly understood the raised concern, and that this might go some way toward addressing it.

from nix.

@fricklerhandwerk thanks for your response.
on the derivations, i think you're right my terminology was off - i guess i intended to say the cache should know the evaluated (json?) input that would bring about the cached build result.

i agree a generic caching mechanism could help facilitate this, given a nix cli (i think this isn't specific to nixpkgs) and cache implementing it.

from nix.