andreaskurth / memora-rs Goto Github PK

To cache partial builds in Memora, the get command needs to be able to optionally get the "nearest" entry instead of the entry for the current commit (which is not cached yet).

This would then enable memora get --nearest in CI scripts such as

if ! memora get compiler; then
  # no up-to-date build found
  if ! memora get --nearest compiler then
    # no nearest found, clean build
    make -C compiler all
  else
    # found nearest, do incremental build
    make -C compiler incremental
  fi
  memora insert compiler
fi

The "nearest commit" has to be defined more precisely (it might be especially tricky after merges?) -- but we might be able to say "the nearest commit is only defined if it is unique". If the nearest commit is not defined, memora get --nearest would fail.

Required objects are currently computed using git log, which stops at submodule boundaries.
Thus, for a repository that looks like

Memora.yml
some_dir/
submodule/
  bar/
  foo/

an artifact that depends on submodule/foo will miss in the cache also when submodule/bar changes. This leads to "false misses".

Instead, Memora should recognize submodule as submodule, descend into it, find the required commit for foo, and then find out which commit corresponds to this commit in the root repository. The last part might be the trickiest, because there is not necessarily a one-to-one correspondence between commits in the submodule and commits in the root repository.

Essentially the opposite of memora get <artifact>, a command such as memora rm <artifact> would locally (i.e., in the current working repository) remove the files listed under outputs for an artifact.

Currently, defining the list of outputs of an artifact in the Memora manifest is a manual procedure of the following steps (taking an install directory shared with other artifacts as example):

1. find install | sort > pre-some-artifact.txt
2. make artifact
3. find install | sort > post-some-artifact.txt
4. diff {pre,post}-some-artifact.txt
5. Go through diff, pruning irrelevant files and reducing some directories to a single entry (for the directory).
6. Replace the outputs of the artifact in the Memora manifest with the result of step 5.

Steps 1-4 would be easy to automate. I think we would keep steps 1 and 2 manual, then combine steps 3 and 4 into a Memora command that takes the output of step 1 as before state for (diff). (Sorting could be done internally to make it more robust.)

Automating step 5 would require sideloading some configuration. I could imagine adding another section to each artifact of a Memora manifest, where users can specify which outputs should be ignored and which paths should be treated as entire directory.

This could be implemented as a new Memora command:

memora update-outputs <artifact> <pre-artifact-file-list>

If the outputs of the artifact should be updated over an existing installation, it might be necessary to remove the outputs of the artifact before step 1. For this, memora rm <artifact> (#3) would be useful.