andreaskurth / memora-rs Goto Github PK
View Code? Open in Web Editor NEWMemora: Build Artifact Cache for Git Repositories
License: Apache License 2.0
Memora: Build Artifact Cache for Git Repositories
License: Apache License 2.0
Currently, defining the list of outputs of an artifact in the Memora manifest is a manual procedure of the following steps (taking an install
directory shared with other artifacts as example):
find install | sort > pre-some-artifact.txt
make artifact
find install | sort > post-some-artifact.txt
diff {pre,post}-some-artifact.txt
outputs
of the artifact in the Memora manifest with the result of step 5.Steps 1-4 would be easy to automate. I think we would keep steps 1 and 2 manual, then combine steps 3 and 4 into a Memora command that takes the output of step 1 as before state for (diff
). (Sorting could be done internally to make it more robust.)
Automating step 5 would require sideloading some configuration. I could imagine adding another section to each artifact of a Memora manifest, where users can specify which outputs should be ignored and which paths should be treated as entire directory.
This could be implemented as a new Memora command:
memora update-outputs <artifact> <pre-artifact-file-list>
If the outputs of the artifact should be updated over an existing installation, it might be necessary to remove the outputs of the artifact before step 1. For this, memora rm <artifact>
(#3) would be useful.
To cache partial builds in Memora, the get
command needs to be able to optionally get the "nearest" entry instead of the entry for the current commit (which is not cached yet).
This would then enable memora get --nearest
in CI scripts such as
if ! memora get compiler; then
# no up-to-date build found
if ! memora get --nearest compiler then
# no nearest found, clean build
make -C compiler all
else
# found nearest, do incremental build
make -C compiler incremental
fi
memora insert compiler
fi
The "nearest commit" has to be defined more precisely (it might be especially tricky after merges?) -- but we might be able to say "the nearest commit is only defined if it is unique". If the nearest commit is not defined, memora get --nearest
would fail.
Required objects are currently computed using git log
, which stops at submodule boundaries.
Thus, for a repository that looks like
Memora.yml
some_dir/
submodule/
bar/
foo/
an artifact that depends on submodule/foo
will miss in the cache also when submodule/bar
changes. This leads to "false misses".
Instead, Memora should recognize submodule
as submodule, descend into it, find the required commit for foo
, and then find out which commit corresponds to this commit in the root repository. The last part might be the trickiest, because there is not necessarily a one-to-one correspondence between commits in the submodule and commits in the root repository.
Essentially the opposite of memora get <artifact>
, a command such as memora rm <artifact>
would locally (i.e., in the current working repository) remove the files listed under outputs
for an artifact.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.