Garbage collection for cache directory about wireit HOT 13 OPEN

Hi @aomarks - this issue piqued my curiosity. I am wondering what the current status is? I'm interested in giving it a shot, but it has been sitting for awhile so I figured I would check before digging in.

from wireit.

Hi @joshkraft! Contributions would be more than welcome.

Lately I've been wondering if local caching is something we really want in general. One thing it's nice for is when switching back and forth between git branches, or undos. But I'm not sure the complexity and additional latency of maintaining the local cache is worthwhile. What do you think? Have you been relying on local caching? Did you find that the cache folder was getting too big?

from wireit.

That is a great question. I actually just started using wireit, so unfortunately I have no real opinion on the utility of the local caching.

If the goal is keeping additional complexity/latency low, one interesting approach I noticed in another project is to inject some randomness into the process. Rather than keeping track of every get/put to the cache, each execution calls a random() function that returns true 1%/5%/etc. of the time. If it returns true, kick off a 'cache cleaning' process.

Cache cleaning could be done in a number of ways:

• delete random cache entries until we are beneath the desired limit (could also consider the 2-random approach)
• delete oldest cache entries (based on stats.mtimeMs) until we are beneath the desired limit
• delete all cache entries that are older than some time (could be user defined)

I am curious if you think this approach would be a good fit here? I'm happy to do some experimentation and see what it would look like in this project.

from wireit.

Hi @aomarks,

I've had a chance to start using Wireit personally and I have found local caching to be useful, as someone who is a serial branch switcher.

I did some POC work implementing a few different eviction mechanisms. My guess is that the additional latency introduced here would largely be driven by the time it takes to delete evicted entries from the cache. I'm curious if you have suggestions on some parameters I can use to benchmark the different approaches? For example:

1. What would be a reasonable cache limit to default to? 100, 1000?

2. What might a 'practical worst-case scenario' output folder look like? Interested in size, structure, etc.

Also, feel free to let me know if adding this functionality doesn't align with the project's goals. I'm happy to investigate some options and report back with benchmarking results, but if the value added here isn't worth the added complexity I'm happy to direct my effort towards another issue.

from wireit.

How is the mangement of the default cache and the cache that GitHub Actions might you different currently? In particular, I'm leveraging CircleCI, for which I actually turn off the default CI cache mechanism, because I couldn't get it to actually build/save a cache in that context, however that cache does grow indefintiely. Is there something about the GH Actions cache approach that could be leveraged in "local" mode? If not, then raising the priority of this issue would be really nice to see and support expanded consumption of Wireit across projects I work with at Adobe.

from wireit.

Hi @aomarks || @rictic - I'm thinking about doing some work on this. Do you think this would still be a useful addition to wireit, or not worth the added complexity?

from wireit.

I do think this is helpful. for cache size, one idea would be for the size target to be a multiple of the size of the live entries in the cache. something like 3x the size, to comfortably fit a few branches (likely more than 3, because they'll share many cache entries)

WDYT about a design for this @aomarks? I'm thinking a wireit vacuum command that would analyze the current package and each referenced package and do GC over each of them in parallel

from wireit.

Hi @aomarks || @rictic - I'm thinking about doing some work on this. Do you think this would still be a useful addition to wireit, or not worth the added complexity?

Yeah, I do think that we should either implement garbage collection, or get rid of local caching.

I am a little unsure if local caching is worthwhile, so I'd be interested to hear @joshkraft and @rictic 's thoughts on that. Remote caching for CI is clearly useful, and it does its own GC. Local caching feels a bit less useful, and it comes with a cost (latency & storage).

WDYT about a design for this @aomarks? I'm thinking a wireit vacuum command that would analyze the current package and each referenced package and do GC over each of them in parallel

That sounds cool. Are you thinking that it's up the user to run the vacuum command? Would we also run it automatically? Or maybe prompt the user to run it sometimes?

from wireit.

@aomarks in a large monorepo like lit I get a lot of mileage out of local caching. Being able to run npm test at the top level and it only runs tests that potentially changed since my last run saves a good bit of time

from wireit.

Are you thinking that it's up the user to run the vacuum command? Would we also run it automatically? Or maybe prompt the user to run it sometimes?

Yeah, start manual, and then as we gather experience and performance info we could run it automatically sometimes (maybe randomly, every Nth run?)

from wireit.

+1, I also enjoy having the local caching when working in beefy monorepos. Starting with the manual approach seems like a good idea before weaving GC deeper into the workflow.

On the UX side, how would users invoke this command manually? Are we thinking something like npx wireit vacuum, or is there a more elegant approach?

from wireit.

On the UX side, how would users invoke this command manually? Are we thinking something like npx wireit vacuum, or is there a more elegant approach?

Maybe it could be similar to --watch? E.g. npm run foo --vacuum would mean "vacuum the cache for foo and all of its transitive dependencies" (would it also mean run it? I don't know, that's maybe why it could be confusing).

npx wireit vacuum could be good too, I suppose that would mean "vacuum the cache for everything in this package"? But would it also apply to all transitive dependencies?

from wireit.

In the short term, I was personally thinking that a simple solution would be to just have the vacuum/eviction command work at the project level, cleaning up all caches present in the .wireit folders. Perhaps delete all entries that are older than N days?

As far as ongoing cache management, I'm intrigued by the idea of basing size on a multiple of live entries. I'm also wondering if having some sort of user-defined TTL (w/ a sane default) would effective and perhaps simpler... in my use of wireit, I've found that older cache entries are rarely used, though 'older' is relative so it might be nice to have a configurable value.

from wireit.