Requesting clarification in documentation about colcon-cache HOT 2 OPEN

Hey @zflat , neat to hear your use of this extension to speed up CI. If you like to see a real world example using this extension, we've been using it rather heavily for the Nav2 project for over a year or so:

• ros-planning/navigation2#2343

Can you please explain the purpose of the --packages-select-cache-key? For example, where would I look to see what the default key is and why would I need to override it?

The purpose of the --packages-select-cache-key is to override the default cache key selected by the verb handler. For example, the verb handler for the "colcon list", it uses the same verb handler for "colcon cache", which defines what lockfiles are to be used for comparison or reference. When used with package selection, this can be helpful when listing what caches are still valid for other colcon verbs (e.g. not just for "colcon list").

• colcon-cache/setup.cfg

  Line 94 in 0a23776

  list = colcon_cache.verb_handler.cache:CacheVerbHandler

• colcon-cache/colcon_cache/package_selection/valid.py

  Lines 56 to 58 in 0a23776

  	verb_name = args.packages_select_cache_key
  	if not verb_name:
  	verb_name = args.verb_name

• colcon-cache/colcon_cache/package_selection/valid.py

  Lines 82 to 85 in 0a23776

  	verb_lockfile = verb_handler_extension\
  	.get_current_lockfile(package_build_base)
  	reference_lockfile = verb_handler_extension\
  	.get_reference_lockfile(package_build_base)

In Nav2 for example, we do this to list packages with invalidated caches for specifically the "colcon build" verb, so that we can determine the package builds that must be subsequently cleaned:

https://github.com/ros-planning/navigation2/blob/5547a20ae711417af90768d9d594a4524064e1ca/.circleci/config.yml#L163-L169

Also, the lock workflow is not intuitive to me.

The level of abstraction with caching results for colcon verbs, verbs that transitively depend on other verbs, makes this difficult to describe. The hashing of source files (or query to detected revision control) is only invoked with the "colcon cache lock" subverb. I suppose this could have been abbreviated to the root verb "colcon cache", but I suspected I may want to add more extension subverbs in the future, so I pushed that hashing funatilty into a "lock" subverb, as "colcon cache cache" seemed too peculiar.

The updating of lockfiles is currently simply a copy operation upon a successful task for the package, where the invoked colcon verb triggers an event handler that copies the lockfile from the verb it transitively depends upon. E.g. the build verb depends upon the cache verb, and the test verb depends upon the build verb. This pattern could be extended for any other verbs via more registered event handlers.

I think that a more detailed explanation (maybe annotating the example) of what creates and or modifies which lock files and what checks against the lock files are performed would be helpful.

I can add a "How does it work" section to the readme, but I'd first like to finish refactoring where the lockfiles are written and read from in the colcon workspace, as to store the lockfiles closer to the artifacts they claim to represent:

• #42

from colcon-cache.

Thanks @ruffsl for the explanation! The example of using a different --packages-select-cache-key makes it more clear to me now.

It is helpful to know that the build verb lock file is just copied from the cache verb. And I think that a "how it works" section would be very helpful.

from colcon-cache.