It would be nice if on top of the package selection option (e.g packages-up-to), we could specify which dependency types to consider. Example: I do not want to build the packages with tag <test_depend> when building for production

Title says it.

$ colcon build --symlink-install --packages-select-regex robot
[0.152s] ERROR:colcon.colcon_core.package_selection:Exception in package selection extension 'select_skip': module 're' has no attribute 'Pattern'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/colcon_core/package_selection/__init__.py", line 198, in _check_package_selection_parameters
    retval = extension.check_parameters(args=args, pkg_names=pkg_names)
  File "/usr/lib/python3/dist-packages/colcon_package_selection/package_selection/select_skip.py", line 58, in check_parameters
    if isinstance(pattern, re.Pattern):
AttributeError: module 're' has no attribute 'Pattern'
                     
Summary: 0 packages finished [0.08s]

Tried on Kubuntu 18.04.5 LTS with ROS2 Dashing from Debian, apt upgraded everything to latest.
Python 3.6.9 (default, Jul 17 2020, 12:50:27)
colcon-package-selection 0.2.9

Issue description

When I call...
$ colcon build --packages-select <my_package>/
I find that I get a warning saying that the package is not recognised and the package is not built. When I remove the trailing slash then all is good and the package builds. Please note that this is unexpected since linux tab-completion will add a trailing slash for a directory.

Steps to reproduce the issue

1. call $ colcon build --packages-select <my_package>/
2. call $ colcon build --packages-select <my_package>

What's the expected result?

Both calls should be legal and both build the package.

What's the actual result?

Call (1) should fail. Call (2) should build

Additional details / screenshot

Note that these calls were made on a jetson-tx2 running balena os with a ROS2 eloquent installation.

Here is some of the output from python3 -m pip freeze...

When making changes to a package that has a lot of dependent packages, it would be great to run builds/tests on all packages that depend on it. However, you still need to build all the packages up to that target package.

If I run the following, the result is the intersection of the two (rclcpp).

colcon build --packages-up-to rclcpp --packages-above rclcpp

In particular for CI purposes, it would be great for there to be an option that is the union of the two. Basically the equivalent of running following sequentially but as a single command:

colcon build --packages-up-to rclcpp
colcon build --packages-above rclcpp

--packages-below-depth DEPTH [PKG_NAME …]
Select the packages with the passed names as well as their recursive dependencies down to the given depth.

It could also be called --packages-up-to-depth
Did I miss another obvious way to achieve this? Thanks!

As discussed in colcon/colcon-core#213 , I think it would be useful to add a selection argument to filter packages by those were it's source code has changed with respect to what was used in the previous workspace build. This could be achieved by augmenting the verbs to audit the source folder of a package when specified, and later comparing the source tree with the previous audit file when determining selecting conditions. It could also be version control aware to ignore specific files/folders.

An example workflow could be as follows:

# setup workspace
mkdir -p ~/ws/src && cd ~/ws
vcs import src < ros2.repos

# build workspace with audit
colcon build --audit

# make source changes 
touch src/foo/foo.txt

# gather list of changed packages
PKGS_CHANGED=$(colcon list --packages-skip-source-unchanged | xarg)

# rebuild only changed packages and above
colcon clean --build --install --packages-above $PKGS_CHANGED
colcon build --audit --packages-above $PKGS_CHANGED --cmake-clean-cache
colcon test  --packages-above $PKGS_CHANGED

# rinse and repeat
touch src/bar/bar.txt
PKGS_CHANGED=$(colcon list --packages-skip-source-unchanged | xarg)
colcon clean --build --install --packages-above $PKGS_CHANGED
colcon build --audit --packages-above $PKGS_CHANGED --cmake-clean-cache
colcon test  --packages-above $PKGS_CHANGED
...

So far, I'm thinking of adding two arguments, one that can be used with existing verbs to audit the source and store the audit file to the package build folder for later use. The second would be used to filter packages by investigating deltas between the audit file and current package's source tree.

Before starting a PR, I'd like to iterate on a few things:

Terminology

I've used the term audit as similar defined in https://github.com/jessek/hashdeep CLI, but perhaps this argument should be made more specific? e.g. --audit-packages?

For the filter argument, given the AND logic of all conditions, I see there is a pattern for having an inclusive and exclusive conditions for setting selected = False assignment. E.g. packages_select_test_failures vs. packages_skip_test_passed. In a similar style, perhaps we could use the names:

• packages_select_audit_changed
• packages_skip_audit_changed

or

• packages_select_audit_changed
• packages_skip_audit_unchanged

I guess it was intentional by design not to make select/skip_test purly symmetric, as packages_skip_test_passed includes both failed and untested packages, while packages_select_test_failures only includes failed packages. What would be the appropriate dualism here?

Extension

Taking a look at the colcon-package-selection extensions, I think adding the feature here might be most appropriate for code reuse. However, if we wanted to add dependencies like gitpython to be version control aware for ignoring files, should we then seek to break this out into its own extension?

Persisting Audits

When the build verb is specified to audit, should the audit file only be over-written to the build folder upon a successful build? I was thinking if the audit file was always updated, regardless of the success of the build, then what would be the cleanest order of commands to rebuild packages matching (audit-changed OR build-not-passed)?

What does it mean to the filter aurments when the audit file does not exist? Should unchanged only be interpreted by the existence of a colcon_audit that matches the audit at runtime, similar to how a package build is only considered as passed if the colcon_build.rc exists and only contains 0? This might be a consideration for the asymmetric duality in select/skip arguments.

I frequently build a workspace with --packages-up-to A and later more packages with --packages-up-to B. To speed up the second invocation the packages already built during the first invocation should be skipped. While that is possible with an invocation like this:

colcon build --packages-up-to B --packages-skip $(colcon list -n --packages-up-to A)

it would be nice to have a convenience option for this which doesn't require a second nested call.