RAPIDS nightly wheels have migrated from the old, private nightly index to a new, public nightly index hosted on anaconda.org. The old index is currently still live, but we should plan to remove it from the rapids-build-utils feature's post-attach command by 24.04 or so since that index is now outdated and will soon be removed altogether. Since the new nightly index is public, every repo's dependencies.yaml file can be relied upon to include the necessary index information.

Currently, the containers that are built and published are specified in the matrix.yaml file and follow a format like:

- os: "ubuntu:18.04"
  images:
  - features: [*gcc_6, *cuda_prev_min, *python, *lit]
  - features: [*gcc_7, *cuda_prev_min, *python, *lit]
  - features: [*gcc_8, *cuda_prev_min, *python, *lit]
  - features: [*gcc_9, *cuda_prev_min, *python, *lit]
  - features: [*llvm_9, *cuda_prev_min, *python, *lit]
  - features: [*oneapi_2022, *cuda_prev_min, *python, *lit]

Where we specify the base OS container and then a list of features to install in each generated image.

I would like to extend this to add the ability to specify custom environment variable definitions in each container environment as well.

I imagine we could do this by adding an env field like this:

- os: "ubuntu:18.04"
  images:
  - features: [*gcc_6, *cuda_prev_min, *python, *lit]
    env: {my_var: "my_value"}

This would then translate down into the generated devcontainer.json as:

"containerEnv": {
  "my_var": "my_value"
}

In the CCCL devcontainers, we would like to have the latest version of clang-format installed in every image.

We could do that today by installing the latest version of the llvm feature, but that would unnecessarily bloat the size of the image.

Ideally there would be finer grain control over which LLVM packages are installed by the llvm feature.

The feature uses the llvm.sh install script provided by the LLVM project. This script already provides some control over which packages are installed via the all argument passed to the script. If all is not specified, it only installs clang, lldb, lld, and clangd. Otherwise it installs the remaining LLVM packages:

I see a few options:

1. Update the devcontainer feature to add a packages: option that controls which packages are installed. Then to install just clang-format we could add the feature with packages: clang-format.
2. Update the llvm.sh script such that when all isn't specified, it will still install clang-format.

Option 1 would be the most flexible, but would require the most modification to the llvm.sh script. Option 2 is probably easier, but provides less control and may still install unnecessary packages.

clone-* does not checkout the tag dictated by the manifest. Instead, the main branch is checked out.

There is a lot of functionality in the scripts in the rapids-build-utils feature. It would be nice to have documentation of what this feature provides and how to use it.

The readme doesn't give much info.

https://github.com/rapidsai/devcontainers/blob/branch-23.12/features/src/rapids-build-utils/README.md

I'm doing a devcontainer build without GitHub auth. In ~coder/.bashrc SCCACHE_BUCKET and SCCACE_REGION are unset, but CUDAARCHS is left set as "RAPIDS" causing my build to target all architectures. I think CUDAARCHS should either be unset or set to "NATIVE" in this case.

We should filter the *-all scripts based on whether the associated *-build, etc, scripts are found.

if type $SCRIPT-${lib} >/dev/null 2>&1; then
    ...
fi;

Is your feature request related to a problem? Please describe.

More developers are using dev containers for RAPIDS development. We can build most of RAPIDS from the provided dev containers. But when changes need to be tested for impacts on the Java bindings, we have to use a completely separate process.

This would also enable experimenting with cuDF / RAPIDS changes while running Spark workloads.

Describe the solution you'd like

Add the proper dependencies and scripts for building and testing JNI.

Describe alternatives you've considered

• Use the documented process for building libcudf for Spark and then install JDK and maven and build them.
• Use Spark-RAPIDS containers for testing.

Both of these require me to commit my changes from the dev container, and then checkout the branch inside the Spark environment in order to build and test. An integrated environment will be more productive.

Additional context

I have successfully installed JDK and Maven in a dev container with cuDF, but was unable to build cuDF because of a CMake error.

[exec] CMake Error at /home/coder/cudf/java/target/cmake-build/_deps/rapids-cmake-src/rapids-cmake/find/package.cmake:125 (find_package):
     [exec]   By not providing "Findcudf.cmake" in CMAKE_MODULE_PATH this project has
     [exec]   asked CMake to find a package configuration file provided by "cudf", but
     [exec]   CMake did not find one.

It would be nice to have nsight compute and nsight system pre-installed in the devcontainers. Benchmarking kernels is common for rapids developers.

I would like a devcontainer feature to install the Grace clang compiler: https://developer.nvidia.com/grace/clang/downloads

I would also like a CCCL image added for using Grace clang as a host compiler.

For Python builds driven by scikit-build-core, the correct way to set the build type is using scikit-build-core's own variable SKBUILD_CMAKE_BUILD_TYPE. Setting CMAKE_BUILD_TYPE directly is not supported. That argument will be filtered out and it will raise a warning.

A previous implementation of this script can be found here: https://github.com/trxcllnt/rapids-compose/blob/main/scripts/git-checkout-same-branch.sh

Currently just about every time I push to CI my PRs fail clang-format checks because the clang-format in the devcontainer (cuSpatial) is v16, and RAPIDS pre-commit uses something like v11. Small things like bracket wrapping and comment alignment have changed between versions.

Would be nice for these to align.