ros2 / ros_buildfarm_config Goto Github PK

We recently added Foxy to our buildfarm. All ROS distros build their SRC and BIN jobs correctly, except for Foxy (our first ROS2 distro in the farm). When jobs are updated from the reconfigure jobs, the ssh-agent credentials appear to not be set properly. This leads to an immediate failure of the job:
java.io.IOException: [ssh-agent] Could not find specified credentials

If I edit the job configuration, the proper credentials are auto-filled. Just opening and saving the job again, lets the jobs run properly.

I'm not sure if the config is the right place for this issue, or if it's another repo.

This is a subset of the greater Rolling transition to 22.04 ros2/ros2#1221

• Disable management jobs during target transitions:
  • rolling_rosdistro-cache
  • rolling_failing-jobs
  • Rci_reconfigure*
  • Rdev_reconfigure*
  • Rdev_trigger*
  • Rdoc_reconfigure*
  • Rdoc_trigger*
  • Rrel_*
• Disable jobs running on Ubuntu Focal
  • default and ufv8 release jobs
    • Rsrc_uF__*
    • Rbin_uF64__*
    • Rbin_ufv8_uFv8__*
  • Devel and PR jobs
    • Rdev__*
    • Rpr__*
  • Doc jobs
    • Rdoc__*
  • Not disabling CI jobs for now.
• Release builds
  • Update release build target platforms (Rsrc and Rbin jobs) #209
  • Merge ros/rosdistro#32036 updating rolling/distribution.yaml
  • Re-deploy rolling release management jobs
  • Re-deploy release trigger upload jobs
  • Wait for ros-rolling-ros-workspace to become available in testing on Jammy
• Source builds
  • Update source build target platforms (Rdev and Rpr jobs) #212
  • Re-deploy rolling source management jobs
• Doc builds
  • Update doc build target platform (Rdoc jobs) #213
  • Re-deploy rolling doc management jobs
• Ci builds
  • Update ci build target platforms (Rci jobs) #217
  • Re-deploy rolling CI management jobs

Since this will be the first time I've updated the platform of an existing rosdistro there are a few things I'm trying to work out to smooth this out going forward.

The target platform and architecture is embedded into all job names, but is only part of the Jenkins view name for release jobs.
Non-default build files have the build file name embedded in the view but default jobs do not. This means that when we bring up the Jammy source (Rdev) jobs they'll create doubled jobs for everything on the build farm or de-list the old focal jobs. I haven't yet checked how the views are updated to check if the old jobs will just remain or be removed. Since we aren't supporting the focal jobs any more the natural thing to do would be to delete them clearing space for the new jammy jobs. However that also means that we lose the build history of those jobs immediately and there is probably value in having a sunset period where the jobs are disabled but not deleted just like the release jobs will be. The same issue true of Rpr, Rci, and Rdoc jobs.
I see a couple of methods to resolve this:

1. We could update ros_buildfarm to include a target identifier in all views regardless of job type. I've not yet looked to see how feasible this would be to change. Sometimes ros_buildfarm changes are small thanks to templating and code re-use and sometimes the same function is implemented across several different layers or in python and bash or groovy. This change would also affect all distributions (and ROS 1 distros) not just Rolling which may or may not be desirable.

2. We could add the Jammy jobs by introducing new non-default targets which would put them in new views based on the build file identifier used in the config index. If we did this I'd suggest that for Rolling we abandon the use of default build files for all build types except release build files where the target is encoded in the view already. This has the benefit of requiring no changes in ros_buildfarm but the downside of creating separate conventions for release build files and every other build file as well as for Rolling compared to every other distribution where there is less of an issue using default build files because the platform list is less likely to change during the life of the distribution.

3. After updating build files and deploying the new jobs we manually move the focal jobs into manually created views which we can clean up at any later date. This is pretty straightforward to do with a Groovy script or Jenkins configuration UI but it's a chore and will have to be maintained separately.

4. We could do nothing and let the jobs fall where they will, and see how much, if any trouble the extra jobs give us.

5. We could delete outright the focal jobs with no sunset period in order to clear the way for the Jammy jobs and not change anything else.

marti_common was added to the blacklist for Humble and Rolling because of build failures (swri-robotics/marti_common#665). We've updated for Humble, and it should be working now. Could we get removed from the blacklist because we've got downstream users wanting to install this package in Humble.

This is a tracking issue for the investigation in CI build performance.

• ros-infrastructure/ros_buildfarm#645 was leveraged to change the MAKEFLAGS to allow further parallelization within a project's build. Results showed that -j1 this was not substantially contributing to the performance shortcomings.
• #31 is an experiment tweaking the host and executor configuration. To do this, we need to tie the CI builds to a specific agent label. This change should be reverted when the experiment is complete.

I've been thinking of this as heterogeneous build agents since the agents can disparate types of jobs but the actual important part is returning to a homogeneous agent pool. So I put the Latin down and called this issue "unified" build agents.

This rollout is happening in a number of stages:

• Support heavy-job plugin in ros_buildfarm ros-infrastructure/ros_buildfarm#839
• Assign a job weight of 4 on all CI jobs #140
• Re-deploy CI agents so they have 4 executors, allowing them to run heavy CI jobs
• Update Jenkins label expression and priorities for CI jobs so they can run on buildagents.
• Determine if it will be necessary to run one build-only and one ci-only agent with only the elastic agents being unified or if we can keep a fully unified agent pool.

Background: When we first put the CI jobs on the ROS build farm we ran into the issue that one CI job from ci.ros2.org which was the prototype for them is used to having a full build agent's resources rather than 1/4th of a build agent and we ran into performance issues as a result.

ros-infrastructure/ros_buildfarm#839 added support for the heavy job plugin which allows jobs to assign a "weight" which is the number of executors required to host that job.

The Jenkins job priorities have previously been allocated such that certain job types are prioritized above other job types, and within each job type, more recent ROS distributions are prioritized above older ones.

We need to increase the spacing between these job types to avoid priority collisions when we add jobs for Foxy.

Additionally, we should consider adding a group for the manual CI jobs (overlay) which would be prioritized above the nightly jobs.

Current allocations:

45
  jenkins_pull_request_job_priority::eloquent/source-build.yaml
46
  jenkins_pull_request_job_priority::dashing/source-build.yaml
47
  jenkins_pull_request_job_priority::crystal/source-build.yaml
48
  jenkins_job_priority::eloquent/ci-nightly-connext.yaml
  jenkins_job_priority::eloquent/ci-nightly-cyclonedds.yaml
  jenkins_job_priority::eloquent/ci-nightly-debug.yaml
  jenkins_job_priority::eloquent/ci-nightly-extra-rmw-release.yaml
  jenkins_job_priority::eloquent/ci-nightly-fastrtps-dynamic.yaml
  jenkins_job_priority::eloquent/ci-nightly-fastrtps.yaml
  jenkins_job_priority::eloquent/ci-nightly-navigation2-prerequisites.yaml
  jenkins_job_priority::eloquent/ci-nightly-navigation2.yaml
  jenkins_job_priority::eloquent/ci-nightly-opensplice.yaml
  jenkins_job_priority::eloquent/ci-nightly-release.yaml
  jenkins_job_priority::eloquent/ci-overlay.yaml
49
  jenkins_job_priority::dashing/ci-nightly-connext.yaml
  jenkins_job_priority::dashing/ci-nightly-cyclonedds.yaml
  jenkins_job_priority::dashing/ci-nightly-debug.yaml
  jenkins_job_priority::dashing/ci-nightly-extra-rmw-release.yaml
  jenkins_job_priority::dashing/ci-nightly-fastrtps-dynamic.yaml
  jenkins_job_priority::dashing/ci-nightly-fastrtps.yaml
  jenkins_job_priority::dashing/ci-nightly-opensplice.yaml
  jenkins_job_priority::dashing/ci-nightly-release.yaml
  jenkins_job_priority::dashing/ci-overlay.yaml
50
  jenkins_job_priority::crystal/ci-nightly.yaml
  jenkins_job_priority::crystal/ci-overlay.yaml
55
  jenkins_commit_job_priority::eloquent/source-build.yaml
56
  jenkins_commit_job_priority::dashing/source-build.yaml
57
  jenkins_commit_job_priority::crystal/source-build.yaml
75
  jenkins_source_job_priority::eloquent/release-bionic-arm64-build.yaml
  jenkins_source_job_priority::eloquent/release-bionic-armhf-build.yaml
  jenkins_source_job_priority::eloquent/release-build.yaml
76
  jenkins_source_job_priority::dashing/release-bionic-arm64-build.yaml
  jenkins_source_job_priority::dashing/release-bionic-armhf-build.yaml
  jenkins_source_job_priority::dashing/release-build.yaml
77
  jenkins_source_job_priority::crystal/release-bionic-arm64-build.yaml
  jenkins_source_job_priority::crystal/release-build.yaml
79
  jenkins_binary_job_priority::eloquent/release-bionic-arm64-build.yaml
  jenkins_binary_job_priority::eloquent/release-build.yaml
80
  jenkins_binary_job_priority::dashing/release-bionic-arm64-build.yaml
  jenkins_binary_job_priority::dashing/release-build.yaml
81
  jenkins_binary_job_priority::crystal/release-bionic-arm64-build.yaml
  jenkins_binary_job_priority::crystal/release-build.yaml
99
  jenkins_binary_job_priority::dashing/release-bionic-armhf-build.yaml
  jenkins_binary_job_priority::eloquent/release-bionic-armhf-build.yaml