Increasing available memory for release builds [Pinocchio] about buildfarm_deployment HOT 6 OPEN

Certainly blacklisting 32-bit builds will stop the current failure. I'll not that the 4.11GB peak for the build is something that will be truely a blocker for 32bit systems as that's over the size of the max addressable memory in 32 bit space

We don't have the ability to add more memory for a specific job. And in general using this size of compiler object is a problem for many users. I would also recommend simply separating out your system into slightly more smaller compile units. If your system has enough resources it can compile them in parallel. And on smaller systems you can do one at a time. Whereas with the large build at the moment with -j1 there is no way for someone to use it on a smaller platform.

We had similar problems with pcl early in its development. It was regularly causing crashes on developer machines due to going OOM. It too is also highly templated however by reviewing the include orders we were able to significantly reduce the memory usage to keep it from overwhelming systems.

We run 8GB VMs but they run 4 jobs at a time so our primary specification is 2GB per job. We don't currently enforce it but I would request that you try to respect that. If for example your releases for 2 different platforms ended up on the same executor at the same time it would likely run out of memory as both platforms typically peak at the same time. And there's potentially 2 other jobs as well as our system overhead. We've seen this sort of simultaneous peaking actually taking our executors offline in the past: ros-infrastructure/ros_buildfarm#265

from buildfarm_deployment.

@wxmerkt What I've found in the past is that having a lot of code in a single compilation unit tends to be the culprit for excessive memory usage. Thus, the solution usually resolves around splitting the code up into multiple compilation units, running the whole thing serially (i.e. make -j1), and then linking them together at the end. I'm not sure if that will work in this particular case, but that's the avenue that I would explore.

from buildfarm_deployment.

Our amd64 build agents have been going offline due to memory exhaustion when building ros-eloquent-pinocchio, and ros-foxy-pinocchio, which were added in ros/rosdistro#26391 and ros/rosdistro#26391 respectively. @wxmerkt, have you made any improvement on the memory consumption? If not, do you mind reverting the PRs until the memory consumption is below 2GB?

from buildfarm_deployment.

I was hoping that similar to ROS1 they'll finish one build by build after a series of failures. This does not seem to be the case :-(. I am okay with reverting the eloquent/foxy releases.

@azeey, what's the current memory limit for the amd64 buildfarm for ROS2? I saw that some builds hung in the tests but I thought I explicitly disabled tests via a patch as in ROS1 (since we know they exhaust memory). Is there another patch we could use to disable tests?

Regarding reducing memory requirements: When speaking to @jcarpent, I seem to recall that reducing memory requirements for compilation of the Python bindings (expose-aba-derivatives etc.) was not straight-forward or possible. I don't have any insights here, sorry.

cc: @Rascof, the ROS2 releases for Pinocchio are likely to be reverted. Could you perhaps look into the possibility of reducing memory requirements? The alternative would be to release without Python bindings, but that'd be quite limiting.

from buildfarm_deployment.

Thanks for the creating the PRs.

what's the current memory limit for the amd64 buildfarm for ROS2?

The build agents have 8GB, but as @tfoote mentioned, each agent runs 4 jobs in parallel, so a limit of 2GB per job is recommended.

Is there another patch we could use to disable tests?

Unfortunately, I'm not familiar with release process to answer that question.

from buildfarm_deployment.

cc: @Rascof, the ROS2 releases for Pinocchio are likely to be reverted. Could you perhaps look into the possibility of reducing memory requirements? The alternative would be to release without Python bindings, but that'd be quite limiting.

I will be looking for a solution but I don't know if I could be of any help since I don't know in detail the Pinocchio package.
I asked if It would work without the python bindings. It would but it will not be useful. I can wait to release the packages dependent on Pinocchio for foxy and eloquent.

from buildfarm_deployment.