Comments (6)
Certainly blacklisting 32-bit builds will stop the current failure. I'll not that the 4.11GB peak for the build is something that will be truely a blocker for 32bit systems as that's over the size of the max addressable memory in 32 bit space
We don't have the ability to add more memory for a specific job. And in general using this size of compiler object is a problem for many users. I would also recommend simply separating out your system into slightly more smaller compile units. If your system has enough resources it can compile them in parallel. And on smaller systems you can do one at a time. Whereas with the large build at the moment with -j1 there is no way for someone to use it on a smaller platform.
We had similar problems with pcl early in its development. It was regularly causing crashes on developer machines due to going OOM. It too is also highly templated however by reviewing the include orders we were able to significantly reduce the memory usage to keep it from overwhelming systems.
We run 8GB VMs but they run 4 jobs at a time so our primary specification is 2GB per job. We don't currently enforce it but I would request that you try to respect that. If for example your releases for 2 different platforms ended up on the same executor at the same time it would likely run out of memory as both platforms typically peak at the same time. And there's potentially 2 other jobs as well as our system overhead. We've seen this sort of simultaneous peaking actually taking our executors offline in the past: ros-infrastructure/ros_buildfarm#265
from buildfarm_deployment.
@wxmerkt What I've found in the past is that having a lot of code in a single compilation unit tends to be the culprit for excessive memory usage. Thus, the solution usually resolves around splitting the code up into multiple compilation units, running the whole thing serially (i.e. make -j1
), and then linking them together at the end. I'm not sure if that will work in this particular case, but that's the avenue that I would explore.
from buildfarm_deployment.
Our amd64 build agents have been going offline due to memory exhaustion when building ros-eloquent-pinocchio
, and ros-foxy-pinocchio
, which were added in ros/rosdistro#26391 and ros/rosdistro#26391 respectively. @wxmerkt, have you made any improvement on the memory consumption? If not, do you mind reverting the PRs until the memory consumption is below 2GB?
from buildfarm_deployment.
I was hoping that similar to ROS1 they'll finish one build by build after a series of failures. This does not seem to be the case :-(. I am okay with reverting the eloquent/foxy releases.
@azeey, what's the current memory limit for the amd64 buildfarm for ROS2? I saw that some builds hung in the tests but I thought I explicitly disabled tests via a patch as in ROS1 (since we know they exhaust memory). Is there another patch we could use to disable tests?
Regarding reducing memory requirements: When speaking to @jcarpent, I seem to recall that reducing memory requirements for compilation of the Python bindings (expose-aba-derivatives etc.) was not straight-forward or possible. I don't have any insights here, sorry.
cc: @Rascof, the ROS2 releases for Pinocchio are likely to be reverted. Could you perhaps look into the possibility of reducing memory requirements? The alternative would be to release without Python bindings, but that'd be quite limiting.
from buildfarm_deployment.
Thanks for the creating the PRs.
what's the current memory limit for the amd64 buildfarm for ROS2?
The build agents have 8GB, but as @tfoote mentioned, each agent runs 4 jobs in parallel, so a limit of 2GB per job is recommended.
Is there another patch we could use to disable tests?
Unfortunately, I'm not familiar with release process to answer that question.
from buildfarm_deployment.
cc: @Rascof, the ROS2 releases for Pinocchio are likely to be reverted. Could you perhaps look into the possibility of reducing memory requirements? The alternative would be to release without Python bindings, but that'd be quite limiting.
I will be looking for a solution but I don't know if I could be of any help since I don't know in detail the Pinocchio package.
I asked if It would work without the python bindings. It would but it will not be useful. I can wait to release the packages dependent on Pinocchio for foxy and eloquent.
from buildfarm_deployment.
Related Issues (20)
- Bionic builds for Arm{64,hf} sometimes fail with a corrupt package HOT 2
- Ubuntu trusty containers need 4.15 kernel to avoid performance hit HOT 1
- 'virtual memory exhausted: Operation not permitted' on 32-bit Trusty (ROS Indigo) HOT 13
- git config missing on ROS2 farm
- Some jobs on build.ros.org seem to be failing with no apparent reason HOT 3
- Apparent race condition with connext generator causes occasional failures HOT 4
- import_upstream job is taking much longer than it used to. HOT 3
- Jenkins admin password not injected at deployment HOT 17
- "Proxy setup broken" after master deployment HOT 5
- install rebuild plugin
- Jenkins build priority strategy not set by the deployment
- Add rsync settings for upload job
- backport reprepro 5.3 for the repository HOT 1
- configure_git_user script returns error HOT 10
- Enable connection from agents on deploy
- Shell escaping password HOT 1
- Incorrect exit code running reconfigure.bash HOT 1
- enable cgroup memory management
- Build command from Jenkins GUI failed cause "Could not find specified credentials"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from buildfarm_deployment.