This comes in very handy if we need to pin to a nameless commit like fix/moveRawAccess-429-gb66dda2 (see ethz-asl/minkindr#31).

Do we ( @dymczykm , @schneith, and me ) have all necessary information?

I'm missing, probably :

• where are the macs (physically)?
• who is renting (formally) and how the virtual servers?
• are there any backup systems configured (workers: ubuntu, mac os x; Jenkins project configurations)

This should then be used instead of the hand-maintained list of packages to build. Run the script on the repo itself (without deps) s.t. unit-tests etc get run.

http://129.132.38.183:8080/job/aslam_vcharge/36/console
Any ideas?

#### Found a catkin repo
Checking whether we should run the tests...
Will first build aslam_vcharge aslam_vcharge_python and then run tests.
Traceback (most recent call last):
  File "/usr/local/bin/catkin", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2749, in <module>
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 444, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 725, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 628, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: catkin-tools==0.0.0

Apparently something is broken with the ros setup on the-mac-mini. It fails to start. One is still running (on the default port). But starting a new one is failing:

aslw039050:~ macmini-admin$ export ROS_MASTER_URI=http://aslw039050.ethz.ch:12000/
aslw039050:~ macmini-admin$ roscore -p 12000
... logging to /Users/macmini-admin/.ros/log/6db237a1-bafb-11e6-abb0-0c4de9ce5a87/roslaunch-aslw039050.ethz.ch-79942.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt

started roslaunch server http://aslw039050.ethz.ch:50928/
ros_comm version 1.11.20


SUMMARY
========

PARAMETERS
 * /rosdistro: indigo
 * /rosversion: 1.11.20

NODES

auto-starting new master
process[master]: started with pid [79949]
ROS_MASTER_URI=http://aslw039050.ethz.ch:12000/

setting /run_id to 6db237a1-bafb-11e6-abb0-0c4de9ce5a87
Traceback (most recent call last):
  File "/opt/ros/indigo/lib/python2.7/site-packages/roslaunch/__init__.py", line 307, in main
    p.start()
  File "/opt/ros/indigo/lib/python2.7/site-packages/roslaunch/parent.py", line 279, in start
    self.runner.launch()
  File "/opt/ros/indigo/lib/python2.7/site-packages/roslaunch/launch.py", line 654, in launch
    self._setup()
  File "/opt/ros/indigo/lib/python2.7/site-packages/roslaunch/launch.py", line 631, in _setup
    self._launch_core_nodes()
  File "/opt/ros/indigo/lib/python2.7/site-packages/roslaunch/launch.py", line 506, in _launch_core_nodes
    name, success = self.launch_node(node, core=True)
  File "/opt/ros/indigo/lib/python2.7/site-packages/roslaunch/launch.py", line 534, in launch_node
    process = create_node_process(self.run_id, node, master.uri)
  File "/opt/ros/indigo/lib/python2.7/site-packages/roslaunch/nodeprocess.py", line 139, in create_node_process
    args = create_local_process_args(node, machine)
  File "/opt/ros/indigo/lib/python2.7/site-packages/roslaunch/node_args.py", line 251, in create_local_process_args
    matches = roslib.packages.find_node(node.package, node.type, rospack=rospack)
  File "/opt/ros/indigo/lib/python2.7/site-packages/roslib/packages.py", line 400, in find_node
    return find_resource(pkg, node_type, filter_fn=_executable_filter, rospack=rospack)
  File "/opt/ros/indigo/lib/python2.7/site-packages/roslib/packages.py", line 497, in find_resource
    source_path_to_packages = rospack.get_custom_cache('source_path_to_packages', {})
AttributeError: 'RosPack' object has no attribute 'get_custom_cache'
[master] killing on exit

Any ideas? Somebody willing to spend time here?

Would it be possible to fail immediately if a test crashes (crash != fail, missing xml)? This should save unnecessary jenkins compute and would have the advantage that the segfault stack trace could be easily found by inspecting the tail of the console output.

http://129.132.38.183:8080/job/aslam_vcharge/91/console

Given package 'baz' is not in the workspace
Build step 'Execute shell' marked build as failure
Recording test results

Any ideas?

Filter the auto-discovered list against the output of catkin list Using:

listcontains() {
  for word in $1; do
    [[ $word = $2 ]] && return 0
  done
  return 1
}
list="11 22 33"
if listcontains "$list" 1; then echo Y; else echo N; fi
if listcontains "$list" 22; then echo Y; else echo N; fi

http://129.132.38.183:8080/job/aslam_cv2/84/console

This builds locally but does not build on the build server. Should we upgrade catkin?

Why:

1. a permanently running roscore is not robust ( #51 ) and it is conceptually not able to support multiple tests in parallel
2. it contradicts the principle of unit testing in a fresh environment because it has memory
3. as soon as we launch individual containers per job, it will be quite a waste
4. without that we would need to implement a specific solution for the docker slaves, which could be almost as much work

Where and how:
In principle this could already be done with the prepare script ( #79 ) on a per repository basis.

But as this seems to be a common need we might want to directly support it through the run_build.sh script.
In any case we must not forget to make sure that the core dies at the end of the job.
To support multiple tests in parallel and therefore multiple roscors we need to find a way to allocate and assign individual ports. Having that things get easy: http://wiki.ros.org/roscore.

@eggerk , would you like to work on this? Or anybody else?

shouldn't we set flags explicitly if we want to build tests / perform cpp check in https://github.com/ethz-asl/continuous_integration/blob/master/run_build_impl.sh#L17

also, isn't the standard --no-tests instead of --no_tests ?

just took me some time to figure this out ...

I'm recently having these errors in euroc_simulator and mav_planning_gui. Everything seems to pass, then the errors below (actually many more) occur. Has anyone else had this? Or what am I doing wrong?

15/15 files checked 100% done
FATAL: null
java.lang.StackOverflowError
    at java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3383)
    at java.util.regex.Pattern$Curly.match0(Pattern.java:3777)
    at java.util.regex.Pattern$Curly.match(Pattern.java:3761)
    at java.util.regex.Pattern$GroupHead.match(Pattern.java:4185)
    at java.util.regex.Pattern$Neg.match(Pattern.java:4615)

This can have performance implications on the long run (because /tmp/ needs to be fast).

Any objections against adding a cron job on each node running the following on a daily basis:

find  /tmp/ -mtime +1 -name \*jenkins\* -exec rm {} \;

This leads to an error state of the build at least if Schweizer-Messer is involved (as in http://129.132.38.183:8080/job/SchweizerMesser/136/label=ubuntu/parsed_console/).

Doxygen is using latex for html output (for formulas). Therefore it does not help that latex output is disabled.

Is it much effort to install enough latex for that on the the-mac-mini (and mc-jenkins)?

Or shall we try to disable doxygen for Jenkins builds?

Any opinions?

Such as https://github.com/ethz-asl/rotors_simulator/blob/master/rotors_simulator/package.xml.

After that add it again to http://129.132.38.183:8080/job/mav_integration_test/configure.

There seems to be a bug in ccache: it currently assumes a cache size of 5.5 G while the folder .ccache has a total size of 49 G:

jenkins@asls06:~/.ccache$ ccache -s
....
cache size                           5.5 Gbytes
max cache size                      30.0 Gbytes
jenkins@asls06:~/.ccache$ du -hs .
49G .

Therefore it cannot respect the maximal cache size.

Does somebody know something about this? Or even how to fix it?
Otherwise we might need to purge the cache on a regular basis.

@HannesSommer Is there a reason for only using 5 workers?

Originally posted by @floriantschopp in #75 (comment)

In e.g. http://129.132.38.183:8080/job/aslam_incremental_calibration/292/console a test clearly fails but the final status is SUCCESS. I checked on Jenkins. The test log xml file contains the failures and is copied properly to aslam_nonparametric_estimation/workspace/test_results.

jenkins@asls06:~/jobs/aslam_incremental_calibration/workspace/test_results$ find
.
./incremental_calibration
./incremental_calibration/gtest-incremental_calibration_test.xml
jenkins@asls06:~/jobs/aslam_incremental_calibration/workspace/test_results$ grep failures -r *
incremental_calibration/gtest-incremental_calibration_test.xml:<testsuites tests="5" failures="2" disabled="0" errors="0" timestamp="2014-07-13T12:29:25" time="0.416" name="AllTests">
incremental_calibration/gtest-incremental_calibration_test.xml:  <testsuite name="AslamCalibrationTestSuite" tests="5" failures="2" disabled="0" errors="0" time="0.416">

Any ideas?

The lines in the log like

Fetching changes from the remote Git repository
Fetching upstream changes from [email protected]:ethz-asl/aslam_cv.git

do not what I would expect. After those lines the ref "remotes/origin/master" can still be outdated! (found so e.g. in aslam_cv).

When I log into the build server and run manually as jenkins "git fetch origin" this will also not update remotes/origin/master! This has the expected bad impact on when a build of "origin/master" is issued (including the regular periodic builds..): It will build an outdated commit.

I believe the problem is that the work trees are in detached mode.

One can manually fix that on command line with e.g. :

git fetch origin master:remotes/origin/master

So I would suggest that we do that instead. Unfortunately I don't know which code is doing producing the upper log lines and doing the fetch. (it does not seem to be part of this repository). So I cannot fix that. Can somebody point me to the right code or fix it?

It would be great if we could have a method of testing a whole collection of packages with tags or branches specified. I think the best way to do this would be to have something like a configuration file for a build job that specified all of these things. This would let us do things like use a command line tool to launch a job on the build server with our current collection of packages, then get the link from this job and paste this link in to the individual pull requests to show that it all builds together.

It seems that the nosetests installation is not proper on Jenkins (since the new setup? ):

jenkins@asls06:/tmp/nosetest$ nosetests
Traceback (most recent call last):
  File "/usr/local/bin/nosetests", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/dist-packages/setuptools-2.0.2-py2.7.egg/pkg_resources.py", line 2709, in <module>
    parse_requirements(__requires__), Environment()
  File "/usr/local/lib/python2.7/dist-packages/setuptools-2.0.2-py2.7.egg/pkg_resources.py", line 571, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: nose==1.1.2
jenkins@asls06:/tmp/nosetest$ /usr/bin/nosetests

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
jenkins@asls06:/tmp/nosetest$ which nosetests
/usr/local/bin/nosetests

Maybe we should just uninstall the one in /usr/local/bin? (the other is from the debian package python-nose)

Apparently on the master Jenkins instances there is latex installed. On W540 it isn't. I wouldn't mind at all if not this way sm_doc (current master HEAD) fails to "build" on W540 but succeeds on the master instances.

On the other hand doxygen should not even touch latex, because it is disabled in the config file (see. https://github.com/ethz-asl/Schweizer-Messer/blob/master/sm_doc/doc/doxygen.config.in#L1020).

So naturally to fix that would be the better option than installing latex on W540 (and maybe the Macs).
But I have no idea how to fix it. Somebody any idea / memory what could override GENERATE_LATEX= NO? (I've checked the file on jenkins it contains that line).

See http://129.132.38.183:8080/job/SchweizerMesser/label=ubuntu/99/consoleFull for how it goes wrong, or this extract

/usr/bin/doxygen /var/lib/jenkins/jobs/SchweizerMesser/workspace/label/ubuntu/src/schweizer_messer/sm_doc/doc/doxygen.config
..............
This is pdfTeX, Version 3.1415926-2.5-1.40.14 (TeX Live 2013/Debian)
 restricted \write18 enabled.
entering extended mode
(./_formulas.tex
LaTeX2e <2011/06/27>
Babel <3.9h> and hyphenation patterns for 78 languages loaded.

(btw. when I build sm_doc on my machine, no latex is used)

this line fails:
https://github.com/ethz-asl/continuous_integration/blob/master/catkin_make_repo#L77
seems to happen if no packages (i.e. build all) are specified when calling run_build.sh .

Do we need another variable to pass to run_build to specify which packages should be tested? right now it seems like packages_to_build==packages_to_test, which is probably not always the case.

In the past we switched from Debug build to Release because some tests would run for too long in Debug build. Unfortunately compromises are not that easy.

So far I see the following options:

1. introduce a specific macro to enable assertions in spite of NDEBUG. Something like ASSERTIONS_ON. Of course this would then not enable the c "assert" macro. (this can be kind of hacked for many cases, though)
2. create a custom build type e.g. as described here http://stackoverflow.com/questions/11437692/how-to-add-a-custom-build-type-to-cmake-targetting-make. Should be possible via command line arguments.
3. try to change the compiler flags for an existing build type but that seems to be troublesome as well (http://stackoverflow.com/questions/28732209/change-default-value-of-cmake-cxx-flags-debug-and-friends-in-cmake).

In any case we could add command line options for the continuous integration scripts such that the Jenkins configuration can switch between different testing modes. But difficult decision would then be if we should change the default from the current Release to something else ..

Any comments or votes?

I would vote for yes per default because a) assertions help finding bugs in the non assertion code and b) assertions can contain bugs (actually sometimes they do not even compile without somebody realizing for a while!)

• Install matlab on Jenkins and mac mini
• http://ch.mathworks.com/help/matlab/matlab_prog/write-script-based-unit-tests.html
• Look for tests in specified matlab folders

I'm just adding this as a placeholder so I don't forget.

When the build server runs with packages specified, the dependencies don't get installed in the right way. This causes, for example, the error "no rule to make target libglog.so".

The workaround was to add glog_catkin and gflags_cakin to the list of packages.

see aslam_vo for an example on the build server.