The doxygen generated documentation available at http://robotology.github.io/robot-testing/index.html is currently outdated (it contains the documentation of the 1.0 version and it was generate the last time the 5th of January).

There is still a compilation problem on windows due to lack of proper configuration (dllexport) of the shared libraries. It can be fixed as described here http://www.cmake.org/Wiki/BuildingWinDLL.

We need some helpers for testing approximate equality of floating point values.

An important feature to have is to print the values when the comparison fails, for easy debugging of failing tests (as in gtest EXPECT_NEAR, ASSERT_NEAR in https://code.google.com/p/googletest/wiki/AdvancedGuide ),

If I have a Syntax Error in my test, the test fails with the following message:

1: Test case geometry.py started...
1: [INFO]  (geometry.py) reports: Running test testInnerProductInvariance
1: [INFO]  (geometry.py) reports: Running test testTransformInverse
1: [ERROR] (geometry.py) asserts error with exception: Cannot call the run() method because Syntax Error!
1: Test case geometry.py failed!

I still have to understand why it does not print a more helpful message.

Hi,

The links under Tutorials and Examples are broken.

Cheers,
Michele

I had a fatal error of this type:

pegua@pareto:~/src/icub-tests/suits$ testrunner --suit basics-icubSim.xml --verbose
Loading basics-icubSim.xml
Staring test runner.
Test suit Basic Tests Suite started...
[INFO]  (Basic Tests Suite) reports: yarpmanager is setuping the fixture...
Returning true from yarpManager setup*** stack smashing detected ***: testrunner terminated
Aborted (core dumped)

This is probably caused by some wrong parameters passed to the yarpmanager fixture plugin. I have disabled the GCC stack protection compiling RTF with the -fno-stack-protector, and I still get some error because it is not able to find the configuration files.

Different robots either real or simulated (ODE or gazebo) have different functionality. Even in case in which functionalities are the same, tests may need different parameters (thresholds, speeds etc).

#5 addresses how to propagate variables from the main configuration file to individual test files. This is for sure useful to propagate the name of the robot as a prefix for port names.

At the moment using the environment functionality (#5) we can use different configuration files (test-ode.ini/test-gazebo.ini)

I wonder if in the end we will need to build different directories/contexts, i.e.:

app/iCubSim --> test ode simulator
app/icub --> test real robot
app/icubGazeboSim --> gazebo simulator

Example: if I write a fixture like this one:

<application>
    <name>iCub Simulator</name>
    <description>A fixture to prepare iCub Simulator for the test cases</description>
    <version>1.0</version>
    <authors>
        <author email="[email protected]">Ali Paikan</author>
    </authors>
    <module>
        <name>iCub_SIM</name>
        <parameters></parameters>
        <node>localhost</node>
        <ensure>
            <wait>1</wait>
        </ensure>
    </module>
    <module>
        <name>yarpview</name>
        <parameters></parameters>
        <node>localhost</node>
        <ensure>
            <wait>100</wait>
        </ensure>
        <dependencies>
          <port timeout="5.0">/dsaadsdada</port>
        </dependencies>
    </module>    
</application>

Nor the ensure tag neither the dependency tag are considered for yarpview, and yarpview is simply launched immediately and the fixture is considered loaded.

...for logging purposes please use yInfo(), yWarning(), yError(), including < yarp/os/LogStream.h > and < yarp/os/Log.h > as appropriate.

By writing a class similar to TextOutputter but that dumps the test results in a format compatible with Jenkins.

It seems that tests may fail because when they start, fixture is not ready yet to execute the test.
Example: I start a motor test on the robot simulator. Even if the simulator is immediately launched (and control ports are responsive), it take approx 10 seconds to perform an initial "calibration". PositionMove() commands are not correctly executed until the parts reach their final home position.
Of course this issue could be addressed by improving the state-machine of the simulator, however I'm wondering if it worth implementing a parametric delay after the fixture startup for applications (e.g. third party modules) which do not provide a better handshaking mechanism...

Hi @apaikan , thanks for providing this tool.

I'm just curious why in some cases the expression "suit" is used (here), whereas other times it's "suite" (here).

If it was just a typo, perhaps it's worth changing it to be consistent everywhere?

Feel free to close if you consider this a minor/irrelevant issue. :)

Similar to how it is done for YARP : https://github.com/robotology/yarp/tree/master/doc/release .

Currently in TestFTSensors we are reading just the sensor port, we should also add the test of use the AnalogSensorClient to read its value, check its state and the timestamps. Reading the raw port should however remain because it is how it is used in a lot of code.

cc @barbalberto any documentation on how to properly use AnalogSensorClient ?

If I tried to load with the yarpmanager fixture plugin an application with a name (contained in the xml <name> tag) longer or equal to 16 characters, I got a failure message:

pegua@pareto:~/src/icub-tests/build$ testrunner --suit ../suits/basics-icubGazeboSim.xml --verbose
Loading ../suits/basics-icubGazeboSim.xml
Staring test runner.
Test suit Basic Tests Suite started...
[INFO]  (Basic Tests Suite) reports: yarpmanager is setuping the fixture...
[ERROR] (Basic Tests Suite) asserts error on (ret) with exception: yarpmanager cannot setup the fixture because Application is not loaded.
[INFO]  (Basic Tests Suite) reports: yarpmanager is tearing down the fixture...
Test suit Basic Tests Suite failed!
Ending test runner.

Need to discuss how to place this c++11 tag.

For some application, it is convenient to set a specific environmental variable for just a test, and do not have this environmental variable set to the rest of the fixture in the suite or for other tests.

cxxabi.h, included by robot-testing\src\testrunner\include\cmdline.h is not standard (gcc only?). testrunner does not compile on windows VS2010.

It will be useful (and in some cases necessary) to have multiple fixture manager for each suites. Example:

<suit name="Basic Tests Suite">
    <description>Testing basic features</description>
    <fixture param="--param myparam"> fixturemanager1 </fixture>
    <fixture param="--param myparam"> fixturemanager2 </fixture>
    ...

    <!-- tests -->
    ...
</suit>

It will be very useful to have timeout for running the tests using testrunner.

Problem

As it is for now, if one of the test cases gets blocked, the testrunner is blocked too and no more tests are running until the CTRL+C is pressed from the testrunner terminal.

Suggestion

1. to add --timeout <seconds> param to the testrunner
2. Add the `timeout' property /tag to the test suite xml file:

<?xml version="1.0" encoding="UTF-8"?>

<suit name="my suit">
    <description> a simple example </description>

   <!-- default timeout for all tests -->
    <timeout> 5.0 </timeout>

    <test type="dll"> mytest2 </test>
   ...
   
   <!-- timeout for a specific test -->
    <test type="dll" timeout="10.0"> mytest1 </test>
    ...
</suit>

RTF_ADD_PLUGIN should force the exporting of the factory symbols

Working on the tests, I realized that in some error situations, I would not only stop the current test, but avoid that the framework continues to launch tests of the same suit.

Therefore I propose to add a new error type in the framework for satisfy this need.
we could call it RTF_ESSERTER_SUIT_ERROR or RTF_ASSERT_CRITICALERROR.

Anyone has any suggestions?

In this Travis Job https://travis-ci.org/robotology/gazebo-yarp-plugins/jobs/62666628 :

testrunner --verbose --suit ../../icub-tests/suits/basics-icubGazeboSim.xml
Loading ../../icub-tests/suits/basics-icubGazeboSim.xml
[testrunner] cannot load plugin FtSensorTest.so; error (load) : FtSensorTest.so: cannot open shared object file: No such file or directory
[testrunner] cannot load plugin FtSensorTest.so; error (load) : FtSensorTest.so: cannot open shared object file: No such file or directory
[testrunner] cannot load plugin FtSensorTest.so; error (load) : FtSensorTest.so: cannot open shared object file: No such file or directory
Staring test runner.
Test suit Basic Tests Suite started...
[INFO]  (Basic Tests Suite) reports: yarpmanager is setuping the fixture...
||| did not find fixtures/icubgazebosim-fixture.xml
[ERROR] (Basic Tests Suite) asserts error on (appfile.size()) with exception: yarpmanager cannot find apllication file icubgazebosim-fixture.xml. Is it in the 'fixtures' folder?
[INFO]  (Basic Tests Suite) reports: yarpmanager is tearing down the fixture...
Test suit Basic Tests Suite failed!
Ending test runner.
-------- results ---------
Total number of tests : 0
Number of passed tests: 0
Number of failed tests: 0

The command "testrunner --verbose --suit ../../icub-tests/suits/basics-icubGazeboSim.xml" exited with 0.

The testrunner is returning 0 because the number of failed tests is zero, but we should make him aware of the ERROR in [ERROR] (Basic Tests Suite) asserts error on (appfile.size()) with exception: yarpmanager cannot find apllication file icubgazebosim-fixture.xml. Is it in the 'fixtures' folder? and return a value different from 0.

On Windows the test FixturePluginLoader get stuck if RTF is compiled in Debug.
We should also enable tests on appveyor and the compilation in debug.

This code

    if(true)
        RTF_TEST_REPORT(Asserter::format("Starting with SystemClock"));
    else
        RTF_TEST_REPORT(Asserter::format("Starting with Network Clock from port %s", clockPort_EnvName.c_str()));

Give this compilation error:

In member function ‘virtual void ClockTest::run()’:
/usr/local/src/robot/icub-src/icub-tests/src/clock/ClockTest.cpp:84:5: error: ‘else’ without a previous ‘if’
     else
     ^

While using brackets it works fine

   if(true)
    {  RTF_TEST_REPORT(Asserter::format("Starting with SystemClock"));   }
    else
    {   RTF_TEST_REPORT(Asserter::format("Starting with Network Clock from port %s", clockPort_EnvName.c_str()));  }

• This is more related to the use of the RsourceFinder in the middleware implementation

To integrate nicely with the traditional CMake/CTest add_test command, we need that the testrunner returns a number different from 0 on failure.
For this are necessary a bunch of modifications:

• For reporting failed test, the return 0 in https://github.com/robotology/robot-testing/blob/master/src/testrunner/src/main.cpp#L232 should be substituted with return collector.failedCount();
• For loading errors (i.e. failure of a load* method) I guess it make sense to simply return 1.

All the RTF headers are installed in the include directory, so they are used as:

#include <TestCase.h>
#include <TestAssert.h>
#include <Arguments.h>

Could it make sense to move them in include/rtf (or include/RTF), so we will have:

#include <rtf/TestCase.h>
#include <rtf/TestAssert.h>
#include <rtf/Arguments.h>

cc @apaikan

https://travis-ci.org/robotology/robot-testing/jobs/96977269

Actual error:

[ 60%] Building CXX object src/plugins/python/CMakeFiles/RTF_python.dir/src/PythonPluginLoader.cpp.o
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp: In member function ‘RTF::TestCase* RTF::plugin::PythonPluginLoaderImpl::open(std::string)’:
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:112:49: error: ‘PyString_FromString’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:135:52: error: ‘Py_InitModule4’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp: In member function ‘std::string RTF::plugin::PythonPluginLoaderImpl::getPythonErrorString()’:
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:183:37: error: ‘PyString_Check’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:184:44: error: ‘PyString_AsString’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:191:37: error: ‘PyString_Check’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:192:60: error: ‘PyString_AsString’ was not declared in this scope
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp: In member function ‘virtual bool RTF::plugin::PythonPluginLoaderImpl::setup(int, char**)’:
/home/travis/build/robotology/robot-testing/src/plugins/python/src/PythonPluginLoader.cpp:231:52: error: ‘PyString_FromString’ was not declared in this scope
make[2]: *** [src/plugins/python/CMakeFiles/RTF_python.dir/src/PythonPluginLoader.cpp.o] Error 1
make[1]: *** [src/plugins/python/CMakeFiles/RTF_python.dir/all] Error 2
make: *** [all] Error 2

Googling for similar errors it appears that the bug is caused by a mismatch between the Python versions:
https://bugs.launchpad.net/opencog/+bug/1063039
http://sourceforge.net/p/robotcub/mailman/robotcub-hackers/thread/[email protected]/

Probably now the Travis images have been update to have multiple versions of Python, and there is a bug in the CMake code handling Python versions. I will look into it.

In case you are debugging a failure of setting up a fixture using the yarpmanager, it is quite difficult to understand what is going on. This because the yarpmanager fixture manager does not print any information on which application it is launching, on which dependency is waiting, etc etc. It does not print anything even if the testrunner was launched with the verbose option.
This does make sense if using the yarpmanager application (because the user can check on the gui what is going on) but is a bit more complicated for use in rtf.
cc @apaikan What is a good strategy to add some (optional) verbose print to yarpmanager ? I saw that libYARP_manager is relativly yarp-indipendent, so I don't know it it make sense to use the yarp output macros in it.

Ref: robotology/icub-tests#8
Example in Travis: https://travis-ci.org/robotology/gazebo-yarp-plugins/jobs/62661440

According to @traversaro it would be important to control the order of execution of both tests and fixtures.

While usual unit testing has to be order independent, in a robot testing scenario it would be useful to test small components at first (like encoder response, plugin response etc) and then larger behaviors like for instance navigation and grasping.

For fixtures, it is necessary where some fixtures (speaking of the yarp framework a yarp device, yarp application or the yarpmanager itself for example) need other utilities (like yarp server) in order to start correctly.

As far as I know, this functionality is not available in RTF. The way to implement it is the following:
In my opinion, the simplest and most intuitive solution from both the user and the development side is to maintain the order of the xml tags (top-bottom). This can be done by simply change the tests and fixtures data storage for from std::set to std vector. While the loss in terms of performance is negligible in my opinion an inconsistency with the xml parsing policy of yarp (where only the depth order of the tags is relevant). However being RTF and YARP two completely separate frameworks this shouldn't be a problem.

TLDR

There is no way to control the fixture or test execution order in RTF while in some cases this is a requirement. Let's discuss on how to implement it.

If one of the modules of a fixture is not launched because its <dependencies> are not satisfied, yarpmanager consider the fixture to be successful anyway, launching the tests.

The description of this issue is still be completed.

Four tests:

1. joint position vs. motor position * Tmatrix
2. joint velocity vs. motor velocity * Tmatrix
3. joint position (derived) vs joint velocity
4. motor position (derived) vs motor velocity

Robots verification table:
http://wiki.icub.org/images/c/c5/Signs.xls

Matlab code:
available in icub-main\app\robots\experimentalSetups\test_utils\matlab\verify_encoders.m

For now, yarp middleware plugin in RTF depends on yarp. This means that that it will be very complicated to use RTF to develop units tests within yarp simply because of mutual dependency!

1) The solution suggested by @drdanz is to move the RTF yarp middleware plugin to the yarp repository. In this way one can:

• Install RTF on a machine
• Install yarp with RTF enabled (can be automatically done by cmake)
• compile and run yarp test units which are developed using RTF (e.g. make tests)

2) The second option is to have the yarp unit tests in a separate repository. This needs a bit of more work to compile and run the tests especially if we want to run the new tests (i.e. written using RTF) along with the available tests in YARP.

To simplify adding tests using testrunner to CTest using the add_test cmake command, the CMake infrastructure of RTF should provide a variable containing the absolute path to the testrunner binary.

Helper CMake functions to automatically add a CTest test from a suite would also be convenient.

cc @S-Dafarra

Coverall has been already setup for robot-testting (see .coveralls.yml). However the Travis file and the related compile flags should be configured to run the code coverage!

For now robot-testing supports just test that are hard coded (and directly linked) into the main application robot-testing.

This is undesirable for the following reasons:

• the users may want to test an own library, without linking it directly to robot-testing.
• the user have to modify the main cpp file for each test that they had, that is a bit awkward .

yarp has a nice feature of allowing dynamically loaded device drivers. I guess we can borrow most of this yarp device drivers infrastructure/logic, in a way that robot-testing will play a role similar to what yarpdev is doing for devices.
In a nutshell, the ideal scenario is the one where the CI jobs will simply call robot-testing testListSimulation.ini .
Similarly a human user that want to test a real robot will call robot-testing testListRobot.ini.

cc @lornat75 @apaikan

While configuring CMake, I get the following warning:

CMake Warning (dev) at src/rtf/CMakeLists.txt:8 (project):
   Policy CMP0048 is not set: project() command manages VERSION variables.
   Run "cmake --help-policy CMP0048" for policy details.  Use the cmake_policy
   command to set the policy and suppress this warning.

   The following variable(s) would be set to empty:

     RTF_VERSION
     RTF_VERSION_MAJOR
     RTF_VERSION_MINOR
     RTF_VERSION_PATCH
     RTF_VERSION_TWEAK
 This warning is for project developers.  Use -Wno-dev to suppress it.

During generation I get:

Policy CMP0042 is not set: MACOSX_RPATH is enabled by default.  Run "cmake
   --help-policy CMP0042" for policy details.  Use the cmake_policy command to
   set the policy and suppress this warning.

   MACOSX_RPATH is not specified for the following targets:

    RTF
    RTF_dll

 This warning is for project developers.  Use -Wno-dev to suppress it.

System: macOS 10.11.6 (El Capitan), CMake 3.7.0.

While compiling under macOS the following warning are raised:

src/rtf/src/Arguments.cpp:17:17: warning: using directive refers to implicitly-defined namespace
      'std'
using namespace std;
                ^

src/testrunner/src/SuitRunner.cpp:170:44: warning: comparison of unsigned expression >= 0 is
      always true [-Wtautological-compare]
                    if (endptr == 0 && rep >= 0) {
                                       ~~~ ^  ~

I am wondering that no one has wondered where are RTF self tests? 😜

A unit testing framework without unit tests .... 😶

TODO:

• Prepare structure (folders) to add unit test for RTF
• Prepare Cmake file so that make tests can run tests using testrunner
• Write some specific tests to test testrunner without testrunner!
• Add unit tests...

the RTF_TEST_FAIL_IF macro fail if the condition is false. example:
int a = 5;
int b = 2;
RTF_TEST_FAIL_IF(a > b) //this call does not fail as it should..

In a nutshell, I propose we move the test name from the main configuration file, as in:

[REFERENCES]
user "alessandro scalzo"
comment "this is a wild bunch of tests"
outfile iCubTestReport.xml

[TESTS]
TestCamera right_camera.ini
TestCamera left_camera.ini
TestMotors test_left_arm.ini
TestMotors test_right_arm.ini
TestFTSensors test_ft_left_arm.ini
TestFTSensors test_ft_right_arm.ini
TestFTSensors test_ft_left_leg.ini
TestFTSensors test_ft_right_leg.ini
TestFTSensors test_ft_left_foot.ini
TestFTSensors test_ft_right_foot.ini

to the test configuration file, so we will have:

[REFERENCES]
user "alessandro scalzo"
comment "this is a wild bunch of tests"
outfile iCubTestReport.xml

[TESTS]
right_camera.ini
left_camera.ini
test_left_arm.ini
test_right_arm.ini
test_ft_left_arm.ini
test_ft_right_arm.ini
test_ft_left_leg.ini
test_ft_right_leg.ini
test_ft_left_foot.ini
test_ft_right_foot.ini

This will permit to reuse more easily the same .ini file in multiple main configuration files, and is more consistent with the device use (test and device share a lot of commonality in my opinion).

RTF.testCheck in the binding language needs to be modified to come up with the recent change in RTF_TEST_CHECK macro from C++.

Just cloned the clean repo and I get 2 compilation errors:

1. src/UnitTest.c -> fabs not defined

2. TestMotors.cpp:365:52: error: ‘isTrue’ was not declared in this scope
   doneAll=isTrue(done_vector, m_NumJoints);

Am I the only one? :$

If the testrunner fails to find the plugin for a given tests, it prints the following erro message:

2: [testrunner] cannot load plugin myTest.so; error (load) : myTest.so: cannot open shared object file: No such file or directory

But the final output is:

2: ---------- results -----------
2: Total number of test suites  : 1
2: Number of passed test suites : 1
2: Number of failed test suites : 0
2: Total number of test cases   : 0
2: Number of passed test cases  : 0
2: Number of failed test cases  : 0

A test that was impossible to load should count as a failed test.

E.g.:

RTF_TEST_CHECK(value, "checking value")

should be logged as:

checking value...OK

or

checking value... NOT OK

At the moment the macro prints the string only if the tests fails.