Giter VIP home page Giter VIP logo

colcon-test-result's Introduction

colcon-test-result

An extension for colcon-core to provide information about the test results.

colcon-test-result's People

Contributors

cottsay avatar dirk-thomas avatar jpsamper2009 avatar mikaelarguedas avatar nuclearsandwich avatar pbaughman avatar rotu avatar

Stargazers

avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar

Forkers

apexai pbaughman aws-ros-dev roverrobotics-forks hakuturu583 claireyywang tier4 serafadam maspe36 andre-nguyen arpitjain799 rhaschke

colcon-test-result's Issues

Add --clean option

Motivation

  • It would be nice to have a shorthand to delete test result files
  • With --result-files-only option, it is possible to pipe the list of files into rm; however, this now produces a warning: e.g.
[0.560s] WARNING:colcon.colcon_cmake.test_result.ctest:Skipping 'build/rcutils/Testing/TAG': could not find latest XML file 'build/rcutils/Testing/20190614-2050/Test.xml'
  • A --clean option would take care of cleaning up files like build/rcutils/Testing/TAG in additional to the result files

Acceptance Criteria

  • A way to clean up test result files

colcon test-result 'skipped' count is inaccurate

When you run colcon test-result you get a little summary of all the tests run:

pete.baughman@ade:~/ws$ colcon test-result
Summary: 7644 tests, 0 errors, 4 failures, 1 skipped

The 'skipped' test field misses pytests skipped with the @skip decorator, and gtests skipped with the GTEST_SKIP macro.

The 1 skip in the above report is from test_interfaces_cpp.gtest.xml. You can see from the XML, this is actually a 'disabled' test

<testsuites tests="21" failures="0" disabled="1" errors="0" timestamp="2019-04-22T12:32:50" time="0" name="AllTests">
  <testsuite name="Test_rosidl_generator_traits" tests="2" failures="0" disabled="0" errors="0" time="0">
    <testcase name="has_fixed_size" status="run" time="0" classname="rosidl_generator_cpp.Test_rosidl_generator_traits"/>
    <testcase name="has_bounded_size" status="run" time="0" classname="rosidl_generator_cpp.Test_rosidl_generator_traits"/>
  </testsuite>
  <testsuite name="Test_messages" tests="19" failures="0" disabled="1" errors="0" time="0">
    <testcase name="primitives_static" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="primitives_static_arrays" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="primitives_bounded" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="primitives_unbounded" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="static_array_static" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="static_array_bounded" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="static_array_unbounded" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="bounded_array_static" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="bounded_array_bounded" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="bounded_array_unbounded" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="unbounded_array_static" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="unbounded_array_bounded" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="unbounded_array_unbounded" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="primitives_constants" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="primitives_default" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="string_arrays_default" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="DISABLED_Test_bounded_strings" status="notrun" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="Test_string" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
    <testcase name="Test_string_array_static" status="run" time="0" classname="rosidl_generator_cpp.Test_messages"/>
  </testsuite>
</testsuites>

This particular test is skipped by prefixing the name with DISABLED_
from rosidl_generator_cpp/test/test_interfaces.cpp

 // TODO(mikaelarguedas) reenable this test when bounded strings enforce length
TEST(Test_messages, DISABLED_Test_bounded_strings) {
  rosidl_generator_cpp::msg::StringBounded message;
  TEST_STRING_FIELD_ASSIGNMENT(message, string_value, "", "Deep into")
  std::string tooLongString = std::string("Too long string");
  message.string_value = tooLongString;
  tooLongString.resize(BOUNDED_STRING_LENGTH);
  ASSERT_STREQ(tooLongString.c_str(), message.string_value.c_str());
}

Skipped pytests have XML that looks like this:

<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="0" name="pytest" skipped="6" tests="175" time="40.009">
  . . .
  <testcase classname="rclpy.test.test_timer" file="test/test_timer.py" line="150" name="test_timer_zero_callbacks100hertz" time="0.001">
    <skipped message="&lt;Skipped instance&gt;" type="pytest.skip">/usr/local/lib/python3.5/dist-packages/_pytest/nose.py:29: &lt;Skipped instance&gt;</skipped>
  </testcase>
  . . .
</testsuite>

Suggested Fix

Tests that are skipped with pytest @unittest.skip have an additional child element under the skipped testcase that says the skipped reason. I have a tool that contains logic sort of like this:

# Things get a little messy down at the test case level.  Some test cases have a "status"
# attribute we can inspect - but it's optional
if 'status' in case.attrib:
    status = case.attrib['status']
    if status == 'notrun' or status == 'skipped':
        return True
# Pytest XML has a 'skipped' element inside the test case
if case.findall('skipped'):
    return True
return False

We could also look for the skipped attribute on the testsuite element and only drill down to the test cases if it's not present.

Add option to get list of results file

Motivation

  • It would be nice to have a way to get a list of all the test result files that colcon test-result uses, such that the list can be fed into other tools that parse them
  • colcon test-result --all already gives a list of the XML files with some information about them, so it shouldn't be too hard to add, say, a --result-files-only which will print the files only and return with exit code 0
  • Our current workaround (which isn't elegant) is to use:
(colcon test-result --all || true) | grep xml | cut -d":" -f1

Goal

  • Provide a simple way to get a list of test result files

test-result produces incorrect counts

Hi, I'm running colcon test-result in my workspace but it looks like tests are counted multiple times.

$ colcon build
$ colcon test

then

$ colcon test-result --test-result-base ./_ws/build
_ws/build/hello_world/Testing/20211120-1117/Test.xml: 5 tests, 0 errors, 1 failure, 0 skipped
_ws/build/hello_world/test_resultshello_world/test_hello.gtest.xml: 1 test, 0 errors, 1 failure, 0 skipped

Summary: 19 tests, 0 errors, 2 failures, 0 skipped

My workspace only has 5 tests and only 1 of them is failing.

similarly other commands

$ colcon test-result --test-result-base ./_ws/build --all
_ws/build/hello_world/Testing/20211120-1117/Test.xml: 5 tests, 0 errors, 1 failure, 0 skipped
_ws/build/hello_world/test_results/hello_world/cppcheck.xunit.xml: 4 tests, 0 errors, 0 failures, 0 skipped
_ws/build/hello_world/test_results/hello_world/cpplint.xunit.xml: 4 tests, 0 errors, 0 failures, 0 skipped
_ws/build/hello_world/test_results/hello_world/lint_cmake.xunit.xml: 1 test, 0 errors, 0 failures, 0 skipped
_ws/build/hello_world/test_results/hello_world/test_hello.gtest.xml: 1 test, 0 errors, 1 failure, 0 skipped
_ws/build/hello_world/test_results/hello_world/uncrustify.xunit.xml: 4 tests, 0 errors, 0 failures, 0 skipped

Summary: 19 tests, 0 errors, 2 failures, 0 skipped

If i inspect the test logs however

$ cat _ws/log/latest_test/hello_world/stdout.log
.....
The following tests passed:
	cppcheck
	uncrustify
	cpplint
	lint_cmake

80% tests passed, 1 tests failed out of 5

Label Time Summary:
cppcheck      =   0.50 sec*proc (1 test)
cpplint       =   0.46 sec*proc (1 test)
gtest         =   0.23 sec*proc (1 test)
lint_cmake    =   0.42 sec*proc (1 test)
linter        =   1.80 sec*proc (4 tests)
uncrustify    =   0.42 sec*proc (1 test)

Total Test time (real) =   2.04 sec

The following tests FAILED:
	  5 - test_hello (Failed)

use '.' instead of build for default base_path

  • Inconvenient for me as a rarely use build as the name of the build folder
  • It diverges from ament tools
  • It diverges from the build and test verbs (that uses '.' by default)

Alternative: Update the readthedocs "ament" documentation to reflect the change in behavior

I don't feel strongly about which approach we take and the second one may be the easiest to change

If passing verbose and result-files-only, colcon test-result still shows the test results

I expect colcon test-result --result-files-only --verbose to show only the results file, but it lists the result files and the errors as well.

This limits the usefulness of --result-files-only when you have verbose: true in your defaults.yaml.

colcon test-result --result-file --verbose            
build/nav2_map_server/Testing/20190730-0331/Test.xml
build/nav2_map_server/test_results/nav2_map_server/test_occ_grid.gtest.xml
- nav2_map_server.MapLoaderTest loadInvalidFile
  <<< failure message
    /home/snuc/ros2_ws/src/navigation2/nav2_map_server/test/unit/test_occ_grid.cpp:218
    Expected: map_loader_->loadMapFromFile(loadParameters) throws an exception of type std::runtime_error.
      Actual: it throws a different type.
  >>>
build/nav2_map_server/test_results/nav2_map_server/uncrustify.xunit.xml
- nav2_map_server.uncrustify src/occ_grid_loader.cpp
  <<< failure message
    Diff with 26 lines
  >>>

Full help text isn't available if result dir doesn't exist

Reproduce by invoking colcon test-result --help in an empty directory.

Expected:

$ colcon test-result --help
usage: colcon test-result [-h] [--test-result-base TEST_RESULT_BASE] [--all]
                          [--result-files-only] [--verbose] [--delete]
                          [--delete-yes]

Show the test results generated when testing a set of packages.

optional arguments:
  -h, --help            show this help message and exit
  --test-result-base TEST_RESULT_BASE
                        The base path for all test results (default: build)
  --all                 Show all test result files (even without errors /
                        failures)
  --result-files-only   Print only the paths of the result files. Use with
                        --all to include files without errors / failures
  --verbose             Show additional information for errors / failures
  --delete              Delete all result files. This might include additional
                        files beside what is listed by --result-files-only. An
                        interactive prompt will ask for confirmation
  --delete-yes          Same as --delete without an interactive confirmation

Actual:

$ colcon test-result --help
usage: colcon test-result [-h] [--test-result-base TEST_RESULT_BASE] [--all]
                          [--result-files-only] [--verbose] [--delete]
                          [--delete-yes]
colcon test-result: error: argument --test-result-base: Path 'build' does not exist

Cannot set `--test-result-base` via mixin

Steps to Reproduce

I'm trying to use a mixin for the test-result verb that sets --test-result-base argument. When running colcon test-result in my workspace, the mixin value for the argument is ignored and colcon exits early.

linux.mixin

{
    "build": {
        "linux": {
            "build-base": linux/build,
            "install-base": linux/install,
        }
    },
    "test": {
        "linux": {
            "build-base": linux/build,
            "install-base": linux/install,
            "test-result-base": linux/build,
        }
    },
    "test-result": {
        "linux": {
            "test-result-base": linux/build,
            "verbose": true
        }
    }
}

I have tried with and without this argument in the test verb, as well as having this arg present in test but not in test-result. They both produce the error. Not 100% sure what the difference is but it doesn't seem to affect this issue. I see this PR and linked issue but it doesn't shed much light to me as to why this arg is different from test-results

defaults.yaml

{
    "build": {
        "symlink-install": true,
        "mixin": ["linux"],
    },
    "test": {
        "mixin": ["linux"],
    },
    "test-result": {
        "mixin": [ "linux" ],
    }
}

My mixin is loaded correctly via colcon mixin

$ colcon mixin list
my_ws: file:///opt/mixins/index.yaml
- /home/docker/.colcon/mixin/my_ws/linux.mixin

And finally running colcon test-result

$ colcon test-result
usage: colcon test-result [-h] [--test-result-base TEST_RESULT_BASE] [--all] [--result-files-only] [--verbose] [--delete] [--delete-yes] [--mixin-files [FILE ...]]
                          [--mixin [mixin1 [mixin2 ...]]]
colcon test-result: error: argument --test-result-base: Path 'build' does not exist

If I set the value via my defaults.yaml everything works.

My Root Causing

Now I've done a bit of root causing and I believe the issue is because after parsing the defaults.yaml colcon will attempt to parse the current args before loading the mixin arg values.

This error is then being thrown because in this package, --test-result-base is registered with a custom type which throws an exception if a path doesn't exist

colcon-test-result/colcon_test_result/verb/test_result.py

Line 30 in 5bbb7bf

type=_argparse_existing_dir,

I did a search on github to see if any other colcon path arguments had this same issue and it seems like this one is unique
https://github.com/search?q=org%3Acolcon+type%3D_argparse_existing_dir%2C&type=code

It seems like the fix is to remove this type parameter when registering the type, but I'm not sure if test-result depends on that behavior somehow.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.