Perform optional and opt-in DMN engines TCK Runners during Continuous Integration about tck HOT 6 CLOSED

Discussed. Need to get input from Camunda on why the runner is not reporting failures

from tck.

The default in our runner is that failing tests are reported as skipped, because we know that one test case is currently failing because of an unsupported feature and three or four others are limitations of our current runner, but not of the engine. This style of reporting is actually helpful in order to not fail the validation of a new test because of known issues with our runner or engine. Feel free to exclude our runner from the continuous integration until we changed the runner. The best solution for including the our runner in the continuous integration seems to me that we only report the known issues as skipped and fail the validation for unknown issues which then would be related to the new or changed test of the pull request.

from tck.

@falko thanks for the feedback, I would like to be sure I fully understand your proposal, in order to attempt to include from the beginning all runners in the CI with confidence; how do you define:

known issues

, please? Is there a way these known issues are currently marked in the code base, can you kindly explain in a bit more detail or provide some reference/example ? Thanks!

from tck.

Known issues in the runner mean that we know from previous runner generations that the engine passes the tests, but a refactoring in the current runner makes the tests fail. The fix for these tests is unfortunately not easy with the new runner architecture.

However, we do not report success for these known issues because the current runner cannot prove it. Instead, we report skipped and unfortunately is done for the new test case in the pull request.

You can see that in the JUnit output:
master branch test result: Tests run: 1158, Failures: 0, Errors: 0, Skipped: 7
pull request test result: Tests run: 1162, Failures: 0, Errors: 0, Skipped: 8

Skipped tests do not affect the overall build result, that's why Travis shows the build as successful.

Let me rephrase the last sentence of my last sentence: The best solution for including our runner in the continuous integration seems to me that we only report the test cases known to fail with Camunda as skipped and fail the validation for new test cases which then would be related to the new or changed test of the pull request.

from tck.

@tarilabs I changed our runner as discussed.

from tck.

Resolved with PR #166 merged.

from tck.