Comments (6)
Discussed. Need to get input from Camunda on why the runner is not reporting failures
from tck.
The default in our runner is that failing tests are reported as skipped, because we know that one test case is currently failing because of an unsupported feature and three or four others are limitations of our current runner, but not of the engine. This style of reporting is actually helpful in order to not fail the validation of a new test because of known issues with our runner or engine. Feel free to exclude our runner from the continuous integration until we changed the runner. The best solution for including the our runner in the continuous integration seems to me that we only report the known issues as skipped and fail the validation for unknown issues which then would be related to the new or changed test of the pull request.
from tck.
@falko thanks for the feedback, I would like to be sure I fully understand your proposal, in order to attempt to include from the beginning all runners in the CI with confidence; how do you define:
known issues
, please? Is there a way these known issues are currently marked in the code base, can you kindly explain in a bit more detail or provide some reference/example ? Thanks!
from tck.
Known issues in the runner mean that we know from previous runner generations that the engine passes the tests, but a refactoring in the current runner makes the tests fail. The fix for these tests is unfortunately not easy with the new runner architecture.
However, we do not report success for these known issues because the current runner cannot prove it. Instead, we report skipped and unfortunately is done for the new test case in the pull request.
You can see that in the JUnit output:
master branch test result: Tests run: 1158, Failures: 0, Errors: 0, Skipped: 7
pull request test result: Tests run: 1162, Failures: 0, Errors: 0, Skipped: 8
Skipped tests do not affect the overall build result, that's why Travis shows the build as successful.
Let me rephrase the last sentence of my last sentence: The best solution for including our runner in the continuous integration seems to me that we only report the test cases known to fail with Camunda as skipped and fail the validation for new test cases which then would be related to the new or changed test of the pull request.
from tck.
@tarilabs I changed our runner as discussed.
from tck.
Resolved with PR #166 merged.
from tck.
Related Issues (20)
- Clarify mixed type workings of `+` operator HOT 3
- Clarify `null` handling in `FunctionInvocation` HOT 6
- Cover comparisons
- Create transitive import test cases based on snippets in DMN15-58 HOT 2
- Doubtful typing on overloaded built-in function in test HOT 2
- Report does not work under windows
- The test cases table breaks up around Compliance-Level-3
- New PRs ... but deleting them. The PRs have dragged in CL/LF differences HOT 2
- Patches to test cases HOT 4
- Confused about decision service encapsulatedDecisions - ambiguous - is ch11 example correct? HOT 5
- Task list for DMN 1.6
- Optional fn params vs null values HOT 4
- Question to test case `decision_015` in `0084-feel-for-loops.dmn` HOT 1
- Quesion to test case `nested009` in `1146-feel-context-put-function.dmn` HOT 2
- 0014-loan-comparison assumes that sorting keeps the original order HOT 2
- breaking change?: what do do with range equality? What is 'undefined'? HOT 4
- Is `[100] = 100` false or null? HOT 4
- Does @"P0D" = "P0Y" give `true` or `null` (or false?) ? HOT 3
- The new text for type conversions in "10.3.2.9.4 Type conversions" misses some use cases HOT 2
- Spec permits ranges to use new `=` and `!=` comparison operators - but it doesn't make much sense (to me) HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tck.