Comments (6)
We should change the term "skip" for "not implemented"
This way the semantic will be clear.
from tck.
"skipped" is currently used if the vendor doesn't support this feature. Therefore we should rename it to "unsupported". "not implemented" would be also okay.
Additionally we can introduce a new category: "currently under development" or "available in future releases" This category can be used to signal that the vendor has planned to support this feature in the near future.
If a vendor does not publish results for all test cases, unpublished test results are currently marked as "skipped". What about using the new category "unknown"?
Overview of all categories:
- unsupported
- unknown
- available in future releases
- succeeded
- failed
from tck.
I agree that we should re-vamp the statuses and I like the suggestions from @DanielThanner , except for the "available in future releases". Unfortunately, I am afraid that this would then become the new go to status and would just replace "skipped" for all tests not supported by a vendor.
I think the following list provides proper reflection of what the current status of each product is:
- unsupported
- unknown
- succeeded
- failed
from tck.
As discussed in the call, the following statuses will be supported going forward:
- result missing: the vendor did not submit a result for the test
- succeeded: the test runs successfully
- failed: the test fails on execution
- not supported: the test is not supported
from tck.
Submitted PR #189 to implement the change discussed here. Please note that at the moment I did not make any change to the way vendors report results, as it wasn't necessary. I just changed the way the website reported the results:
- succeeded
- failed
- not supported
- missing
For not supported tests, vendor will continue to report the status on the results file as "IGNORED", to avoid changing their runners and old reported files. On the website, these tests will show as "not supported".
Please review the PR and if approved we can close this ticket.
from tck.
PR has been merged that takes care of this.
from tck.
Related Issues (20)
- Clarify mixed type workings of `+` operator HOT 3
- Clarify `null` handling in `FunctionInvocation` HOT 6
- Cover comparisons
- Create transitive import test cases based on snippets in DMN15-58 HOT 2
- Doubtful typing on overloaded built-in function in test HOT 2
- Report does not work under windows
- The test cases table breaks up around Compliance-Level-3
- New PRs ... but deleting them. The PRs have dragged in CL/LF differences HOT 2
- Patches to test cases HOT 4
- Confused about decision service encapsulatedDecisions - ambiguous - is ch11 example correct? HOT 5
- Task list for DMN 1.6
- Optional fn params vs null values HOT 4
- Question to test case `decision_015` in `0084-feel-for-loops.dmn` HOT 1
- Quesion to test case `nested009` in `1146-feel-context-put-function.dmn` HOT 2
- 0014-loan-comparison assumes that sorting keeps the original order HOT 2
- breaking change?: what do do with range equality? What is 'undefined'? HOT 4
- Is `[100] = 100` false or null? HOT 4
- Does @"P0D" = "P0Y" give `true` or `null` (or false?) ? HOT 3
- The new text for type conversions in "10.3.2.9.4 Type conversions" misses some use cases HOT 2
- Spec permits ranges to use new `=` and `!=` comparison operators - but it doesn't make much sense (to me) HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tck.