What is the meaning of "Skipped?" about tck HOT 6 CLOSED

We should change the term "skip" for "not implemented"
This way the semantic will be clear.

from tck.

"skipped" is currently used if the vendor doesn't support this feature. Therefore we should rename it to "unsupported". "not implemented" would be also okay.

Additionally we can introduce a new category: "currently under development" or "available in future releases" This category can be used to signal that the vendor has planned to support this feature in the near future.

If a vendor does not publish results for all test cases, unpublished test results are currently marked as "skipped". What about using the new category "unknown"?

Overview of all categories:

• unsupported
• unknown
• available in future releases
• succeeded
• failed

from tck.

I agree that we should re-vamp the statuses and I like the suggestions from @DanielThanner , except for the "available in future releases". Unfortunately, I am afraid that this would then become the new go to status and would just replace "skipped" for all tests not supported by a vendor.

I think the following list provides proper reflection of what the current status of each product is:

• unsupported
• unknown
• succeeded
• failed

from tck.

As discussed in the call, the following statuses will be supported going forward:

• result missing: the vendor did not submit a result for the test
• succeeded: the test runs successfully
• failed: the test fails on execution
• not supported: the test is not supported

from tck.

Submitted PR #189 to implement the change discussed here. Please note that at the moment I did not make any change to the way vendors report results, as it wasn't necessary. I just changed the way the website reported the results:

• succeeded
• failed
• not supported
• missing

For not supported tests, vendor will continue to report the status on the results file as "IGNORED", to avoid changing their runners and old reported files. On the website, these tests will show as "not supported".

Please review the PR and if approved we can close this ticket.

from tck.

PR has been merged that takes care of this.

from tck.