Need an indicator for ERROR test cases about tck HOT 9 CLOSED

Arthur mentioned that in their case, if the execution hits an error, that the test case will be marked as FAILED. I think this is a bug. If the test cases says it is expecting null, then hitting an error should not cause the test to be marked failed.

That is not a fault of the TCK, but the way how we implemented the logic to mark tests as SUCCESSFUL or FAILED.
I will change our TckRunner implementation to remove the check for runtime errors repoerted by the engine in our TckRunner.
Therefore it is not a problem of the TCK, but of our implementation.

from tck.

We had yet another chat about that topic in the office today.

Daniel and I came to the conclusion that error handling and identification of error scenarios is highly vendor-specific as the spec does not really mention error scenarios at all.

There are just two small hints in the spec regarding error handling:

10.3.2.13 Error Handling
When a built-in function encounters input that is outside its defined domain, the function SHOULD report or log diagnostic information if appropriate, and SHALL return null.

and

pattern, replacement, and flags SHALL conform to the syntax and constraints specified in clause 7.6 of XQuery 1.0
and XPath 2.0 Functions and Operators. Note that where XPath specifies an error result, FEEL specifies a null result

That opens up a huge space for vendors to come up with own interpretations what error scenarios are and what to do in error-cases.
A vendor could even decide to silently return null in case of errors and do no reporting about the cause at all. That wouldn't be ideal for business users, but still spec-compliant.

Therefore it is vendor-specific what actual error scenarios are: One vendor could allow cases where another interprets same or similar case as an error.

Conclusion: I thought the error flag would be useful. However, what we interpret as an error scenario might not be interpreted as such by other vendors.
Therefore we came to the conclusion that introducing an error flag in the TCK is still useful. But as of now only really useful for built-in functions where error conditions are clearly specified in the spec. Other than that, the goal of the TCK should be to check for spec compliance and help to clarify situations where the spec is not clear enough. I am not sure if the error-handling domain is a candidate for that. I guess it should rather be fixed in the spec itself.

In my opinion, the lack of specification ragarding error scenarios in DMN is a hindrance for true execution-compatibility between vendors.

from tck.

@arthware I mostly agree with you, but there is the possibility of us creating the flag to tests stating: the TCK expects an error to be raised. The vendor-specific runner can check, in whatever vendor-specific way, if there was an error raised or not.

Not saying we must to, but it is possible to do.

from tck.

I am not seeing the huge space for controversy. When the spec says that a function "SHALL return null" it is pretty clear that this means that the point of call RECEIVES a null, and carries on processing with the null, as if null had been a correct result.

The spec says that the function "SHOULD report or log diagnostic information if appropriate" it seems pretty clear to me that this diagnostic information should not change the behavior of the call site. Nothing is required to be reported, so it is clear that an engine does not need to report anything. More important is that it seems clear that this report is not supposed to be something that the caller can use to modify behavior. "diagnostic" seems like something for a system administrator or a programmer to look at, but not something that can be programmatically consumed.

To be clear, I don't like this behavior. I would prefer "exceptions" that stop all execution at the point of problem. But the spec does not say or imply this.

For the discussion today, we should consider a specific error situation that is a couple levels of nesting deep. We also should consider a decision table that has lots of expressions, some of which hit error conditions, and the effect on the evaluation of that table.

If my understanding is correct, then whether an error occurs or not should not change the flow of execution in any way. There should be a way, after the evaulation has finished, to ask the question "did an error occur during the course of evaulation" and get a boolean. It is also possible that we should be able to ask the question "how many errors occurred" and get a count. The engine should provide some diagnostic info on each error, however there is no way we can check that the diagnostic is "correct" or not. The test, however, should be able to state check to see that "at least one error occurred" and fail if it does not find any error occurred.

from tck.

The test, however, should be able to state check to see that "at least one error occurred" and fail if it does not find any error occurred.

Yes, that is what I meant on my comment.

For the discussion today, we should consider a specific error situation that is a couple levels of nesting deep.

Is there any difference between an error in a top level decision or nested inside boxed expressions/decision tables? Once an error occurs, the whole computation becomes invalid.

from tck.

Here is what we decided in the meeting today:

(1) we will add a bit of metadata that will indicate that the test is expected to encounter an error given the model and the set of inputs.
(2) The test runner will ignore this marker, and will not behave any differently regardless of whether an error was encountered or not. While the spec says that the engine SHOULD report an error, there will be no testing of whether the engine makes any indication at all of an error. We will not make a set of tests that test whether the engine does so, or does not do so. Specifically: if the test is marked as producing and error, and the engine does not produce an error, OR if the test is not marked as producing an error, and the engine does report an error, in NEITHER case will the reporting of an error be considered. The only thing tested will be the values that come out.
(3) People who are running the tests might decide to skip tests that encounter errors.
(4) The metadata will be a boolean value indicating that "at least one" error was encountered, and will not record the number of errors that are expected to occur during the running of the test.

from tck.

As we are currently working on the schema and marshaller module, we can can take care of updating the schema file to include the error boolean on testCase element.

from tck.

How about having more info about the error(s)? A list of error messages seems a bit more generic to me.

I believe, DMN has annotations at decision level to track the execution (see DMN metamodel). AFAIK the annotations are collected across the entire decision tree.

from tck.

The original suggestion was to add a boolean indicator and that was completed.

Today we discussed briefly the idea of also having / collecting / maintaining specific error messages, list or otherwise. Nobody in the meeting wanted to go in that direction. The spec itself does not state any specific error message details, and it is not clear what is to gain by the TCK developing a standardized list of error messages.

from tck.