TCK Observations / notes / questions about tck HOT 13 CLOSED

I tested a few DMNs against the schema and some failed

Can you kindly list actually which ones, please?
As we have Travis CI performing schema check for all DMN models in the suite, and no error is currently reported as far as I know. Thanks.

from tck.

Hi tarilabs, a 5 year absence from the java ecosystem has my Java/xsd kung-foo a little rusty. But both xmlspy and a small 'check a schema' app fail on '0003-iteration.dmn'. The 'variable' element for the bkm "PMT2" is out of sequence. It is the first I one checked at random. Like I said, a bit out-of-practice on Java but I thought it worth noting - could be me and my setup.

Greg.

EDIT: I should note I am using DMN12.xsd against the current dmn.

from tck.

EDIT: I should note I am using DMN12.xsd against the current dmn.

That is the problem. Currently TCK suite is based on DMNv1.1 and the migration work is currently in-progress on a PR branch here: #191

Can you kindly confirm there is NO schema validation problem using the correct XSD, meaning DMNv1.1 as embedded here: https://github.com/dmn-tck/tck/blob/master/TestCases/dmn.xsd
or available here: https://www.omg.org/spec/DMN/1.1
(is the same dmn.xsd file)

please?

from tck.

Thanks, (Matteo?),

That looks like it. Using 1.1 passes. Thanks.

Greg.

from tck.

0003 has a typeRefs of 'tck:tEmploymentStatus' and 'tns:tLoan' but they are not actually defined. There is a type 'tEmploymentStatus' and a 'tLoan' defined, but neither bound to the 'tck' namespace.

I cannot locate tck:tEmploymentStatus in any file of TestCases/compliance-level-3/0003-iteration directory.

Then to what concerning: tns:tMonthlyPaymentList and tns:tLoan etc. I will make example with tns:tMonthlyPaymentList:

This typeref:

is using (DMNv1.1) prefix tns: which for this model is bound to:

which is the same as this model DMN's namespace.

This means it is indeed referencing the DMN ItemDefinition defined in this model:

Please notice because in this DMN XML file the default XML namespace is bound to:

you MUST use the tns: prefix otherwise it would need to look-up ItemDefinition in the default namespace which would be the DMN-specification definition namespace, and not this model namespace.

Does this clarify, please?

• Matteo

from tck.

Matteo, thank you very much. Yes, that does clarify. I am certainly going to owe you some beers after this I can tell.

The 0003 is my mistake. tns:tLoan is in 0003 in L3, and tck:tEmploymentStatus is in 0003 in L2. I got them 'together' in my notes. But, many thanks.

That also does clear some fog around how namespaces are used in DMN. I had been looking and looking but missed the relationship between the definition 'namespace' attribute and another xmlns prefix value.

Thank you again.

May I ask what is the policy when a typeRef does not exist at all? L3 test '0031-user-defined-functions' the typeRef 'ns1:tFnInvocationResult' is used twice. The 'ns1' parts defined as per your explanation above, but the 'tFnInvocationResult' is not defined anywhere - namespace or not.

Greg.

from tck.

Greg, thank you for the extensive analysis that you did. This is very useful to start addressing some of the gaps we have in the tests and Matteo already started looking into some of the cases you mentioned.

Just to catch you up on the current status of things, the goal of this TCK is to promote compatibility between vendor implementations, and while the DMN spec is quite good overall, there are some problems on it that we (vendors) found while implementing the spec. These are being addressed in the RTF (Review Task Force) group, but meanwhile, it makes it impossible for us to test.

Also, the spec is not clear on how to report errors, so we made some assumptions and agreed on ways to handle it here in the TCK, but as you noticed the main focus of the TCK has been on the positive cases. We recently started adding negative cases to some areas, but we are a bit more careful on those because there is wiggle room in the spec on how vendors should handle such cases.

Finally, as Matteo already mentioned, we are at the end of the DMN 1.1 cycle, but all our tests are currently targeting DMN 1.1. We are in progress of migrating to DMN 1.2, and our goal is to start working with DMN 1.2 in January/2019.

Regarding some of your observations on test coverage:

• no DecisionService coverage
  DecisionService was not implementable in DMN 1.1 (bug in the spec). We will add coverage as we move to DMN 1.2 where this was fixed.

• no equivalence coverage
  What are you referring to? If this is about the singleton list equivalence, a.k.a., the a = [a] equivalence, it was found to be ambiguous and removed in DMN 1.2, so we avoid testing it in DMN 1.1.

• no conformance coverage
  What do you mean by conformance? We found that conformance level 1 and 2 are pretty much useless for real use cases other than for documentation purposes. We do have some conformance level 2 tests, but honestly, conformance level 3 is what really matters for all practical purposes.

• no import coverage
  Import was not implementable in DMN 1.1 (bug in the spec). We will add coverage as we move to DMN 1.2 where this was fixed.

• very little add/subtract coverage beyond numbers and strings
  Agreed, we need to add coverage on this.

Finally, as you know, this TCK is community driven. We will start adding tests for the gaps you mentioned, but it would be great if you can help as well sending PRs with tests to help cover the gaps you mentioned.

We also have a weekly call where we discuss next steps and pending PRs/tickets. You are welcome to join us (if you haven't done it yet).

from tck.

Hi Edson,

Thanks for the response, and the background. Important stuff.

Re errors: Yes, I had noticed that. In the absence of specific guidance of things like that then perhaps 'convention' plays a part. As implementors, if we agree on a common way then almost as good. Another example of this is the string() function and what it outputs. Not in the spec, but likely worth establishing a convention for. Perhaps a third test folder called 'conventions' that are tests for things outside the spec but a there in good faith to help with interoperability.

Re the audit: Thanks. My intention here is not to drop work on everybody's lap! I'll be doing some of those in time (I still have some other things to get done - likely porting the prototype from Typescript to Java). As I've not been involved as yet I didn't want to start with a bunch of PRs without first verifying that the work wasn't already underway, or I wasn't being stupid. I am happy to contribute - it is why I've been buggin' you all. :-)

To the specifics:

• Decision Service. Cool. Thanks.

• equivalence - 1.2 tables 47 and 48 "Equality and Inequality". I believe wherever the '=' is used, FEEL is to perform a deep equality check, not a shallow check like a '==' in (say) Java. Lists and contexts are compared 'deeply'. I believe this should be in effect for the '=' operator in FEEL but also for other stuff like list functions:

these should all use deep equality and not '==' style checking:
"indexof"
"contains
"distinct value"
"union"

Likely most Java implementors have used .equals() for same, but I was noting there is no coverage asserting deep equality, only shallow.

Re 'a = [a]'. Thanks, I thought that was in!! And yes, it creates nasty ambiguities. But ... the coercion of singleton list to a value where a list is not expected is still in yes? Like uppercase(["foo"]) should result in "FOO".

• Conformance

1.2 section "10.3.2.9.2 Type Conformance". Types conform to each other (or not!). I may be wrong but my take on it is this: If, say, an Invokable formal param is for type 'X' then I believe the spec says I should be to pass it 'X' or a conforming type as a param. For example, if it requires a context param type like context<age:number, name:string> and I wish to invoke it passing in a context param like {name: 'foo', age: 10, bar: 'bar'} then that is okay, but passing it {name: 'foo'} is not - it does not meet the type expectations of the Invokable param.

I had a feeling it may have also be required to keep typed lists homogeneous - and thus used in a few list operation like 'insert before' and so on to enforce type safety. But as lists are immutable I think the emphasis for type checking is at the contractual level of invocation and return types - that is if (say) the formal params of an Invokable says it requires a list then that needs to be checked. Likewise, if the invokable variable type is a list them the data going into that variable needs to be checked for type conformance. There is a test that verifies that a singleton list is unwrapped to conform to a variable type but nothing deeper there.

This is what I mean. Hopefully that makes sense.

Re L1/l2 vs L3 'conformance'. Agree. I wouldn't be spending time on anything by L3.

• Re import: great, coz I dont quite 'get' it all as yet. :-)

• Re the weekly call. Thanks. Keith has added me to that. I think it is at 3am on Saturday morning my time .....

Again, many thanks.

Greg.

PS. I bought the book. Well done to you both. Thanks.

from tck.

Greg,

I think I understand now, you were looking at the DMN 1.2 spec. :)

• equivalence: ok, we need to improve on that area then

Re 'a = [a]'

yes, what is defined in the 1.2 spec will be supported and needs to have tests added, but that is quite different from what was in 1.1, and so we avoided testing it in 1.1. We actually removed a couple tests we had for that when some vendors complained.

Type Conformance

this is all new to 1.2. We need to add tests for that once we move to DMN 1.2

"I think it is at 3am on Saturday morning my time"

:) ouch, where are you based? Australia/New Zealand?

PS. I bought the book. Well done to you both.

Thanks! :)

from tck.

Thanks Edson, yes - 1.2 all the way. I started reading the 1.1 spec then saw then was a 1.2 up for release and got that. I never looked at 1.1 again. :-) I didn't realise there were such deep differences.

Glad about a = [a] going. Too confusing. Though singleton array coercion is not too bad.

Is the embedding of a anon function definition (lambda?) 'inline' in a literal expressions one of those differences? As I read it, the 1.2 spec grammar has it that a function definition is not in a textual expression, only a boxed expression - but there are tests for sort() in literal expressions passing anon lambdas. As far as I can tell, and I am no expert, that makes parsing a sort() expression darn hard if not impossible when the type is not known and the lambda has var names with spaces, '+' signs etc. Even more so when the params are named, and the 'list' param is last ...

Re the book. It is well done and a good read - I learned a lot from it and my copy has notes in the margins re things I didn't quite get in the spec. It also shows what a power combination FEEL and boxed-expressions can be. Thank you.

I am in Melbourne, Australia.

Greg.

from tck.

Is the embedding of a anon function definition (lambda?) 'inline' in a literal expressions one of those differences?

No, anonymous functions are supported and have not changed since 1.1. The only place they are used in the core API though is in the sort() function. It is possible for a user to assign a name to a function as part of a context and pass it around, the same way that you would do in a functional language.

It is possible as well to pass a BKM name (that is a function after all) as the parameter for the sort() function. For instance, in this old example:

http://blog.athico.com/2017/01/dmn-runtime-example-with-drools.html

"passenger priority" is a BKM (i.e. a function) with a decision table that defines the relative priority between 2 passengers. The "Prioritized Waiting List" decision uses that to invoke the sort() function like:

sort( Waiting List, passenger priority )

We don't have many tests for inline functions because most vendors are still working on its support, and it is somewhat an edge feature. The idea is to add coverage over time.

parsing a sort() expression darn hard

It is not that hard, once you get the grammar in place.

As I read it, the 1.2 spec grammar has it that a function definition is not in a textual expression, only a boxed expression

That is just the name of the grammar rule. FEEL supports any syntax that is parseable by its grammar.

I learned a lot from it and my copy has notes in the margins re things I didn't quite get in the spec

Glad to hear. If you have specific questions feel free to send me an e-mail.

from tck.

Thanks Edson, I get the lamda, context function, BKM as function thing - no issues there - maybe apart creating stuff that is hard for people to follow. sort() is easy to parse when things are 'knowable' I guess. Rather than overload this thread, I'll start a new one.

from tck.

closing. Still lots lots do, but this will be here in issue history. I have started on some if the things noted here.

from tck.