Comments (7)
Perhaps the test runner should be standalone code that communicates with the engine-under-test via http. So, the engine-under-test would need to spin up a small http server and accept incoming standardised models & requests, and provide a response.
I guess we'd need to establish what that request/response protocol looked like as it'd need to take into account providing a 'package' of models to test 'imports', not just standalone models.
from tck.
There is this, which may be used https://jpy.readthedocs.io/en/latest/
from tck.
This is exactly the same case like in DMNTK (Rust).
DMNTK engine loads all compatibility tests and exposes endpoints to evaluate them (there is no difference if one, two or hundreds of models are loaded).
Then, the separate test runner (separate application) reads the test cases and evaluates them, just like production deployed models. It happens all over HTTP using JSON API.
For DMNTK we have implemented such protocol like the one mentioned by @StrayAlien. It is currently based on test file schema. If you think it could be a good starting point, then I can share a JSON schema for this protocol and some explanations how it is done in DMNTK.
from tck.
Hi @dmntk , first of all, congratulations and well done on the Rust impl of DMN. Something I have long been considering doing. It ain't easy I imagine. I've done a Java and a Javascript (Typescript) impl and have really found the spec to be rather, well, Java centric. Date handling and BigDecimal in particular. I had often wondered whether the date/time/duration handing in Rust was capable. Seems maybe it is. Nice. But, that is another conversation. Nice to meet you, even by way of github.
Expressing an opinion here, but I think if we created a 'test runner' that invoked a runtime via a standardised (http) API spec then we'd only ever need one test runner for all languages and have a single way to test and report compliance.
I was imagining the API would accept the test model(s) also execute tests against it.
You mention JSON? The spec is XML Also, the 'standardarised' API would need to handle multiple models being transmitted to handle 'imports'. For the sake of naming things, maybe let's call it a 'package'. I had a quick stab a getting multiple XML models into a single XML 'package' file and it proved troublesome. Namespace collisions, id collisions. It was hard to verify with the schema. Etc.
I ended up thinking that multi-part mime might be the most flexible way to handle transmission of multiple models. Either that or a encoded zip. I'm not sure either are very tasteful, but given the schema complications of jamming multiple XML model definitions and associated namespaces into a single textual XML, other ways have to be considered. I say XML as we have no established standard for the representation of a DMN model as JSON.
That is, unless you are considering the API does not involve transmission of models, just requests and responses.
At any rate, glad to kick of discussions. And again, well done on DMNTK. Btw, It's great you're publishing your metrics.
from tck.
Hi @StrayAlien, thanks for your congratulations, there is a long way behind and there is still a long way ahead before production readiness (current version is 0.2.0, so I may say 20% is ready ;-)). Anyway, thanks again!
Few versions ago in DMNTK, I had already such implementation as you have decribed above. I was sending testing models to execution engine, deployed them and then tested with the separate runner via HTTP and JSON. Sending tested models was as simple as converting XML (it is just some text ;-)) to Base64 encoded string and POST'ing them as JSON payload. I have used this API for over two years during development.
But currently I dropped this solution for few reasons:
- security - when someone forgets to block such endpoint on production, then everyone may deploy any number of own models,
- performance - sending and deploying models takes more time than just loading them during server startup,
- rustification - Rust prevents many "Java natural" operations on data - so the code was too complicated to maintain REST deployable models (I was dreaming about turning back to Java then ;-)).
So, current solution in DMNTK is very simple (will be officially available in version 0.3.0):
- you start the execution engine pointing a directory with test cases, like ~/tck/TestCases,
- the engine loads all models (*.dmn files) and is ready for testing,
- test runner reads the test cases from all *.xml files and evaluates them.
The protocol I had in mind is the JSON format for transferring invocable's input data from test runner to execution engine and of course getting back the results.
So the general rule is:
- the execution engine is responsible for loading tested models and exposing dedicated endpoint for testing,
- the test runner with standarized protocol tests (evaluates) all deployed invocables.
With such approach, we could test any execution engine with a single test runner, that generates unified stats for all vendors ;-).
Anyway, this is just the way we did it in DMNTK, it may be useful for other engines (or not), but I guess, it is always a good thing to share some ideas ;-)
from tck.
Hi @dmntk , I had figured that is the approach you might be taking if not sending models. I've no arguments with it and seems reasonable. Though, we're not all Rustaceans (yet!) maybe an on-development of the existing test runner might be more accessible? Though, Tbh, I've never really looked at the existing Java test runner. I have my own, but I think a standardised one is the right thing to do as more languages seem to come on board.
from tck.
The common runner does not have to be written in Rust. Just the common API must be agreed between DMN engine implementations, but this is out of the scope of this discussion I guess. ;-)
from tck.
Related Issues (20)
- Clarify `null` handling in `FunctionInvocation` HOT 6
- Cover comparisons
- Create transitive import test cases based on snippets in DMN15-58 HOT 2
- Doubtful typing on overloaded built-in function in test HOT 2
- Report does not work under windows
- The test cases table breaks up around Compliance-Level-3
- New PRs ... but deleting them. The PRs have dragged in CL/LF differences HOT 2
- Patches to test cases HOT 4
- Confused about decision service encapsulatedDecisions - ambiguous - is ch11 example correct? HOT 5
- Task list for DMN 1.6
- Optional fn params vs null values HOT 4
- Question to test case `decision_015` in `0084-feel-for-loops.dmn` HOT 1
- Quesion to test case `nested009` in `1146-feel-context-put-function.dmn` HOT 2
- 0014-loan-comparison assumes that sorting keeps the original order HOT 2
- breaking change?: what do do with range equality? What is 'undefined'? HOT 4
- Is `[100] = 100` false or null? HOT 4
- Does @"P0D" = "P0Y" give `true` or `null` (or false?) ? HOT 3
- The new text for type conversions in "10.3.2.9.4 Type conversions" misses some use cases HOT 2
- Spec permits ranges to use new `=` and `!=` comparison operators - but it doesn't make much sense (to me) HOT 3
- TCK is using yearMonthDuration/dayTimeDuration/dateTime typeRefs HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tck.