Improved Unit Testing Feedback about opencl-icd-loader HOT 3 OPEN

I like the way you are testing the loader here. I think I have an idea to completely automate the generation of 95% of the test.
I have a couple of questions though, and one remark. Questions:

• Is there any reason to avoid generating a mockup ICD? This would allow testing a 100% of the API while staying generated. I imagine the problem would lie with Windows, as on unixes I don't see any obstacles.
• We could also use the same approach to create a test layer to check cases where the call can't dispatch, and the layer mechanism.
• Do you mind if I pick-up where you left it?

Remark:

• Using the same framework to generate both the loader and the tests, what is the intrinsic value of the tests? Wouldn't the tests have the same blind spots as the generated loader?

from opencl-icd-loader.

• Do you mind if I pick-up where you left it

No, not at all! I'm also happy to help if needed, just let me know what I can do.

• Using the same framework to generate both the loader and the tests, what is the intrinsic value of the tests? Wouldn't the tests have the same blind spots as the generated loader?

This is a very interesting question now that we're generating most of the loader...

from opencl-icd-loader.

Using the same framework to generate both the loader and the tests, what is the intrinsic value of the tests? Wouldn't the tests have the same blind spots as the generated loader?

One alternative is to test the generator itself, e.g. for input X does it produce expected output Y. The variety of patterns in cl.xml, however, means it would be hard to write tests with good coverage. Also if the bug is in cl.xml itself no test generation approach will catch it.

The current testing is pretty minimal. Pragmatically I think generated testing could still provide more safety than that, even with the risk of blind spots. For hand-written testing there is still the CTS providing more coverage, just with unknown timescales to discover issues because it's not suitable for use in CI here.

from opencl-icd-loader.