convert our model(s) from onnx to tf about dianna HOT 9 CLOSED

use
https://github.com/onnx/onnx-tensorflow/graphs/contributors
first try it on our models.

from dianna.

Explored it with a single model (leafsnap) and predictions were equivalent to the original onnx. Script can be found here:
https://github.com/dianna-ai/dianna-exploration/blob/56-onnx-to-tf-exploration/onnx_conversion_scripts/onnx_to_tensorflow.ipynb

from dianna.

Can somebody who has time, please test the Tutorial notebook of onnx-tensorflow executing exactly the same notebook and just appending it with the shap step to see if it works?

from dianna.

That tutorial didn't work for me even without changing anything, the model they provide fails when running prepare (see also the discussion in the teams channel). I couldn't find which versions of e.g. tensorflow they used.

I did do some tests together with Chris and Yang, see this notebook: https://github.com/dianna-ai/dianna-exploration/blob/test-shap-tf-onnx/onnx_conversion_scripts/test_shap_tf_onnx.ipynb
We're getting further, i.e. we can generate an explainer object, but getting the shap values still fails.
shap also fails on a native keras model (see the same notebook), it looks like the DeepExplainer is not (fully) compatible with tensorflow 2, or at least not with recent versions.

from dianna.

2 remarks:

• Make a draft PR with a failing test at https://github.com/slundberg/shap/pulls.
• Post error messages with version details here to get help easier

from dianna.

Note that the relevant branches are in dianna-exploration. If it results in a PR we should not forget it in the stand-ups.

from dianna.

My interpretations of the errors we got in https://github.com/dianna-ai/dianna-exploration/blob/test-shap-tf-onnx/onnx_conversion_scripts/test_shap_tf_onnx.ipynb:

For our model- the warning after cell [21] about exporting a tensor dict might be related to the error later after cell [24]:

TypeError: Expected any non-tensor type, got a tensor instead.

Looks like versioning issues indeed. But maybe we can try not with mnist. but another or our models? I recall @geek-yang saying the mnist had a different structure after conversion than the others.

For the native Keras model: I can report our error also in their issue, hoping someone will pay attention. Also worth looking at how the skomatin guy changed his model and avoided the error (post from 11 Sep 2020). He thinks shap does not support TF 2.0 :-( @loostrum what shap and TF versions did you use?

from dianna.

One more tough: if DeepExplainer attempts ultimately fail, any point of trying the slower Shap's KernalExplainer?

https://github.com/slundberg/shap

from dianna.

I tried the latest version of shap (0.40), a slightly older version (0.38) and the latest master branch.
Using tensorflow 2.6

from dianna.