Validation accuracy not improving? about tensorflow-directml HOT 7 OPEN

@jstoecker thanks for the answer really appreciate it. Unfortunately i cannot share the dataset because of restriction. i tried it as well with tensroflow-rocm and get result closer to cpu tensorflow. I'll try this again once new release is up and give a followup

from tensorflow-directml.

Longer train session

ResNet50 with tensorflow 2 cpu:

ResNet50 with tensorflow 1.15 gpu:

from tensorflow-directml.

I would not say it's expected or normal to see this kind of difference. We've fixed a number of bugs since the last package of tensorflow-directml on pypi.org, and we still have more to go through, so it's possible this will be resolved in the next release (hopefully quite soon, i.e. weeks). It would be great if you could try this again when we have the new release, which also has many performance improvements, since this type of issue can be quite challenging to debug without the same data you're using.

We're also ramping up our own internal conformance testing to hopefully catch more issues like this.

from tensorflow-directml.

Thanks @aquilaadrian. It's really helpful that you file these types of issues regardless, as it's something we'll try to look at (with our own data) at some point. :)

from tensorflow-directml.

Hi, I am running tensorflow-directml==1.15.7 (Windows) on my RX6600XT 8GB

I use Xception to train dogs-vs-cats classification. (train:test=9:1 by split Kaggle train dataset)

The validation accuracy is equal to 0.5 on first two epochs

But val_acc going up after third epochs...

It seems validation model is not the same with training model due to there is larger than 0.6 train_acc in first one epoch.

My GPU driver is AMD Software Adrenalin Edition==22.10.01.03 (2022/5/5)

Any suggestion?

from tensorflow-directml.

Hi @CardLin,

Is this a problem that you were having with tensorflow-directml==1.15.5, or only 1.15.7?

from tensorflow-directml.

Hi @CardLin,

Is this a problem that you were having with tensorflow-directml==1.15.5, or only 1.15.7?

I have fix this issue by add GAP and Dense layer before Xception...

But it is strange that why there have different result compare to tensorflow-gpu==2.8.0 with CUDA runs on RTX2070?

I use 299x299x15 as input that should using weights=None. (random initialization)

Maybe the random initialization of the weights is different?

from tensorflow-directml.