Output accuracy on a test set about digits HOT 6 CLOSED

I'm sorry @gheinrich but this is inconvinient. I am currently running digits on a server, so when I use the option Upload image list for the Classify Many functionality I get to choose a file from my local machine, so I have to download it and upload it again.

If a test set is available for this dataset and no image list is uploaded, one would expect Classify Many to run over the test set.

from digits.

From https://groups.google.com/forum/#!topic/digits-users/JnXVuckUUG0:

It would be good if DIGITS can draw a confusion matrix at the end of training...
something which needs to run things again from outside using scripts...

from digits.

My plan for this is to only calculate the final accuracy and confusion matrix if the user provides a test dataset in addition to the training and validation sets. Once I implement this feature, I'll change the default folder splits to something like 60/20/20 (train/val/test) so that people will see it by default.

from digits.

Default folder splits sound perfect. In terms of the user flow I would envisage something along the lines of a dropdown/checkboxes for potential tests e.g. accuracy, confusion matrix, top-k matches etc., combined with the existing area to upload a test set specification file.

from digits.

Implemented with #608

Just use the test.txt file from your dataset job folder.

from digits.

My plan for this is to only calculate the final accuracy and confusion matrix if the user provides a test dataset in addition to the training and validation sets. Once I implement this feature, I'll change the default folder splits to something like 60/20/20 (train/val/test) so that people will see it by default.

It will be great if the same function can also be applied to object detection. Currently I can only get the predicted bbox location and confidence score(still dont know how it is calculated) for one test image using python or REST_API. I am wondering why DIGITS can't output the same accuracy visualization for test images as it does for val dataset. I am sure it is necessary for many user to check the generlaization of trained model on additional images.
Thanks!

from digits.