More input signals for evaluation about clrs HOT 9 CLOSED

Hi, thank you for raising this. The current dataset on GCP should have the correct shapes and amount of samples now. Could you check that that is the case? If you are using examples/run.py You will need to remove your local dataset copy to force a download (which is by default stored in /tmp/CLRS30)

from clrs.

Thanks for your reply. The issue seems to be still existing. I double-checked that and only 32 data points are loaded into the validation/test datasets (take the minimum algorithm for instance). The content of dataset_info.json also suggests that there are only 32 data points in the dataset.

from clrs.

Thank you for following up so quickly! It indeed was a bug that had gone under the radar. This should be fixed now + dataset has been re-generated. Could you take a look at this new version?

from clrs.

Yup, looks perfect. Thanks for the fix.

from clrs.

Just wanted to let you know that in your run.py, only one batch (=32) of test data (and validation data in each iteration) is being evaluated. Full batch leads to high GPU memory consumption for test data, and average over batches would not give accurate results (e.g., F1 metric for masks).

from clrs.

Hi Sadegh, thank you so much for spotting this!

Indeed, the example script needs to be patched, and we will do this shortly after the Easter holiday.

The issue you point out about properly aggregating batches at test time is very important, especially when datasets are manually generated, and we will make sure to address it! Although for the specific CLRS-30 dataset we generated, this would only really impact tasks with graph mask outputs (eg. segments-intersect); in all other mask output settings we do not need to go beyond 32 trajectories, and for types other than mask I think simple averaging across batches will be sufficient.

Lastly, thank you very much for your general interest in the library and all the great pointers you've had! We'd be delighted to add you to our paper's acknowledgements when it is ready for release :)

from clrs.

Thank you for the detailed explanation!

from clrs.

Hi Sadegh, the issue of evaluating/testing on multiple batches should now be resolved by #70.
Please let us know if any issues remain! :)

from clrs.

Great! Thanks for the fix :)

from clrs.