what is the test dataset about redtail HOT 4 CLOSED

Hello,
We split images from training set into train/val and used images from 003, 008 and 010 as test set. Images from folders 011 - 014 are special: they were recorded by original authors to test performance of DNNs for extreme scenarios like left and right turns (see info.txt file in each of those folders for more information).

As for the accuracy: as mentioned in our paper/slides, the accuracy alone is not a very good indicator of how DNN will perform for navigation purposes in the real world. For example, ResNet-18 CE (i.e. ResNet-18 with standard cross-entropy loss) model achieved around 92% accuracy which was the highest among all the models we trained, but did not do very well on the test trail - its autonomy score was around 88% (see slide 21 from our GTC talk).
Unfortunately, we could not find a good, reproducible and generic way to measure autonomy score, it's all pretty subjective.

from redtail.

Thanks for answering my question!
So you mean the images from 003,008 and 010 as the test set to get accuracy. However, in the wiki you set 003,008 and 010 as the validation set. i am confused about this. Are the 003,008 and 010 the test set or the validation set? If they are validation set, what dataset is the test set?
By the way, I also want to train the TrailNet from the scratch, but it’s difficult for me to create a dataset. So, could you provide the lateral offset dataset?
Thank you for your time!

from redtail.

We started with 003,008 and 010 as our test set and 85%/15% as train/val split from our train set. In ideal world you train and fine-tune your model (hyperparameters search etc) on train/val split only and then run the "final" model on the test set only once. In real world you keep changing and improving the models so your test set eventually becomes validation so that's why we decided to call it a validation in our scripts.
For every model we did hyperparameter tuning only on train/val split and tested on test set once, but we had a lot of models.

As for the lateral translation dataset: we haven't released the data mostly due to internal review process which is more involved for the data than for the code. However, we released complete instructions and scripts which should allow to collect your own dataset and train translation head of the model. There is nothing special in our dataset - you can collect the similar data in your nearest park or forest.

from redtail.

You can collect data with a rig similar to this one: https://github.com/NVIDIA-Jetson/redtail/wiki/Datasets
that uses GoPro 4/5 cameras. Please note - for better results and be less camera dependent (in your runtime), you need to calibrate your rig cameras and un-distort collected footage before training (see the link). This way your robot camera does not need to be the same as dataset collection camera (have same camera model)

from redtail.