run_metric_caching.sh error on test split about navsim HOT 9 CLOSED

Hi @ucwxb ,
this is unfortunately an issue with the test dataset that has been uploaded to huggingface.
We are aware of this and in the process of uploading a fixed version of the dataset.
In the meantime, We recommend to run training / evaluation on the mini split.

I will follow-up here, once the new version of the dataset is pushed.

Sorry for any inconvenience.

from navsim.

Hi @ucwxb , this is unfortunately an issue with the test dataset that has been uploaded to huggingface. We are aware of this and in the process of uploading a fixed version of the dataset. In the meantime, We recommend to run training / evaluation on the mini split.

I will follow-up here, once the new version of the dataset is pushed.

Sorry for any inconvenience.

Thanks for your reply!
And I also notice that the process still fails on trainval set:

from navsim.

Additionally, when I run the default run_cv_pdm_score_evaluation.sh on the mini split, a series of AssertionError occur. Is this normal? Do I need to further supplement the agent part of the code?

The front part of the log appears to be normal, but the later scenes have no values and display false.

from navsim.

Hi,
I assume you mean the mini split as we haven't published the trainval split yet.
Can you please share which version of the dataset you are using to run the metric caching (i.e., when and how did you download it)?

from navsim.

Hi, I assume you mean the mini split as we haven't published the trainval split yet. Can you please share which version of the dataset you are using to run the metric caching (i.e., when and how did you download it)?

Hi, I follow the download instructions in OpenScene: https://github.com/OpenDriveLab/OpenScene/blob/main/docs/getting_started.md#download-data
And I download the trainval split on the provided OpenDriveLab website: https://openxlab.org.cn/datasets/OpenDriveLab/OpenScene/tree/main/openscene-v1.1

from navsim.

Hi,
thanks for the clarification. The devkit does not yet officially support the trainval set since we first need to handle some annotation issues in the original trainval split. Hence, the behavior you see is somewhat expected, i.e., for some scenes the metric cache cannot be generated due to route annotation issues. In consequence, the evaluation will fail for these tokens, which is why you see the assertion error. Thus, for these tokens, the CSV contains no results and the second column (that describes if the results for this token are valid) is set to False.

We will soon release the next version of the devkit together with the trainval dataset on huggingface.

However, as far as I can see, you still get valid results for the remaining Scenes (see the "front part" of your CSV).
So, in the meantime, you could still run your agent on the trainval split, but be aware that results might change slightly once we address all the dataset issues.

I am closing this issue for now. Feel free to re-open or open a new issue once you face more problems.

from navsim.

Thanks for your reply!

from navsim.

Hi,
we uploaded the corrected version of the test dataset (which includes the annotations).
You can download it directly from Huggingface or with the download script.

Besides, the latest fix we pushed today should solve the issue you had with the trainval set (however, no guarantee yet that everything works as its not officially supported yet).

from navsim.

Hi, we uploaded the corrected version of the test dataset (which includes the annotations). You can download it directly from Huggingface or with the download script.

Besides, the latest fix we pushed today should solve the issue you had with the trainval set (however, no guarantee yet that everything works as its not officially supported yet).

Thank you for your timely reply. Please let me know when all datasets are ready. Thanks!

from navsim.