Invalid submission for WOSAC about waymo-open-dataset HOT 16 CLOSED

One possible reason might be duplicated rollouts for the same scenario as I don't filter out the repeated rollouts for the same scenario.

from waymo-open-dataset.

Hi @pengzhenghao,

Sorry about the cryptic error messages, these are not properly propagated by the server.
I've looked at your submission, and I can see the error is of type "Sim agents {228, 213} are missing from the simulation.", which means not all of the required objects are there (or you're not populating the IDs correctly?).
Anyway, you can verify that the submission is correct by running

Let me know if you cannot reproduce

from waymo-open-dataset.

Hi @nicomon24

Thanks for the quick response! Would you mind inform me the scenario ID that yields this error?

from waymo-open-dataset.

I suspect this is due to the filtering of agents. In one scenario there are 300+ agents. My preprocessor will filter and keep only 128 agents because I thought in tfrecord there are maximally 128 agents.

from waymo-open-dataset.

I can check the scenario IDs for you tomorrow. In the meantime:
The scenario proto can indeed have more than 128 agents but the tf.Example dataset does not, which is also why we have removed the scenarios with >128 from the evaluation server (so this should not be the problem).

from waymo-open-dataset.

Thanks for the response!

Could you elaborate more on we have removed the scenarios with >128? You mean even though my submission for those scenarios > 128 agents might have only 128 agents, this will not triger the Sim agents {228, 213} are missing from the simulation. error?

from waymo-open-dataset.

Basically, to have a fair comparison between the Scenario and tf.Example dataset, Scenarios with too many objects will not be evaluated at all, so those scenarios (if I remember correctly, it's roughly 400 of them) are discarded (and so they should not raise that error)

from waymo-open-dataset.

I tried to remove the filtering mechanism in my preprocessor and made a new submission. It turns out that no error message can be seen and the webpage just breaks: https://waymo.com/open/challenges/sim-agents/results/92d57ec4-798b/1712098821307000/

from waymo-open-dataset.

Thanks @pengzhenghao, sorry for the disruption. The page is blank because the error is not propagated properly, but it's still returning an error.
I'm now running a more detailed eval so I can give you example of scenarios where there error is and how many there are. I should have some results in a couple of hours, hang tight :)

from waymo-open-dataset.

Hi @pengzhenghao,

An updated from your submission: there are a total of 44097 scenarios in your submission (timestamp 1711985043511000). Of these, 41161 are correct, while there are a total of 2535 that have missing sim agents.

Some examples:
Scenario 891805f154b4f0dd: Sim agents {1178} are missing from the simulation.
Scenario de8c427e65487b93: Sim agents {432, 569, 554} are missing from the simulation.
Scenario 386f0b2faebe74af: Sim agents {3361, 3332, 3399, 3371, 3375, 3311, 3345, 3346, 3378, 3319, 3352, 3388} are missing from the simulation.
Scenario cd861218ceb2dc1e: Sim agents {2145, 4805, 2150, 2123, 2162, 2131, 4730, 2139} are missing from the simulation.
Scenario 46a12cf2da1fdda8: Sim agents {1540, 1544, 4507, 4514, 4545, 1608, 1616, 1623, 1626, 1627, 1500, 1630, 1631, 1632, 1634, 4450, 1636, 4599, 1638, 1639, 4455, 4583, 4585, 4586, 1515, 4461, 1649, 4469, 4597, 4598, 4477} are missing from the simulation.

(The numbers between the brackets are the missing sim agent IDs, as reported in the Scenario proto).

Let me know if I can help you in some other way!

from waymo-open-dataset.

Hi @nicomon24

Thank you so much for your help! I take those scenarios out and test my model in it and find all agents are simulated. So I reran my pipeline and made a new submission and it works now!

In the old submission some processes to generate the submission dead so those scenarios were broken.

Hope this issue can help improve the error handling and logging in WOSAC server.

Thanks!

from waymo-open-dataset.

@nicomon24 Hi, I have uploaded the sim agents test set submissions for more than 30 hours, but it still shows "This submission could not be found. If it was recently uploaded, it may still be processing. Check back in a few minutes", could you please check it? My account is [email protected], submission time is 5/10/2024 1:38 PM.

from waymo-open-dataset.

Hi @rainmaker22, sorry that was a problem on our part, I was not monitoring the service during the weekend. Could you please check now? It seems that your submission resulted in an error because of the new fields missing (we made these required a couple of weeks ago, see https://github.com/waymo-research/waymo-open-dataset/blob/master/src/waymo_open_dataset/protos/sim_agents_submission.proto)

from waymo-open-dataset.

Hi, @nicomon24 I just realized the proto is updated from last year, I will resubmit it, how many hours will it take to finish the evaluation?

from waymo-open-dataset.

Hi @rainmaker22, yeah exactly, this was a new change this year. Please also make sure you have the pip package up to date (should be version 1.6.4 now https://pypi.org/project/waymo-open-dataset-tf-2-12-0/).

Evaluation should take 2-3 hours usually, but if you post your submission ID here I can make sure this is evaluated asap.

from waymo-open-dataset.

Hi @nicomon24 I have submitted the results, here is the submission page
https://waymo.com/open/challenges/sim-agents/results/4561ad1e-c697/1715522617206000/

from waymo-open-dataset.