Official server run SimAgents over tracks_to_predict or all valid agents? about waymo-open-dataset HOT 3 CLOSED

Hi @pengzhenghao,

The sim agents that are actively evaluated are the tracks_to_predict, but every other agent is part of the same "scene", which means all the interactive metrics for "evaluation sim agents" (== tracks_to_predict + SDC, see

The rationale for this choice is that objects not in tracks_to_predict are less consistent in time and produce a much noisier estimate. Happy to discuss this more if you want

from waymo-open-dataset.

Hi, in the tutorial it seems that the Sim Agent metrics are computed only over tracks_to_predict agents. In the validation_interactive dataset I saw the number of these agents in a scenario might be really small, might as small as 2 agents.

So I want you to clarify whether we should run SimAgents over tracks_to_predict or all valid agents at current step?

Hi @pengzhenghao, the answer is 'run SimAgents over all valid agents at current step.'

Just to clarify a bit more -- all objects that are valid at the 'current' step (effectively the last step of history, just before the user-provided simulation should commence) should be simulated and their trajectories should be included in the user submission.

Here is a link to the relevant code: https://github.com/waymo-research/waymo-open-dataset/blob/master/src/waymo_open_dataset/wdl_limited/sim_agents_metrics/metric_features.py#L117-L243

Note the two subsets in the code -- evaluated_sim_agent_ids vs. non_evaluated_sim_agent_ids. The former (forming evaluated_trajectories) comes from tracks_to_predict (https://github.com/waymo-research/waymo-open-dataset/blob/master/src/waymo_open_dataset/utils/sim_agents/submission_specs.py#L101), and is only used for the kinematic features. The latter (forming simulated_trajectories, i.e. all other objects valid at the current step) is used in the offroad and collision metrics. In short, all objects are 'evaluated' for certain metrics, but only the objects with the highest-fidelity ground truth are used in the kinematic feature comparison (likelihood scoring).

Please let us know if you have remaining questions.

from waymo-open-dataset.

Hi @jlambert-waymo thanks for the explaination! Don't have any question at this moment.

from waymo-open-dataset.