Mis-matching between Trajectory and gt_graph about vidsgg-big HOT 5 CLOSED

Hi, I find that the cat_ids of the trajectory proposal is different from the traj_cat_ids of the paired gt_graph. For example: proposal.cat_ids: tensor([ 4, 4, 24, 4, 31, 31]), gt_graph.traj_cat_ids: tensor([11, 11, 7, 35, 35]) There is no common trajectory categories between each video-gt_graph pairs. But I think there should be the same category. So I want to know how to obtain the concrete catetory of a trajectory proposal or gt_graph. Thanks!

This is true, before doing bipartite matching of predicate nodes, we first match the entity nodes, and we do entity matching by the func:enti_viou_align, i.e.,

as you can see, in the func:enti_viou_align, we only consider the viou_matrix to align proposal traj and the gt traj.
Therefore, if the aligned proposal.traj and the gt.traj has different cat_ids, then it means that the proposal has the wrong object classification results (i.e., the object detector makes some mistake), since they are close (their viou is high enough).

Certainly, apart from the self.positive_vIoU_th, you can add additional constraints like the aligned trajs should have the same category. By doing this, the number of aligned traj pair will decrese
But in my opion, i think this is not necessary, since if the aligned trajs are close, then they are relaible, and the unrelaible factor is the proposal traj's category

A good approach to solve this is train another enitity classification head on top of the Transformer encoder, based on the aligned proposal.traj & gt.traj, since they are close, they deserve a new (and more correct) category.

Hope my answer is helpful to you :) and if there is any problem in my above analysis, please point it out, thank you.

from vidsgg-big.

A good approach to solve this is train another enitity classification head on top of the Transformer encoder, based on the aligned proposal.traj & gt.traj, since they are close, they deserve a new (and more correct) category.

, which we have tried before, but we gave up for some reasons, and it was not used in the final paper. I forgot whether there was any performance improvement.

from vidsgg-big.

Hi, I find that the cat_ids of the trajectory proposal is different from the traj_cat_ids of the paired gt_graph. For example: proposal.cat_ids: tensor([ 4, 4, 24, 4, 31, 31]), gt_graph.traj_cat_ids: tensor([11, 11, 7, 35, 35]) There is no common trajectory categories between each video-gt_graph pairs. But I think there should be the same category. So I want to know how to obtain the concrete catetory of a trajectory proposal or gt_graph. Thanks!

This is true, before doing bipartite matching of predicate nodes, we first match the entity nodes, and we do entity matching by the func:enti_viou_align, i.e.,

VidSGG-BIG/models/model_0v7.py

Line 564 in eaf7578

def enti_viou_align(self,gt_adj,proposal,gt_graph):

as you can see, in the func:enti_viou_align, we only consider the viou_matrix to align proposal traj and the gt traj. Therefore, if the aligned proposal.traj and the gt.traj has different cat_ids, then it means that the proposal has the wrong object classification results (i.e., the object detector makes some mistake), since they are close (their viou is high enough).

Certainly, apart from the self.positive_vIoU_th, you can add additional constraints like the aligned trajs should have the same category. By doing this, the number of aligned traj pair will decrese But in my opion, i think this is not necessary, since if the aligned trajs are close, then they are relaible, and the unrelaible factor is the proposal traj's category

A good approach to solve this is train another enitity classification head on top of the Transformer encoder, based on the aligned proposal.traj & gt.traj, since they are close, they deserve a new (and more correct) category.

Hope my answer is helpful to you :) and if there is any problem in my above analysis, please point it out, thank you.

Thank you for your anwsering!
But I notice that when generating gt_graph in

from vidsgg-big.

Thank you for your anwsering! But I notice that when generating gt_graph in

VidSGG-BIG/dataloaders/dataloader_vidvrd.py

Line 356 in eaf7578

category_id = vidvrd_CatName2Id[category_name]

the traj_cat_ids of a gt_graph come from /utils/categories_v2.py, while the the cat_ids of a trajectory come from the loaded file.

VidSGG-BIG/dataloaders/dataloader_vidvrd.py

Line 261 in eaf7578

cat_id = box_info[7]

If I want to know the categories of a trajectory, like 'person' and 'shirt', what should I do? And how can I obtain the word embeddings when I only have the traj_cat_ids of the gt_graph?
Thank you very much!

(1) the cat_id loaded from box_info is the cat_id from the object detector, and we use the same category id order in this project and our object detector (MEGA). e.g., you can check the code of our detector's dataloader to make sure, here:
https://github.com/Dawn-LX/VidVRD-tracklets/blob/695e49c5fe43d99d37ab357a7cdaa82309f55af9/MEGA/mega_core/data/datasets/vidor.py#L20
, and compare the above category order with that in utils/categories_v2.py

Therefore, you can just index the category name using vidvrd_CatId2name or vidor_CatId2name in utils/categories_v2.py

(2) for word embedding, refer to tools_draft/construct_CatName2vec.py and issue #7

from vidsgg-big.

And how can I obtain the word embeddings when I only have the traj_cat_ids of the gt_graph?

just index them back using CatId2name !

from vidsgg-big.