The meaning of label '4' about actionformer_release HOT 3 CLOSED

Good question! In THUMOS14 dataset, the action category "Cliff Diving" (ID: 4) is a subset of the category "Diving" (ID: 7). Namely, all "Cliff Diving" instances are also annotated as "Diving", yet not all "Diving" instances are marked as "Cliff Diving". This brings an issue for the implementation of label assignment during training.

Specifically, our current implementation assumes that a moment can be matched to at most one ground truth (GT) action instance, and thus could not handle the case where there are two GT action instances with exactly the same temporal boundaries (yet different labels).

Our solution, somewhat hacky, is to remove all instances from the action category "Cliff Diving" during training by masking out those instances. Note that (1) all instances from "Diving" still remain; and (2) this is only done during training. At test time, our model is still evaluated on all categories. The downside of this solution is that our model will not be able to recognize "Cliff Diving" on the test set, as it never saw an training sample from this category. Part of this discrepancy is resolved by fusing external classification scores. To verify this, you can disable score fusion and print the mAP scores per category, and the mAP for "Cliff Diving" (ID: 4) should be close to 0.

from actionformer_release.

Let me know if further clarification is needed. Otherwise, I will mark this issue as resolved.

from actionformer_release.

Let me know if further clarification is needed. Otherwise, I will mark this issue as resolved.

Good question! In THUMOS14 dataset, the action category "Cliff Diving" (ID: 4) is a subset of the category "Diving" (ID: 7). Namely, all "Cliff Diving" instances are also annotated as "Diving", yet not all "Diving" instances are marked as "Cliff Diving". This brings an issue for the implementation of label assignment during training.

Specifically, our current implementation assumes that a moment can be matched to at most one ground truth (GT) action instance, and thus could not handle the case where there are two GT action instances with exactly the same temporal boundaries (yet different labels).

Our solution, somewhat hacky, is to remove all instances from the action category "Cliff Diving" during training by masking out those instances. Note that (1) all instances from "Diving" still remain; and (2) this is only done during training. At test time, our model is still evaluated on all categories. The downside of this solution is that our model will not be able to recognize "Cliff Diving" on the test set, as it never saw an training sample from this category. Part of this discrepancy is resolved by fusing external classification scores. To verify this, you can disable score fusion and print the mAP scores per category, and the mAP for "Cliff Diving" (ID: 4) should be close to 0.

Thanks for your detailed explanation! This solution is reasonable and effective. Nice work!

from actionformer_release.