Comments (6)
Our code allows a single moment to be matched to multiple overlapping GT segments. L517-526 in the same file handles this.
from actionformer_release.
Hi, I saw your issues #43 before. My answer currently remains the same. The code you mentioned is that each feature point will only have one regression target. And ActionFormer will perform class-agnostic boundary regression.
For EPIC-Kitchens 100, the action is composed by a verb plus a noun. Though it will have some overlap between different actions, since our center sampling strategy, we can still achieve good performance on EPIC-Kitchens dataset since most feature point (which represents the center area of an action) will only have one class.
If you want to tweak ActionFormer for multiclass datasets, you may need to perform class-aware regression, i.e., in the code you mentioned, you keep the regression length for multiple length, and perform per-class boundary regression.
from actionformer_release.
OK. I see the confusion here. In short, I don't think there is a need for class-aware regression. An important detail is that when decoding the action instances, our code considers every category in each feature slot as a candidate.
To further clarify, let us consider the following cases.
- There are multiple GT segments that are not overlapping. This will not bring any issue.
- There are multiple GT segments with minor overlap. Each segment will be assigned to a different feature slot, as only those slots around the center of a segment are considered as positive. Again, there is no issue here.
- There are multiple GT segments with major overlap, where multiple segments might be assigned to the same set of feature slots. Our current implementation will consider a multi-class classification for those feature slots, yet will only regress the closest temporal boundary for each slot. At inference time, if some segments have major overlap, our model will predict one of the boundaries yet multiple categories at the same slot, leading to multiple detected action instances with the same temporal span. This should be totally fine in practice, as again, the boundaries of those segments are similar anyway.
- There are multiple GT segments with the same span. This is the case on THUMOS'14 and is handled similar to 3.
from actionformer_release.
For case 3, a major failure mode will lies in the sub-action action pattern, i.e., one sub-action is localized in the center of another action.
For example, one action is composed by three subactions, and the second subaction is localized at the center of the action. In that case, ActionFormer will fail. These cases can be found in complex datasets like FineGym.
from actionformer_release.
In the example you mentioned, the second subaction, co-centered with the action, is likely distributed to a different level on FPN. A corner case will be that two GT segments are co-centered on the same FPN level. But again, predicting any of the two boundaries will lead to a pretty high tIoU (>0.5) for the other, due to the design of the FPN.
from actionformer_release.
closed due to inactivity.
from actionformer_release.
Related Issues (20)
- Possible to get rid off regression head? HOT 4
- Use which checkpoint to report performance when evaluating HOT 1
- EMA model not working HOT 2
- anet_1.3_i3d.tar.gz HOT 3
- May I ask why there is warn: "No predictions of label '16' were provdied". HOT 5
- question about input i3D features HOT 5
- Missing video in THUMOS14 I3D features HOT 3
- Error "predicted offsets must be non-negative" HOT 1
- multigpu training HOT 2
- How to extract features of an action after transformer encoder in pyramid architecture? HOT 4
- No module named 'nms_1d_cpu HOT 2
- SlowFast feature extraction HOT 7
- What's the unit of the maximum input sequence (max_seq_len) HOT 7
- Visualization appendix D HOT 1
- Including SlowFast in LocPointTransformer HOT 7
- Why replace the predicted labels? HOT 2
- Can we run evaluate script on multiple GPUS? HOT 1
- Regarding Epic-Kitchen Combination of Noun and Verb HOT 2
- Inference on single video HOT 2
- Guidelines on setting up config for custom datasets? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from actionformer_release.