Comments (5)
@Kratos-Wen Hi, I just followed the instructions given by @mmaaz60, and changed num_frm from 100 to 200.
But this change seems had no improvement on understanding longer videos.
If a video is about 30 minutes, 100 frames and 200 frames could have no difference.
I guess changing the frame extracting method may work well such as extracting 1 frame per second in one video.
from video-chatgpt.
Thank you for your interest in our work. Currently the codebase is configured to use 100 frames only from a video. Given 100 video frames, features are extracted using pretrained CLIP model and the -2nd
layer outputs are considered. This gives us a tensor of shape t, s, c
where t=100
, s=256
is the number of spatial tokens and c=1024
is the feature dimensions.
We construct spatiotemporal features as by taking average across t
and s
dimensions.
temporal features -> average(t, s, c, dim=s) -> (t, c) -> (100, 1025)
spatial features -> average(t, s, c, dim=t) -> (s, c) -> (256 x 1024)
all features = 100 + 256 -> (356 x 1024)
As can be seen from the above explanation, the total number of tokens for 200 frames would be 200 + 256 = 456
and so on.
And in order to implement this change in the code base, we have set the number of total features to 456 at a few places. For example changing 356
to 456
here and changing the padding size accordingly here. Similarly, during training the changes should be made at here as well.
Please let me know if it solves the issue. Thanks
from video-chatgpt.
Thanks so much. It works right now. It's very kind of you to help us with this.
Wish you happiness every day!
from video-chatgpt.
Thanks so much. It works right now. It's very kind of you to help us with this. Wish you happiness every day!
Hi @wang9danzuishuai , I'm interesting about your attempt, how does the frame number adjustment work in your tests? Did it improve on understanding longer videos?
from video-chatgpt.
@wang9danzuishuai , hi, I also think so. how to change the code to extract the 1 frame or 2 frames per second?
from video-chatgpt.
Related Issues (20)
- Same output for any system prompt HOT 1
- Adding Modality HOT 1
- Question about the semi-automatic dataset creation process HOT 1
- RuntimeError: Parent directory ./Video-ChatGPT_7B-1.1_Checkpoints/checkpoint-3000 does not exist. HOT 5
- Why is the <video> tag is needed in training json? HOT 1
- How to load the tuned backbone? HOT 1
- Why does "No module named 'video chatgpt'" appear? What should I do next? HOT 3
- How to download the vidoes? HOT 1
- Single Node Training HOT 3
- GoogleDrive of Clip-Features HOT 3
- 13B, 70B models HOT 1
- RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())] HOT 5
- I encountered an error while using "Run the Demo". HOT 3
- Insufficient VRAM on a single card, how to utilize multiple cards for inference HOT 2
- Question about consistency evaluation HOT 3
- The responses from the offline_demo are garbled HOT 2
- Why TGIF videos missed and can't be processed HOT 3
- How to download the ready LLaVA-Lightening-7B weights HOT 5
- Download videos HOT 1
- How many frames are sampled per video for the training and testing process? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from video-chatgpt.