cg1177 / videollm Goto Github PK

I find your paper and methodology very intriguing and would like to use it for my research. Could you please provide a specific timeline for when the source code and pretrained models will be made publicly available?

I have no idea about you training schedule( 2 stage or 1 stage), what text encoder and visual encoder used, and what decoder used, although you have done great ablation study.

Hi,

First of all, thank you for this wonderful contribution, your results are really impressive!
I had a question regarding Table 4 in your paper. For action segmentation, can you please share the train/test split you used for the breakfast dataset? and did you use raw labels for training or you modified the dataset into chat style?
I know you plan to release the dataset soon but if you can answer the above, that would be really helpful!
Thanks

Dear authors,
thank you for your will to open-sourcing your code and models. May I ask how heavy is the training resources needed? How many gpu required for videollm?

Hi, I am really interested in your fantastic work.
I want to follow your work and investigate further.
May I know when you plan to release the code and the pretrian model?
That will inspire all the researchers in the video area.
Thanks

Hi! I just went through your preprint, and here are my two quick reactions, if you don't mind:

Typo in the Figure 3 caption of the preprint

(b) “Unssen tokens” are data units that have not yet arrived, and predicting their attributes or when they appear in the future usually belongs to future prediction tasks.

It should be “Unseen tokens”.

Possibility of token-retrieval pretraining

VideoLLM, especially the use of linear projector to map video tokens to tokens for the LLM, reminds me of https://github.com/kohjingyu/fromage. However, there is no equivalent of the Image-text retrieval pretraining task: i.e., given a description of the video, train the LLM to retrieve the correct video tokens in the correct order among all the video tokens in the same batch. Can it be a useful pretraining task here?