chenhsing / awesome-video-diffusion-models Goto Github PK

Hi, thanks for collecting a list of benchmark and metrics. Would you consider adding the Text-to-Video benchmark "VBench"?

VBench: Comprehensive Benchmark Suite for Video Generative Models
arXiv: https://arxiv.org/abs/2311.17982
website: https://vchitect.github.io/VBench-project/
code: https://github.com/Vchitect/VBench
video: https://www.youtube.com/watch?v=7IhCC8Qqn8Y
huggingface: https://huggingface.co/spaces/Vchitect/VBench_Leaderboard

Evalcrafter is open-sourced, it’s nice of you to update. Thanks!

Webpage: https://evalcrafter.github.io
GitHub: https://github.com/evalcrafter/EvalCrafter

Many thanks for your awesome list!!!

Congrats on the impressive work.

Our DiffusionRet has been accepted by ICCV 2023. The code is available at Code.

Could you please update the information on our paper?

https://github.com/fudan-generative-vision/champ

Hi I'm the author of StableVideo @ICCV 2023
code: https://github.com/rese1f/StableVideo
website: https://rese1f.github.io/StableVideo

you do a very great survey, many thanks!

Dear Authors,

I hope this email finds you well. I recently came across your comprehensive survey paper with this companion GitHub repository, which I found them to be an insightful and valuable resource for the community.

However, I noticed that our paper, "STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models" (https://arxiv.org/abs/2403.09669), which was published on arXiv in March and has been accepted to ICLR 2024, is not included in your survey nor in the Github. Given its relevance, I believe it would be a valuable addition to your survey.

TL;DR:
Our paper introduces STREAM, the first video evaluation metric designed to independently evaluate spatial and temporal aspects, offering comprehensive analysis unconstrained by video length. STREAM addresses limitations of current metrics like FVD, providing more stable and human-aligned evaluations.

Abstract of our paper:

Image generative models have made significant progress in generating realistic and diverse images, supported by comprehensive guidance from various evaluation metrics. However, current video generative models struggle to generate even short video clips, with limited tools that provide insights for improvements. Current video evaluation metrics are simple adaptations of image metrics by switching the embeddings with video embedding networks, which may underestimate the unique characteristics of video. Our analysis reveals that the widely used Frechet Video Distance (FVD) has a stronger emphasis on the spatial aspect than the temporal naturalness of video and is inherently constrained by the input size of the embedding networks used, limiting it to 16 frames. Additionally, it demonstrates considerable instability and diverges from human evaluations. To address the limitations, we propose STREAM, a new video evaluation metric uniquely designed to independently evaluate spatial and temporal aspects. This feature allows comprehensive analysis and evaluation of video generative models from various perspectives, unconstrained by video length. We provide analytical and experimental evidence demonstrating that STREAM provides an effective evaluation tool for both visual and temporal quality of videos, offering insights into area of improvement for video generative models. To the best of our knowledge, STREAM is the first evaluation metric that can separately assess the temporal and spatial aspects of videos. Our code is available at this URL.

We would be grateful if you could consider including a reference to our work in your survey. We believe that it will provide additional value and a more comprehensive overview of the current state of video generative model evaluation metrics.

Thank you for your consideration, and I look forward to your response.

Best regards,
Jaejun

Not sure Stable Video Diffusion is a T2V model or I2V model.

DragVideo- Interactive Drag-style Video Editing 2312.02216

Drag-A-Video- Non-rigid Video Editing with Point-based Interaction 2312.02936

Looking forward to increasing the realization of image generation and video related projects

We have released a text-2-video dataset named ECTV, please take a look at:
Dataset: https://huggingface.co/datasets/RaphaelLiu/EvalCrafter_T2V_Dataset
Paper: https://arxiv.org/abs/2310.11440
GitHub: https://github.com/evalcrafter/EvalCrafter

Many thanks for your awesome list!!!

Thanks a lot for your efforts in categorizing and staying tuned even for recently published papers.

I would like to ask you to add github url to VMC paper.
In category of Motion-guided Video Generation
The paper VMC: One-Shot Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models

The github link is this: https://github.com/HyeonHo99/Video-Motion-Customization

Thank you!

Thank you for your excellent work for VDM! Out of respect for your efforts, I translated it on Zhihu. Sincerely hope that your work garners further recognition and appreciation.
https://zhuanlan.zhihu.com/p/661860981