chenhsing / awesome-video-diffusion-models Goto Github PK
View Code? Open in Web Editor NEW[Arxiv] A Survey on Video Diffusion Models
[Arxiv] A Survey on Video Diffusion Models
Hi, thanks for collecting a list of benchmark and metrics. Would you consider adding the Text-to-Video benchmark "VBench"?
VBench: Comprehensive Benchmark Suite for Video Generative Models
arXiv: https://arxiv.org/abs/2311.17982
website: https://vchitect.github.io/VBench-project/
code: https://github.com/Vchitect/VBench
video: https://www.youtube.com/watch?v=7IhCC8Qqn8Y
huggingface: https://huggingface.co/spaces/Vchitect/VBench_Leaderboard
Evalcrafter is open-sourced, it’s nice of you to update. Thanks!
Webpage: https://evalcrafter.github.io
GitHub: https://github.com/evalcrafter/EvalCrafter
Many thanks for your awesome list!!!
Congrats on the impressive work.
Our DiffusionRet has been accepted by ICCV 2023. The code is available at Code.
Could you please update the information on our paper?
Hi I'm the author of StableVideo @ICCV 2023
code: https://github.com/rese1f/StableVideo
website: https://rese1f.github.io/StableVideo
you do a very great survey, many thanks!
Dear Authors,
I hope this email finds you well. I recently came across your comprehensive survey paper with this companion GitHub repository, which I found them to be an insightful and valuable resource for the community.
However, I noticed that our paper, "STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models" (https://arxiv.org/abs/2403.09669), which was published on arXiv in March and has been accepted to ICLR 2024, is not included in your survey nor in the Github. Given its relevance, I believe it would be a valuable addition to your survey.
TL;DR:
Our paper introduces STREAM, the first video evaluation metric designed to independently evaluate spatial and temporal aspects, offering comprehensive analysis unconstrained by video length. STREAM addresses limitations of current metrics like FVD, providing more stable and human-aligned evaluations.
Abstract of our paper:
Image generative models have made significant progress in generating realistic and diverse images, supported by comprehensive guidance from various evaluation metrics. However, current video generative models struggle to generate even short video clips, with limited tools that provide insights for improvements. Current video evaluation metrics are simple adaptations of image metrics by switching the embeddings with video embedding networks, which may underestimate the unique characteristics of video. Our analysis reveals that the widely used Frechet Video Distance (FVD) has a stronger emphasis on the spatial aspect than the temporal naturalness of video and is inherently constrained by the input size of the embedding networks used, limiting it to 16 frames. Additionally, it demonstrates considerable instability and diverges from human evaluations. To address the limitations, we propose STREAM, a new video evaluation metric uniquely designed to independently evaluate spatial and temporal aspects. This feature allows comprehensive analysis and evaluation of video generative models from various perspectives, unconstrained by video length. We provide analytical and experimental evidence demonstrating that STREAM provides an effective evaluation tool for both visual and temporal quality of videos, offering insights into area of improvement for video generative models. To the best of our knowledge, STREAM is the first evaluation metric that can separately assess the temporal and spatial aspects of videos. Our code is available at this URL.
We would be grateful if you could consider including a reference to our work in your survey. We believe that it will provide additional value and a more comprehensive overview of the current state of video generative model evaluation metrics.
Thank you for your consideration, and I look forward to your response.
Best regards,
Jaejun
Not sure Stable Video Diffusion is a T2V model or I2V model.
DragVideo- Interactive Drag-style Video Editing 2312.02216
Drag-A-Video- Non-rigid Video Editing with Point-based Interaction 2312.02936
Looking forward to increasing the realization of image generation and video related projects
We have released a text-2-video dataset named ECTV, please take a look at:
Dataset: https://huggingface.co/datasets/RaphaelLiu/EvalCrafter_T2V_Dataset
Paper: https://arxiv.org/abs/2310.11440
GitHub: https://github.com/evalcrafter/EvalCrafter
Many thanks for your awesome list!!!
Thanks a lot for your efforts in categorizing and staying tuned even for recently published papers.
I would like to ask you to add github url to VMC paper.
In category of Motion-guided Video Generation
The paper VMC: One-Shot Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models
The github link is this: https://github.com/HyeonHo99/Video-Motion-Customization
Thank you!
Thank you for your excellent work for VDM! Out of respect for your efforts, I translated it on Zhihu. Sincerely hope that your work garners further recognition and appreciation.
https://zhuanlan.zhihu.com/p/661860981
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.