Comments (6)
Hi, the path "prompts/prompts_per_dimension" contains only a part of evaluation dimensions. So, if i want to evaluation the dimension of "background_consistent", and there is no proper prompt text file (maybe background_consistent.txt) for me to generation videos. So, what can i do to perform evaluation on "background_consistent"?
@GFENGG Hi , you can refer to this file. On the left are the corresponding evaluation dimensions, and on the right is the prompt list of left dimension.
For example, if you want to test background_consistency
, you only need to sample the video of scene
.
VBench/dimension_to_folder.json
Lines 2 to 17 in 8046fcb
from vbench.
@GFENGG Sorry for the late reply. In all_category
, there are approximately 100 prompts for each category. We conducted VBench evaluation for all prompts in subject consistency
, background consistency
, motion smoothness
, dynamic degree
, aestheic quality
, image quality
and overall consistency
without further screening corresponding relationships.
from vbench.
And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?
from vbench.
And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?
@GFENGG all_category
is not a standalone dimension, so it will not appear in the json
.
prompts/all_category.txt
is a collection of video prompts for all dimensions that need to be tested.
By sampling all prompts from prompts/all_category.txt
into videos, VBench
will automatically assign corresponding evaluation dimensions to each sampled video for evaluation and provide results.
from vbench.
And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?
@GFENGG
all_category
is not a standalone dimension, so it will not appear in thejson
.
prompts/all_category.txt
is a collection of video prompts for all dimensions that need to be tested. By sampling all prompts fromprompts/all_category.txt
into videos,VBench
will automatically assign corresponding evaluation dimensions to each sampled video for evaluation and provide results.
Thanks for your reply. In which file (maybe a .json file like "VBench_full_info.json") can i find the relation between the prompts in "prompts/all_category.txt" and corresponding evaluation dimensions?
from vbench.
Hi, we're closing this issue as it appears your questions have been addressed. However, feel free to open a new issue or reopen this one, if you have further questions or if anything else comes up related to this issue.
from vbench.
Related Issues (20)
- When evaluate aesthetic_quality, 'CUDA out of memory' always occur for multiple videos HOT 2
- failed to get the output result HOT 1
- Vbench Leaderboard HOT 2
- Where is the VLM-tuning model (VideoChat finetuned using human annotation)? HOT 2
- Great works. A question about detailed inference settings for animatediffv2. HOT 9
- How to calculate total score? HOT 1
- Can we withdraw the results in leaderboard? HOT 5
- How to solve the problem "av.error.FileNotFoundError: [Errno 2] No such file or directory” HOT 4
- unzip missing HOT 4
- VBench-Long Leaderboard HOT 2
- Difference between "overall_consistency" and "temporal_style"? HOT 2
- Can vbench_i2v support customized video evaluation?
- Can not download pretrained model for "overall consistency" evaluation HOT 4
- Implementation Details HOT 5
- Duplicate prompts in overall_consistency.txt and full.json HOT 3
- Empirical Min HOT 1
- What is the role of the trustworthiness folder at the same level as per_dimension and per_category?
- Cogvideo2B score HOT 7
- Unrecognized arguments: --mode=custom_input when running vbench HOT 2
- error with torch.distributed for multiprocesssing HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vbench.