Goal: given v2d format of

 ├── 00000.tar
 |     ├── 00000.mp4
 |     ├── 00000.txt
 |     ├── 00000.json
 |     ├── 00001.mp4
 |     ├── 00001.txt
 |     ├── 00001.json
 |     └── ...
 |     ├── 10000.mp4
 |     ├── 10000.txt
 |     ├── 10000.json
 ├── 00001.tar
 |     ├── 10001.mp4
 |     ├── 10001.txt
 |     ├── 10001.json
 │     ...
 ...

produce a video_description/ modality data folder of the following format:

root/video_description/shard-00000.tar
 |     ├── 00000.jsonl # this corresponds to one video. each line within it corresponds to one subsequence of frames.
 |     ├── 00001.jsonl
 |     └── ...

Each jsonl should look something like

[
            {
                "description": "here's a description",
                "start_frame_index": 0,
                "end_frame_index": 5,
            },
            {
                "description": "here's another description",
                "start_frame_index": 5,
                "end_frame_index": 12,
            } 
]

Note that the txt/jsons in the v2d might not correspond exactly to the representation we want here (e.g., we might need some logic to determine the start/end frame indices from timestamps).

Where are these descriptions coming from? Do we pseudolabel them out with another description model? @smontariol?
Child issue of #3.

Goal: given v2d format of

 ├── 00000.tar
 |     ├── 00000.mp4
 |     ├── 00000.txt
 |     ├── 00000.json
 |     ├── 00001.mp4
 |     ├── 00001.txt
 |     ├── 00001.json
 |     └── ...
 |     ├── 10000.mp4
 |     ├── 10000.txt
 |     ├── 10000.json
 ├── 00001.tar
 |     ├── 10001.mp4
 |     ├── 10001.txt
 |     ├── 10001.json
 │     ...
 ...

produce a metadata/ modality data folder of the following format:

root/metadata/shard-00000.tar
 |     ├── 00000.json # this corresponds to one video.
 |     ├── 00001.json
 |     └── ...

Each json should look something like

{
    "video": {
        "fps": 10,
        "resolution": (512, 512),
        "dataset": "howto100m"
    },
},

(exact format/required keys TBD, since there probably should be more than just video metadata here? like maybe caption quality or something would be nice? 1st person vs 3rd person? what else?)

Given data from:

video_description modality

root/video_description/shard-00000.tar
 |     ├── 00000.jsonl # this corresponds to one video. each line within it corresponds to one subsequence of frames.
 |     ├── 00001.jsonl
 |     └── ...

and the video_metadata modality

root/metadata/shard-00000.tar
 |     ├── 00000.json # this corresponds to one video.
 |     ├── 00001.json
 |     └── ...

and maybe video_transcript modality

root/video_transcript/shard-00000.tar
 |     ├── 00000.jsonl # this corresponds to one video. each line within it corresponds to one subsequence of frames.
 |     ├── 00001.jsonl
 |     └── ...

generate a global description (one string) for each video. The output format should look like

root/video_transcript/shard-00000.tar
 |     ├── 00000.txt # this corresponds to the global description for one video.
 |     ├── 00001.txt
 |     └── ...

And xxxxx.txt just looks contains a string, e.g.

This is a global description of a video!

According to https://docs.google.com/presentation/d/1AY3QV1N_hoi9aXI1r8QTqrNmDK9LyorgJDQMPWb8hBo/edit#slide=id.g2e696416940_0_144, we have to add the temporal/frame encoding to IMAGE-based modality embeddings (but not sequence based ones).

A good starting point: check out this

Things to consider: make sure the embedding for temporal frame doesn't interfere with the positional patch embedding somehow?

Definition of Done: all image based encoder embeddings are augmented with a temporal embedding.

@vesteinn @garjania

Implement the model according to this design: https://docs.google.com/presentation/d/1AY3QV1N_hoi9aXI1r8QTqrNmDK9LyorgJDQMPWb8hBo/edit#slide=id.g2e696416940_0_144.

This includes (at least) several steps, each which will be detailed in its own github issue/PR:

• Determine the correct format for storing each video modality and implement pseudolabelers/data downloaders/etc. to get the video data stored in parallel directories as usable by 4M and video2dataset. "Definition of done" here means we have the data in the right directories and we can load them in the correct format. (#3)
• Implement modality_info and modality_transforms to map the new video modalities to their transformations which prepare them from the input filetype to be passable downstream to the model. (WIP PR: #1)
• Implement the encoder embeddings to be encode frame position (in addition to the patch position and modality embeddings). (#4)
• Implement a masking strategy which masks consistently across temporal frames (i.e., if you mask out patch position 7 for one frame, do it for all frames in that clip). (#5)
• TODO? @garjania what other steps are required here? anything for decoder embeddings?

Given a path to a raw dataset of v2d tar files, filter out all files within the tar that are "bad", i.e., don't have an mp4.
E.g., given the path raw/howto100m containing:

raw/howto100m
|    | - shard-00000.tar
|    | - | - 00000.mp4
|    | - | - 00000.m4a
|    | - | - 00000.json
|    | - | - 00002.mp4
|    | - | - 00002.m4a
|    | - | - 00002.json
|    | - | - 00010.json
|    | - | - 00011.json
|    | - | - 00012.json

we should end up with filtered_raw/howto100m containing:

filtered_raw/howto100m
|    | - shard-00000.tar
|    | - | - 00000.mp4
|    | - | - 00000.m4a
|    | - | - 00000.json
|    | - | - 00002.mp4
|    | - | - 00002.m4a
|    | - | - 00002.json

The video_rgb and other modalities should then all be drawn/pseudolabeled from filtered_raw/howto100m.

How consistent are pseudolabels for consecutive frames of a video? Take a couple random videos and spot-check the yolo outputs and identify whether it feels consistent.

definition of done: write up your vibe check along with some example images here.

Related to #3.

Goal: given v2d format of

 ├── 00000.tar
 |     ├── 00000.mp4
 |     ├── 00000.txt
 |     ├── 00000.json
 |     ├── 00001.mp4
 |     ├── 00001.txt
 |     ├── 00001.json
 |     └── ...
 |     ├── 10000.mp4
 |     ├── 10000.txt
 |     ├── 10000.json
 ├── 00001.tar
 |     ├── 10001.mp4
 |     ├── 10001.txt
 |     ├── 10001.json
 │     ...
 ...

produce a video_transcript/ modality data folder of the following format:

root/video_transcript/shard-00000.tar
 |     ├── 00000.jsonl # this corresponds to one video. each line within it corresponds to one subsequence of frames.
 |     ├── 00001.jsonl
 |     └── ...

Each jsonl should look something like

[
            {
                "transcript": "here's a transcript",
                "start_frame_index": 0,
                "end_frame_index": 5,
            },
            {
                "transcript": "here's another transcript",
                "start_frame_index": 10,
                "end_frame_index": 13,
            } 
]

Note that the txt/jsons in the v2d might not correspond exactly to the representation we want here (e.g., we might need some logic to determine the start/end frame indices from timestamps).

We may also want to run whisper to get improved transcripts.

Child issue of #3.

Download HowTo100M and Hd-vila datasets in video2dataset format using the scripts/methods described in https://github.com/swiss-ai/video2dataset/tree/main/swiss_ai.

This will be downloaded in this format:

 ├── 00000.tar
 |     ├── 00000.mp4
 |     ├── 00000.txt
 |     ├── 00000.json
 |     ├── 00001.mp4
 |     ├── 00001.txt
 |     ├── 00001.json
 |     └── ...
 |     ├── 10000.mp4
 |     ├── 10000.txt
 |     ├── 10000.json
 ├── 00001.tar
 |     ├── 10001.mp4
 |     ├── 10001.txt
 |     ├── 10001.json
 │     ...
 ...

Child issue of #3.

Goal: given a video_rgb/ modality data folder of the following format:

root/video_rgb/shard-00000.tar
 |     ├── 00000.mp4 # this corresponds to one video.
 |     ├── 00001.mp4
 |     └── ...

produce a video_det/ modality folder of the following format:

root/video_det/shard-00000.tar
 |     ├── 00000.jsonl # this corresponds to one video. each line within it corresponds to one frame.
 |     ├── 00001.jsonl
 |     └── ...

For now, let's use YOLO pseudolabeler.
This should involve calling the YOLO pseudolabeler on the videos in video_rgb, generating outputs, and moving it into the right directory paths. Each jsonl corresponds to one video. Each line in the jsonl corresponds to one frame.

Example jsonl:

[
        # FRAME 0 Bounding boxes
        {
            "num_instances": 5,
            "image_height": 512,
            "image_width": 906,
            "instances": [
                {
                    "boxes": [
                        0.4229210317134857,
                        0.00020096010121051222,
                        0.5715101361274719,
                        0.13699540495872498
                    ],
                    "score": 0.9029952883720398,
                    "class_id": 74,
                    "class_name": "clock",
                    "segmentation": [
                        [
                            0.5055187637969095,
                            0.1337890625,
                            ...
                        ]
                    ]
                },
                {
                    "boxes": [
                        ...
                    ],
                    ...
                },
                    ...
            ]
        },
        # FRAME 1 Bounding boxes
        {
            "num_instances": 5,
            "image_height": 512,
            "image_width": 906,
            "instances": [
                ...,
            ],
            ...
        }
]

Child issue of #3.

TODO: Modify and run save_vq_tokens.py to tokenize RGB videos.

save_vq_tokens.py is the file which you run in 4M to pretokenize images, e.g., to go from images of the modality rgb to examples of the modality tok_rgb. It takes a pretrained tokenizer and input dataset directory (among other things) and applies the tokenizer to the images in the input dataset to create the tokens in a new output dataset directory.

We want to get tokens for the rgb videos, so going from and input directory of root/video_rgb to the tokenized examples in the output directory root/video_tok_rgb.

The steps to do this are to modify save_vq_tokens.py to have the following capabilities:

1. It can load video files from the dataset folder in webdataset format (see the structure under video_rgb modality directory proposed in this post #3 (comment)).
2. you can run the pretrained rgb tokenizer on each frame of each video.
3. It saves the tokens as .npy files in webdataset format in the directory root/video_tok_rgb.

Proposed input directory format:

root/video_rgb/shard-00000.tar
 |     ├── 00000.mp4 # this corresponds to one video.
 |     ├── 00001.mp4
 |     └── ...

Proposed output directory format:

root/video_tok_rgb/shard-00000.tar
 |     ├── 00000.npy # this corresponds to one video. shape: something like (num_frames, H, C, W)
 |     ├── 00001.npy
 |     └── ...

Definition of Done:

• we have a script which can, given an input directory (e.g. video_rgb), pretrained tokenizer (e.g., from https://huggingface.co/collections/EPFL-VILAB/4m-tokenizers-66019388bda47e9bcff3f887), and output directory (e.g., video_tok_rgb), generate the tokenized representations of those videos according to the structure above saved to the output directory.
• This script is run and we actually have tokenized rgb videos in root/video_tok_rgb.

(This is a subtask of #3)

i.e., if you mask out patch position 7 for one frame, do it for all frames in that clip. The motivation here is that if you randomly mask patches then the model can probably "cheat" to predict the patch from the adjacent frames, as discussed with @garjania.

def of done (DoD): implement this masking strategy, ideally test it too

We want to get video RGB, video RGB tokens, video bounding boxes, video transcriptions, and video descriptions downloaded in a format that matches what 4M expects. Maybe that's something like

root/video_rgb/shard-00000.tar
root/video_rgb/shard-00001.tar
root/video_rgb/shard-00002.tar

root/video_tok_rgb/shard-00000.tar
root/video_tok_rgb/shard-00001.tar
root/video_tok_rgb/shard-00002.tar

root/video_det/shard-00000.tar
root/video_det/shard-00001.tar
root/video_det/shard-00002.tar

root/video_transcript/shard-00000.tar
root/video_transcript/shard-00001.tar
root/video_transcript/shard-00002.tar

root/video_description/shard-00000.tar
root/video_description/shard-00001.tar
root/video_description/shard-00002.tar

except I'm not sure because maybe the text should just be like jsonlines or something? This is very much just a suggestion. First task is to decide what makes the most sense, second is to then implement it. Keep an eye also on #1 because that loads the data and that PR will need to be fixed to correspond to the decisions made here (e.g., rn it assumes text is saved as JSONL, which I picked kinda arbitrarily and is def up for change).

To get video_rgb, we just need to download using video2dataset, probably, with some file saving shenanigans to make it fit our naming/path formats/requirements.
To get video_tok_rgb, we need to run (for now) a pretrained tokenizer on the video_rgb files and save it appropriately with right filetype and names/paths/etc.
To get video_det, we need to run the YOLO pseudolabeler on the video_rgb files and save appropriately (maybe as JSONL?)
To get video_description, we need to run ???something??? on the video_rgb files and save appropriately (maybe as JSONL?)
To get video_description, we need to run whisper on the video_rgb files and get the transcripts appropriately (maybe as JSONL?). (we can also start with the default youtube captions as an easier thing so we don't bring whisper in the mix yet.)

Thank you @yahya for taking the lead on implementing these steps. @garjania if you could provide feedback/suggestions on the right format for saving these things/how this corresponds with video2dataset that'd be super helpful! I think one concrete unknown to pursue in making the decision is to first look at how video2dataset stores files and decide whether we bend more to follow video2dataset or if we use that as an intermediary to extract the captions, etc. and form them into this format for 4M. also @vesteinn if you're familiar with v2d?

Goal: given v2d format of

 ├── 00000.tar
 |     ├── 00000.mp4
 |     ├── 00000.txt
 |     ├── 00000.json
 |     ├── 00001.mp4
 |     ├── 00001.txt
 |     ├── 00001.json
 |     └── ...
 |     ├── 10000.mp4
 |     ├── 10000.txt
 |     ├── 10000.json
 ├── 00001.tar
 |     ├── 10001.mp4
 |     ├── 10001.txt
 |     ├── 10001.json
 │     ...
 ...

produce a video_rgb/ modality data folder of the following format:

root/video_rgb/shard-00000.tar
 |     ├── 00000.mp4 # this corresponds to one video.
 |     ├── 00001.mp4
 |     └── ...

Option 1: This should mostly just involve extracting the mp4/video files from the video2dataset format and moving it into the right directory paths.

Option 2: We can use v2d now to normalize the videos by making them same number of frames, etc.

We choose option #2 because by the time we get something in a modality folder, it should already be the last preprocessing step before pseudolabeling for aligned data.

Child issue of #3.

Given a folder data/raw/... where ... = the downloaded v2d datasets, we need a script to partition (move) them into subdirectories called train, val, and test.

This is because the 4M folder directories expect the train/val/test to precede the modality, e.g.

train/video_rgb/class0/000.tar
train/video_det/class0/000.tar
train/video_transcript/class0/000.tar
```.

Then we can just run the merge_data.sh to go from raw to 4m-data 3 times (one for each split).

For now we can use a split of 70/10/20.

1. cropping? color jitter? etc.
2. For now we ignore just to get MVP. but we should probably implement some sort of aug. (WIP)