Questions about the files - which files to download about macaw-llm HOT 7 CLOSED

Also, several functions are never called in the script, e.g.,:
preprocess_vqa2_to_val_dataset()
preprocess_avsd_to_val_dataset()
resize_image()

Are these functions abandoned, or they are called but not illustrated in the python script?

from macaw-llm.

Also, in preprocess_avsd_to_tensor_dataset(), the audios and frames extracted from videos are actually not utilized. Is this correct?

from macaw-llm.

My last question is regarding the filtering: in preprocess_avsd_to_tensor_dataset(), we don't have a filtering step based on the input text length. In other preprocess functions (for vqa and alpaca), we have filtering steps based on the length of input text without answer. Why we filter based on the input text rather than the full_text? And why the avsd preprocessing don't have a filter step?

from macaw-llm.

Hi, sorry for the late reply. For downloading corresponding files, they are also used in preprocessing steps (vqa and avsd dataset for supervised data and our instruction dataset for unsupervised data). The metadata and raw images/videos of vqa and avsd are used in supervised data whereas only raw images/videos of vqa and avsd are used in unsupervised data with our instruction data. The image_path is the directory to the coco images.

from macaw-llm.

Also, several functions are never called in the script, e.g.,: preprocess_vqa2_to_val_dataset() preprocess_avsd_to_val_dataset() resize_image()

Are these functions abandoned, or they are called but not illustrated in the python script?

The first two functions are used to process validation dataset for evaluation and inference. The last one is used to resize images as well as video frames.

from macaw-llm.

Also, in preprocess_avsd_to_tensor_dataset(), the audios and frames extracted from videos are actually not utilized. Is this correct?

Usually we do not directly store image pixels in tensor dataset as it would incur significant memory use, instead we keep the index of the image and load it during training.

from macaw-llm.

My last question is regarding the filtering: in preprocess_avsd_to_tensor_dataset(), we don't have a filtering step based on the input text length. In other preprocess functions (for vqa and alpaca), we have filtering steps based on the length of input text without answer. Why we filter based on the input text rather than the full_text? And why the avsd preprocessing don't have a filter step?

This is because when the instruction is longer than the maximum length then no response will be included in the sequence, thus model can not learn how to generate response.

from macaw-llm.