🤗Space Link

1. Model used: Microsoft Phi2
2. Dataset used: Tiny Stories dataset(100k samples) & Realtime data(100k samples) from finetuned Phi2 model via Ollama
3. Pretraining approach: Pretraining using QLoRA

1. LLM Backbone: Phi2
2. Vision Tower: clip-vit-large-patch14-336
3. Audio Model: Whisper
4. Pretraining Dataset: LAION-CC-SBU dataset with BLIP captions(200k samples)
5. Finetuning Dataset: Instruct 150k dataset based on COCO

Question Asked: How many people are there in this image? On HF Space:

1. Full Training: Here I have pretrained using 200k samples of LAION-CC-SBU dataset with BLIP captions, though its giving good results, full dataset would make it still better.
2. Captions for Finetuning: I have used Instruct 150k dataset for finetuning the model, original Llava model was finetuned also on blip captions(558k) which would again improve the model capabilities.
3. Latency Reduction / Model Optimization: Model could be quantized probably with GPTQ or AWQ to reduce the latency and make the model run faster on CPU too.
4. Audio Adapter: There is abundant data available for Whisper pretraining / finetuning, so could give it a try to use an audio adapter too and finetune a complete multimodal llm.
5. Lighter Variant of ClIP?: For audio I have used Whisper Tiny and still getting good results at minimal latency, it would be interesting to see if I could use a lighter variant of CLIP as well to reduce the latency.