๐คSpace Link
- Model used: Microsoft Phi2
- Dataset used: Tiny Stories dataset(100k samples) & Realtime data(100k samples) from finetuned Phi2 model via Ollama
- Pretraining approach: Pretraining using QLoRA
- LLM Backbone: Phi2
- Vision Tower: clip-vit-large-patch14-336
- Audio Model: Whisper
- Pretraining Dataset: LAION-CC-SBU dataset with BLIP captions(200k samples)
- Finetuning Dataset: Instruct 150k dataset based on COCO
Question Asked: How many people are there in this image?
On HF Space:
- Full Training: Here I have pretrained using 200k samples of LAION-CC-SBU dataset with BLIP captions, though its giving good results, full dataset would make it still better.
- Captions for Finetuning: I have used Instruct 150k dataset for finetuning the model, original Llava model was finetuned also on blip captions(558k) which would again improve the model capabilities.
- Latency Reduction / Model Optimization: Model could be quantized probably with GPTQ or AWQ to reduce the latency and make the model run faster on CPU too.
- Audio Adapter: There is abundant data available for Whisper pretraining / finetuning, so could give it a try to use an audio adapter too and finetune a complete multimodal llm.
- Lighter Variant of ClIP?: For audio I have used Whisper Tiny and still getting good results at minimal latency, it would be interesting to see if I could use a lighter variant of CLIP as well to reduce the latency.