Supply a notebook with open source llm (e.g. llama 2) about ragas HOT 5 CLOSED

Hey Eyal, thanks for bringing this up

We do have plans for this and is in our roadmap but it might take a while. All our testing is with OpenAI so we can't ensure accuracy with opensource models. That being said I was wondering what were the reasons you wanted open-source models and how big of a model can you run at your end. Also, would you be okay running a ~30B model for evaluations, do you have a hosted version available right now?

This will help us get more sense of how to prioritise this

from ragas.

Hello,
I am in a similar situation, as I am building a RAG pipeline but I cannot use the OpenAI key (due to cost reasons); I am using a quantized version of Llama2 that fits in the free version of Colab.
The idea I had was to use the same model for generation and evaluation, aware that the results won't be that accurate as in case I was using GPT3.5/GPT4, but still in some ways informative for optimization.
Looking here (https://github.com/explodinggradients/ragas/blob/main/docs/guides/llms.ipynb) I thought it was possible to do that by changing the llm called by the metrics, but looking at your comment I don't know if it is convenient (?)
Do you have any suggestions?

from ragas.

Hey @Robs1999, full support for opensource LLMs will be coming shortly, I'll update this once its in

from ragas.

Thanks a lot for the update @jjmachan! I was wondering if you could give us a rough idea of when we might expect full support for open-source language models? Even a rough estimate would be really helpful!
Thanks again, and keep up the great work! 😊🚀

from ragas.

appologies for the delay but we have shipped this feature you can check the docs here https://docs.ragas.io/en/latest/howtos/customisations/llms.html#evaluating-with-open-source-llms

do check it out and feel free to open it this doesn't address the issue completely.

from ragas.