[Feature] vLLM Integration about canopy HOT 3 OPEN

This would also get me to adopt canopy.
I have set my base_url and api_key to a local instance to an LLM, but I am not able to continue - even though the LLM uses the Open AI module.

I get a: TypeError: 'NoneType' object is not subscriptable error; however, when I use the real base url and api key, it works just fine.

Would love to see this adopted.

from canopy.

I have set my base_url and api_key to a local instance to an LLM, but I am not able to continue - even though the LLM uses the Open AI module.

I get a: TypeError: 'NoneType' object is not subscriptable error; however, when I use the real base url and api key, it works just fine.

@rachfop can you please elaborate or provide a repro code?
For any model that follows the OpenAI APIs, this should actually work out of the box.

from canopy.

Totally think this should be considered! For two reasons:

1. One shouldn't just rely on paid models where local, fine-tuned models will perform just as good or better.
2. OpenAI embeddings are not great which can affect the accuracy of your RAG setup https://huggingface.co/spaces/mteb/leaderboard

Also perhaps the use of Ollama will be good as this creates a local server for the LLM.

Thoughts?

from canopy.