Multi-lora support about text-generation-inference HOT 5 OPEN

Any update on this feature request?

from text-generation-inference.

This is definitely on our roadmap and will be tackled in the coming weeks.
Here are the priorities right now:

1. re-write the scheduling code and cache multi-turn conversations. This will have a huge impact on latency and throughput.
2. better support for grammar/tools
3. multi-modal
4. mulit loras

All of this will land in the coming month.

What are your use case for multi-loras btw? We are wondering if we should support them on the api-inference free tier.

from text-generation-inference.

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

from text-generation-inference.

x2

from text-generation-inference.

This is definitely on our roadmap and will be tackled in the coming weeks. Here are the priorities right now:

1. re-write the scheduling code and cache multi-turn conversations. This will have a huge impact on latency and throughput.
2. better support for grammar/tools
3. multi-modal
4. mulit loras

All of this will land in the coming month.

What are your use case for multi-loras btw? We are wondering if we should support them on the api-inference free tier.

@OlivierDehaene Amazing for this plan, and is there any concrete roadmap for production ready multi-lora support ?

from text-generation-inference.