[Feature]: Support for local models about rivet HOT 8 CLOSED

@abrenneke you could simply redirect the request towards an OpenAI compatible API, like the one present in Oobabooga, so you basically maintain everything the same, just allow to redirect the calls for the API to OpenAI Oobabooga API, that's a quick and easy way to support Local models with GPU support :)

from rivet.

Something like this?

from rivet.

That's something I'd like to support too! I'll check out that project for tips on how to accomplish it.

from rivet.

Yep that's the one I tested with @andraz. This has been pushed to main so I'll close this issue, should be available when 1.2.0 comes out soon.

from rivet.

Yes I have just finished the first version. It allows for model loading and checking the loaded model. Any requests welcome, right now it has some chat params, and model loading is done using your saved model loading defaults in oobabooga.

https://github.com/hushaudio/rivet-oobabooga-plugin

Import link
https://cdn.jsdelivr.net/gh/hushaudio/rivet-oobabooga-plugin@main/dist/oobabooga-rivet.js

from rivet.

Oh that's a good idea. You'd have to run an instance of whatever API that is but that would be really easy to implement on the Rivet side.

from rivet.

Looks great, it can be plug and play with this project too: https://github.com/go-skynet/LocalAI

from rivet.

Also, we have someone working on a deeper oobabooga integration plugin

from rivet.