Can we have support for GGML as triton backend about lmdeploy HOT 12 CLOSED

but having ggml also can attract more contributors, also i'm curious whether inferLLM support cuda, doesn't see that on the page

from lmdeploy.

Indeed llama.cpp/GGML is famous (about 30k star), but the code is really hard to read.

InferLLM may be a better choice.

from lmdeploy.

wow what is it can you elaborate a bit more, to what i understand it's just that you need to implement the ggml itself into the triton backend and later on you can reuse the triton backend and have the benefit of ggml update after every iteration

from lmdeploy.

will not be re-using fastertransformer i guess but more about how the ggml itself can be integrated with tritonbackend

from lmdeploy.

will not be re-using fastertransformer i guess but more about how the ggml itself can be integrated with tritonbackend

Here is my opinion, when you integrate a inference backend, you have to make sure of service quality.
Let's take a assumption, after integration, if ggml has a bug, lmdeploy has responsibility to locate it or fix it.

The greater cost of software is maintenance, so we need to consider the code complexity of ggml.

from lmdeploy.

About InferLLM https://github.com/MegEngine/InferLLM

from lmdeploy.

saw enable gpu on the cmake file still

from lmdeploy.

but having ggml also can attract more contributors, also i'm curious whether inferLLM support cuda, doesn't see that on the page

yes, attracting more contributors is a good reason to integrate ggml.

from lmdeploy.

but having ggml also can attract more contributors, also i'm curious whether inferLLM support cuda, doesn't see that on the page

InferLLM cuda part is WIP MegEngine/InferLLM#27

from lmdeploy.

InferLLM will have many language barriers for non-chinese speakers contributors also, which is a very large audience, for most of the open source implementation it depends on how many contributors you have.

from lmdeploy.

InferLLM will have many language barriers for non-chinese speakers contributors also, which is a very large audience, for most of the open source implementation it depends on how many contributors you have.

Already passed on to InferLLM team .. QvQ

from lmdeploy.

Close since no more activity over two weeks. Feel free to reopen it if it is still an issue

from lmdeploy.