HTTP client question about lmdeploy HOT 5 CLOSED

像 openai RESTful API 一样？

Yes, this kind of regular HTTP client would be very convenient for use and integration. Also, as a suggestion, the preprocess and postprocess models can be implemented with the Triton framework ensemble mode, so that all steps can be completed with just one request to the interface. Just like the GPT example in the FasterTransformer project.
https://github.com/triton-inference-server/fastertransformer_backend/tree/main/all_models/gpt

from lmdeploy.

like openai RESTful APIs?

from lmdeploy.

We are working on it now.

from lmdeploy.

Hi I opened the --allow-http=1 option before running the service_docker_up.sh, but "curl -v http://0.0.0.0:8000/v2/models/llama2" always return "unknown model"（grpc interface works well), does it because restful api is not supported now? thank you.

from lmdeploy.

Hi I opened the --allow-http=1 option before running the service_docker_up.sh, but "curl -v http://0.0.0.0:8000/v2/models/llama2" always return "unknown model"（grpc interface works well), does it because restful api is not supported now? thank you.

The restful APIs were added. But they are not through triton server. For more details please refer to https://github.com/InternLM/lmdeploy/blob/main/docs/en/restful_api.md

from lmdeploy.