AI Services streaming support about langchain4j HOT 12 CLOSED

@breezexiao what is your use case? Please share the code of how you expect to use this new API. Thanks

from langchain4j.

@breezexiao what is your use case? Please share the code of how you expect to use this new API. Thanks
@langchain4j when using AIServices to develop applications, may be we need add some tool chains and EmbeddingStoreRetriever support, and the whole process will take a long time, so hope the streaming output can also be supported in AIServices, and then we push to frontend through sse etc .Thanks

from langchain4j.

@breezexiao I am afraid streaming will not work with tools as OpenAI does not support that. But we can add streaming support for the final response from the LLM (after the tool has been used and execution result was sent back to LLM). I would like to understand a bit better which libs/frameworks are you using and how streaming will be sent to the FE. Thank

from langchain4j.

@langchain4j get it, streaming support for the final response is a good idea. we send stream data through ResponseBodyEmitter, for example: ResponseBodyEmitter emitter = new ResponseBodyEmitter(); emitter. send(message); emitter. complete(); Thanks

from langchain4j.

@breezexiao ok, thanks. Will check today

from langchain4j.

@breezexiao ok, thanks. Will check today

@langchain4j hi, I found that streaming also supports function calls, such as this output message, hope it can help you
{"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"role":"assistant","content":null,"function_call":{"name":"getWeather","arguments":""}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"{\n"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"arg"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"0"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"\":"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"31"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"."}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"230"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"4"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":",\n"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"arg"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"1"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"\":"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"121"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"."}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"473"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"7"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":",\n"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"arg"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"2"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"\":"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"1"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"\n"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"}"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{},"finish_reason":"function_call"}]} [DONE]

from langchain4j.

@breezexiao wow that's cool! Too bad we still can't call the function before we get all arguments...
Regarding the streaming support in general, I think we can introduce some class that you can define as a return type of a service method and then subscribe on it, something like this (not a final version, just an idea);

interface ServiceWithStreaming {

        Streamer chat(String message);
}

Streamer streamer = service.chat("hello");

streamer.onNewToken(token -> emitter.send(token))
        .onError(e -> emitter.completeWithError(e))
	.onComplete(() -> emitter.complete())

Much simpler option would be to support returning ResponseBodyEmitter/SseEmitter directly from the service, like this:

interface ServiceWithStreaming {

        ResponseBodyEmitter chat(String message);
}

This way you won't have to re-map from Streamer to Emitter, but I have to check if this is possible to do without introducing lots of spring dependencies to the library...
If you have ideas, please share.

from langchain4j.

@langchain4j em I have an idea, can we extend StreamingResultHandler, such as add a subclass to proxy StreamingResultHandler, etc StreamingFunctionCallResultHandler, then add a method to handle functionCall, such as onFunctionCall(AIMessage), when the client.chatCompletion(request).onPartialResponse() method is executed, we judge whether it is a function_call according to the function_call field in the returned delta. If so, temporarily collect store the function name and arguments of the function_call. When onComplete called, if it is a function call, execute onFunctionCall (AIMessage), in onFunctionCall Execute the relevant logic of the tool chain, similar to AiServices, else execute the Proxied StreamingResultHandler logic. For ServiceWithStreaming, maybe we can define the method as void chat(String message, StreamingResultHandler handler), and the streaming is processed in the handler. thanks

from langchain4j.

@breezexiao I am not sure I understand your goal. Do you want to stream the function call response (if yes, please explain the reason)? Or you want to get a callback when the function call is triggered by LLM and be able to intrude? Thank you!

from langchain4j.

@breezexiao I am not sure I understand your goal. Do you want to stream the function call response (if yes, please explain the reason)? Or you want to get a callback when the function call is triggered by LLM and be able to intrude? Thank you!

@langchain4j
@breezexiao I am not sure I understand your goal. Do you want to stream the function call response (if yes, please explain the reason)? Or you want to get a callback when the function call is triggered by LLM and be able to intrude? Thank you!

@langchain4j em i just want use AIServices then send data to fe use streaming，don't want to block waiting for return, thanks

from langchain4j.

@breezexiao ok got it, will add this option asap (hopefully this week)

from langchain4j.

@breezexiao I have added streaming support to AI Services in 0.18.0:

interface Assistant {

    TokenStream chat(String message);
}

Assistant assistant = AiServices.create(Assistant.class, model);

assistant.chat("Tell me a joke")
    .onNext(token -> ...)
    .onComplete(() -> ...)
    .onError(e -> ...)
    .start();

from langchain4j.