Comments (12)
@breezexiao what is your use case? Please share the code of how you expect to use this new API. Thanks
from langchain4j.
@breezexiao what is your use case? Please share the code of how you expect to use this new API. Thanks
@langchain4j when using AIServices to develop applications, may be we need add some tool chains and EmbeddingStoreRetriever support, and the whole process will take a long time, so hope the streaming output can also be supported in AIServices, and then we push to frontend through sse etc .Thanks
from langchain4j.
@breezexiao I am afraid streaming will not work with tools as OpenAI does not support that. But we can add streaming support for the final response from the LLM (after the tool has been used and execution result was sent back to LLM). I would like to understand a bit better which libs/frameworks are you using and how streaming will be sent to the FE. Thank
from langchain4j.
@langchain4j get it, streaming support for the final response is a good idea. we send stream data through ResponseBodyEmitter, for example: ResponseBodyEmitter emitter = new ResponseBodyEmitter(); emitter. send(message); emitter. complete(); Thanks
from langchain4j.
@breezexiao ok, thanks. Will check today
from langchain4j.
@breezexiao ok, thanks. Will check today
@langchain4j hi, I found that streaming also supports function calls, such as this output message, hope it can help you
{"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"role":"assistant","content":null,"function_call":{"name":"getWeather","arguments":""}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"{\n"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"arg"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"0"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"\":"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"31"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"."}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"230"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"4"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":",\n"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"arg"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"1"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"\":"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"121"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"."}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"473"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"7"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":",\n"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" \""}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"arg"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"2"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"\":"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":" "}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"1"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"\n"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{"function_call":{"arguments":"}"}},"finish_reason":null}]} {"id":"chatcmpl-7cxMqb5Q4SmmQBdRQOWnk48aI3c1l","object":"chat.completion.chunk","created":1689519028,"model":"gpt-3.5-turbo-16k-0613","choices":[{"index":0,"delta":{},"finish_reason":"function_call"}]} [DONE]
from langchain4j.
@breezexiao wow that's cool! Too bad we still can't call the function before we get all arguments...
Regarding the streaming support in general, I think we can introduce some class that you can define as a return type of a service method and then subscribe on it, something like this (not a final version, just an idea);
interface ServiceWithStreaming {
Streamer chat(String message);
}
Streamer streamer = service.chat("hello");
streamer.onNewToken(token -> emitter.send(token))
.onError(e -> emitter.completeWithError(e))
.onComplete(() -> emitter.complete())
Much simpler option would be to support returning ResponseBodyEmitter/SseEmitter directly from the service, like this:
interface ServiceWithStreaming {
ResponseBodyEmitter chat(String message);
}
This way you won't have to re-map from Streamer to Emitter, but I have to check if this is possible to do without introducing lots of spring dependencies to the library...
If you have ideas, please share.
from langchain4j.
@langchain4j em I have an idea, can we extend StreamingResultHandler
, such as add a subclass to proxy StreamingResultHandler, etc StreamingFunctionCallResultHandler
, then add a method to handle functionCall, such as onFunctionCall(AIMessage)
, when the client.chatCompletion(request).onPartialResponse()
method is executed, we judge whether it is a function_call
according to the function_call
field in the returned delta
. If so, temporarily collect store the function name and arguments of the function_call
. When onComplete
called, if it is a function call, execute onFunctionCall (AIMessage)
, in onFunctionCall Execute the relevant logic of the tool chain, similar to AiServices
, else execute the Proxied StreamingResultHandler
logic. For ServiceWithStreaming
, maybe we can define the method as void chat(String message, StreamingResultHandler handler)
, and the streaming is processed in the handler. thanks
from langchain4j.
@breezexiao I am not sure I understand your goal. Do you want to stream the function call response (if yes, please explain the reason)? Or you want to get a callback when the function call is triggered by LLM and be able to intrude? Thank you!
from langchain4j.
@breezexiao I am not sure I understand your goal. Do you want to stream the function call response (if yes, please explain the reason)? Or you want to get a callback when the function call is triggered by LLM and be able to intrude? Thank you!
@langchain4j
@breezexiao I am not sure I understand your goal. Do you want to stream the function call response (if yes, please explain the reason)? Or you want to get a callback when the function call is triggered by LLM and be able to intrude? Thank you!
@langchain4j em i just want use AIServices then send data to fe use streaming,don't want to block waiting for return, thanks
from langchain4j.
@breezexiao ok got it, will add this option asap (hopefully this week)
from langchain4j.
@breezexiao I have added streaming support to AI Services in 0.18.0:
interface Assistant {
TokenStream chat(String message);
}
Assistant assistant = AiServices.create(Assistant.class, model);
assistant.chat("Tell me a joke")
.onNext(token -> ...)
.onComplete(() -> ...)
.onError(e -> ...)
.start();
from langchain4j.
Related Issues (20)
- [FEATURE] The PgVectorEmbeddingStore enables query operations that exclusively leverage metadata for filtering criteria HOT 4
- [BUG] AzureAiSearchEmbeddingStore stopped retreiving Embedding (vector) HOT 3
- Dimension for RedisEmbeddingStore should not be required if the index exists HOT 2
- [BUG] Expected a SETTINGS frame but was WINDOW_UPDATE HOT 2
- [FEATURE] Support for custom max tool sequential requests HOT 2
- [FEATURE] OpenAI: Use provided token usage instead of manually calculated
- Use LLM to Generate SPARQL Queries from Natural Language Queries on DBpedia LOD HOT 1
- [BUG] For vector search, we get a higher result score than for native SDK search HOT 4
- "Cannot resolve symbol OpenAiChatModelName" in IntelliJ HOT 3
- NVIDIA NIM support HOT 2
- [FEATURE] Milvus - upgrade to Milvus SDK 2.3.9 HOT 3
- langgraph HOT 1
- How to customize and add LLM models HOT 7
- [FEATURE] Get rid of Lombok
- [FEATURE] Supports sorting of data in the Elasticsearch vector database HOT 10
- Return citations in AiMessage HOT 5
- Raise the baseline to Java 17 HOT 2
- [BUG] Application crashes on using AllMiniLmL6V2EmbeddingModel embedding when started in docker container with openjdk:8 HOT 1
- How to call the relationship between tools
- [FEATURE] Jlama: Add Tool Support
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from langchain4j.