consuming streaming response will block until SSE stream ends about google-generative-ai-rs HOT 4 CLOSED

I guess Google use batch generating. multiple requests are concurrently join and send to llm in a batch . outputing is still streamly but nearby tokens in stream may join together into chunk

I wrote a Python cli for Gemini,which confirmed Gemini supports output in stream

there could be a solution that moving original response into
stream-gemini-response and letting user code parse response stream lazily,make async iteration possible

from google-generative-ai-rs.

@hitsmaxft As per our conversations in #20 and #21 we've moved this to a branch, streamed_response, for more comprehensive testing. I'm a bit time poor right now. But to summarise, there are two endpoints, the Public and that of the Vertex AI API service. I just want to make sure that all the use cases are covered as there are some subtle differences between each (which is so Google). I use this code in production... 😟

I wrote the original client code the day after the API was released. I notice that they have updated a few things last week, so hopefully the two endpoints are now the same. I need to test and also fix #12 (which should easy). I really need some CI tests to run too 😉

Holding this here for records.

from google-generative-ai-rs.

I agree the current implementation is not great. It suits my needs though ;). I'll look into a 'third' path to release the returns as they come rather than 'packing' into a struct list - basically opening up the client for each response. See note that it will still be blocked to some degree due to the design decisions made by Google.

Not sure of your use case, but the Gemini API doesn't really stream the output - i.e. it does not give the output token by token / character by token, but gives it out in 'chunks', including harm and other data. It is a strange design decision by Google IMO, but I get the control over the harm and exit conditions, rather than hiding it all as OpenAI does and saying the request failed...

from google-generative-ai-rs.

@hitsmaxft found some time today as I needed to implement the countTokens handling.

I've merged your code into main as well as the countTokens functionality. Your changes broke the usage of Vertex AI endpoint, but it was an easy fix.

I bumped the version to 0.2.1 to show that it was breaking changes, as well as addition functionality.

Thanks for your help.

from google-generative-ai-rs.