Streaming response about llm-as-chatbot HOT 7 CLOSED

I am planning to add streaming feature soon! :)
in that case batch requests handling feature will be disabled (well will remain as an option)

from llm-as-chatbot.

check this out
: https://twitter.com/algo_diver/status/1638079375085305856?s=20

from llm-as-chatbot.

just updated the repository and experimentally running here
: https://notebookse.jarvislabs.ai/BuOu_VbEuUHb09VEVHhfnFq4-PMhBRVCcfHBRCOrq7c4O9GI4dIGoidvNf76UsRL

from llm-as-chatbot.

Can you elaborate on how you would implement it ? Maybe I can help. From what I understand it helps reducing the latency in a way such that the user wont have to wait until the full response is returned.

from llm-as-chatbot.

Thanks @alexanderfrey

Hugging Face library does not support streaming feature out of the box, so I thought I need a sort of money patch. Luckily, I found one from different open source project
: hyperonym/basaran#57

if it turns out hard for me to implement streaming feature in the current version, I will let you know!

from llm-as-chatbot.

@alexanderfrey

in the streaming, most of the parameters in GenerationConfig are not supported. Do you think you can make that happen?

from llm-as-chatbot.

let me have a look tonight. can not promise but definitely interested to contribute

from llm-as-chatbot.