Script allows sending 'infinite' amounts of text, but without context/memory about infinitegpt HOT 4 OPEN

if you are using the chatgpt endpoint and set a conversation token, it should be capable of continuing the same conversation within its own memory context. i have tested this in previous integrations and it works. haven't tried this project yet though.

but you are right. i glanced at the script and was trying to understand the value-add. perhaps some examples of a supported example usage would be good to put in the readme.

from infinitegpt.

I had the same thought. I believe the endpoint is stateless. Happy to be corrected.

from infinitegpt.

if you are using the chatgpt endpoint and set a conversation token, it should be capable of continuing the same conversation within its own memory context. i have tested this in previous integrations and it works. haven't tried this project yet though.

but you are right. i glanced at the script and was trying to understand the value-add. perhaps some examples of a supported example usage would be good to put in the readme.

Yes, if you use something like conversationbuffermemory in the script, you can create a form of memory for the API, but it is done by sending the entire conversation back through the API after each completion, and is limited by the max_tokens of the engine, so 4K/8K/32K(comingsoon-tm-). You can never extend this memory beyond the API token limit for prompt/completion combined. So anything of any significant length will very quickly run up that memory, not to mention that each call would cost incrementally more for each call.

Having written a couple of scripts, trying different forms of memory and context methods, I found it to be reliable, but costly, and that is not counting the 32K model which would push the costs even higher.

I believe that open source llm models will very shortly be both readily available, have a lower 'bar' required to effectively train and use them, and as already shown in many open source llm's, very capable on a fraction of the computing power. That's when we will be able to start using near unlimited size of data,, limited by local compute power and effective memory storage solutions. For now, I have a tough time thinking of a use case for this method, as it is not very effective, highly limited and costly.

I still appreciate the authors idea and work though, and hope it will help inspire others!

from infinitegpt.

In my experience it helps to try to be explicit if you wish to carry state forward (e.g. this kind of thing)

One idea would be to ask for a summary, or compression, of previous steps. Though that is a bit orthogonal to the functionality provided here.

from infinitegpt.