Support ollama's new `keep_alive` request parameter to prevent model unloading about continue HOT 7 CLOSED

30 minute is definitely better than 5, but honestly my use case is probably not the typical one. I have a RTX 3090, so plenty of VRAM to have a 6-7b model loaded all the time. The ollama devs justified their default of 5 minutes to not hog VRAM on smaller GPUs, which is probably the more common use case.

For me I would prefer to disable unloading entirely (keep_alive: -1), since I can just stop the ollama docker container when I'm done with my development session. I imagine it would be useful to configure this per-model, since the size of the model is probably the biggest factor affecting this setting. For example, just exposing this as a configuration option in config.json for ModelDescription would be super useful for me.

Thanks for your work on this extension, it's been great watching its development and has been fun to use!

from continue.

Love the idea! Working on tab-autocomplete release probably for the next couple of days, but will add this later this week

from continue.

@noahhaon @johnozbay I can't believe it took me this long to make such a simple commit, but here it is!: 2614465. Will be in next pre-release

I was a bit concerned about adding tons of parameters, but to be honest this one feels pretty fundamental. In the future if there are any others you want that we don't support, we also recently added requestOptions.body to config.json, so you can add arbitrary params to the HTTP POST request body

Now we can all sip piña coladas by the pool while the GPUs/CPUs are writing our code : )

from continue.

Joining @noahhaon here to say that, I too have plenty of vram sitting around idle sipping piña coladas, and would love to make better use of it! 🍹

Having a keep_alive : -1 or a similar config parameter that allows keeping the model loaded for at least 6 - 8 hours to cover a working day would be amazing. Otherwise, going to a 30min meeting and coming back to the desk requires a much longer wait time for the model to load up. It would improve Continue's responsiveness by a very large margin.

— btw, thanks a lot for making continue @sestinj! This is a total game changer, especially for coders who have businesses or clients etc that are more privacy-sensitive. 🙏🏻

from continue.

@sestinj You're the legend! Thank you so much!
Also, love the requestOptions addition idea, will definitely help a lot of users!

from continue.

😍

Thank you @sestinj ! I thought it would be about as simple as your commit when I first looked through the code, but I didn't want to assume 😅 This will be very valuable, and give some nice forward compatibility as ollama develops too. Thanks!!

from continue.

@noahhaon out of curiosity, what would you plan on setting this to? I am going to set a higher default even if the user doesn't configure it, and if this would be enough for your case, I tend to like avoiding too much extra configuration if it's not necessary. I was thinking 30min for default.

Either way will add this default today and if config is necessary, then will do that later

changes: 2fab0e2

from continue.