Feature request: configurable resource usage about serge HOT 3 CLOSED

args = (
"llama",
"--model",
"/usr/src/app/weights/" + model,
"--prompt",
prompt,
"--n_predict",
str(n_predict),
"--temp",
str(temp),
"--top_k",
str(top_k),
"--top_p",
str(top_p),
"--repeat_last_n",
str(repeast_last_n),
"--repeat_penalty",
str(repeat_penalty),
"--threads",
"4",
"--n_parts",
"1",
)

I was wondering why it wasn't using my full CPU. The constraint seems to come from api/utils/generate.py line 46 where the args are defined. Modifying the the number below "--threads" allows you to use more cores. Until there's a GUI option for it, you can modify the file directly.

from serge.

Yes I need to add more options to the UI, especially the threads!

from serge.

I added an option to change the number of threads in the UI!

For multiple instances in parallel, it's a much bigger change that's a bit out of scope for the moment. I think this is okay for now.

from serge.