Option for pre-loading specific models into memory about serge HOT 8 CLOSED

Here are some thoughts :

• tweet from Justine tunney about loading model with mmap
• using mmap in python

But maybe a simpler way would be to declare a path as a tmpfs in the docker-compose file, and have the api code copy the files into that tmpfs location at startup.

Another thought : using https://github.com/hyperonym/basaran

I'm curious what other ideas people will come with on that matter

from serge.

Hi, on my side I already tested by mounting in tmpfs the /var/lib/docker directory and the repository directory (manually via the unix system), and it's hardly faster if you already have an nvme, in any case I didn't notice much difference.

from serge.

On the other hand I wonder if increasing the priority of the processes at startup when it generates a discussion would be interesting ex: "chrt -f 90 llama".
And above all, wouldn't it be more efficient to run the process directly outside of docker.

from serge.

Just clone and build branch with mmap allocation, and yes is faster than main branch, is like instant when process is running.
https://github.com/ggerganov/llama.cpp/tree/mmap
I try some modification for allocation on memory in main.cpp.

from serge.

Here is main version :

from serge.

And version custom with mmap allocation (i added some code to enable back avx512 on this version).

from serge.

Sorry here is last main version previous screenshot is an older main version than 2 days before i think.

from serge.

Closed via #129

from serge.