llama-cli: command not found about localai HOT 5 CLOSED

Hi @tjthejuggler !

Indeed the first command starts a new instance each time, and it is used or for troubleshooting and/or automating things by piping commands to it. To have a long-running instance, start it in API mode:

docker run -p 8080:8080 -ti --rm quay.io/go-skynet/llama-cli:v0.1 api

And in another terminal run inferences with curl:

curl --location --request POST 'http://localhost:8080/predict' --header 'Content-Type: application/json' --data-raw '{
    "text": "What is an alpaca?",
    "topP": 0.8,
    "topK": 50,
    "temperature": 0.7,
    "tokens": 100
}'

The API will keep the model loaded into memory, and it's a long running process

from localai.

hey @mudler, thanks so much for the help! I've got another question, everytime I run it, it makes me download that 3.839gig file again. I don't know where it is downloading it to, I can't seem to find any file that size on my HD. I assumed with it's size that it was the 7B model, so I tried pointing it that model which I already have downloaded, but it still wants to download the 3.839gig file again.

$ sudo docker run -p 8080:8080 -ti --rm quay.io/go-skynet/llama-cli:v0.1 api --model /models/ggml-alpaca-7b-q4.bin
Unable to find image 'quay.io/go-skynet/llama-cli:v0.1' locally
v0.1: Pulling from go-skynet/llama-cli
32fb02163b6b: Already exists
167c7feebee8: Already exists
d6dfff1f6f3d: Already exists
e9cdcd4942eb: Already exists
543368fb39ee: Already exists
5898d990df6b: Already exists
9602be2ba0fe: Already exists
dda7abc9e477: Pull complete
13679b03456b: Pull complete
c5704ac31306: Pull complete
8f2899c04205: Downloading 11.34MB/68.38MB
c829f586020d: Download complete
0837277f1cf1: Downloading 10.27MB/3.839GB
05ea17c3de8f: Download complete

Thanks again, I really appreciate your time and effort!

from localai.

looks like there is something wrong in your docker installation, images shouldn't be cleaned up between calls, how did you installed docker?

from localai.

@mudler

I have no experience with it, I hadn't even heard of it until setting up your project. All I did to set it up was follow these instructions exactly:

https://docs.docker.com/engine/install/ubuntu/

I will look into debugging it knowing that the issue is that the image is being cleaned up between calls. Thank you!

from localai.

The issue has been solved! when I ran 'sudo docker images' I saw that the image was listed in there and tagged 'latest', but when i was running the command it had ':v0.1' at the end, i switched to it ':latest' and it worked beautifully. Thanks so much, I really appreciate it!

from localai.