bug: There is no way to load a model from the docker local server about jan HOT 10 CLOSED

Could you confirm if you want to modify the container's 'models' folder from the host machine? If so, I think you should consider using a Docker Mount Volume.

The data within the container should be sandboxed after it is deployed.

from jan.

It looks like there are API commands to stop and start models, however, my models are still empty

from jan.

Walking through this code to figure out why GET /models returns an empty array. I'll add console.debugs everywhere and post the result

from jan.

Never mind, I cannot even get to that function. I get booted at getBuilder:

The output of that console.debug is the following:

When I docker exec -it <containerId> /bin/bash into the instance and run the following command, the models folder is not there:

Before I ran the docker, I created a symlink in jan/server/build/models to point to jan/models, yet, it appears they did not copy

from jan.

I can finally get the list populated:

cd jan
docker-compose --profile cpu-fs up -d
docker exec -it $(docker ps -q --filter ancestor=jan:latest) /bin/bash

# Once in docker, run:
cd server/build/jan/ && ln -s ../../../models models && exit

Attempting to start the model now...

from jan.

I can finally start the Mistral model with the following command:

curl -X PUT 'http://localhost:1337/v1/models/mistral-ins-7b-q4/start' -H 'Accept: application/json' -H 'Content-Type: application/json' -d '{}'

Output: {"message":"Model mistral-ins-7b-q4 started"}

In summary, to fix this problem at the developer level:

• Add a symlink in /app/server/build/jan called models and point it to /app/models during the Docker build process (Dockerfile and Dockerfile.gpu)
• Update Swagger YAMLs and documentation to expose the PUT /model/:model_id/start and PUT /model/:model_id/stop commands
• Update Getting Started documentation to point out that the models have to be started when in Local Server mode

from jan.

I'll provide updates on how to bake this solution into the Dockerfiles at some point in the close future

And probably pack that into a PR

from jan.

Adding this after the COPY commands in the Dockerfiles should work (Branching appears protected)

RUN cd server/build/jan/ && ln -s ../../../models models

from jan.

Could you confirm if you want to modify the container's 'models' folder from the host machine? If so, I think you should consider using a Docker Mount Volume.

The data within the container should be sandboxed after it is deployed.

I'll check to see if VOLUMES works with and without the symlink

from jan.

Please let us know if this works, @HyperCrowd. We will close it as not planned for now. Please feel free to reopen it.

from jan.