Meilisearch 1.8.0-rc.1 stuck process when adding hugging face and ollama embedders about meilisearch HOT 14 CLOSED

Alright I have done some more investigating and here is what I have found so far. On rc.1 it does complete it is just way slower. I did not add any documents to the index. On rc.1 it takes ~30 seconds (sometimes much longer though), the same thing takes ~0.148 seconds on rc.0.

My current theory is this delay is causing my test suite to time out and error, which would explain why it looked like it never finished processing.

For reference this is the exact JSON that is being sent:

{
  "embedders": {
    "default": {
      "source": "huggingFace",
      "model": null,
      "revision": null,
      "documentTemplate": null,
      "distribution": null
    }
  }
}

The same thing happens when I try ollama. As far as I can tell these are the only two that do it.

from meilisearch.

Updating with a new log message I just saw while running tests on rc1 again. I assume this is related because it stalled at this point.

WARN milli::vector::rest: Failed: runtime error: could not reach embedding server: http://localhost:11434/api/embeddings: Connection Failed: Connect error: Connection refused (os error 111)

from meilisearch.

I just realized it also explains why I couldn't find it with a bisect. This first time it would be slow, then the model would be cached so it wouldn't happen again until I removed the volume and tried again.

from meilisearch.

hello @sanders41 thanks for the report.

the team will work on it as soon as possible! 😊 We will fix this before v1.8.0

from meilisearch.

Hello @sanders41,

I don't manage to reproduce your bug exactly,
Did you put any documents in your index? If so, could you let me know if you inserted your documents before changing the settings?

I've tried the following process:

1. Put documents

curl -XPOST localhost:7700/indexes/movies/documents -H "Content-type: Application/json" -d @movies.json

2. change settings

curl \
  -X PATCH 'http://localhost:7700/indexes/movies/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "embedders": {
      "default": {
        "source": "huggingFace"
      }
    }
  }'

Is it right? or do I miss something?

Thank you for your report!

from meilisearch.

Oh! Ok!
So it's a slowness. Thank you for the precisions I am currently investigating the issue.
I plan to fix it for v1.8 and maybe for the next rc.

from meilisearch.

Hello @sanders41,
I made some improvements on the embeddings part, but I am not 100% sure if it fixes your issue.
I built a docker image so that you can try it on your side:

docker pull getmeili/meilisearch:prototype-embeddings-opti-0

Let me know if it improves your use case.

from meilisearch.

@ManyTheFish I tried this container and the numbers I'm seeing are inconsistant. The first time I ran it it took around 60 seconds for completion, the fastest run I have seen is 8 seconds.

Now here is the even weirder part. I tried to do a git bisect on the release-v1.8.0 branch starting at the rc0 tag to see where the issue started. Doing this I wasn't able to find the exact commit that caused the issue because sometimes the issue happens and sometimes it doesn't. I tried using your branch directly and I seem to get around 2 seconds consistantly.

One thing I did notice while running the bisects is when things are slow pytorch_model.bin always shows up in the log as the thing taking all the time.

from meilisearch.

Ok, it's weird,
so, it means that it's not completely related to the changes between rc0 and rc1 :/

Let's keep this issue open. I will ask for a second eye in the engine team.

Thank you for the report!

from meilisearch.

Hello @sanders41,
Because the bug could come from the external embedder API and because I made some minor enhancements in rc.2,
I close this issue, don't hesitate to reopen it or open a new one if I am wrong,

thank you again for the report!

See you

from meilisearch.

Hi @ManyTheFish with further testing I can say that my test suite fails every time with rc.1 and rc.2, but never fails with rc.0 (no changes to the code, only which docker image is used) so I still think something is wrong. I also verifed with curl that I have the same issue. The only way I can get my test suite to pass is to disable any test that updates the settings for HuggingFace or Ollama so I have done that.

So with that said I can get my tests to pass by ignoring these embedders so closing is OK with me.

from meilisearch.

Pinging @dureuill here to be sure you did not miss this discussion. Can interest you when you come back 😊

from meilisearch.

Hello @sanders41 👋

Up to RC0, we would download the model from the Hugging Face Hub and make a call to ollama only on adding the first document needing to be embedded.

Starting with RC1, we perform these operations right when updating the settings. If the model from the Hugging Face Hub is not in the local cache of the machine, performing this download operation is going to take time.

Performing these operations when updating the settings allow for more intuitive errors. It also does not pessimize performance in the general case, because that cost already existed before RC1, it was just delayed up to the first document addition.

Thank you for your report ❤️

from meilisearch.

Thanks @dureuill this definitely explains the issue I was seeing. Knowing this I can either patch the route in tests, or really skipping HuggingFace and Ollama should be fine in my tests now that I know the reason and that settings updates are working.

from meilisearch.