[New Model]: mistralai/Codestral-22B-v0.1 about vllm HOT 3 OPEN

@eduardozamudio can you please share the version of vLLM used and the error of why the model won't load? It is a MistralForCausalLM, so I would expect it to run as Mistral models do.

from vllm.

Hi @eduardozamudio (Hola Eduardo!)

mistralai/Codestral-22B-v0.1 worked for me using vllm 0.4.3.

Regards
Matias

from vllm.

Hi @eduardozamudio (Hola Eduardo!)

mistralai/Codestral-22B-v0.1 worked for me using vllm 0.4.3.

Regards Matias

Does it work for "fill in the middle" https://huggingface.co/mistralai/Codestral-22B-v0.1#fill-in-the-middle-fim? I image there would be some work required to support the prefix and suffix params both in the rest API and core APIs...

I haven't dug into the https://github.com/mistralai/mistral-inference code yet, but I think it's just uses special tokens to mark prefix, suffix and middle, so it can probably also be implemented outside of vllm and just passed as the normal input...

from vllm.