Phi 3 128 about mlc-llm HOT 11 OPEN

r u run phi3 ok?

from mlc-llm.

r u run phi3 ok?

What do you mean?

from mlc-llm.

The 128k version could allow us to analyze a pdf from the phone. the 4k version would probably be less cpu intensive and would work out of the box.

from mlc-llm.

sorry.i want to known that are yiu run phi3 mini model on the phone?i i run it with onnxruntime genai but it's too slow

from mlc-llm.

Well I use phi2 all the time using ml. Android on my phone with Snapdragon 2nd gen and it works really good and smooth. That is why I would like to know how I can put phi 3 on mlc.

from mlc-llm.

Have you tried following the instructions for building new models?

from mlc-llm.

Tried following the instructions, and got the error:

ValueError: Unknown model type: phi3. Available ones: ['llama', 'mistral', 'gemma', 'gpt2', 'mixtral', 'gpt_neox', 'gpt_bigcode', 'phi-msft', 'phi', 'qwen', 'qwen2', 'stablelm', 'baichuan', 'internlm', 'rwkv5', 'orion', 'llava', 'rwkv6', 'chatglm', 'eagle']

from mlc-llm.

I would try to add the string "phi3" to the list

from mlc-llm.

@kripper thanks for your advice, the MODELS is not really just a list with a model name, it is a dict with the model name as key and the corresponding loader and implementation.

I've tried specify the --model-type phi-msft but without luck.

from mlc-llm.

I would first research if the model loader is compatible with previous versions. If so, map the ID to the existing loader. Otherwise, create a new loader and have fun hacking, ie:

• research what changed by comparing (diff) the official loader for the previous and new versions.
• replicate those changes in mlc-llm's loader

Always prefere to share a same code base for the old and new model and use conditions (if(version == 1) {...} else if(version == 3) {...}).

from mlc-llm.

Once this is figured out please update the android app apk with phi3. ❤️

from mlc-llm.