[Bug] Token IDs not accepted by JSON grammar about mlc-llm HOT 7 CLOSED

thanks @dtkettler can you also let us know what is the model and prompt? having an example script would be helpful

from mlc-llm.

Sure. I have run into this with every prompt I have tried, but here's a very simple example adapted from the basic usage in the MLC-LLM documentation:

from mlc_llm import LLMEngine

# Create engine
model = "HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC"
engine = LLMEngine(model)

# Run chat completion in OpenAI API.
response = engine.chat.completions.create(
    messages=[{"role": "user", "content": """
Produce a JSON object with a list of US states and the largest cities in those states
The list should have the following keys:
'state' - Name of the state
'cities' - Another list containing city names"""}],
    model=model,
    response_format={"type": "json_object"},
    stream=False
)

print(response)

engine.terminate()

from mlc-llm.

Thanks for the report @dtkettler Currently there are several issues with llama3 because it changes the tokenizer a lot. That will be fixed soon in these days

from mlc-llm.

I'm seeing a similar error when using phi-2-q4f16_1-MLC

InternalError: Check failed: (accepted) is false: Token id 0 is not accepted by the grammar state matcher

from mlc-llm.

llama3 should be fixed by latest version

from mlc-llm.

Hi,

I tried it out and yes Llama 3 runs without error now. However I still run into a pretty serious issue with it. If I specify a schema in the response_format it just...returns the schema back to me. Like, the output is literally just the schema I gave it, not an actual response. If I just change the model to something else with no changes to my code I do not encounter this behavior so it's something specific to Llama 3.

from mlc-llm.

Thanks @dtkettler , there are some fixes that just come in now, so please wait another day for the effect to kick into nightly.

when runnning json mode, likely you want to prompt your model and ask for it to output json

from mlc-llm.