generalized constrained token inference interface about llm-vm HOT 11 CLOSED

After doing a bit more research I now understand that compiling regex to DFAs is not as straight-forward as I thought. However, I've read through the lmql paper and they expound on the idea of partial evaluation and final semantics to evaluate the state of a generated sequence with regard to a given constraint expression. Using this concept we can eagerly prune a subset of valid tokens in the crude_filter_set that guaranteed to violate the expression. Will try to implement these ideas for generating the final mask for allowed_transitions_from_state. Feel free to correct me if there's something I'm missing

from llm-vm.

derp, posted before done editing

from llm-vm.

anyways, this is just my fuzzy thoughts, @itsmeashutosh43 and @VictorOdede , honestly you guys can drive the figuring this out design wise

from llm-vm.

I think it would be more intuitive to implement 2 separate classes for generate_simple and generate_with_constraints where they both inherit from the same parent class. Also would this mean a rewrite of the current onsite_llm implementation?

from llm-vm.

For the TokenConstraint class, what other types of constraints are we looking to add apart from regex?

from llm-vm.

Everything else is kinda straight forward since it can all be done in parallel with next token generation using a logit processor. I just need some clarification on what you mean by state_type and copy_state?

from llm-vm.

generic interface which those would be examples of.

I'd like to point out that "those being examples" of a generic interface does by no means imply this is trivial. Compiling regex to DFAs is a pain. These languages are important.

from llm-vm.

I currently have a partial solution for this issue but I need some clarification on implementing the language constraints. Is the process to parse cfg to regex then compile regex to dfa then use the dfa to filter accepted tokens?

from llm-vm.

Hey @cartazio, I found a way to implement cfg constraints without needing to scan the entire vocabulary for each generation. Since the outlines lib doesn't support this yet, can I add it to Abhighya's guided generation implementation?

from llm-vm.

@VictorOdede Is this done?

from llm-vm.

Yes. This was addressed in #154

from llm-vm.