Lexical Tie-In Support? about jison HOT 3 CLOSED

Sort of. You can set flags during semantic actions for the lexer to observe and obey. The yy object can be used to share state within actions. Jison does not support actions mid-rule, though, so it would have to be broken up.

I've done similar things on the scanner level, where scanning certain tokens changes the state to modify the result of the next scan, but at the parser level you do get more contextual power.

But there's also a catch 22 situation that is related to issue #12. The lookahead token is used to determine if the parser should reduce, but the semantic action of the reduction may modify the state such that the next token is actually something different, as with lexical tie-ins.

Currently, any state modifications by the semantic action won't be seen by the lexer on the next token it shifts, because it was already shifted for the lookahead token and cached. Only on the token after next will the lexer observe the new state, so you can sort of do this today with a sort of token padding.

Perhaps the solution to this as well as #12 is to simply rescan the input after the actions runs. It just warps my mind that the lookahead token could be different than next actual token. I'll probably ask on the Bison list how this works also.

Hrmm, anyway, I'll work on this while resolving issue #12.

from jison.

Ah, I received a nice explanation from the Bison list. The key is that the lookahead token is not needed when there is only one possible action at a given state. So for lexical tie-ins, there should only be one possible action – a reduction – so the semantic action could execute before the lexer scans for the next token. I'll work on this tonight.

from jison.

Jison now supports proper lexical tie-ins since cf6a3b1

from jison.