= and += about jison HOT 9 CLOSED

The lexer will return the first rule to match, so if "+" comes before
"+=", it will return the "+" token and then "=" as separate tokens.
Put the longer rules before the shorter ones to avoid that.

On Sat, Aug 20, 2011 at 12:53 PM, ProjectMoon
[email protected]
wrote:

This almost assuredly a problem where I don't understand quite how to use the lexer. My lexer specifies "=" as '=' and "+=" as 'PLUSEQUAL'.

However, the parser will fail with stuff like:

x += 5;

It will say expecting an = sign instead. x = 5; works fine. I'm guessing this has something to do with precedence that I'm missing. I'm using the jscore example. I added a lexer to the file.

The grammar is here: https://gist.github.com/1159333

Reply to this email directly or view it on GitHub:
#58

Zach Carter

from jison.

I see. Is this in the documentation at all?

from jison.

This has fixed the problem.

from jison.

This is a good point, since flex will use the longest matching rule. I may expose an option to have it behave as flex does. Maybe in the future that behavior will be the default.

from jison.

I agree that the lexical part should match the longest rule. Since this makes grouping the same attribute easier.

Flex

%%
i(sabella)?  { printf("noun"); }
is           { printf("verb"); }
s            { printf("alpb"); }
%%

Both i and isabella return noun, while is return verb.

Jison

("i"("sabella")?)   alert("noun")
("is")              alert("verb")
("s")               alert("alpb")

Both i and isabella still return noun, but is return noun and then alpb. To fix this, I need to forget the powerful regex and rewrite this code like:

("isabella")        alert("noun")
("is")              alert("verb")
("i")               alert("noun")
("s")               alert("alpb")

That's make the lexical part too heavy.

from jison.

Maybe this issue should be reopened since people are commenting on it.

from jison.

Yes, I'll leave this open until we have an option for longest rule matching.

from jison.

There's now a "flex" mode where the lexer uses the rule with the logest match. Just put:

%options flex

in your lexer grammar before the rules section.

from jison.

wow! thank you for improving this. :)

from jison.