zaach / jison Goto Github PK

Hi,

I've just downloaded a clean copy of 0.2.9 and am getting the following error:

Thu May 12 2011 13:54:47 GMT+0300 (EEST) [error] Could not load package 'jison'. TypeError: Cannot find function forEach in object [object Object].
Error: require error: couldn't find "jison/cli-wrapper"

lib/jison/cli-wrapper.js is there, so I'm not sure what's causing it. Any ideas?

It was working before I updated.

Thanks,

Marcus.

Instead of writing grammars in JSON, an easier to author format could compile to JSON. It should look similar to Bison's .y files.

When finding the lookahead token, the input should not be advanced. This way, semantic actions can interact with the input stream as it should be, unaltered by future tokenization.

http://dinosaur.compilertools.net/flex/flex_11.html

An interesting suggestion that may be possible to implement, though probably not efficiently.

After install 0.2.5 on npm 1.0pre, the jison folder only include these files:

Makefile README.md package.json

Remove the "files" key in package.json will solve this bug.

It appears that if there are extra brackets at the end of code executed when a state is reached, even if they are properly balanced, they are not parsed correctly unless there is a space before them.

e.g.

TEST
    :
        {{if(true); else {} }}
    ;

works, but

TEST
    :
        {{if(true); else {}}}
    ;

results in a parse error.

Cheers,

Marcus.

When I try to to use @$ within a parser rule to refer to the location of a match, the javascript compiler just tells me I have an unexpected token ILLEGAL.

The same happens with @1, @2 etc...

I am using jison programatically, if that matters, and not generating a parser from a bison-style file.

I am using CoffeeScript (w/ Jison) on AppEngine/JVM/Rhino. The generated CoffeeScript parse table is too large, and causes coffee-script.js to fail loading on Rhino due to a JVM limit that methods can be 64k max in size.

Would it make sense to somehow break larger parse tables into 2 or more tables, so Jison, and projects that run Jison, can be safely loaded on Rhino/JVM/AppEngine?

Hi Zach.

Upgrading from Jison 0.2.0 to 0.2.2 and rebuilding the CoffeeScript compiler causes the generated parser to fail. Looking at the diff, it appears that an lstack array was introduced, into which is pushed two different types of objects: the yyloc location, and the yyval._$ object, which have different properties. The first time the _$ is created, it assumes the existence of a previous object in the stack, causing things to break like so:

TypeError: In Cakefile, Cannot read property 'first_line' of undefined
    at Object.parse (/Users/jashkenas/Desktop/Projects/coffee-script/lib/parser.js:621:63)
    at /Users/jashkenas/Desktop/Projects/coffee-script/lib/coffee-script.js:26:22
    at Object.run (/Users/jashkenas/Desktop/Projects/coffee-script/lib/coffee-script.js:55:28)
    at /Users/jashkenas/Desktop/Projects/coffee-script/lib/cake.js:41:20
    at path.js:271:19

Here's the diff of the generated parser.js:

https://gist.github.com/878709

Hope that helps.

Hi,

Just looking at the docs, so haven't used it yet, but seems it might be nice to allow functions as an option in place of evaluable strings... For example, in place of "$$ = $1 + $3;" allowing:

function ($1, $2, $3) { return $1 + $3; }

Easier to format, play around with, take advantage of syntax highlighting, and error-checking, etc.

Brett

Hi,

I'm seeing the following error when running the latest master using Narwhal:

Wed Jul 27 2011 23:58:06 GMT+0300 (EEST) [error] Could not load package 'jison'. TypeError: Cannot find function forEach in object [object Object].
Error: require error: couldn't find "jison/cli-wrapper"

The error appears to be caused by the 'dependencies' field in the package.json file. This caused a similar error in a previous version.

When I remove the dependencies field, I get the following error:

Error: require error: couldn't find "nomnom"

Cheers,

Marcus.

It's referred to in examples/classy.jison and examples/classy_ast.jison.

The Jison will be used wildly, so add WScript/CScript support please.
It can be used on a command line without installing some modules.

Possible ways to trim bytes from the generated parser:

• collapse duplicate semantic actions
• use nested ternaries instead of a switch statement for actions
• pass semantic values as arguments instead of the whole stack to performAction (same for location values). This would get rid of dynamic $$[$0-n] expressions
• remove error recovery code from parsers that don't need it

check to see that performance is not negatively impacted by these.

Like in lexical tie-ins in Bison.
Does Jison support these?

I tried a lexer pattern "\u03c0" but jison didn't understand that this was a unicode character.

Hi, Im implementing an CSS parser with Jison, but sometimes I will need to use just a context of grammar, instead it all.

For example, a full parse works ok with this:

body { background: #fff }

but sometimes I just wanna parse the Attributes context, like:

background: #fff

Will be nice to be able to do something like:

attributes = parser.parse("background: #fff", "Attributes")

This almost assuredly a problem where I don't understand quite how to use the lexer. My lexer specifies "=" as '=' and "+=" as 'PLUSEQUAL'.

However, the parser will fail with stuff like:

x += 5;

It will say expecting an = sign instead. x = 5; works fine. I'm guessing this has something to do with precedence that I'm missing. I'm using the jscore example. I added a lexer to the file.

The grammar is here: https://gist.github.com/1159333

The included RegExp lexer basically wraps JavaScript regular expressions in a Lex-like API. A real, DFA based lexer would provide numerous benefits, namely streaming support for parsing.

The generated parsers test for the existence of require. If require is defined, generated parsers immediately use exports without checking if exports is also defined.

From Handlebars.js@398:

if (typeof require !== 'undefined') {
exports.parser = handlebars;

If I am also using RequireJS (included in a previous <script /> tag or packaged up in the same packaged script together with Handlebars) then require will be defined but exports will not be defined.

RequireJS uses an alternative system for defining and requiring modules, which, unlike the standard system that Node.js uses, does not rely on exports. This is because RequireJS is optimized for the browser.

As a simple fix, Jison should check for the existences of both require and exports before using exports.

Cheers!

Hi,

I've now got used to where conflict errors are likely to happen, but I find the conflict error messages rather opaque at times: e.g.

Conflict in grammar (state:99, token:catch)
   reduce by CATCHES -> 
   shift 230

The thing that can sometimes be opaque, is the state number. It would seem more useful to me to have a message that gave say the name of the pseudo-token that is being checked, perhaps with the number of the alternative. I realise that this may well add extra code to the resulting parser, it only needs to happen at build-time, and I think would make it much easier to find the sources of errors for some people.

Also what do the 'shift' messages mean?

Thanks,

Marcus.

I would like the ability to recursively call parse from within the code hooks in my bnf.

My use case is implementing string interpolation in a programming language. My lexer will recursively tokenize the contents of embedded code in a string and give yyval the embedded token stream. I then want to recursively call parser.parse from within the string token in my bnf.

I can't seem to find a way to get at the current parser instance from within the callback. The this scope is just an object with one key $. Would it be possible to add a reference to the current parser instance to the this scope.

Also, while not required, it would help me a lot if I could specify the starter non-terminal for this recursive parse. I don't want to start at my root node, but at my expression node for the embedded code.

We're adding (line, col)-to-(line, col) mappings in CoffeeScript. [1] [2]

It would be nice if Jison had yycolno available.

[1] https://github.com/jashkenas/coffee-script/issues#issue/558

[2] https://github.com/andrewschaaf/coffee-script/commits/master

We could convert EBNF grammars to BNF. For this, we need an EBNF parser, a JSON representation, and a translation algoritm to convert the EBNF JSON to BNF JSON.

If I put an if...else inside a rule's command, jison chokes on the extra braces. I can work around this by moving the code to a function, but it's annoying.

It seems the code on
http://zaach.github.com/jsonlint/

fails on inputs that aren't inside arrays or objects, such as just a single Number. For example, successful parses using JSON from node.js,

JSON.parse("123")
JSON.parse('"xyz"')

etc. If I'm running things correctly, this also fails on the example in jison repo.

Reduce the footprint of free "yyvar" variables by reusing the yy shared scope variable to store them. This diverges visually from Bison actions, but semantically it is more representative of how Jison works, i.e. non-global.

I think it would be nice if the default error message for a ParseError mentioned the unexpected token that was encountered, instead of the list of expected tokens. For me, it almost always ends up being half the grammar, like this:

Error: Parse error on line 4. Expecting: 'OUTDENT', 'STRING', 'JS', 
'REGEX', 'BREAK', 'CONTINUE', 'TRUE', 'FALSE', 'YES', 'NO', 'ON', 
'OFF', 'ASSIGN', 'RETURN', 'COMMENT', '!', '!!', '-', 
'NOT', '~', '--', '++', 'DELETE', 'TYPEOF', '*', '/', '%', 
'<<', '>>', '>>>', '&', '|', '^', '<=', '<', '==', '!=', 
'IS', 'ISNT', '&&', '||', 'AND', 'OR', '?', '-=', '+=', '/=', 'THROW', 
'WHILE', 'FOR', 'OF', 'LEADING_WHEN', 'IF', 'ElsIf', 'UNLESS'
9

Instead of what would really be helpful to see, which is:

Error: Parse error on line 4. Unexpected 'COMMA'

When I run this command jison calculator.jison I get this error

        throw e; // process.nextTick error, or 'error' event on first tick
        ^
Error: Parse error on line 2:
\s+                 ...
-----^
Expecting '<', '(', '/', '/!', '.', '^', '$', '{', 'ANY_GROUP_REGEX', 'ESCAPE_CH
AR', 'STRING_LIT'
    at Object.parseError (/usr/local/lib/node/.npm/jison/0.2.7/package/lib/jison
/util/lex-parser.js:109:11)
    at Object.parse (/usr/local/lib/node/.npm/jison/0.2.7/package/lib/jison/util
/lex-parser.js:185:22)
    at Object.parse (/usr/local/lib/node/.npm/jison/0.2.7/package/lib/jison/jiso
nlex.js:6:19)
    at /usr/local/lib/node/.npm/jison/0.2.7/package/lib/jison/bnf.js:37:21
    at Object.addDeclaration (/usr/local/lib/node/.npm/jison/0.2.7/package/lib/j
ison/bnf.js:12:23)
    at Object.anonymous (/usr/local/lib/node/.npm/jison/0.2.7/package/lib/jison/
util/bnf-parser.js:16:30)
    at Object.parse (/usr/local/lib/node/.npm/jison/0.2.7/package/lib/jison/util
/bnf-parser.js:237:40)
    at Object.parse (/usr/local/lib/node/.npm/jison/0.2.7/package/lib/jison/bnf.
js:4:54)
    at processGrammar (/usr/local/lib/node/.npm/jison/0.2.7/package/lib/jison/cl
i-wrapper.js:72:36)
    at Object.main (/usr/local/lib/node/.npm/jison/0.2.7/package/lib/jison/cli-w
rapper.js:48:46)```

Hi,

I can use RegExpLexer with some lex rules and some input and a callback such that the callback gets called for every token that is found:

  tokens = (lexData, input, cb) ->
      lexer = new RegExpLexer lexData, input
      while (tok = lexer.lex()) and tok isnt 1
          cb tok

But if I want to have bison type functionality as well and use the bnf, and have a Parser, I lose this ability. I've tried to figure out how to do this, but to no avail

As in, all I can do is this

parsed = (grammar, input, cb) ->
     cb new Parser(grammar).parse()

Which only calls the callback with the final result (which I also want :p)

Is there an easy way of hooking into the parser, such that I can make it call some callback with each token it finds ?

Thankyou.

Hi,

I'm trying to implement a parser for a custom programming language using Jison. I'm trying to implement a statement like

return  [val]

which could also just be

return

The problem I'm having, though, is that the [val] part of the return statement could be a variable (or other constructs), which could be the beginning of the next statement.

I've not looked into the source of Jison yet, but from what I can make out, I'm guessing that the resulting parser can only look one token ahead. Am I correct in this? In order to be able to differentiate between the two above alternatives, in some situations, it would need to look two tokens ahead.

I can get around the problem by using

ret  [val]

and

return

on its own, but it would obviously be preferable to not have to do this.

I've posted my two spl.jison and spl.jisonlex files here: https://gist.github.com/955096. (Note: the SINGLE_STR_REPLACE and DOUBLE_STR_REPLACE in the spl.jisonlex file are replaced using a shell script because of issues mentioned in previous posts.)

Is there any way of being able to resolve this issue? Is this a restriction that you find in Bison too?

I'd be very grateful for any assistance/comments.

Thanks,

Marcus.

I created the following .jisonlex file:

NUM [0-9]+("."[0-9]+)?


%%
(
{NUM}\s*"'"\s*{NUM}\s*"\""\s*{NUM}\s*"s"
|{NUM}\s*"'"\s*{NUM}\s*"\""\s*
|{NUM}\s*"'"\s*{NUM}\s*"s"
|{NUM}\s*"\""\s*{NUM}\s*"s"
|{NUM}\s*"'"\s*
|{NUM}\s*"\""\s*
|{NUM}\s*"s"
)
 {return 'FIS';}
{NUM}\b  {return 'NUMBER';}
\s+                   {/* skip whitespace */}
"*"                   {return '*';}
"/"                   {return '/';}
"-"                   {return '-';}
"+"                   {return '+';}
"("                   {return '(';}
")"                   {return ')';}
<<EOF>>               {return 'EOF';}

%%

When I ran it (with the associated .jison file) I found that my macro had not been expanded.

On going through the source, I noticed line 18 in lib/jison/lexer.js:

 m = m.split("{"+k+"}").join(macros[k]);

I believe this line should be:
m = m.split("{"+macros[k][0]+"}").join(macros[k][1]);

Either that, or macros is being incorrectly defined somewhere else

Hi, Im trying to use jison2json utility, but it's not on path, and when I try to run it from cloned source I got:

env: narwhal: No such file or directory

How I solve it? Can you make the jison2json and json2jison to be available after installing by npm?

Hi,

The following regex (defined in the top section of a jisonlex file)

test_string                (|a)

gives the following error when running Jison:

Error: Parse error on line 8:
...ing                (|a)expr           
-----------------------^

This particular example is just a simplified one, obviously, and you can always immitate the blank behaviour by doing something like ({other_code})? rather than (|{other_code}), but since it's valid regex code, you might want to add this syntax too.

This would be huge, but awesome. http://en.wikipedia.org/wiki/GLR_parser

Tokens could be inferred from the grammar rules themselves. Bison needs token declarations to generate constants for lexing and allows special types to be associated through the declarations. JavaScript is flexible enough that these processes don't seem to be necessary.

I get this on both Node.js or Narwhal (the narwhal stack trace is java polluted).

node.js:63
throw e;
^
TypeError: Cannot call method 'setInput' of undefined
at Object.parse (/Users/lucian/School/dissertation/interp/js/jscore.js:33:16)
at Object.commonjsMain as main
at Object. (/Users/lucian/School/dissertation/interp/js/jscore.js:205:11)
at Module._compile (node.js:462:23)
at Module._loadScriptSync (node.js:469:10)
at Module.loadSync (node.js:338:12)
at Object.runMain (node.js:522:24)
at Array. (node.js:756:12)
at EventEmitter._tickCallback (node.js:55:22)
at node.js:772:9

Using jscore.js generated from examples/jscore.jison.

I cannot tell if it is GPL 2 or GPL 3 or some other opensource one. Where is the complete license file

Narwhal-jsc doesn't seem to have a print() function that accepts multiple arguments. Here is a patch:

--- lib/jison.js~       2010-04-24 15:40:45.000000000 -0400
+++ lib/jison.js        2010-04-24 15:37:47.000000000 -0400
@@ -304,7 +304,8 @@
 generator.trace = function trace () { };

 generator.warn = function warn () {
-    Jison.print.apply(null,arguments);
+    var args = Array.prototype.slice.call(arguments,0);
+    Jison.print.call(null,args.join(""));
 };

 generator.error = function error (msg) {

Bison specification describes the possibility of "mysterious conflicts" in LALR parsers:

http://www.gnu.org/software/bison/manual/bison.html#Mysterious-Conflicts

However, the solution presented does not currently work in Jison. It continues showing the same reduce/reduce conflicts.

I just ran in to this while hacking on CoffeeScript, whose lexer doesn't expose the proper information. It would be nice if the interface were documented. I'm going to dig in to the code now, but I can't guarantee a pull request.

Hello!

I installed JISON from NPM, and got the following warning messages in the process:

npm WARN [email protected] package.json: 'modules' object is deprecated
npm WARN prefer global [email protected] should be installed with -g

I'm running NPM v1.0.24. It seems JISON is working fine, but I'm kind of concerned things might stop working on a future version of node.js / npm.

Thanks

Jison's symbol for EOF is 1, and when you end a token stream too early -- that's what you get: Unexpected '1'... This isn't too helpful, so it would be great if the parser could return Unexpected EOF, or Unexpected end of file, instead.

Are there any plans to support advanced Javascript Regular Expression syntax, as listed here

It's not an idle request, I have need of support for (?=.), which is the zero-width positive lookahead. In particular, this is useful for regexes which (amongst other things) match a null length string. Details of that scenario can be found here

see https://github.com/wycats/handlebars.js/issues#issue/32

at 81400fa#L0R859 in the browser, this breaks handlebars if requirejs is included. it would probably fix it if you tested for exports as well
if(typeof require !== 'undefined' && typeof exports !== 'undefined') {

CoffeeScript uses Jison.

Hi,

Thanks again for making Jison. Overall it seems to be an excellent bit of coding - well done! I have, however, come across some problems with entering a few regexes. As far as I can tell, these issues affect both defining regexes in the regex section and in the main lex section.

1. backslashes in regexes seem to be completely deleted:

Entering the regex:

['](\\[']|.)*?[']

results in it becoming

[']([']|.)*?[']

However many backslashes I tried, they were all removed (I only tried even numbers, up to 8).

2. Quotes don't play nicely:

Both

'(\\'|.)*?'

and

\'(\\\'|.)*?\'

fail.

Using ['] as above at the top succeeds, though, but it's a pain to have to put all quotes inside square brackets.

3. Regexes with (?: fail

e.g.

['](?:\\[']|.)*?[']

despite it being a valid syntax.

I'm not totally familiar with how Bison/YACC handle regexes, but it would seem to me that the best approach would be just to copy the code exactly as it's written, wouldn't it? If you need to distinguish between quoting some text and having a single/double quote at the beginning of the line, then a single escape at the beginning of the line would be sufficient.

I'll let you know about any more issues I come across.

Btw, jison.org appears to be down.

Cheers,

Marcus.

Currently Jison generates both parser and lexer at the same time. Once it is possible to combine commonjs modules easily, parser and lexer modules could be generated separately and combined only when needed.

An easy way to combine commonjs modules would also allow the generic parser module to be decoupled from parse tables. Then, only the parse table would be necessary output of the generation process; the parser module could simply require the needed parse table.

The generator uses the jison base filename as the variable name. Whenever the filename uses '-', it creates an invalid variable name.

So having as a filename my-file.jison

will generate a variable

var my-file = {...}

which crash JavaScript