Unless I'm not understanding something:

{
  tokens = [ MY_TOKEN = "something" ]
}

rule ::= something

This will generate an element type:

IElementType MY_TOKEN = new TokenType("something");

...but on the parser and Psi implementations, SOMETHING will be used instead of MY_TOKEN

please add support for expanding lexical rules in Live Preview like in JFlex, here simple example what I mean

tokens=[
  Digit = "regexp:[0-9]"
  Integer = "regexp:{Digit}*"
  Float = "regexp:{Integer} \\. {Integer}"
]

I try implement this behaviour like below, but it have different behavior rather than JFlex lexical rules, I think because JFlex used RegEx to NFA converter

package org.intellij.grammar.livePreview;
...
public class LivePreviewLexer extends LexerBase {
...
}
public LivePreviewLexer(Project project, final LivePreviewLanguage language) {
...
Map<String, String> regexPatternName2Definition = new LinkedHashMap<String, String>();
for (String pattern : map.keySet()) {
          tokens[i++] = new Token(pattern, map.get(pattern), tokenConstantPrefix, language, regexPatternName2Definition);
        }
...
static class Token {
...
  Token(String pattern, String mappedName, String constantPrefix, LivePreviewLanguage language, Map<String,          String regexPatternName2Definition) {
  ...
  if (ParserGeneratorUtil.isRegexpToken(pattern)) {
  ...
          String patternText = ParserGeneratorUtil.getRegexpTokenRegexp(pattern);

          Pattern ruleName = Pattern.compile("(\\{(.*?)\\})");
          Matcher ruleMatcher = ruleName.matcher(patternText);
          while(matcher.find())
          {
              String patternName = matcher.group(2);

              if (regexPatternName2Definition.containsKey(patternName)) 
                  patternText = ruleMatcher.replaceAll(regexPatternName2Definition.get(patternName));
          }
          this.pattern = ParserGeneratorUtil.compilePattern(patternText);
         ...
         }
   ...
}
...
}
}

I think must be possible define private rules, which would be components of other complex rules, in the case of JFlex I can define complex rule consisting of private rules for input entry I will expect

.bfn file

tokens=[
    Lowercase="regexp:[a-z]"
    Uppercase="regexp:[A-Z]" 
    Letter="regexp:{Uppercase} | {Lowercase}
    Letters="regexp:{Letter}*"    
]

root ::= Letters

.preview file

   someword here

As stated above, the issue is that parser files are generated in the default source files root ('src' in this case) instead of being generated in into the 'gen/' folder as was the previous behaviour.

This is very easy to recreate, just go ahead and create a new project, add any bnf file and run the parser generator. The files should appear in the src folder, instead of creating a new 'gen' folder. This same issue also occurs if there is already a 'gen' present.

This is with IntelliJ Ultimate 14.0.1 with GrammarKit 1.2 on OS X 10.10.1 .

Hi

I've just discovered a bug in error handling in the latest version of Grammar-Kit (although it might've been there for a long time).
Please find example project here (just a fork of SimplePlugin from custom language plugin tutorial).
I added new token types just to ilustrate my problem in this commit.
Then I added a compound type (element type which causes usage of overloaded version of nextTokenIs; instead of nextTokenIs(PsiBuilder builder_, IElementType token) which was used before now nextTokenIs(PsiBuilder builder_, String frameName, IElementType... tokens) is being used). It causes issues with marking the area that is affected by error and the error message is incorrect.
The last commit shows that after additionally adding pinning the error handling on the end of the file is completely broken = it's not marking any errors, treats code as valid even if it's broken.

For all these commits the behaviour of the editor which is displaying file with the contents below is completely different:
special word

Hope this makes any sense...

Here's a cut and paste from some tutorial about writing IntelliJ plugins:

%public
%class MyLexer
%implements FlexLexer
%function advance
%type IElementType
%unicode
%eof{  return;
%eof}

Grammar-Kit gives a red squiggly mark right after %eof{ and claims that there must be a newline there. JFlex does not complain about that when generating the code, and the JFlex output looks as expected.

I am trying to create an ant task to generate my parser with grammar-kit and compile my project; however, I am running into an issue where grammar-kit warns class not found for the psiImplUtilClass. This seems similar to #24.

I can get around this by adding the output directory to the classpath, but this is error-prone since the class definition may have been updated since it was last compiled. I could generate the parser and ignore the warning, compile, then re-generate the parser so it can catch the psiImplUtilClass, then re-compile so it can catch the new parser. But that seems incredibly wasteful.

What is the right way to solve this problem?

When rebuilding the parser class from the BNF, the generated GrammarParser.java is not compilable because the method signatures of
GeneratedParserUtilBas.consumeToken
and
GPUB.enterErrorRecording --> enterErrorRecordingSection

have changed.

From what I see in the git-history it has been like this since the first revision: generateConsumeToken has 3 arguments in the Generator-class but just two in the GeneratdParserUtilsBase.
So the generatored grammar which is in the git has been created before the intial import.

In my grammar the automatically generated getter seems to be producing a slightly odd result for getUpperBound, which is marked as @nullable, when it should be @NotNull just like its twin getLowerBound.

range ::= LSB string RANGE_OPERATOR string RSB {
    methods = [
        getLowerBound="/string[0]"
        getUpperBound="/string[1]"
    ]
}

generates:

@NotNull
List<EBNFString> getStringList();

@NotNull
EBNFString getGetLowerBound();

@Nullable
EBNFString getGetUpperBound();

For the interface and unless I'm mistaken that last method should definitely be @NotNull like the lower bound is.

Using Grammar-Kit 1.0.9, no error shown in the grammar file, but the generated parser does not compile (else without if):

  public static boolean expression_0(PsiBuilder builder_, int level_, int priority_) {
    if (!recursion_guard_(builder_, level_, "expression_0")) return false;
    boolean result_ = true;
    while (true) {
      Marker left_marker_ = (Marker) builder_.getLatestDoneMarker();
      if (!invalid_left_marker_guard_(builder_, left_marker_, "expression_0")) return false;
      Marker marker_ = builder_.mark();
      else {
        marker_.rollbackTo();
        break;
      }
    }
    return result_;
  }

Please tell me if you need more specific data.

This would allow to integrate in Ant build, for example.

Please change the semantics of recoverWhile attribute to be activated only when pin attribute was activated. recoverWhile doesn't make sense when the rule was not pinned first. Also, it should be allowed only on rules that have pin.

I've spent 2 days debugging my simple grammar because I was stupid enough to use recoverWhile. Partially my problems were because I forgot that recoerWhile is always activated. The other part of the problem came from 2 bugs in the implementation of recoverWhile. If the restriction requested above would be enforced, those 2 bugs would not even exist.

I'm pasting my definition below. Please take a look at least at the first bug. The second one I'm not sure if is valid.

file inotes.bnf:

{
parserClass="ro.uaic.fmse.inotes.parser.InotesParser"
parserUtilClass="com.intellij.lang.parser.GeneratedParserUtilBase"

extends="com.intellij.extapi.psi.ASTWrapperPsiElement"

psiClassPrefix="Inotes"
psiImplClassSuffix="Impl"
psiPackage="ro.uaic.fmse.inotes.psi"
psiImplPackage="ro.uaic.fmse.inotes.psi.impl"

elementTypeHolderClass="ro.uaic.fmse.inotes.psi.InotesTypes"
elementTypeClass="ro.uaic.fmse.inotes.psi.InotesElementType"
tokenTypeClass="ro.uaic.fmse.inotes.psi.InotesTokenType"
}

inotesFile ::= (LINE_TERMINATOR | day_entry | entry_1)*

day_entry ::= date (text_line | LINE_TERMINATOR+) (time_entry | entry_1)* {pin=2 /recoverUntil=not_day_entry_end/}
private not_day_entry_end ::= !date
date ::= INTEGER_LITERAL "/" INTEGER_LITERAL ("/" INTEGER_LITERAL)?

// Grammar Kit bug: Instead of (entry_1 | LINE_TERMINATOR) we should have used entry_1
// This is a hack to avoid a bug caused by recoverUntil on entry_1
// Example to reproduce the bug:
//1/2
//3/4
//5.6
//7/8
time_entry ::= time (text_line | LINE_TERMINATOR+) (entry_1 | LINE_TERMINATOR)* {pin=2 /recoverUntil=not_time_entry_end/}
private not_time_entry_end ::= !(date | time)
time ::= INTEGER_LITERAL "." INTEGER_LITERAL

//because this rule have recoverUntil it should always be the last in any | - sequence
entry_1 ::= entry_header_1 (text_line | LINE_TERMINATOR+) ((not_entry_end text_line) | entry_2)*
{pin=1 recoverUntil=not_entry_end}
entry_2 ::= entry_header_2 (text_line | LINE_TERMINATOR+) ((not_entry_end text_line) | entry_3)*
{pin=1 recoverUntil=not_entry_end}
entry_3 ::= entry_header_3 (text_line | LINE_TERMINATOR+) (not_entry_end text_line)*
{pin=1 recoverUntil=not_entry_end}

private entry_end ::= date | time | entry_header_1 | entry_header_2 | entry_header_3
private not_entry_end ::= !entry_end
entry_header_1 ::= (WORD | INTEGER_LITERAL)+ ":" !":"
entry_header_2 ::= (WORD | INTEGER_LITERAL)+ "::" !":"
entry_header_3 ::= (WORD | INTEGER_LITERAL)+ ":::" !":"

// Grammar Kit bug: Looks like there is a parser bug with this definition,
// the part "!(date | entry_header)" is incorrectly optimized
//text_line ::= !(date | entry_header) (INTEGER_LITERAL | WORD |OTHER)* LINE_TERMINATOR+
// Looks like restrictions like !(foo | bar) work incorrectly just at the beginning of the rule,
// in the middle of the rule they work ok.
text_line ::= text_item* LINE_TERMINATOR+
private text_item ::= INTEGER_LITERAL | WORD |OTHER | COLON | STRING | quantity | "(" text_item* ")" | keyword
quantity ::= "(" INTEGER_LITERAL text_item* ")"
keyword ::= "'" (WORD | INTEGER_LITERAL)+ "'"

// Used to generate InotesTypes entries for all token types produced by the lexer
private aux_all ::= LEFT_PAREN | RIGHT_PAREN | BRACE | VERTICAL_BAR | COLON | QUOTE | WORD | INTEGER_LITERAL COMMENT
| LINE_TERMINATOR | STRING | OTHER

file inotes.flex:

package ro.uaic.fmse.inotes;

import com.intellij.lexer.FlexLexer;
import com.intellij.psi.tree.IElementType;
import ro.uaic.fmse.inotes.psi.InotesTypes;
import com.intellij.psi.TokenType;

%%

%public
%class InotesLexer
%implements FlexLexer
%unicode
%function advance
%type IElementType
%eof{ //return;
%eof}

LineTerminator = \r|\n|\r\n
InputCharacter = [^\r\n]
WhiteSpace = ([ \t\f])+

/* comments */
Comment = {TraditionalComment} | {EndOfLineComment} | {DocumentationComment}

TraditionalComment = "/" [^_] ~"/" | "/_" "*"+ "/"

//EndOfLineComment = "//" {InputCharacter}* {LineTerminator}
EndOfLineComment = "//" {InputCharacter}*

DocumentationComment = "/*" {CommentContent} ""+ "/"
CommentContent = ( [^_] | _+ [^/_] )_

Word = [:jletter:] [:jletterdigit:]*

DecIntegerLiteral = [0-9]+

String = "([^\n\r"] | \t | \n | \r | \" | )*"

%%

{

"(" {return InotesTypes.LEFT_PAREN;}
")" {return InotesTypes.RIGHT_PAREN;}
// "{" | "}" {return InotesTypes.BRACE;}
// "|" {return InotesTypes.VERTICAL_BAR;}
":" {return InotesTypes.COLON;}
"'" {return InotesTypes.QUOTE;}

/* identifiers */
{Word} { return InotesTypes.WORD; }

/* literals */
{DecIntegerLiteral} { return InotesTypes.INTEGER_LITERAL; }

/* comments */
{Comment} { return InotesTypes.COMMENT; }

/* not whitespace */
{LineTerminator} { return InotesTypes.LINE_TERMINATOR; }

/* whitespace */
{WhiteSpace} { return TokenType.WHITE_SPACE; }

{String} { return InotesTypes.STRING; }

}

/* simple text */
. { return InotesTypes.OTHER; }

file test.inotes:

1/2
3/4
5.6
7/8
0.0 //comment
x1:
x2:: zzz
x3::: xxx
0.0
x1: 'keyword'
x2::
x3:::
9/10

Having the following grammar:

expression ::= SOME_TOKEN
{
  mixin='com.example.MyExpressionMixin'
  methods=[sampleMethod]
}

and following code:

abstract class MyAbstractExpression {
  void sampleMethod() {}
}
class MyExpressionMixin extends MyAbstractExpression {
  @Override
  void sampleMethod() {}
}

generated MyExpression contains unnecessary annotation

interface MyExpression {
  @Override
  void sampleMethod();
}

which leads to uncompilable code

This is important, when you rename some rule. In this case previously generated Psi will be still in directory -> broken compilation, as visitor doesn't have some appropriate method.

I can see gen is hardcoded as the output directory, it would be great if it could be customized.

Using 1.2.0.2 prerelease I tried "inception" scenario

1. Open Grammar.bnf
2. Open Live Preview
3. Paste Grammar.bnf content into Live Preview

PsiErrorElement: "{' unexpected'

Intellij IDEA 12.0.4
Build #IU-123.169
Build on February 13, 2013
JDK: 1.6.0_43

Hello, I want create syntax highlighter for Pawn language and installed latest version of plugin 1.1.1 from repo, and got red highlight on his name in plugins list, screen here

, when I tried to install previous version 1.1.0 - I got Live Edit broken, screen here

, I tried install this on three machines, iMac, MacBook Pro - Lion 10.8, PC - Windows 7 and plugin not work

When I click "Run JFlex Generator" I get error message:

/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -Xmx512m -Dfile.encoding=UTF-8 -classpath /Users/artem/Work/idea-nsis/JFlex.jar JFlex.Main -sliceandcharat -skel /Users/artem/Work/idea-nsis/idea-flex.skeleton -d /Users/artem/Work/idea-nsis/src/com/krylysov/nsisplugin /Users/artem/Work/idea-nsis/src/com/krylysov/nsisplugin/Nsis.flex
Exception in thread "main" java.lang.UnsupportedClassVersionError: JFlex/Main : Unsupported major.minor version 52.0

I tried to set JDK 1.7 and JDK 1.8 for the project, but it didn't help.

I get a compilation error in my parser from the following code:

import ::= var
         | tycon ["(..)" | cnames]
         | tycls ["(..)" | vars]

Renaming "import" to "importt" makes the problem go away. The generated java Parser contains a function called "import".

I've found quite a few times that the bnf preview would parse my code correctly, while the generated grammar would fail to do so.

One of the things that make it fail seems to be these 4 expressions:

var_1_expstm ::= var_expr assign_op_atom expr
var_2_expstm ::= var_expr LBRACK aidx_expr RBRACK assign_op_atom expr

slot_acc_1_expstm ::= slot_acc_expr assign_op_atom expr
slot_acc_2_expstm ::= slot_acc_expr LBRACK aidx_expr RBRACK assign_op_atom expr

Which generate the following 2 error messages:

00:21:33 only first definition will be used for 'assign_op_atom': [BINARY, N_ARY, POSTFIX]
00:21:33 only first definition will be used for 'LBRACK': [BINARY, N_ARY, POSTFIX]

And the issue seems to have a manual fix by copy-pasting the following code-block and implement it for the slot_acc statements:

      else if (priority_ < 2 && ((LighterASTNode)left_marker_).getTokenType() == VAR_EXPR && assign_op_atom(builder_, level_ + 1)) {
        result_ = report_error_(builder_, expr(builder_, level_, 2));
        marker_.drop();
        left_marker_.precede().done(VAR_1_EXPSTM);
      }
      else if (priority_ < 2 && ((LighterASTNode)left_marker_).getTokenType() == VAR_EXPR && consumeTokenSmart(builder_, LBRACK)) {
        result_ = report_error_(builder_, expr(builder_, level_, 2));
        result_ = report_error_(builder_, var_2_expstm_1(builder_, level_ + 1)) && result_;
        marker_.drop();
        left_marker_.precede().done(VAR_2_EXPSTM);
      }

Ofc I don't want to do this fix everytime.

The full BNF can be found here: http://pastebin.com/7ZRAcCtj although it's pretty ugly atm because of my debugging trip.

Its been a few years since my formal languages class in college, but I'm challenging myself and try to define a grammar for the android disassembly language: smali

I'm having some issues with words in my grammar that are java identifiers, What am I doing wrong?

In this screenshot, a short snippet of smali code is attempted to be parsed, it should be ".class" followed by a modifier "public", "private", "static" etc. so I wrapped my head around the pins and the recoverWhile and broke out the keywords as tokens and I get an error that is shown below:

Strangely, if I pretend that the smali language uses ".public" instead of "public" and then change my parser to look for ".public" the parsing succeeds:

Is this a bug, or do I have to do something special for parsing identifiers that are also java identifiers?

For some reason, the commenttext token from my grammar file is no longer generating a COMMENTTEXT ElementType. Not sure if there is some setting I'm omitting to provide to enable this.

I'm using IntelliJ 13.1 with Grammar Kit 1.2.0.1.

I am trying to add custom methods to a grammar type:

{
  // ...
  psiImplUtilClass="net.frapontillo.idea.plugin.prolog.psi.PrologPsiImplUtil"
}

// ...

functor ::= atom PARENTHESIS_LEFT predicate_list PARENTHESIS_RIGHT {
mixin="net.frapontillo.idea.plugin.prolog.psi.impl.PrologNamedElementImpl" 
implements="net.frapontillo.idea.plugin.prolog.psi.PrologNamedElement" 
methods=[getName setName getNameIdentifier]}

// ...

where the Util class is correctly implemented:

public class PrologPsiImplUtil {
    public static String getName(PrologFunctor element) {
        ASTNode keyNode = element.getNode().findChildByType(PrologTypes.ATOM);
        if (keyNode != null) {
            return keyNode.getText();
        } else {
            return null;
        }
    }

    // ...

Still, the generated methods don't match the parameters and the return types, as I get something like:

public void getName() {
    PrologPsiImplUtil.getName(this);
}

What can cause this?

Having the following grammar

...
expression ::= something
{ 
  mixin='com.example.ExpressionMixin' 
  methods=[ multiResolve ]
}
...

and supposing ExpressionMixin contains following method:

class ExpressionMixin implements Expression {
  List<? extends Container<? extends Item>> multiResolve() { /* code */ }
}

the generated Expression contains same method with altered signature

interface Expression {
    List<Container<Item>> resolveItemContaners();
}

which obviously fails to compile.

When using this config:

generateStubParser=true
stubParserClass="net.nicoulaj.idea.byteman.lang.parser.BytemanParserUtil"

The stub parser is not generated. I work this around by manually copying GeneratedParserUtilBase and creating an empty BytemanParserUtil extending it, but it should be generated, right ? Or am I missing something ?

It seems that grammar kit 1.1.10 no longer generates the get*List method for meta rules. Here's an example -

private impspec ::= [i "hiding"] '(' [<<sequence importt>>] ')' | impempty {pin(".*")=2}
private meta sequence ::= <<p>> (',' <<p>>)*  {pin(".*")=1}

(full source)

Now generates this -

public List<HaskellImportt> getImportt()

Instead of this -

public List<HaskellImportt> getImporttList()

(full source)

For now I've reverted to Grammar Kit 1.1.9.

Version: 1.1.2

For grammar

{
  tokens = [
    FOO = 'regexp:"foo"'
    BAR = '"bar"'
  ]
}

file ::= FOO | BAR

GK produces a JFlex spec containing

FOO="foo"

and

  ""bar""            { return BAR; }

instead of

FOO=\"foo\"

and

  "\"bar\""            { return BAR; }

Given this simple grammar, why numbers get parsed correctly but identifier does not?

123
aa1

grammar ::= identifier | number

identifier ::= letter+digit*
number ::=digit+
digit ::=('0'|'1'|'2'|'3'|'4'|'5'|'6'|'7'|'8'|'9')
letter ::= ('a'|'b')

Consider the following grammar:

{...
  tokens = [
       LBRACE='{'
       RBRACE='}'
       START='start'
       STOP='stop'  ] }

simpleFile ::= element*

element ::= '{' item* complex_stuff '}'
item ::= 'start' ID* 'stop' { recoverUntil=item_recover }

private item_recover ::= !'start' // we don't want to recover with
                                  // item_recover ::= !('start'|complex_stuff)
                                  // because it is too similar to item's content 

complex_stuff ::= ID+ // let assume it really complex

And the following input text:

{
 start a b c stop
 start d e stop
 f
}

The text does not parse unless we remove { recoverUntil=item_recover }.
Errors occurs on f.
Is it really necessary to run the recovering rule even when there is no parsing error, as it is done now?

I noticed that If I replace in GeneratedParserUtilBase

        if (!state.suppressErrors && eatMore != null) {

by

        if (!result && !state.suppressErrors && eatMore != null) {

It works well both in correct parsing cases, and in parsing error cases.
Is there any hidden detail that I missed, that could wake the Parsing Beast up?

When the same method is injected into two rules, which produce inheritance-related classes:

// ...
extends("let_.*_bind")=let_bind
// ...
let_bind ::= let_tuple_bind | let_struct_bind | let_variable_bind | let_unit_bind | let_self_arg {methods=[getBindedNames]}
let_variable_bind ::= variable_signature [':' type] {methods=[getBindedNames]}
// ...

// RustPsiImplUtil.java
// ...
public static Collection<PsiElement> getBindedNames(RustLetBind bind) {
    return Collections.emptyList();
}
public static Collection<PsiElement> getBindedNames(RustLetVariableBind bind) {
    return Collections.<PsiElement>singleton( bind.getItemName() );
}
// ...

the editor shows warning on let_variable_bind's getBindedNames: "Unresolved method reference".

When I try to generate a lexer from an existing .flex file (with ctrl-shift-g), the plugin brings up a file chooser with the title, "Downloaded files will be copied to selected directory". There's no indication of what it's downloading, why it would need to download something, or where would be a sensible place to put it. After choosing a directory, I get the above error message.

I've tried choosing the root of my project directory, and tools/lexer/ and tools/lexer/jflex-1.4/ in the root of my existing checkout of the intellij-community source. I'm on IntelliJ IDEA 13.1.0.

I have been trying to pin expression 1 and 4 simultaneously without success. I looked at the Erlang grammar for hints. I found this line in one commit:

try_catch ::= catch try_clauses [after try_expressions_clause] | after try_expressions_clause {pin(".*")="catch|after"}

So I tried:

block ::= "begin" statements "end" {pin(".*")="begin|end"}

I don't get any warnings, but the resulting parser is without pinned blocks. Here's other syntactically legal things I've tried:

block ::= "begin" statements "end" {pin(".*")="1|3"}
block ::= "begin" statements "end" {pin(".*")=[1 3]} 

They result in a generated parser that doesn't pin anything, and the way to find this out is to look at the diff.

There are some syntactical warnings already though, for example for this pin:

block ::= "begin" statements "end" {pin(".*")=1,3}

I've read README, HOWTO and TUTORIAL, but I'm still not sure what the regular expression in the parenthesis means, and what I can put after the equals that generates a parser with meaningful pins.

I have a parser with the following rules:

triggerDefinition ::= TRIGGER identifier ON type triggerDmlEvents codeBlock { pin=1 }
triggerDmlEvents ::= LPAREN dmlEventList RPAREN { pin=1 }
private dmlEventList ::= dmlEvent (COMMA dmlEvent)* { pin=1 recoverWhile=dmlEventListRecover }
private dmlEventListRecover ::= !(RPAREN)
private dmlEvent ::= dmlEventTiming dmlEventOperation { pin=1 }
dmlEventTiming ::= BEFORE | AFTER
dmlEventOperation ::= DELETE | INSERT | UNDELETE | UPDATE | UPSERT

This parses valid code properly, but unfortunately it also shows some invalid states as valid, e.g.,

trigger MyTrigger on MyObject {}

and:

trigger MyTrigger on MyObject (before) {}

The first example is missing the mandatory triggerDmlEvents portion and the latter is missing the mandatory dmlEventOperation portion.

If I don't pin, these are properly flagged as invalid, but obviously I lose the advantages of recovery while typing.

I've tracked this down to the generated parser class and its corresponding rules. When code of the following form is called with pinned_=true and result_=false:

result_ = pinned_ && report_error_(builder_, <rule>(builder_, level_ + 1)) && result_;

it in turn calls GeneratedParserUtilBase.report_error_() which doesn't seem to exhibit the correct behavior here:

int position = rawTokenIndex(builder);
if (frame.errorReportedAt < position && getLastVariantPos(state, position + 1) <= position)
{
  reportError(builder, state, frame, null, true, advance);
}

The conditional evaluates to false because getLastVariantPos() > position.

Is there some problem with the way I've expressed my grammar rules or tried to use pinning here? For the most part my grammar and its pinning work quite well, but obviously failing to report errors for malformed syntax seems like a problem.

Thanks in advance!

I can't run Grammar-Kit in standalone mode. Using Grammar-Kit.zip (1.1.9) I get this error:

java -jar grammar-kit.jar my\gen my.bnf
java.lang.NoSuchFieldError: HARD_REF_TO_DOCUMENT_KEY
    at org.intellij.grammar.LightPsi$Init.initPsiFileFactory(LightPsi.java:248)
    at org.intellij.grammar.LightPsi$MyParsing.<init>(LightPsi.java:156)
    at org.intellij.grammar.LightPsi.<clinit>(LightPsi.java:98)
    at org.intellij.grammar.Main.main(Main.java:64)

Using Grammar-Kit-13.1.zip I get another error:

java -jar grammar-kit.jar my\gen my.bnf
java.lang.ExceptionInInitializerError
    at org.intellij.grammar.Main.main(Main.java:64)
Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: @NotNull method com/intellij/openapi/components/ServiceManager$1.fun must not
    at org.intellij.grammar.LightPsi.<clinit>(LightPsi.java:101)
    ... 1 more
Caused by: java.lang.IllegalStateException: @NotNull method com/intellij/openapi/components/ServiceManager$1.fun must not return null
    at com.intellij.openapi.components.ServiceManager$1.fun(ServiceManager.java:55)
    at com.intellij.openapi.components.ServiceManager$1.fun(ServiceManager.java:51)
    at com.intellij.openapi.util.NotNullLazyKey.getValue(NotNullLazyKey.java:37)
    at com.intellij.openapi.project.DumbService.getInstance(DumbService.java:154)
    at org.intellij.grammar.LightPsi$Init.initExtensions(LightPsi.java:207)
    at org.intellij.grammar.LightPsi$MyParsing.<init>(LightPsi.java:157)
    at org.intellij.grammar.LightPsi.<clinit>(LightPsi.java:98)
    ... 1 more

Using IDEA versions 13.1.5, 13.1.6, 14.
JDK: 1.6, 1.7, and 1.8 (Using default options; aggressive options appear to make it worse).
Grammar-Kit versions: 1.2.0, 1.2.0.1 (Did not occur on 1.1.x versions.)
OS: Windows 8.1, Linux (Ubuntu 14.04, CentOS 6.5)

The Intellij-haxe project is seeing an inconsistency in output when generating code from within the Intellij IDE (Ctrl+Shift+G). Sometimes the (consistent with v1.1.x) output looks like:
HaxeFunctionType getFunctionType();
and sometimes:
List<HaxeFunctionType> getFunctionTypeList();
We say "sometimes" because the bug exhibits itself intermittently: at times you can regenerate for hours and not see the bug, and other times, you will hit it the first time you generate. (The recovery mechanism is to exit and restart IDEA and /usually/ you get the expected result.)

We've been trying to automate generation, so that we don't have to check in the gen/* files, but command-line generations always give the second (unexpected) result. If you want to try using our scripts, you can find them in the tivo repository.

The relevant BNF is here. Since the bug is intermittent, it will be very difficult to create a minimal example.

The output diff is:

diff --git a/gen/com/intellij/plugins/haxe/lang/psi/HaxeTypeTag.java b/gen/com/intellij/plugins/haxe
index 8309947..3bf6410 100644
--- a/gen/com/intellij/plugins/haxe/lang/psi/HaxeTypeTag.java
+++ b/gen/com/intellij/plugins/haxe/lang/psi/HaxeTypeTag.java
@@ -25,10 +25,10 @@ import com.intellij.psi.PsiElement;

 public interface HaxeTypeTag extends HaxePsiCompositeElement {

-  @Nullable
-  HaxeFunctionType getFunctionType();
+  @NotNull
+  List<HaxeFunctionType> getFunctionTypeList();

-  @Nullable
-  HaxeTypeOrAnonymous getTypeOrAnonymous();
+  @NotNull
+  List<HaxeTypeOrAnonymous> getTypeOrAnonymousList();

 }
diff --git a/gen/com/intellij/plugins/haxe/lang/psi/impl/HaxeTypeTagImpl.java b/gen/com/intellij/plu
index 8e20f0e..4580244 100644
--- a/gen/com/intellij/plugins/haxe/lang/psi/impl/HaxeTypeTagImpl.java
+++ b/gen/com/intellij/plugins/haxe/lang/psi/impl/HaxeTypeTagImpl.java
@@ -40,15 +40,15 @@ public class HaxeTypeTagImpl extends HaxePsiCompositeElementImpl implements Haxe
   }

   @Override
-  @Nullable
-  public HaxeFunctionType getFunctionType() {
-    return findChildByClass(HaxeFunctionType.class);
+  @NotNull
+  public List<HaxeFunctionType> getFunctionTypeList() {
+    return PsiTreeUtil.getChildrenOfTypeAsList(this, HaxeFunctionType.class);
   }

   @Override
-  @Nullable
-  public HaxeTypeOrAnonymous getTypeOrAnonymous() {
-    return findChildByClass(HaxeTypeOrAnonymous.class);
+  @NotNull
+  public List<HaxeTypeOrAnonymous> getTypeOrAnonymousList() {
+    return PsiTreeUtil.getChildrenOfTypeAsList(this, HaxeTypeOrAnonymous.class);
   }

 }

I'm currently trying to figure out how the error recovery works and therefor i just copied the exmple given within TUTORIAL.md.

The issue now is that the shown errors within the live preview don't match the ones expected.

After some digging in the code in the code I found that the actual test expects this behaviour:

LivePreviewTutorial.txt#L56

So I dug some more, I found the following:

• The screenshot in the tutorial is outdated and still shows recoverUntil instead of recoverWhile and also has the attribute at anoter element (property vs root_item)
• The version used in the tests and in the tutorial text have a bad error recovery as I relly expected the error shown in the screenshot. This fixed the example for me:

root ::= root_item *
private root_item ::= !<<eof>> property ';' {pin=1}

property ::= id '=' expr  {pin=2 recoverWhile=property_recover}
private property_recover ::= !(';' | id '=')

Maybe I'm on the wrong track and missed something obvious, but I would really appriciate it if the code in the tutorial, the screenshot and the test would match and convey the same thing.

If my solution sounds reasonable I'd be happy to help you out with a PR.
Just let me know.

Live Preview is a great feature; it'd be even more useful if you could configure an existing lexer rather than approximating an existing one within the BNF.

This issue was prompted by:
http://devnet.jetbrains.com/message/5521947#5521947

How are you supposed to generate parser code on the command line? It looks like issue #4 indicates this is possible, however, I'm not sure how to apply it. It seems that I am supposed to indicate a class path.

JAR=~/Library/Application\ Support/IdeaIC13/GrammarKit/lib/grammar-kit.jar
BNF=src/com/haskforce/Haskell.bnf
GEN=gen
java -jar $JAR $BNF $GEN

Exception in thread "main" java.lang.NoClassDefFoundError: com/intellij/lang/ParserDefinition                          │cli-parser-1.1.jar                 jna-utils.jar                      resources_en.jar
        at java.lang.Class.getDeclaredMethods0(Native Method)                                                          │commons-codec-1.8.jar              jna.jar                            rhino-js-1_7R4.jar
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2531)                                                  │commons-httpclient-3.1-patched.jar jps-launcher.jar                   rngom-20051226-patched.jar
        at java.lang.Class.getMethod0(Class.java:2774)                                                                 │commons-logging-1.1.1.jar          jps-server.jar                     rt
        at java.lang.Class.getMethod(Class.java:1663)                                                                  │commons-net-3.1.jar                jsch-0.1.50.jar                    sanselan-0.98-snapshot.jar
        at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)                                          │ecj-4.2.1.jar                      jsr166e.jar                        serviceMessages.jar
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)                                       │extensions.jar                     jsr173_1.0_api.jar                 snappy-java-1.0.5.jar
Caused by: java.lang.ClassNotFoundException: com.intellij.lang.ParserDefinition                                        │forms_rt.jar                       junit-4.10.jar                     swingx-core-1.6.2.jar
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)                                                      │grammar-kit.jar                    junit.jar                          trang-core.jar
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)                                                      │groovy                             jzlib-1.1.1.jar                    trove4j.jar
        at java.security.AccessController.doPrivileged(Native Method)                                                  │groovy-all-2.0.6.jar               libpty                             trove4j_src.jar
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)                                                  │gson-2.2.3.jar                     log4j.jar                          util.jar
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)                                                       │guava-14.0.1.jar                   markdownj-core-0.4.2-SNAPSHOT.jar  velocity.jar
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)                                               │icons.jar                          microba.jar                        winp-1.17-patched.jar
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)                                                       │idea.jar                           miglayout-swing.jar                xbean.jar
        ... 6 more

I am looking for JSON grammars/fuzzers for an evaluation and stumbled upon your Json.bnf example. Is it intentional that Json.bnf is incomplete?

e.g.

• true, false and null
• Unicode escapes
• character escapes e.g. \n \r and \t
• whole number spectrum of JSON
• whole visible character spectrum of JSON for strings and object members

It would be nice to be able to generate lexers to specified directory. In the other case source and generated source files are mixed in a mess.

Grammar .bnf file part:

argument_name ::= IDENTIFIER
{
  implements='com.example.MyReferenceElement<com.example.MySymbol>'
  mixin = 'com.example.MyArgumentNameImpl'
}

Generated MyVisitor:

public void visitArgumentName(@NotNull MyArgumentName o) {
  visitSymbol>(o);
}
...
public void visitSymbol>(@NotNull MySymbol> o) {
  visitPsiElement(o);
}

ExpectedResult:

public void visitArgumentName(@NotNull MyArgumentName o) {
  visitReferenceElement(o);
}

I have an IntelliJ plugin, containing two bnf's, two languages in it. The first language correctly generates to the gen/ folder, but the second bnf seems to want to generate to the src/ folder instead. I don't see what I did differently and I can't seem to find a config option for this.

Running ubuntu 14.04, Intellij 14, Grammar Kit 1.2.0.2.

Hi,

I tried to install the Grammar-Kit plugin, but it fails with the message "Plugin Grammar-Kit is incompatible with your current installation". The event log stays blank.

I'm using IDEA 13.1.5 on an Ubuntu 14.04 64bit.

Do I need a special version for this plugin?

Hi,

I'm actually not sure if this is a bug or I'm missunderstanding something.

I've tried the example in the "HOW-TO" to use expression roots. The problem is, the code for BINARY, POSTFIX and N_ARY is never called.

In your example, the generated code will look like this:

public static boolean expr(PsiBuilder builder_, int level_, int priority_) {
  if (!recursion_guard_(builder_, level_, "expr")) return false;
    boolean result_ = false;
    boolean pinned_ = false;
    Marker marker_ = enter_section_(builder_, level_, _NONE_, "<expr>");
    result_ = unary_plus_expr(builder_, level_ + 1);
    if (!result_) result_ = unary_min_expr(builder_, level_ + 1);
    ... // more PREFIX and ATOM statements
    pinned_ = result_;
    result_ = result_ && expr_0(builder_, level_ + 1, priority_);
    exit_section_(builder_, level_, marker_, null, result_, pinned_, null);
    return result_ || pinned_;
}

The problem is (and sorry if I'm not understanding something) that if none of the PREFIX statements match, _result will always be false, and expr_0 will never be called (because of result_ = result_ && expr_0(...)

Is this a bug, or am I using it wrong?

If I use an underline at the end of the rule then wrong method names will be generated.

For example if I have rules like this.

//...
EXPR ::= IDENTIFIER [EXPR_]
EXPR_ ::= 'and' EXPR | 'or' EXPR
//...

The BNF won't complain, it sees it as valid BNF.

But when code is generate from this BNF then the following methods will be generated.

public interface SimpleExpr extends PsiElement {
/*...*/
@Nullable
RubyArg getExpr();
/*...*/
@Nullable
RubyArg getExpr();
/*...*/
}

Rule EXPR and EXPR_ will have the same method name which is uncompilable.

I think the annotated screenshot in the README.md is brilliant, and it's probably the first thing a lot of users will see. The screenshot shows and has an arrow to recoverUntil, but that is deprecated/removed since 1.1.5.

Hi!
I am trying to install the plugin (commit: 0ab6f3e) on Intellij IDEA 12.1.4 Build IU-129.713 using Grammar-Kit.jar from binaries directory. I get the following message: "Plugin Grammar-Kit is incompatible with current installation".

Please help me with this issue.

Hello, how I can provide lexer state or variable to parser while parsing?

When using a double quote character " as a token, the generated JFlex lexer erroneously adds an extra backslash. This happens whether you define the token as '"' or "\"". Workaround is to either manually edit the generated lexer or use regexp.

In the examples below, the offending line in the lexer shows -

"\\""               { return DOUBLEQUOTE1; }

Instead, it should show -

"\""                { return DOUBLEQUOTE1; }

sample.bnf

{
  tokens=[
    doublequote1='"'
    doublequote2='regexp:"'
  ]
}

stuff ::= doublequote1 | doublequote2

_SampleLexer.flex

package generated;
import com.intellij.lexer.*;
import com.intellij.psi.tree.IElementType;
import static generated.GeneratedTypes.*;

%%

%{
  public _SampleLexer() {
    this((java.io.Reader)null);
  }
%}

%public
%class _SampleLexer
%implements FlexLexer
%function advance
%type IElementType
%unicode

EOL="\r"|"\n"|"\r\n"
LINE_WS=[\ \t\f]
WHITE_SPACE=({LINE_WS}|{EOL})+

DOUBLEQUOTE2=\"

%%
<YYINITIAL> {
  {WHITE_SPACE}       { return com.intellij.psi.TokenType.WHITE_SPACE; }

  "\\""               { return DOUBLEQUOTE1; }

  {DOUBLEQUOTE2}      { return DOUBLEQUOTE2; }

  [^] { return com.intellij.psi.TokenType.BAD_CHARACTER; }
}

Using 1.2.0.1

1. Open Grammar.bnf in editor
2. Open Live Preview
3. Paste sample bnf (from tutorial) into Preview
4. PsiErrorElelemnt "{' unexpected'

I would like to have a way to keep the structure view "pinned" to the live preview, so that I can see parsing changes as I edit the BNF. Currently, I have to switch between the editor and preview windows in order to see the changes I'm working on, which is much slower.