roop / citron Goto Github PK

… but http://roopc.net/citron/grammar-file/#token doesn't say so.

Or better yet, there should be a way to request public accessibility for any/all of the components in the generated Swift code. Not having this limits usability.

citron's Package.swift creates library products CitronParserModule and CitronLexerModule. My project's Package.swift has CitronParserModule as a dependency. However the parser generated by citron does not import CitronParserModule and so fails to compile. I manually added the import and my parser compiled OK but manually editing a generated file isn't a good plan.

Is there another way for me to make CitronParserModule available to the generated parser or does citron need fixed?

Hi,

I have a much-improved lexer that I'd like to contribute to this project, but I'm having trouble interpreting the project's license, which looks like MIT, plus, in some files (that appear to be derived from Lemon), "public domain code". My employer allows contribution to MIT licensed project, but not projects with public domain dedications. It would be clarifying if you could add a LICENSE file for the entire project, even if individual files might have additional properties.

Thanks!

It would be really helpful if you could insert #sourceLocation directives corresponding to the #line directives in a lemon-generated parser. There are some annoying restrictions on where you can put them in Swift, but it should work to put one before each generated parse action function, I think.

Hi, i tried to run the expo example by lunching the Makefile, but i got the following errors in output:

mkdir -p ../../bin && clang ../../src/citron.c -o ../../bin/citron
../../bin/citron ArithmeticExpressionParser.y -o ArithmeticExpressionParser.swift
swiftc ../../src/CitronParser.swift ../../src/CitronLexer.swift ArithmeticExpressionParser.swift main.swift -o expr
../../src/CitronParser.swift:56:54: error: 'where' clause cannot be attached to an associated type declaration
associatedtype CitronTokenCode: RawRepresentable where CitronTokenCode.RawValue == CitronSymbolCode
^
../../src/CitronParser.swift:399:14: error: expected '(' for subscript parameters
subscript<I: BinaryInteger>(safe i: I) -> Element? {
^
../../src/CitronParser.swift:399:14: error: expected declaration
subscript<I: BinaryInteger>(safe i: I) -> Element? {
^
../../src/CitronParser.swift:398:9: note: in extension of 'Array'
private extension Array {
^
../../src/CitronParser.swift:50:38: error: use of undeclared type 'BinaryInteger'
associatedtype CitronSymbolCode: BinaryInteger // YYCODETYPE in lemon
^~~~~~~~~~~~~
../../src/CitronParser.swift:51:39: error: use of undeclared type 'BinaryInteger'
associatedtype CitronStateNumber: BinaryInteger
^~~~~~~~~~~~~
../../src/CitronParser.swift:52:38: error: use of undeclared type 'BinaryInteger'
associatedtype CitronRuleNumber: BinaryInteger
^~~~~~~~~~~~~
../../src/CitronParser.swift:139:40: error: use of undeclared type 'BinaryInteger'
enum _CitronParsingAction<StateNumber: BinaryInteger, RuleNumber: BinaryInteger> {
^~~~~~~~~~~~~
../../src/CitronParser.swift:139:67: error: use of undeclared type 'BinaryInteger'
enum _CitronParsingAction<StateNumber: BinaryInteger, RuleNumber: BinaryInteger> {
^~~~~~~~~~~~~
../../src/CitronParser.swift:149:38: error: use of undeclared type 'BinaryInteger'
enum _CitronStateOrRule<StateNumber: BinaryInteger, RuleNumber: BinaryInteger> {
^~~~~~~~~~~~~
../../src/CitronParser.swift:149:65: error: use of undeclared type 'BinaryInteger'
enum _CitronStateOrRule<StateNumber: BinaryInteger, RuleNumber: BinaryInteger> {
^~~~~~~~~~~~~
../../src/CitronParser.swift:159:30: error: cannot invoke 'symbolNameFor' with an argument list of type '(code: Self.CitronTokenCode.RawValue)'
tracePrint("Input:", symbolNameFor(code:symbolCode))
^
../../src/CitronParser.swift:159:30: note: expected an argument list of type '(code: Self.CitronSymbolCode)'
tracePrint("Input:", symbolNameFor(code:symbolCode))
^
../../src/CitronParser.swift:161:26: error: cannot invoke 'yyFindShiftAction' with an argument list of type '(lookAhead: Self.CitronTokenCode.RawValue)'
let action = yyFindShiftAction(lookAhead: symbolCode)
^
../../src/CitronParser.swift:161:26: note: expected an argument list of type '(lookAhead: Self.CitronSymbolCode)'
let action = yyFindShiftAction(lookAhead: symbolCode)
^
../../src/CitronParser.swift:185:26: error: cannot invoke 'yyFindShiftAction' with an argument list of type '(lookAhead: Int)'
let action = yyFindShiftAction(lookAhead: 0)
^
../../src/CitronParser.swift:185:26: note: expected an argument list of type '(lookAhead: Self.CitronSymbolCode)'
let action = yyFindShiftAction(lookAhead: 0)
^
../../src/CitronParser.swift:250:20: error: cannot invoke initializer for type 'Int' with an argument list of type '(Self.CitronStateNumber)'
assert(Int(state) < yyShiftOffset.count)
^
../../src/CitronParser.swift:250:20: note: overloads for 'Int' exist with these partially matching parameter lists: (Int64), (Word), (UInt8), (Int8), (UInt16), (Int16), (UInt32), (Int32), (UInt64), (UInt), (Int), (Float), (Double), (Float80), (String, radix: Int), (CGFloat), (NSNumber)
assert(Int(state) < yyShiftOffset.count)
^
../../src/CitronParser.swift:251:30: error: binary operator '<' cannot be applied to operands of type 'Self.CitronSymbolCode' and 'Int'
assert(lookAhead < yyNumberOfSymbols)
~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~
../../src/CitronParser.swift:251:30: note: expected an argument list of type '(Int, Int)'
assert(lookAhead < yyNumberOfSymbols)
^
../../src/CitronParser.swift:252:31: error: cannot invoke initializer for type 'Int' with an argument list of type '(Self.CitronStateNumber)'
i = yyShiftOffset[Int(state)] + Int(lookAhead)
^
../../src/CitronParser.swift:252:31: note: overloads for 'Int' exist with these partially matching parameter lists: (Int64), (Word), (UInt8), (Int8), (UInt16), (Int16), (UInt32), (Int32), (UInt64), (UInt), (Int), (Float), (Double), (Float80), (String, radix: Int), (CGFloat), (NSNumber)
i = yyShiftOffset[Int(state)] + Int(lookAhead)
^
../../src/CitronParser.swift:254:74: error: binary operator '!=' cannot be applied to two 'Self.CitronSymbolCode' operands
if (i < 0 || i >= yyLookaheadAction.count || actionLookahead != lookAhead) {
~~~~~~~~~~~~~~~ ^ ~~~~~~~~~
../../src/CitronParser.swift:254:74: note: overloads for '!=' exist with these partially matching parameter lists: (Any.Type?, Any.Type?), (UInt8, UInt8), (Int8, Int8), (UInt16, UInt16), (Int16, Int16), (UInt32, UInt32), (Int32, Int32), (UInt64, UInt64), (Int64, Int64), (UInt, UInt), (Int, Int), (ContiguousArray, ContiguousArray), (ArraySlice, ArraySlice), (Array, Array), (T?, T?), (T?, _OptionalNilComparisonType), (_OptionalNilComparisonType, T?), ((A, B), (A, B)), ((A, B, C), (A, B, C)), ((A, B, C, D), (A, B, C, D)), ((A, B, C, D, E), (A, B, C, D, E)), ((A, B, C, D, E, F), (A, B, C, D, E, F)), (LazyFilterIndex, LazyFilterIndex), ([Key : Value], [Key : Value])
if (i < 0 || i >= yyLookaheadAction.count || actionLookahead != lookAhead) {
^
../../src/CitronParser.swift:256:45: error: cannot subscript a value of type '[Self.CitronSymbolCode]' with an index of type '(safe: Self.CitronSymbolCode)'
if let fallback = yyFallback[safe: lookAhead], fallback > 0 {
^
../../src/CitronParser.swift:256:45: note: overloads for 'subscript' exist with these partially matching parameter lists: (Int), (Range), (Range<Self.Index>), (ClosedRange<Self.Index>), (CountableRange<Self.Index>), (CountableClosedRange<Self.Index>)
if let fallback = yyFallback[safe: lookAhead], fallback > 0 {
^
../../src/CitronParser.swift:265:33: error: cannot invoke initializer for type 'Int' with an argument list of type '(Self.CitronSymbolCode)'
let j = i - Int(lookAhead) + Int(wildcard)
^
../../src/CitronParser.swift:265:33: note: overloads for 'Int' exist with these partially matching parameter lists: (Int64), (Word), (UInt8), (Int8), (UInt16), (Int16), (UInt32), (Int32), (UInt64), (UInt), (Int), (Float), (Double), (Float80), (String, radix: Int), (CGFloat), (NSNumber)
let j = i - Int(lookAhead) + Int(wildcard)
^
../../src/CitronParser.swift:275:40: error: cannot invoke initializer for type 'Int' with an argument list of type '(Self.CitronStateNumber)'
return yyDefaultAction[Int(state)]
^
../../src/CitronParser.swift:275:40: note: overloads for 'Int' exist with these partially matching parameter lists: (Int64), (Word), (UInt8), (Int8), (UInt16), (Int16), (UInt32), (Int32), (UInt64), (UInt), (Int), (Float), (Double), (Float80), (String, radix: Int), (CGFloat), (NSNumber)
return yyDefaultAction[Int(state)]
^
../../src/CitronParser.swift:284:16: error: cannot invoke initializer for type 'Int' with an argument list of type '(Self.CitronStateNumber)'
assert(Int(state) < yyReduceOffset.count)
^
../../src/CitronParser.swift:284:16: note: overloads for 'Int' exist with these partially matching parameter lists: (Int64), (Word), (UInt8), (Int8), (UInt16), (Int16), (UInt32), (Int32), (UInt64), (UInt), (Int), (Float), (Double), (Float80), (String, radix: Int), (CGFloat), (NSNumber)
assert(Int(state) < yyReduceOffset.count)
^
../../src/CitronParser.swift:285:32: error: cannot invoke initializer for type 'Int' with an argument list of type '(Self.CitronStateNumber)'
var i = yyReduceOffset[Int(state)]
^
../../src/CitronParser.swift:285:32: note: overloads for 'Int' exist with these partially matching parameter lists: (Int64), (Word), (UInt8), (Int8), (UInt16), (Int16), (UInt32), (Int32), (UInt64), (UInt), (Int), (Float), (Double), (Float80), (String, radix: Int), (CGFloat), (NSNumber)
var i = yyReduceOffset[Int(state)]
^
../../src/CitronParser.swift:288:26: error: binary operator '<' cannot be applied to operands of type 'Self.CitronSymbolCode' and 'Int'
assert(lookAhead < yyNumberOfSymbols)
~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~
../../src/CitronParser.swift:288:26: note: expected an argument list of type '(Int, Int)'
assert(lookAhead < yyNumberOfSymbols)
^
../../src/CitronParser.swift:314:27: error: binary operator '<' cannot be applied to operands of type 'Self.CitronRuleNumber' and 'Int'
assert(ruleNumber < yyRuleInfo.count)
~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~
../../src/CitronParser.swift:314:27: note: expected an argument list of type '(Int, Int)'
assert(ruleNumber < yyRuleInfo.count)
^
../../src/CitronParser.swift:316:54: error: cannot invoke initializer for type 'Int' with an argument list of type '(Self.CitronRuleNumber)'
tracePrint("Reducing with rule:", yyRuleText[Int(ruleNumber)])
^
../../src/CitronParser.swift:316:54: note: overloads for 'Int' exist with these partially matching parameter lists: (Int64), (Word), (UInt8), (Int8), (UInt16), (Int16), (UInt32), (Int32), (UInt64), (UInt), (Int), (Float), (Double), (Float80), (String, radix: Int), (CGFloat), (NSNumber)
tracePrint("Reducing with rule:", yyRuleText[Int(ruleNumber)])
^
../../src/CitronParser.swift:320:35: error: cannot invoke initializer for type 'Int' with an argument list of type '(Self.CitronRuleNumber)'
let ruleInfo = yyRuleInfo[Int(ruleNumber)]
^
../../src/CitronParser.swift:320:35: note: overloads for 'Int' exist with these partially matching parameter lists: (Int64), (Word), (UInt8), (Int8), (UInt16), (Int16), (UInt32), (Int32), (UInt64), (UInt), (Int), (Float), (Double), (Float80), (String, radix: Int), (CGFloat), (NSNumber)
let ruleInfo = yyRuleInfo[Int(ruleNumber)]
^
../../src/CitronParser.swift:384:15: error: binary operator '>' cannot be applied to operands of type 'Self.CitronSymbolCode' and 'Int'
if (i > 0 && i < yySymbolName.count) { return yySymbolName[Int(i)] }
~ ^ ~
../../src/CitronParser.swift:384:15: note: expected an argument list of type '(Int, Int)'
if (i > 0 && i < yySymbolName.count) { return yySymbolName[Int(i)] }
^
../../src/CitronParser.swift:384:68: error: cannot invoke initializer for type 'Int' with an argument list of type '(Self.CitronSymbolCode)'
if (i > 0 && i < yySymbolName.count) { return yySymbolName[Int(i)] }
^
../../src/CitronParser.swift:384:68: note: overloads for 'Int' exist with these partially matching parameter lists: (Int64), (Word), (UInt8), (Int8), (UInt16), (Int16), (UInt32), (Int32), (UInt64), (UInt), (Int), (Float), (Double), (Float80), (String, radix: Int), (CGFloat), (NSNumber)
if (i > 0 && i < yySymbolName.count) { return yySymbolName[Int(i)] }
^
make: *** [expr] Error 1

When using Citron as an SPM package, the CitronParser protocol (which includes the core parsing code) ends up in a separate module from the generated parser. I found that this led to the parser spending most of its time at runtime in Swift metadata lookup (I don't believe it's able to inline specialise the protocol and inline its code across the module boundary).

I don't still have the Instruments trace handy, but one of the examples I saw was lookup for metadata of a tuple, which seemed to be the one for yyStack.

By gratuitously adding @inlinable to CitronParser (in BenHetherington/citron@87d835c), I was able to reduce the runtime of my project's parser by about 8x (in my test, this took it from over 8 seconds to about 1 second, with the parser running on all cores).

I'm not sure if this is an appropriate solution for this project – it required making a lot of things internal to the Citron module rather than private to CitronParser – but I figured it was worth reporting the performance issue all the same!

There are more opportunities for good error reporting by adding more sourceLocations, e.g in this excerpt of generated code,

    enum CitronSymbol {
        case yyBaseOfStack
        case yy0(CitronToken)
        case yy7(EBNF.AltList)
        case yy17(EBNF.Alt)
        case yy26(EBNF.TermList)
        case yy40(EBNF.RuleList)
        case yy46(EBNF.Term)
        case yy52(EBNF.Rule.Kind)
        case yy56(EBNF.Rule)

Each of these case lines corresponds to a place in the original source where the type was declared for a nonterminal. If I got the spellings of those types wrong, I'll see errors in the generated .swift file rather than in the citron input.

The docs say I can write:

%token_set throws_clause Throws | Rethrows.

That doesn't work; you have to leave out the |.

I don't know if something changed in the upstream citron sources you're using, but…

If you check out the non-determinism branch of my project and do make run repeatedly, it will fail to parse some good percentage of the time, and succeed the others.
I'm testing on an M1 mac, FWIW.

It appears SPM has a feature that would support running citron as part of a build. Citron could supply one. Please see https://github.com/apple/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md#example-1-swiftgen

I am currently referencing citron as a Git submodule of my project, but that creates problems for some of the less Git-experienced people on my team. It would be much better if I could simply reference citron in my own Package.swift.

I have a grammar with shift/reduce conflicts I've been unable to resolve using associativity and precedence. The default behavior of citron is to resolve them in the way I want, but when I integrate running citron into my build, the build step that includes generating the .swift file always fails. That's problematic for integration into a larger workflow. I'd like a command line option that lets citron exit with status 0 even when there are conflicts.

results in code like this when the action clause for a rule is empty:

        case 1: /* rule_list ::= */
            func codeBlockForRule01() throws -> EBNF.RuleList {
#sourceLocation(file: "Sources/ebnf-citron/EBNFGrammar.citron", line: 37)

#sourceLocation()
}

That causes the error report to point into the generated .swift file instead of the grammar file, which is not very useful.

Sources/ebnf-citron/EBNFGrammar.citron:37:1: error: missing return in local function expected to return 'EBNF.RuleList' (aka 'Array<(lhs: Substring, rhs: Array<Optional<Array<EBNF.Term>>>)>')
}
^

If you move the #sourceLocation() to just after the closing brace, the report shows up in the right place. This is an important case because parsers will often be developed by capturing the grammar first and using %default_nonterminal_type Void. When making the transition, it doesn't work out so well.

So I did this:

%class_name ValParser
%preface {
  import CitronParserModule
}
%token_type Int
%nonterminal_type input Void
%start_symbol input
input ::= .
{}

Just as a step in getting a new project started. The resulting Swift parser file wouldn't compile.
I know it's not a realistic use case, but it could get in the way of someone just getting started.

I got an error today that pointed into the definition of CitronSymbol in the generated parser code. It would be nice if that could point back into the source file. Thanks!

All the examples seem to get built by SPM… except for that one!