masala / masala-parser Goto Github PK
View Code? Open in Web Editor NEWJavascript Generalized Parser Combinators
License: GNU Lesser General Public License v2.1
Javascript Generalized Parser Combinators
License: GNU Lesser General Public License v2.1
Create a simple markdown parser
It could be nice to accept date('MM-D-YYYY, h:mm:ss a')
. It should not be that complicated for dates that does not require languages.
MMMM is difficult and moment.js uses some localized files, wich is out of masala's scope. https://github.com/moment/moment/tree/develop/locale
Letters is a succession of letters
The Parser must deal with internationalization. There may need to configure the Parser with a builder function.
I started writing my own parser combinators library until I came across this project and Bennu. Bennu hasn't been worked on in a while, but it seems quite mature. Its really lightweight and has all the bells and whistles I could imagine.
Any reason you are deciding to build a library of your own rather than using Bennu? Just curious, what are you guys using this library for?
In the file samples/trivial-test.md, it works if I put:
title1
====
But fails if i put
# title1
The goal is to write easily these often seen structures :
2+2
P.sequence ( this.number(), '+', this.number() )
will result in an array : [2, '+', 2]
So we can write :
P.sequence ( this.number(), '+', this.number() ).map( values => values[0] + values[2] );
Make the parse of a Kiss compiler. The context created must reference variables and scopes. If there is none in this stupid language, use an ultra basic Javascript.
We often use combinator.parse(Stream.of(document) )
. We could add a function combinator.parseString(document)
, or modify the parse(doc | stream)
function to test if argument is a String or a Stream.
Question is the same with the extractor. Should we create textUntil( string | combinator
), or create textUntil(combinator)
and textUntilString(string)
?
It's mainly to avoid emails when we push onb WIP branches. I don't know what is the best practice. I'm pretty sure we don't need emails at midnight to say the build is broken.
The objective is to quickly mix a with b multiple elements.
In Parser class :
thenSpread(p) {
return this.flatmap((a) => p.map((b) => [a, ...b]));
}
Real exemple : We have a paragraph then a succession of others
function paragraphs() {
return P.try(paragraph().thenSpread(followingParagraph().rep()));
}
It will return :
parsing Accept {
offset: 90,
consumed: true,
value:
[ { paragraph: [Object] },
{ paragraph: [Object] },
{ paragraph: [Object] } ],
input:
StringStream {
source: 'Lorem ipsum is a *first* paragraph\nSecond line\n\nThe second paragraph\n\nThe Third paragraph\n' } }
The counterpart return this.flatmap((a) => p.map((b) => [...a,b]));
might also be interesting in some cases.
Write a documentation of most Parser functions with exemple and links to unit tests.
I will open a branch for a text Extractor in standard/
directory. It comes after my "real world mission". Main features will be:
stringIn(['John', 'Jack'])
: like charIn
, but with an array of stringscharNumber
: search for numbers, but returns a string and keep 0 ; useful for dates or phone number ( 0502)simpleWords(separators)
: grab words define as letters
and separated with separators. Separators will default to [' ', '\n']
;textUntil(stop, including:bool = false)
: eats text until stop. Stop can be a combinator, or a text. Set including
flag to true to also eat the stop"We are the 23-09-2013 at noon." : textUntil(looseDate())
will return "We are the "
and offset is put at the beginning of the date. textUntil(looseDate(), true)
would return "We are the 23-09-2013"
Hello @d-plaindoux
I have set ES6 with grunt-babel and jshint does correctly his job. Grunt needs to be at 0.4 old version because of grunt-coverage dependency.
At the end babel produces a dist/app.js ES5 file with sourcemap. Options are defined in the Gruntfile.
When running grunt --gruntfile Gruntfile_Coverage.js
,I have no error but I have seen no coverage information, and no lib-cov/report
directory. What is it supposed to do ?
If it's ok, I will ES6 everything as soon as possible.
What is the proper way to select everything up to a given sequence?
Let's say that I have this source : "wordSTOPwordagainSTOP"
And I want my value to be : ["word","wordagain"]
I have wrote :
function stop(){
return P.string('STOP').rep();
}
function detectStop(){
return P.try(stop().or(P.letters())).rep();
}
function parseText( line, offset=0){
return detectStop().parse(stream.ofString(line), offset)
}
So once the parser enters into letters(), it does not come back to see if there is a STOP, which has however a better priority. How would I do that ?
====
But suppose that the separator is now -stop-
with not only letters
function stop(){
return P.string('-stop-').rep();
}
And we test : test-stop-test-stop-
, then my parser works
It is easy, because the parse will detect a special character '-'. But we don't always want to
introduce a special character.
====
I propose a debug function used like this :
function paragraph() {
// if we have found a line, then we will enter in debug
return line().debug('found a line').thenLeft(eol.opt()).map(paragraphText);
}
Let's have a line
as Markdown input. The console output is :
[debug] : found a line [ { text: 'Lorem ' }, { bold: { text: 'ipsum' } }...]
Because debug
() is put after line()
, the parser will enter in debug. If there was no line, we would not enter.
I send a PR with an exemple of code. If you have more ideas, they are welcome.
The P.letter
parser will not accept accents likeé
or any foreign UTF-8 characters. There is a sufficient for the moment trick:
var firstLetter = name.charAt(0).toUpperCase();
if( firstLetter.toLowerCase() != firstLetter) {
// it's a letter
}
else {
// it's a symbol
}
First solution is to rename it to P.asciiLetter
. Second solution is to use the trick, but it will make it quite slower. Third solution is to redefine the method and use a flag as P.letter(onlyAscii=true)
The question is : When you use optrep(), is it ok to have zero elements ? I think so according to the passing tests.
'expect (optrep) to accepted': function(test) {
test.deepEqual(parser.char("a").optrep().parse(stream.ofString("a"),0).isAccepted(),
true,
'should be accepted.');
},
'expect (optrep) none to accepted': function(test) {
test.deepEqual(parser.char("a").optrep().parse(stream.ofString("b"),0).isAccepted(),
true,
'should be rejected.'); // <===== HERE : did you meant 'Sould be accepted' ?
},
I'm not fond of this semantic : in real language, you repeat when you have at least two elements. optrep() should be ok with one element or more, but not OK with zero elements.
So we need to build something else when you need to test a real repetition (at least two elements). Something like :
P.try(anyTitle()).blankLine().blankLine().optrep().paragraph() // at least one blankline
There is a debug in markdown lib, and still a lot in unit-tests.
A stream can be buffered. For this purpose an entropic cache is build. When a parser is accepted a cut mechanism can be applied in order to flush cache and optimise memory usage.
Creating a simple operation parser mainly for educational purposes
I want to create a class LineParser :
export default class LineParser { // ideally it could extend directly Parser or ParserHelper
textValue(chars) {
return { text: chars.join('').trim() };
}
text(separator) {
if (separator) {
/// vvvv this is null
return P.not(eol.or(P.string(separator))).optrep().map(this.textValue);
} else {....}
}
The problem is that this fonction is extracted from the class with this code :
// (('b -> Parser 'a 'c) * 'b)-> Parser 'a 'c
function lazy(p, parameters) {
// vvvv 'this' will always be null :(
return new Parser((input, index=0) => p.apply(null, parameters).parse(input, index));
}
Do you have any clue to avoid this p.apply(null)
?
This method is unused. I think this stuff is useless since map and flatmap provide the best approach for data transformation.
In the markdown token.js the following function:
function fourSpacesBlock() {
return P.char('\t').or(P.try(P.charIn(' \u00A0').then(P.charIn(' \u00A0'))
.then(P.charIn(' \u00A0')).then(P.charIn(' \u00A0'))));
}
can be simplified and replaced by:
function fourSpacesBlock() {
return P.char('\t').or(P.charIn(' \u00A0').occurrence(4));
}
Expressiveness can be increased using infix operators.
p1 <*> p2 // == p1.then(p2)
p1 <|> p2 // == p1.or(p2)
p1 >>= p2 // == p1.flatmap(p2)
p1 || p2 // == p1.chain(p2)
This can be achieved using Sweet.JS meta language.
operator <*> left 1 = (left, right) => #`${left}.then(${right})`;
operator <|> left 1 = (left, right) => #`${left}.or(${right})`;
operator >>= left 1 = (left, right) => #`${left}.flatmap(${right})`;
operator || left 1 = (left, right) => #`${left}.chain(${right})`;
Since v 0.3, hotelhub automated tests are 10x slower, but the main difference is that their custom code has been wrapped into the Extractor bundle, and that we have split code on bundles.
Here is the current implementation of subStream:
// Stream 'a => [Comparable 'a] -> number -> boolean
subStreamAt(s, index){
for (var i = 0; i < s.length; i++) {
var value = this.get(i + index);
if (!value.isSuccess() || value.success() !== s[i]) { // <=== compareTo
return false;
}
}
return true;
}
Suppose we want to create P.stringIgnoreCase("john doe")
. We could send a compareTo function to subStream(s, index, [compareTo])
.
if (!value.isSuccess() || !compareTo(value.success(), s[i]) ) {
return false;
}
Is it a good idea ?
It's a mapping function to pick first and last element of an array:
var helloParser = C.string("Hello")
.then(C.char(' ').rep())
.then(C.char("'"))
.thenRight(C.letter.rep()) // keeping repeated ascii letters
.thenLeft(C.char("'")); // keeping previous letters
.map (F.last) // keep last letter
The standard directory contains a json parser and a naive markdown parser (obsolete) and a markdown parser. This should be reorganise with sub directories i.e. one per parser.
Be able to parse (2+2)*3
and find 12.
Make it in a feature/operation branch.
Standard libs should not refer to fs. Only unit tests.
This test is passing. It should not, I think.
'test bullet niveau 2': function (test) {
const line = "\t * This is another lvl2 bullet \n ";
testLine(line);
test.deepEqual({bullet:{level:2, content:[{text:'This is another lvl2 bullet '}]}}, value,'probleme test:test bullet Lvl2');
test.done();
},
It should be quite easy to create a stream form RxJS, and as it is a de facto standard, it should be well accepted.
A bit more complex is to allow send Response inside another Rx stream.
Token bundle is not exported and pretty empty.
Suppose there is no bug. What feature do we need for 1.0 ?
Any ideas ?
Instead of
var parsec = require('parser-combinator');
var S = require('parser-combinator').stream;
const document = "Hello World in 2017";
const stream = S.ofString(document);
var P = parsec.parser;
We could just write parser.parseString(document);
or something like that.
To be defined ...
The goal is to parse easily ;
!image: duck.png
Using something like :
P.char('!').then(text()).then(P.char(':')).thenLeft(spaces()).then(text())
We now have a strange soup of arrays. Using :
P.char('!').then(text()).then(P.char(':')).thenLeft(spaces()).then(text()).flattenDeep()
We now have a flat array : ['!','image', ':', 'duck']
It should use npm run xyz
Windows uses a \r\n
for line feed. There is a lot of \n
tests, especially in Markdown, so we may check if \r\n
is also working.
Exemple: markdown/bullet-parser.js
function bulletLv1(){
// TODO: check if T.eol is better on windows
return C.char('\n').optrep()
the git tag
command doesn't give any tags
There is a token class, some token (?) in Parser such as P.letter, and a standard/token file. Maybe the term token for an email is inappropriate.
Making a 2 step integration:
npm run prepublish
will package parser-combinator.js to /pre-integration and verify that it can be usednpm run integration
will download from npm and check that it's ok to be usedThere must be at least:
Then standard/ will use a JsonBundle and MarkdownBundle
I think it's not easy to understand for beginners.
We could change x.thenLeft(y)
by x.thenSkip(y)
and x.thenRight(y)
by x.thenKeep(y)
;
What do you think ?
It's not needed anymore
First and last items of Paragraphs are trimmed, so they may be empty.
Tests are still organizds with previous all in Parser class.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.