Comments (5)
There is no official standalone lexer feature. There is a way, but I'm not convinced it should be official in its current state.
If you want you can look under the hood and see how regex parser implements it's lexer.
from ctpg.
Or did I just completely misunderstood you and you want to just use the regex lexer without the actual parser?
In this case you are out of luck. You could theoretically create a huge regex with all lexical tokens (just separate them with | ) like this:
(token_1_regex)|(token_2_regex)
but unfortunately there is no support for grouping captures.
You won't know which sub-regex was actually matched.
I'm open implementing this feature, this would take some time though.
from ctpg.
Yes, I was hoping to be able to use the lexer without the parser.
If you have the ability for your regexes to have numeric ids you just include that in the end state for each regex. You resolve ambiguity by only setting the id in an end state if one has not already been set.
(As you hinted at your lexer generator should or all the regexes together)
from ctpg.
I guess I could do a standalone lexer feature, it shouldn't be too hard to expose some interface for that. Like you said I already am creating a DFA from all the terminal symbols.
It would be a lexer class that resemble a parser interface but simpler, just the terms(...) call is enough.
And then a 'match' method accepting same kinds of arguments (buffer, options, error stream) and returning an index and a string view of a matched text.
from ctpg.
There are a couple of conventions:
- If there is no match for the lexer at the current position, you usually return a single character (I return an id of ~0 in this case too). This allows you to continue lexing.
- It is customary to return 0 for End of Input.
from ctpg.
Related Issues (20)
- Make tests and examples executables conditional HOT 6
- Constexpr parsing: non-constexpr containers HOT 5
- Parsing exponent notation in json-parser HOT 3
- Ignore all whitespaces but newlines HOT 3
- string_view buffer HOT 2
- [question] Cant Parameterize and "Seperate out" Parser HOT 7
- [question] Trouble Making Argument Lists HOT 1
- [Question] Is it applicable to parsing markdown? HOT 2
- [Question] Did I exceed the maximum number of rules? HOT 1
- Precedence When Matching/Lexing HOT 5
- [Feture Request] Integration with CTRE HOT 3
- how long to next release ? HOT 8
- Maybe it is possible that removing lexer scanner? HOT 2
- Big Parsers? HOT 1
- Constexpr parsing of an array of ints HOT 6
- [Question] how to make the lexer NOT match a particular string HOT 2
- [question solved] constexpr variable 'p' must be initialized by a constant expression HOT 1
- Simple expression example and AST HOT 3
- [MSVC] Compilation error, unable to resolve operator overload of `ctpg::stdex::cbitset<N>::operator==` HOT 6
- Documentation wrong? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ctpg.