ilammy / sash Goto Github PK

Each diagnostic should be identified by a number in the code, instead of plain text strings. This will allow more rigorous testing and is a prerequisite for providing localizable diagnostics.

I have an idea of writing a Sash plugin for IntelliJ IDEA to get IDE support almost for free. The first step to this would be a scanner, so that we have at least basic syntactic highlighting and stuff. We have a scanner, but it certainly needs improvements before we can start using its code.

• Documentation. Lexical syntax of Sash should be documented, as well as have better technical documentation
  • Lexical syntax: documentation in doc directory at root level.
  • Implementation: outline should be described in src/README.md
• Library. The crate should be turned into a library, like libsyntax or something.
• Refactoring. I am not very satisfied with some points in the code and they definitely need some improvement.
  • Readability:
    • Better assertions for expected character sequences
    • Additional comments
    • Rustic style
  • Tracing: add a bunch of debug! calls to trace what happens during scanning.
  • Unit tests:
    • Avoid boilerplate and arbitrary calculations
    • Better failure messages
    • Check for the kind of errors detected
  • Structure: split out token type definitions and diagnostics into separate modules.
• Improvements. Some additional features of the scanner that need to be looked into.
  • Diagnostics:
    • Review and unify error messages.
    • Improve diagnostic levels. They should be clear to use. Look at Clang or whatever.
    • Better interface for diagnostic reporting.
  • Context: the scanner should provide as much information about tokens as possible. E.g., it should tell the type suffix of a literal.
  • Interpreting tokens: the lexer should do some interpreting of the tokens. E.g., normalize identifiers, expand escape sequences in strings. Maybe parse numbers into their values.
  • Coverage: I am absolutely sure that I have missed some cases in unit-tests. They need to be improved for better coverage. Here is how it can be done with kcov. (I do not think that Rust tooling is mature enough to include coverage reports into CI, but it is okay for occasional manual runs.)
• TODOs. There are quite a few sprinkled all over the code. They need to be resolved.
• Identifier normalization. Normalize identifiers and implicit symbols as a part of scanning.
  • Implement Unicode normalization algoritms.
  • Integrate them into scanner.
• Reserved identifiers. Integrate reserved identifeirs into tokens produced by the scanner.

It would be great to have a reference formal grammar of Sash. This will allow us to independently validate scanner and parser implementations, and may be a good starting point for documentation on allowed syntax.