rhaiscript / lsp Goto Github PK

Currently we have almost 0 type information, which sucks. There should be a way for library authors to define types, maybe even extend the language like python or typescript did.

We will probably need some kind of configuration file the same way it is done in typescript.

The config file should set the following:

• what paths to include when looking for rhai scripts, currently we traverse the entire workspace tree
• what paths to exclude

Rhai 1.4.0 added range operators, these need to be added to both the parser and the HIR.

?. augments . (property access)

?[ augments [...] (index access)

?? is a new binary operator similar to && and ||

The LSP cannot be configured right now, so all features are enabled as-is.

Configuration options to consider:

• semantic tokens
• show references
• Rhai panel (vscode)
• Debug information (vscode)
• Log level

We currently make no difference between closures and functions, so the following is valid:

let foo = 2;
fn bar() {
  let baz = foo + 1;
}

We'll simply have to restrict function scopes.

These are some of the issues I found running the LSP on Windows:

1. src/ast/rhai.ungram must have LF line ending. Windows CR+LF line ending is not supported by Rowan. Git for Windows automatically converts all line endings to Windows-style when cloning.

2. All Windows CR+LF line endings in Rhai scripts throw an invalid input error.

3. print seems to be unresolved, and it gives a hint saying maybe it should be print.

4. It no longer works once you start editing the code. Before edit, popup's work fine. After editing anything, no longer.

We currently support one export item per export statement, but the Rhai book has examples for comma separated exports.

How about some instructions to build the language server and use it with VS Code?

I would LOVE to try it out...

this inside functions can be bound to pretty much anything, to find out the type(s) of this, we'll have to look up all the references to the function and see how it is used.

Currently expected tokens are dumped as-is (e.g. expected one of KW_RETURN ), we could name them better in errors.

We should parse in just as an operator for the sake of simplicity, then restrict its usability in the HIR where needed.

Adjust the binding powers as described here: https://rhai.rs/book/engine/custom-op.html#operator-precedence

When an unexpected token is encountered, in some cases the parser will try to parse the next rule without consuming the invalid token. This way the parser can be more tolerant to errors and more valid language elements can be recognized in partially valid code. However in some cases this leads to repeatedly trying to parse the same rule leading to either stack overflows or infinite loops.

The cases where a token can be left unconsumed on error should be thoroughly reviewed and used sparingly.

We should also implement fuzz tests to detect these issues.

We can use engine.gen_fn_metadata_to_json, generate the function and data type structure, and import lsp as the function and structure type registered by the extension developer.

#[export_module]
mod json_module {
    use rhai::{Dynamic, ImmutableString, Map};

    #[rhai_fn(name = "dump", global)]
    pub fn json_dump(json_map: Map) -> String {
        serde_json::json!(json_map).to_string()
    }

    #[rhai_fn(name = "load", global)]
    pub fn json_load(json_string: &str) -> Dynamic {
        let map: Dynamic = serde_json::from_str(json_string).unwrap_or_default();
        return map;
    }
}

• gen_fn_metadata_to_json

{
    "modules": {
        "json": {
            "functions": [
                {
                    "baseHash": 1772367929775027429,
                    "fullHash": 119576746319902270,
                    "namespace": "global",
                    "access": "public",
                    "name": "dump",
                    "type": "native",
                    "numParams": 1,
                    "params": [
                        {
                            "type": "Map"
                        }
                    ],
                    "returnTypeName": "String",
                    "signature": "dump(json_map: Map) -> String"
                },
                {
                    "baseHash": 10649942213899484619,
                    "fullHash": 1440055287418587681,
                    "namespace": "global",
                    "access": "public",
                    "name": "load",
                    "type": "native",
                    "numParams": 1,
                    "params": [
                        {
                            "type": "&str"
                        }
                    ],
                    "returnTypeName": "Dynamic",
                    "signature": "load(json_string: &str) -> Dynamic"
                }
            ]
        }
    }
}

• When I type json::,automatically complete load or dump.

Since Rhai is designed to be embedded in Rust applications, it would make sense to support such applications. The language is very dynamic, even supporting user-provided keywords and operators.

Rhai identifiers are slightly different from Rust. For example, the first non-underscore character must be a letter and not a digit, so _0X is not allowed as an identifier.

Also, numbers cannot start with an underscore, so _10 is a syntax error. Otherwise it is too easy to confuse with -10.

A number's decimal point cannot be followed immediately by a _. Therefore, 10._1 is not allowed, but 10_.1 is OK.

NOTE: Your Rhai panel with the parse tree works wonders in finding out these issues.

Show docs and types when hovering over symbols.

Support throw <Expr>.

Figure out a way to handle symbols and scopes for completions.

We simply have to add the object fields to the visible symbols.

Provide a proper hierarchy of symbols.

fn outer() {
  fn inner() {
  }
}

This is currently accepted, but errors upstream.

We should do some control-flow analysis for type inference (if typeof ...), I imagine this to be pretty hard with full of edge-cases (even for typescript it took a while to do it properly).

Currently all strings are processed as-is, and interpolations are ignored.

I don't know yet where to process these, as they will complicate the parse tree a whole lot.

I started with targeting WASM initially as it can be run in the browsers, and the VSCode extension distribution is trivial.

However there are a few issues:

• build times are worse (due to additional steps) and debugging is awful
• it requires a shim of javascript for i/o
• worse performance (no threads available)

So right now I left the WASM builds as-is, and switched to a native executable for the initial development.
I will also clean up any unnecessary WASM-related build steps and javascript shims to make contributions easier.

The big question is how to port Rhai LSP to browsers. There are several options:

• target WASI, we would have i/o but still no threads, and the state of WASI still seems very experimental
• target WASM with a javascript layer, this is already what we had and turned out to be rather cumbersome
• do not port at all, and always require the LSP to run somewhere outside of the browser or vscode extensions

Provide folding ranges in the LSP server.

Show a dedicated VSCode activity panel for Rhai.

Things to display:

• Syntax tree
• User integrations and context information (#3)

All tokens are available from the Rhai book.

Only very simple code can be parsed right now, a Pratt parsing algorithm is required for expressions (and precedence).

Every part was designed with this in mind however it's still probably a large amount of work.

For the sake of simplicity, constructs like let a = 0 are also expressions that evaluate to ().

As a consequence currently the following is also accepted:

It is roughly equivalent to (this) . (const a = 2), It's not clear whether to leave it as is, or specify stricter rules.

• constants
• functions
• closures

Create a Rhai formatter on top of the current parser, and also integrate it with the LSP.

The HIR currently uses a somewhat ad-hoc architecture for state representation. Right now no partial updates are possible without manual fiddling with internal state.

Maybe we should look at something like salsa (used by rust-analyzer) for partial updates and query caching, it should make our lives easier in the long term.

Such as logging to a console, or a callback, etc.

Go to definiton for symbols.

LSP completions:

• functions
• variables
• #25

For better visibility/debugging.

I haven't been able to reproduce it with unit tests, adding and removing sources eventually lead to panics with missing scopes or symbols.

Find references of declarations. We already have this information in the HIR, we just have to send it via the LSP.

I pulled in the rhai engine in order to test the compliance of the parser (and later the HIR), but Engine::compile(...) seems to fail if the scripts starts with a shebang line with the error of #! being reserved.

According to the book, eval_file supports shebangs, is this a bug or do I need to strip the line manually before passing the script to compile?

I'm not sure if I should open this in the Rhai repo.

Planned language features from easy to hard:

• symbol tree (we need it anyway)
• folding ranges
• hover
• completion
  • functions
  • variables
  • modules (nontrivial)
• go to definition
• semantic tokens
• formatting

AST should be generated from ungrammar. I originally tried copying the AST codegen from rust-analyzer, but unfortunately it is very Rust grammar-specific and spaghetti-like, and it would be better to write the generation code from scratch.

With the current bugs fixed, I think the language server could get into its first shippable state.

Here are the steps that I think need to be done in order to properly ship it:

TODO before shipping

• fix all the breaking bugs in the HIR and language server (#52)
• write a lot more tests for the HIR
• include a default set of hand-written definition files (#35)
• figure out a seamless way for users to load their own metadata for definitions (#35)
• finalize configuration file format and actually use it (#39)
• document the vscode extension

Steps to ship

We'll need to

• either merge this repo into rhai and create a universal CLI that includes the language server
• or create the WASM bindings and use them in the extension

and then

• create CI/CD jobs that builds the vscode extension either for multiple platforms or just one in case the WASM step above was chosen
• publish the extension to marketplaces