phaazon / kak-tree-sitter Goto Github PK

View Code? Open in Web Editor NEW

73.0 5.0 11.0 883 KB

tree-sitter meets Kakoune

License: Other

Rust 66.42% KakouneScript 8.75% Scheme 24.83%

highlighting kakoune text-objects tree-sitter

kak-tree-sitter's Introduction

kak-tree-sitter

This is a binary server that interfaces tree-sitter with kakoune.

Important note: by default, no colorscheme supporting tree-sitter is set for you. You have to pick one or write your own. See this section from the wiki for further information.

Features
Install
Usage
Contributing
Credits

Features

Semantic highlighting.
- Automatically detects whether a buffer language type can be highlighted.
- Removes any default highlighter and replaces them with a tree-sitter based.
Semantic selections (types, functions, parameters, comments, tests, etc.)
- Similar features to f, ?, <a-/>, etc.
- Full object mode support (i.e. <a-i>, {, <a-]>, etc.)
Indents
Indent guidelines
Incremental parsing
Fetch, compile and install grammars / queries with ease (via the use of the ktsctl controller companion)
Ships with no mappings, defined options, but allows to use well-crafted values, user-modes, mappings and commands by picking them by hand.
Transformation-oriented; actual data (i.e. grammars, queries, etc.) can be used from any sources.

Roadmap

See the milestones.

User manual

See the User manual to know how to install, use, configure and get runtime resources.

Contributing

Whether you want to fix a bug, make a feature request, help improving something or add support for a new language by changing the default configuration, you should read the CONTRIBUTING.md file.

Credits

This program was inspired by:

kak-tree-sitter's People

Contributors

Stargazers

Watchers

Forkers

korreman refi64 joncol nuudlman jdugan6240 chriselrod postsolar igor-ramazanov topisani taupiqueur zetashift

kak-tree-sitter's Issues

Installs grammars and queries in the wrong directory

Following instructions from the wiki, it says that it will install the grammars at $XDG_CONFIG_HOME/kak-tree-sitter or ~/.config/kak-tree-sitter/ but it installs it in /run/user/1000/ktsctl.

I am not sure which of it was true

Edit:

Reading the README of ktsctl, it installs at ~/.local/share/kak-tree-sitter of which, ~/.local/share is part of the XDG_DATA_DIRS. But wiki says XDG_CONFIG_HOME. Is this an oopsie? At least grammars and queries are really installed there though

IOError on v0.5.3

Not sure yet why; yes, std::io::Error is super explicit about things.

IO error: Os {
    code: 2,
    kind: NotFound,
    message: "No such file or directory",
}

Packaging: contribution well welcomed

Currently, there is no release channel for people to install the project, besides crates.io. It’s fine for people working with Rust, but I stand by the idea that binaries on crates.io should only be about Rust development, and hence, I don’t think it’s a good release channel.

kak-lsp is in the extra Archlinux repository, for instance, and I think it would be great to have a similar thing for kak-tree-sitter. Same for brew.

I haven’t really thought about how to ease that process to package managers, and it’s a bit early, but it’s never too early to start thinking about that.

On the meantime, I’ll bootstrap a PKGBUILD for the AUR.

Fails to properly run

Adding kak-tree-sitter -k -d in kakoune config results in this output in debug buffer

shell stderr: <<<
running in /tmp/runtime-solitude/kak-tree-sitter
>>>
/home/solitude/.local/etc/kak/kakrc:5:1: 'eval': option not found: 'gray1'. Use declare-option first
/home/solitude/git/app/kakoune/share/kak/kakrc:29:1: 'evaluate-commands': 81:1: 'source': 5:1: 'eval': option not found: 'gray1'. Use declare-option first
error while parsing kakrc:
    1:1: 'source': 29:1: 'evaluate-commands': 81:1: 'source': 5:1: 'eval': option not found: 'gray1'. Use declare-option first

Highlighting ranges don't stop before the newline character

I'd move this discussion from #156 into a separate issue because it existed in prior versions too so it's not a regression introduced by fixes for #156.

The gist of this issue is that highlighting ranges sometimes span onto the newline character which makes it assume their face.

Example with a multi-line string:

Examples from #156:
#156 (comment)
#156 (comment)

A KTS thread gets stuck sometimes

More investigating is needed (maybe it’s what causing #156; maybe not). Basically, sometimes, a buffer stop getting highlighted (and the session freezes). Killing the session and opening a new one get no highlighting, even though KTS responds.

Spawns numerous kak processes

After each "refresh" a kak process is left.

Allow to merge configurations

As reported by @taupiqueur, a user might be interested in tweaking only a small part of the configuration. It could be nice not to have to copy the whole configuration, but instead do what Helix does: inlining the config (include_bytes!("config.toml")), and then loading the user configuration, overriding what’s already there.

That would allow people who do not have any configuration set to get new features as they update kak-tree-sitter. It would still require ktsctl though, but that’s fair for now.

Rewrite highlighting with tree-sitter only

Currently, kak-tree-sitter uses tree-sitter-highlight to get the capture group for us. This was convenient for the first implementation but:

The API is pretty poor for a general tree-sitter experience. Especially, now that highlighting works great, I would like to support semantics objects, and I can’t, because tree-sitter-highlight doesn’t expose the parsed Tree. So it would require a second parse of the buffer, which is a total waste of resources and time.
It doesn’t support textedit objects (i.e. delta edition). For a very big buffer, editing only a few characters at a few different places will generate something akin to O(n²) diffing, which is going to hurt. See https://github.com/phaazon/kak-tree-sitter/discussions/24.

Support local file paths for queries and grammars

People usually have already figured out how to manage their collection of configuration files (i.e. their dotfiles) and move it between systems, and when customizing queries or grammars, adding another few files to that tends to be easier than to ensure that

those customizations exist somewhere they can be cloned from
when that somewhere is private, this tool's internal git clone call knows the credentials for that

Shenanigans with git being able to clone from a local path exist as a workaround, but create an unnecessary copy.

And side note: The doc comments in kak-tree-sitter-config/src/lib.rs explain how LanguageQueriesConfig::url can be omitted to re-use the one from the grammar. The wiki doesn't mention that.

Kakoune becomes unresponsive when using kak-tree-sitter with golang

I am currently testing kak-tree-sitter with golang, and it’s working great! However, I am encountering an issue where kakoune becomes completely unresponsive after running for ~30-45 minutes. In order to resolve this, I have to terminate the shell session it’s running in and then kill both the kak and kak-tree-sitter processes before restarting kakoune.

I would be more than happy to provide any information you need to assist with troubleshooting. It’s worth noting that I usually have a split open with multiple buffers loaded, all of which have tree-sitter enabled when this issue occurs.

Blocks Kakoune session (on crash?)

I'm using commit b310011 (I wonder if there's a relevant difference to master).
I made a text object request in C++ and Kakoune started hanging indefinitely.
There was no kak-tree-sitter process left and also no /tmp/kak-tree-sitter-* where the fifo would live.

I don't know how to unblock Kakoune in this case.
Can we maybe have a control process that unblocks Kakoune if the kak-tree-sitter process dies?
Or provide instructions how to unblock it.
The fifo must still be alive in the kernel because Kakoune is trying to open it, it's only deleted on the filesystem.

“Real” partial updates

We want to use %val{uncommitted_modifications} and %val{history} instead of streaming buffers.

Server is broken since `0.4.0`

Summary

kts_lang is not set automatically on 0.4.0 and 0.4.1 (clean kakrc without -override kak-tree-sitter-set-lang), so calling kak-tree-sitter-highlight-buffer throws unsupported language:
setting kts_lang manually when buffer is opened fixes unsupported language: , but highlighting is not working (no error thrown, but no visual changes present also)
overriding kak-tree-sitter-set-lang and calling kak-tree-sitter-highlight-buffer throws Error: 1:1: 'kak-tree-sitter-highlight-buffer': 2:3: 'evaluate-commands': 2:5: 'echo': : No such file or directory

Reproduction

copy https://github.com/phaazon/kak-tree-sitter/blob/kak-tree-sitter-v0.4.1/config.toml
install toml via ktsctl
make a basic kakrc with eval %sh{ kak-tree-sitter -dks -vvvvv --session $kak_session } line
open a file (the same config.toml for example)

Info

OS: macOS Ventura 13.5.1 (22G90)
kak -version: Kakoune v2023.08.05
ktsctl: 0.3.0 via cargo install
kak-tree-sitter: 0.4.1 via cargo install

Logfile `stdout.txt` with `-vvvvv` when opening a `config.toml` file

2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages] loading configuration for cpp
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/cpp.so
2023-08-28T08:37:08.679Z WARN  [kak_tree_sitter::languages] cannot load grammar for language cpp: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/cpp.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/cpp.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/cpp.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/cpp.so' (no such file)
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages] loading configuration for comment
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/comment.so
2023-08-28T08:37:08.679Z WARN  [kak_tree_sitter::languages] cannot load grammar for language comment: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/comment.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/comment.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/comment.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/comment.so' (no such file)
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages] loading configuration for haskell
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/haskell.so
2023-08-28T08:37:08.679Z WARN  [kak_tree_sitter::languages] cannot load grammar for language haskell: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/haskell.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/haskell.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/haskell.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/haskell.so' (no such file)
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages] loading configuration for git-commit
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/git-commit.so
2023-08-28T08:37:08.679Z WARN  [kak_tree_sitter::languages] cannot load grammar for language git-commit: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/git-commit.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/git-commit.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/git-commit.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/git-commit.so' (no such file)
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages] loading configuration for java
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/java.so
2023-08-28T08:37:08.679Z WARN  [kak_tree_sitter::languages] cannot load grammar for language java: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/java.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/java.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/java.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/java.so' (no such file)
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages] loading configuration for scheme
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/scheme.so
2023-08-28T08:37:08.679Z WARN  [kak_tree_sitter::languages] cannot load grammar for language scheme: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/scheme.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/scheme.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/scheme.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/scheme.so' (no such file)
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages] loading configuration for xml
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/xml.so
2023-08-28T08:37:08.679Z WARN  [kak_tree_sitter::languages] cannot load grammar for language xml: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/xml.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/xml.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/xml.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/xml.so' (no such file)
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages] loading configuration for yaml
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/yaml.so
2023-08-28T08:37:08.679Z WARN  [kak_tree_sitter::languages] cannot load grammar for language yaml: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/yaml.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/yaml.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/yaml.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/yaml.so' (no such file)
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages] loading configuration for c
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/c.so
2023-08-28T08:37:08.679Z WARN  [kak_tree_sitter::languages] cannot load grammar for language c: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/c.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/c.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/c.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/c.so' (no such file)
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages] loading configuration for awk
2023-08-28T08:37:08.679Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/awk.so
2023-08-28T08:37:08.680Z WARN  [kak_tree_sitter::languages] cannot load grammar for language awk: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/awk.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/awk.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/awk.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/awk.so' (no such file)
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages] loading configuration for json
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/json.so
2023-08-28T08:37:08.680Z WARN  [kak_tree_sitter::languages] cannot load grammar for language json: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/json.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/json.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/json.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/json.so' (no such file)
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages] loading configuration for latex
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/latex.so
2023-08-28T08:37:08.680Z WARN  [kak_tree_sitter::languages] cannot load grammar for language latex: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/latex.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/latex.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/latex.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/latex.so' (no such file)
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages] loading configuration for markdown.inline
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/markdown.inline.so
2023-08-28T08:37:08.680Z WARN  [kak_tree_sitter::languages] cannot load grammar for language markdown.inline: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/markdown.inline.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/markdown.inline.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/markdown.inline.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/markdown.inline.so' (no such file)
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages] loading configuration for bibtex
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/bibtex.so
2023-08-28T08:37:08.680Z WARN  [kak_tree_sitter::languages] cannot load grammar for language bibtex: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/bibtex.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/bibtex.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/bibtex.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/bibtex.so' (no such file)
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages] loading configuration for css
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/css.so
2023-08-28T08:37:08.680Z WARN  [kak_tree_sitter::languages] cannot load grammar for language css: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/css.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/css.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/css.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/css.so' (no such file)
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages] loading configuration for bash
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/bash.so
2023-08-28T08:37:08.680Z WARN  [kak_tree_sitter::languages] cannot load grammar for language bash: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/bash.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/bash.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/bash.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/bash.so' (no such file)
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages] loading configuration for go
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/go.so
2023-08-28T08:37:08.680Z WARN  [kak_tree_sitter::languages] cannot load grammar for language go: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/go.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/go.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/go.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/go.so' (no such file)
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages] loading configuration for html
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/html.so
2023-08-28T08:37:08.680Z WARN  [kak_tree_sitter::languages] cannot load grammar for language html: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/html.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/html.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/html.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/html.so' (no such file)
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages] loading configuration for rust
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/rust.so
2023-08-28T08:37:08.680Z INFO  [kak_tree_sitter::languages]   queries directory: /Users/beno/Library/Application Support/kak-tree-sitter/queries/rust
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages] loading configuration for diff
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/diff.so
2023-08-28T08:37:08.798Z WARN  [kak_tree_sitter::languages] cannot load grammar for language diff: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/diff.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/diff.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/diff.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/diff.so' (no such file)
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages] loading configuration for markdown
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/markdown.so
2023-08-28T08:37:08.798Z WARN  [kak_tree_sitter::languages] cannot load grammar for language markdown: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/markdown.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/markdown.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/markdown.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/markdown.so' (no such file)
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages] loading configuration for toml
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/toml.so
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages]   queries directory: /Users/beno/Library/Application Support/kak-tree-sitter/queries/toml
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages] loading configuration for make
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/make.so
2023-08-28T08:37:08.798Z WARN  [kak_tree_sitter::languages] cannot load grammar for language make: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/make.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/make.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/make.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/make.so' (no such file)
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages] loading configuration for python
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::languages]   grammar path: /Users/beno/Library/Application Support/kak-tree-sitter/grammars/python.so
2023-08-28T08:37:08.798Z WARN  [kak_tree_sitter::languages] cannot load grammar for language python: dlopen(/Users/beno/Library/Application Support/kak-tree-sitter/grammars/python.so, 0x0005): tried: '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/python.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/beno/Library/Application Support/kak-tree-sitter/grammars/python.so' (no such file), '/Users/beno/Library/Application Support/kak-tree-sitter/grammars/python.so' (no such file)
2023-08-28T08:37:08.798Z TRACE [mio::poll] registering event source with poller: token=Token(1), interests=READABLE
2023-08-28T08:37:08.798Z TRACE [mio::poll] registering event source with poller: token=Token(2), interests=READABLE
2023-08-28T08:37:08.798Z INFO  [kak_tree_sitter::server] starting server
2023-08-28T08:37:08.798Z DEBUG [kak_tree_sitter::server] waiting on poll…
2023-08-28T08:37:12.056Z TRACE [kak_tree_sitter::server] mio event: Event {
    token: Token(
        2,
    ),
    readable: true,
    writable: false,
    error: false,
    read_closed: false,
    write_closed: false,
    priority: false,
    aio: false,
    lio: false,
    details: kevent {
        ident: 9,
        filter: EVFILT_READ,
        flags: EV_ADD|EV_CLEAR|EV_RECEIPT,
        fflags: (empty),
        data: 97,
        udata: 0x0000000000000002,
    },
}
2023-08-28T08:37:12.056Z DEBUG [kak_tree_sitter::server] waiting for command FIFO…
2023-08-28T08:37:12.056Z DEBUG [kak_tree_sitter::server] command FIFO read
2023-08-28T08:37:12.056Z INFO  [kak_tree_sitter::server] FIFO request: { "type": "highlight", "client": "client0", "buffer": "config.toml", "lang": "", "timestamp": 1 }
2023-08-28T08:37:12.056Z DEBUG [kak_tree_sitter::handler] highlight for session KakSession { session_name: "9052" }, buffer config.toml, lang , timestamp 1
2023-08-28T08:37:12.056Z DEBUG [kak_tree_sitter::handler] waiting for buffer FIFO…
2023-08-28T08:37:12.056Z DEBUG [kak_tree_sitter::handler] buffer FIFO read
2023-08-28T08:37:12.057Z DEBUG [kak_tree_sitter::server] waiting on poll…
2023-08-28T08:37:50.359Z TRACE [kak_tree_sitter::server] mio event: Event {
    token: Token(
        2,
    ),
    readable: true,
    writable: false,
    error: false,
    read_closed: false,
    write_closed: false,
    priority: false,
    aio: false,
    lio: false,
    details: kevent {
        ident: 9,
        filter: EVFILT_READ,
        flags: EV_ADD|EV_CLEAR|EV_RECEIPT,
        fflags: (empty),
        data: 101,
        udata: 0x0000000000000002,
    },
}
2023-08-28T08:37:50.359Z DEBUG [kak_tree_sitter::server] waiting for command FIFO…
2023-08-28T08:37:50.359Z DEBUG [kak_tree_sitter::server] command FIFO read
2023-08-28T08:37:50.359Z INFO  [kak_tree_sitter::server] FIFO request: { "type": "highlight", "client": "client0", "buffer": "config.toml", "lang": "toml", "timestamp": 1 }
2023-08-28T08:37:50.359Z DEBUG [kak_tree_sitter::handler] highlight for session KakSession { session_name: "9052" }, buffer config.toml, lang toml, timestamp 1
2023-08-28T08:37:50.359Z DEBUG [kak_tree_sitter::handler] waiting for buffer FIFO…
2023-08-28T08:37:50.360Z DEBUG [kak_tree_sitter::handler] buffer FIFO read
2023-08-28T08:37:50.370Z DEBUG [kak_tree_sitter::server] waiting on poll…

Support session-less server

The current code always assumes that the server is started from Kakoune, because we rely on the session number to know whether we want to quit the server.

We should simply check whether we were started with --kakoune, and if not, we can ignore the number of sessions, allowing us to keep the server running even if we don’t have any session anymore.

We should also add support for SIGINT signals so that we can gracefully shutdown the server.

After stopping the daemon with kak-tree-sitter-req-stop, Kakoune freezes if we try to highlight

          I noticed if I do `:kak-tree-sitter-req-stop` and then `:kak-tree-sitter-req-highlight-buffer` my Kakoune client freezes completely. Related?

Originally posted by @postsolar in #168 (comment)

Add --version

Should be fairly simple with clap 🙂

Remove %sh{} blocks for protocol-based requests

Currently, any request from Kakoune to KTS requires to go through the shell. We can remove that need by opening a FIFO when a new session is created on the KTS side, and stream requests with this.

Support the « all » language for ktsctl to handle everything at once

Add a command to reload grammars and queries from the filesystem

We don’t even need a notify thing nor even detect which grammar should be reloaded specifically. We can just completely throw away all the runtime objects, and reload everything from disk. It will be much easier and should allow to reload everything at once.

Highlighting breaks when first line is empty

Reproduction: Open kakoune such that it is highlighting a Rust buffer with kak-tree-sitter (fx. ...kak-tree-sitter/src/cli.rs). Verify that the file is being highlighted. Insert a newline at the beginning of the file. The highlighting will disappear or break in other ways. Remove the newline and highlighting returns.

This has been reproduced on Linux with an empty configuration save for a single eval %sh{kak-tree-sitter -dk}. It only seems to be an issue with for grammars.

Lazy commands

@topisani made an interesting commit recently allowing not to run a shell command if the buffer hasn’t changed. It uses a trick to find a function ending with -0, and we pass a timestamp as in place of the 0 so that it can use timestamps to know whether buffers have changed without any condition.

We should visit that idea to optimize.

Theming

We should support theming via KTS directly. The idea would be to define several Kakoune options that themes could replace / set.

Freeze on a specific JS file

KTS version: kak-tree-sitter 0.5.2-dev-7bb2a67

There's a specific JS file on which my Kakoune session freezes as soon as I make any movement in the buffer (e.g. press j, k, e, etc). Interacting with Kakoune in other ways (e.g. the prompt) does not cause freezes. Trying to clear dead sessions afterwards makes that command freeze, as well as trying to connect to frozen session. I have tried it with an empty config file and empty autoload folder as well (just the KTS startup command). Kakoune version is current master.

This file is my config for a GTK widgets lib and isn't suspicious in any way. It behaves fine with Helix and with tree-sitter-cli.

Attached is this file, with changed extension (Github doesn't let .js in).
config.txt

Please look if you can reproduce it, that's very weird indeed

ctrl-c on KTS should send a response to all connected sessions to disable KTS support

We especially need to remove hooks and kts_{buf,cmd}_fifo_path options must be set back to /dev/null to prevent blocking. This is a rather rare situation, but it’s annoying.

Figure out the UX for mappings

We currently run

define-command -hidden kak-tree-sitter-textobjects-enable %{
    map -docstring 'function (tree-sitter)'  buffer object f   '<a-;> kak-tree-sitter-req-textobjects function<ret>'
    map -docstring 'class (tree-sitter)'     buffer object t   '<a-;> kak-tree-sitter-req-textobjects class<ret>'
    map -docstring 'comment (tree-sitter)'   buffer object '#' '<a-;> kak-tree-sitter-req-textobjects comment<ret>'
    map -docstring 'parameter (tree-sitter)' buffer object 'v' '<a-;> kak-tree-sitter-req-textobjects parameter<ret>'
}

while LSP recommends these:

map global user l %{:enter-user-mode lsp<ret>} -docstring "LSP mode"
map global insert <tab> '<a-;>:try lsp-snippets-select-next-placeholders catch %{ execute-keys -with-hooks <lt>tab> }<ret>' -docstring 'Select next snippet placeholder'
map global object a '<a-semicolon>lsp-object<ret>' -docstring 'LSP any symbol'
map global object <a-a> '<a-semicolon>lsp-object<ret>' -docstring 'LSP any symbol'
map global object e '<a-semicolon>lsp-object Function Method<ret>' -docstring 'LSP function or method'
map global object k '<a-semicolon>lsp-object Class Interface Struct<ret>' -docstring 'LSP class interface or struct'
map global object d '<a-semicolon>lsp-diagnostic-object --include-warnings<ret>' -docstring 'LSP errors and warnings'
map global object D '<a-semicolon>lsp-diagnostic-object<ret>' -docstring 'LSP errors'

As far as I know, convention for Kakoune plugins is to provide commands and perhaps user-modes with mappings,
but not add mappings to existing Kakoune modes.
The only exception I can think of is the great kakoune-text-objects but that's kind of the point of that plugin.
LSP also adds some mappings but those are mostly historical.
LSP briefly added some object mappings but later stopped.
The mappings were carefully chosen to not collide with kakoune-text-objects.

I'm absolutely happy to break this convention and have 3rd party plugins add mappings.
But if we do that, we should design them well and make sure that the plugins work together.
I think it would be a disservice to users if plugin load order mattered because they override each other's mappings. We can do better.
One issue with supplying default mappings is that it becomes painful to change them, hence the need for good design.

I'm tempted to change the LSP recommended bindings of e/k to <a-f>/<a-t>, which is easier to remember and matches with tree-sitter.
Of course us taking f and t means that kakoune-text-objects is left behind, unless we figure something out.. I'm sure can, if needed (cc @Delapouite).

map global object a 'lsp-object' -docstring 'LSP any symbol'

I wonder if tree sitter has an equivalent for this? As in, select any syntax node? Probably not very useful since tree-sitter gives many more tokens.

The goals of this issue are:

Figure out whether we should provide default mappings, or whether to add them to the README (or wiki I guess)
Whichever way we go, make sure the LSP and tree-sitter are consistent
For the mappings that both tree-sitter and LSP provide:
1. since tree-sitter is more reliable for this (IME), make it take precedence automatically if both are installed
2. perhaps we should even deprecate LSP here (if it provides no extra functionality)

I currently prefer simply making LSP match tree-sitter with <a-f> and such (because today's inconsistency is super confusing),
then there is also no overlap.

BTW Thanks for the object mode mappings; sorry for the long issue, I mainly don't want to let LSP bitrot here.

ktsctl cannot install parser & grammar

I installed ktsctl and kak-tree-sitter via cargo.I have copy config file to .config/kak-tree-sitter

when i try to install tree-sitter parser i get error: configuration error: cannot read configuration: No such file or directory (os error 2)
I think it tell me there's something wrong in my config file but I don't know what's wrong here.

my os version is: OS: macOS 13.4.1 22F82 x86_64

Semantics objects

Semantics objects will be used to return selections, so that we can map a command to something like mif to « match inside functions », or pretty much any kind of query users would like to.

That must be applied to every Kakoune selections.

Sometimes, highlighting doesn’t happen and some weird hooks are required to trigger them

I’m not sure about this one, so hard to reproduce. It happens at work from time to time, especially after hibernating / sleep. KTS either doesn’t send the responses fast enough, or hooks are slow to ask KTS to highlight, or maybe a mix between both.

I do not see any potential bottlenecks, so I need to investigate a bit more.

ktsctl: Allow multiple compile commands?

In contrast to most other grammars, tree-sitter-python contains both a C and C++ file, which must be compiled with cc and c++ respectively, then linked with c++.

It's likely possible to hack a solution using sh to run both compile commands from ktsctl, but it's probably preferable to properly support separate compilation of the scanner and parser.

Bundle tree-sitter grammars and queries into the repository

First of all thank you for contribution of your work to the open source.

I am fairly new in Kakoune and do not know much about Tree Sitter yet,
but I noticed that comparing to Helix, it is a bit cumbersome to
use ktsctl in order to clone, compile and install tree sitter grammars and queries.

I think it would cool to bundle all grammars in the same way as Helix does (maybe we could can we even borrow their work on that).

So, after installing the kak-tree-sitter it would ready to go immediately.

Also, it would be cool, I think, to avoid cloning Helix multiple times, but do it only once.

The downside is depending on Helix and their list of supported languages,
although, we do not have to use Helix, the main idea is to precompile and bundle all interesting language into the repository.

Any thoughts?

[RFC] Possible merge with kak-lsp to provide a better out-of-the-box experience

(posting to notify people who subscribe here)
please see kakoune-lsp/kakoune-lsp#712 for some thoughts on the future of kak-tree-sitter/kak-lsp

Update the wiki to remove the reference to the milestone

Once the milestone is completed, we must remove that reference; here.

Server claims language isnt supported, when grammar is installed.

# nim
[language.nim.grammar]
url = "https://github.com/alaviss/tree-sitter-nim/"
pin = "ecbd15e1e573bd3c6b3f8098df4a1917be2824ef"
path = "src"
compile = "cc"
compile_args = ["-c", "-fpic", "../scanner.cc", "../parser.c", "-I", ".."]
compile_flags = ["-O3"]
link = "cc"
link_args = ["-shared", "-fpic", "scanner.o", "parser.o", "-o", "nim.so"]
link_flags = ["-O3"]

[language.nim.queries]
url = "https://github.com/helix-editor/helix"
pin = "dbd248fdfa680373d94fbc10094a160aafa0f7a7"
path = "runtime/queries/nim"

adding this definition to config file and succesfully installing it with ktsctl still results in server respoding with Unsupported language: nim.

Support indent guide lines

I think it should be fairly simple to have indent guide lines with tree-sitter.

Support file extension-based highlighting

Currently, highlighting request are made by sending the content of %opt{filetype}. This works for most cases, but not all of them. For instance, sh vs. bash.

It would be great to also support file extensions. The big question is: if we use file extension, do we want to always use it? It doesn’t seem unlikely that a user would like to override the content a buffer to highlight it differently.

Update the code sending commands back to Kakoune to directly connect to the UNIX socket

The initial implementation I tried many commits ago tried to use the socket path directly, but I was constantly failing. The reason was that I was missing the protocol, which, back then, was not documented / hard to find.

Looks like it’s now doable. Context here.

Update the wiki for v0.4

Design Suggestions

Hey, this is a really nice project. I have a few suggestions that I believe would make it more user-friendly.

Map highlighting queries to Kakoune faces instead of options. Essentially, in the kak-tree-sitter config, instead of highlight.groups just being a list of queries you're enabling, you'd have a table matching queries to Kakoune faces, for example matching comment.line, etc to the comment face. This would keep you from having to define an option for each query you want to support, and you'd get #78 for free since you're using the faces colorschemes already use for syntax highlighting. tree-sitter.kak does this, I believe.
Grab query files from the tree sitter grammar repos, or only clone Helix once. As is, ktsctl needs to re-clone Helix (a rather large repository) for each grammar that the user installs. This is a waste of disk space, and it would be better if either only one Helix copy was needed (basically ktsctl would only clone it if it hasn't already), or we could just use the query files that come with the grammars.
Compile grammars without forcing the user to define the compile args. ktsctl should support compiling grammars without the user needing to tell it how. py-tree-sitter (the Python bindings for tree-sitter) defines a method for building the library file by just passing it the output path and the repo path - it handles finding the source files and linking it correctly. I'd imagine something similar could be done in ktsctl (rust's tree-sitter binding doesn't have such a function, unfortunately)
Don't remove the default highlighting by default. Sadly, some grammars, such as the Nim grammar, aren't comprehensive, and this combined with the fact that the user may not have all the necessary queries defined in config.toml means that some holes may be left in kak-tree-sitter's highlighting. It would be nice to give the user the option to keep the default syntax highlighting as a fallback in cases like these.

You don't need to implement all, or really any, of these if they don't match your vision of the project. I have my own (very WIP) implementation here, but I'd like to be able to use yours since Rust is obviously more performant than Python.

Do not remove the default highlighting by default (make it a per-lang option)

This comment explains the rationale.

Helix is to change queries precedence order

See helix-editor/helix#9436

This doesn't affect existing KTS users because queries' versions are pinned, but any updates will require to adapt the ordering as well, and reverse the existing in-house queries.

Highlights break and render escape sequences on Unicode symbols

No KTS:

KTS:

Sorry if it's not related to kak-tree-sitter, I'm new to this editor.

Document highlight queries

A brief few words on which each one is supposed to mean wouldn't hurt. For example what's the difference between keyword.control.import and include is not obvious. If documentation already exists elsewhere, the comment in config.toml should link to it.

More intelligent auto-enable

Currently, opening a file with an unsupported filetype, then changing the filetype to a supported one will not enable highlighting.

In a similar vein, try opening a supported file, then opening an unsupported buffer from the first client (:new edit foo). The client of the second buffer will repeatedly attempt to highlight the buffer with a constant "unsupported language" message.

It would be nice if highlighting was automatically enabled or disabled for every buffer whenever its filetype is set/modified.

ktsctl should check when to clone a git repo and if not needed, use git checkout instead

This comment explains the rationale.

Allow to disable highlighting

Use deterministic FIFO locations (fixes unicode decode problems)

Context

Sometimes, KTS crashes on bad unicode decoding (I’m not sure why yet). In such a tragic situation, every sessions (every one of them) connected to KTS gets frozen, because they all communicate with the server via FIFOs.

Solution

First, we can have a mitigation by not sending the FIFO paths to the Kakoune session anymore, and use a deterministic one. That would allow to reuse the same one and, if KTS breaks, starts a new server and « go on with what was going on before. » That is a pretty important fix to me, as it would allow to unfreeze every session.

Secondly, we could ensure that if the server breaks, it automatically restarts (and ensure that if it broke in the past x seconds, then it’s probably going to go full CLB, and we should just let it die and burn in hell).

Finally, we need to understand what the heck is going on with this unicode horror. I haven’t received a single report about the problem, but I face it at least once a day at work, especially on big buffers, and that’s annoying me a lot.

Which resources should be deleted when KTS shuts down?

Currently, we have log files that shouldn’t be removed (as they are valuable assets to understand a crash). We need to think about whether we even want to clean resources up since they are created in disposable directories (XDG runtime dirs and tmpfs / TMPDIR for platforms without XDG support).

[suggestions] Renaming repo name

Hi
I think the tree-sitter modules should be following the same naming, so could you change it to:

tree-sitter-kak

UPDATE 1

I believe that the naming convention for the tree-sitter parser should be consistent, and therefore, could you please change the name of our repository in GitHub to 'tree-sitter-kak'?

PID detection doesn't allways work

We often end up with many KTS running. This is mostly due to the cleanup option I think. We should probably check whether kak-tree-sitter is in the return of ps in addition to checking the PID file value.