Support `locals` for highlighting about nvim-treesitter HOT 35 OPEN

Semshi author here. Sorry for chiming in late, I didn't work on the plugin for a while.

Does Tree-sitter integration replace Semshi?

No, at least not yet. There is often a fundamental misunderstanding what Semshi does. It's not primarily a "native highlighter". The whole point of Semshi is to base its semantic highlighting on static name resolution. With that, it can indicate whether a particular name is imported, defined globally, possibly undefined, etc. Tree-sitter provides a fantastic incremental parsing engine, but it doesn't automatically replace Semshi's semantic analysis. E.g. names may need to be resolved across multiple scopes...

...and statements like global or nonlocal should affect the name resolution accordingly:

Can Semshi's features be implemented in a way that uses Tree-sitter?

Yes, and they should indeed. Using TS instead of Python's ast module would be much faster and less redundant. However I'm unsure to what degree this can be done with just the grammar and TS queries. There is of course support for tracking locals in TS and the like, but no idea yet if this can be used to completely mimic Python's scoping logic. I'm happy to look into it :)

@kyazdani42

unused arguments would work better with a lsp or a linter.

It's true that the LSP can/should provide that. But if you want that information as you type, as part of the instant code highlighting, LSP and linters will be too slow and not in sync with the main syntax highlighting.

treesitter highlighting [...] makes way more sense when you look at some code than semshi.

That's missing the point a little. As explained above, Semshi doesn't highlight all the different syntax elements, it's meant to highlight names based on scope and context. I hear you if you say you find that distracting - but for me, it helps a great deal to see at a glance where a name comes from or whether I forgot an import.

from nvim-treesitter.

I ported all the completion-treesitter features (that I found) over in #110. Let me know if there is anything that may be missing. Doesn't cover the import scenario though.

from nvim-treesitter.

I guess we almost achieved this goal. A question is how to highlight imported names. When I use the highlights of semshi, nvim-treesitter looks almost the same. It can also highlight the imported names.

from nvim-treesitter.

@Congee treesitter highlighting, when properly configured (not very hard to configure, specially with nvim-treesitter), makes way more sense when you look at some code than semshi. Too much random highlighting can be really distracting, and treesitter highlights in a syntax aware way. Jumping between languages, highlighting stays the same for each syntax concept.

from nvim-treesitter.

@kevinhwang91 we need to add decoration priority in neovim core for that, I hope to look into it soon : )

from nvim-treesitter.

unused arguments would work better with a lsp or a linter.

from nvim-treesitter.

I think this can be done now. But only as opt-in. We do not have to feature this as a pinned issue.

I still need to transition all the API and deprecate old usages of iter_prepared_captures. This is better to do in small steps.

from nvim-treesitter.

What I miss most from Semshi is a way to highlight unused arguments.

from nvim-treesitter.

For Python, I very much like how Neovim treesitter currently styles Python, seems just about ideal to me (using my nightfly theme which is treesitter aware):

These are the Semshi without and with screenshots:

Treesitter looks the best to me. I definitely do not want arguments highlighted in color.

Typos and unresolved names are best highlighted by a linter in my opinion (e.g. ALE or Neovim LSP for example), ideally using virtual text as seen here:

My 2cents.

from nvim-treesitter.

The discussion reads like semantic (or at least scope aware) highlighting should already work. However, I was not able to find any plugin that uses nvim-treesitter to implement something akin to Semshi's semantic syntax highlighting. Is this a mere coincidence and everybody is still using Semshi or am I just unaware of how to enable this feature?

check this out: neovim/neovim#15723

from nvim-treesitter.

@numirias thanks for chiming in. To give you a short update on the state of implementation: we already have a scope analysis and it's already working. We need to make our current implementation faster (nobody has found the time yet to implement it). Essentially, using the nvim queries directly and building up a clever cache based on hash maps instead of looking up in lists. In predicates.lua you can find the is? predicate that can restricte the highlighting on the type of symbol at their declaration (e.g. (is? @namespace "import")).

So we had semantic highlighting for a very short time: #296 . The highlight definition feature is in nvim-treesitter/nvim-treesitter-refactor

A difference betwee semshi and tree-sitter as in-process plugin is that we have very strict deadlines in order to not block the user during typing.

If you have any hints for a possible implementation or if you even want to build a MVP please reach out to us!

from nvim-treesitter.

Just dropping a new note here for others that land here perhaps:

The "highlight semantic matches" treesitter functionality has been moved into a separate repository in this same org (as of this comment). It can be found in this repo: https://github.com/nvim-treesitter/nvim-treesitter-refactor

To enable it you'd need to install this repo and that repo and add the refactor = { highlight_definitions = { enable = true } } bit to your treesitter config setup call.

Of course, I'm still not really following the work going on in the neovim solution itself around semantic token highlighting. That likely depends on a functioning Python LSP server however. Treesitter does not.

from nvim-treesitter.

For the last two points, you have a good point noting that completion-treesitter has this, I will port that to here.

from nvim-treesitter.

Then, should this feature be implemented in completion-treesitter?

from nvim-treesitter.

First the first point, it might just be a matter of having a new query and supporting some kind of #is? @mything include.

from nvim-treesitter.

Then, should this feature be implemented in completion-treesitter?

It shouldn't as it is not related to completion.

from nvim-treesitter.

I think that the import scenario will be added when we will use the is? predicate along with #114

from nvim-treesitter.

That would be quite difficult to implement I guess. One thing though is that, using find_usages you can have something generic to highlight unused symbols.

from nvim-treesitter.

https://github.com/jackguo380/vim-lsp-cxx-highlight is amazing for cpp users

from nvim-treesitter.

But this uses LSP, right ?

from nvim-treesitter.

But this uses LSP, right ?

yup

from nvim-treesitter.

using my nightfly theme which is treesitter aware

ummm, semshi recognizes my colorscheme onedark. And, semshi looks far better than treesitter with richer kinds of identifiers and color palette, in my humble opinion. 👀

from nvim-treesitter.

@Congee tree sitter is just a parser. Defining the colors you want is completely up to you.

from nvim-treesitter.

But this uses LSP, right ?

I think treesitter is the competitor for a native syntax highlight, but not for LSP. For now, semantic highlighting is more accurate than treesitter, I prefer to use treesitter as a fallback for c rather than main use.

How can I lower the priority of treesitter to make semantic highlighting override it?

from nvim-treesitter.

IMO being able to highlight based on what kind of definition is being referenced would be more generally useful than just highlighting imported symbols.

from nvim-treesitter.

@theHamsta Ah that's great to hear! Yes, I'm interested in helping out :)

When you say "we already have a scope analysis and it's already working" and "we had semantic highlighting for a very short time", I'm still not entirely sure, though - do you mean the general language-agnostic support for scope-based highlighting, or an implementation that could already do correct name resolution as in e.g. my examples from above? (I'd be happy to chat about this on the Gitter/Zulip channels)

from nvim-treesitter.

it worked but was too slow i believe https://nvim-treesitter.zulipchat.com/#narrow/stream/252271-general/topic/General/near/215188572

from nvim-treesitter.

• do you mean the general language-agnostic support for scope-based highlighting, or an implementation that could already do correct name resolution as in e.g. my examples from above?

Yes, it was mostly correct. The scopes and definitions are defined by the locals.scm queries for each language.

I created a channel on zulip: https://nvim-treesitter.zulipchat.com/#narrow/stream/252271-general/topic/Semantic.20Highlighting.20Brainstorming

from nvim-treesitter.

@theHamsta do you think this issue is still worth being open ? With the improvements of the locals you did, this is mostly solved now, isn't it ?

from nvim-treesitter.

The discussion reads like semantic (or at least scope aware) highlighting should already work. However, I was not able to find any plugin that uses nvim-treesitter to implement something akin to Semshi's semantic syntax highlighting. Is this a mere coincidence and everybody is still using Semshi or am I just unaware of how to enable this feature?

from nvim-treesitter.

semantic tokens is now supported in core. Is this something we can drop now?

from nvim-treesitter.

@lewis6991 Usually tree-sitter implementations are expected to support locals https://tree-sitter.github.io/tree-sitter/syntax-highlighting#local-variables . locals.lua is just implemented using slow definition look-up and slow legacy querying API.

from nvim-treesitter.

legacy querying API

I thought there was only one query API? What new API should it be using?

from nvim-treesitter.

Also, I should point out that treesitter highlighting has now been moved to core, so any improvement (apart from queries) needs to happen there (and take into account the presence of LSP semantic highlighting).

So I think the issue here should be closed, and a new, more focused, issue opened in Neovim core.

from nvim-treesitter.

I appreciate folks keeping this conversation alive, but I will admit I tried to read along for a bit and wasn't able to follow. I realize this is a lot of fast moving parts though.

Clason mentioned that "treesitter highlighting has now been moved to core". I don't see anything about that in NeoVim 0.8.x release notes, and I don't see anything about that in the root readme of this repo. So I'm guessing that's a near future thing? Maybe a NeoVim 0.9.x thing? And this plugin would still get updated to not do highlighting but to still provide easy language parser access?

And as for this original issue, as mentioned above, the best semshi-like experience I've found that is powered by TreeSitter is over in https://github.com/nvim-treesitter/nvim-treesitter-refactor now. I personally think it would be fair to close this issue, but perhaps a note in the root readme to suggest users also checkout the companion plugins under the nvim-treesitter org on github?

from nvim-treesitter.