neovim / tree-sitter-vimdoc Goto Github PK

Tree-sitter parser for Vim help files

License: Apache License 2.0

JavaScript 86.17% Scheme 13.83%

tree-sitter-vimdoc's Introduction

tree-sitter-vimdoc

This grammar implements the vimdoc "spec" (ref1, ref2). Predictable results are the primary goal, so that output formats (e.g. HTML) are well-formed; the input (vimdoc) is secondary. The first step should always be to try to fix the input rather than insist on a grammar that handles vimdoc's endless quirks.

Overview

block is the main top-level node, delimited by blank line(s) or any line starting with < (codeblock terminator).
- contains line and line_li nodes.
line:
- contains atoms (words, tags, taglinks, …)
- contains headings (h1, h2, h3, column_heading) because codeblock terminated by "implicit stop" (no terminating <) consumes blank lines, so block has no way to end.
line_li ("listitem")
- lines starting with -/• (not +/*) are listitems.
- consumes lines until blank line, codeblock, or next listitem.
- nesting is ignored: indented listitems are parsed as siblings.
codeblock:
- contained by line or line_li, because ">" can start a codeblock at the end of any line.
- contains line nodes without word nodes: it's just the full raw text line including whitespace. This is somewhat dictated by its "preformatted" nature; parsing the contents would require loading a "child" language (injection). See #2.
- the terminating < (and any following whitespace) is discarded (anonymous).
h1 = "Heading 1": ====== followed by text and optional *tags*.
h2 = "Heading 2": ------ followed by text and optional *tags*.
h3 = "Heading 3": UPPERCASE WORDS, followed by optional *tags*, followed by atoms.

Known issues

Input must end with newline/EOL (\n). Grammar does not support files without EOL.
Input must end with a blank line. Though this doesn't seem to matter in practice.
Spec requires that codeblock delimiter ">" must be preceded by a space (" >"), not a tab. But currently the grammar doesn't enforce this. Example: :help lcs-tab.
url doesn't handle surrounding parens. E.g. (https://example.com/#yay) yields word
url doesn't handle nested parens. E.g. (https://example.com/(foo)#yay)
column_heading currently only recognizes tilde ~ preceded by space (i.e. foo ~ not foo~). This covers 99% of :help files.
column_heading children should be plaintext, but currently are parsed as $._atom.
modeline must be preceded by a blank line.

TODO

tag_heading : line(s) containing only tags, typically implies a "heading" before a block.

Release

Steps to perform a release:

Bump and tag the version:

First bump Cargo.toml, pyproject.toml, and Makefile to the new version. Then bump the package:
```
npm version patch -m "release %s"
```
Choose patch/minor/major to indicate query compatibility:
- patch for bugfixes (no changes to queries needed)
- minor for added nodes (queries may need changes to use new nodes but will not error)
- major for removed or renamed nodes (queries will error if not adapted), other breaking changes

Bump to prerelease, without creating a tag:

npm version --no-git-tag-version prerelease --preid dev
git add package*.json Cargo.toml pyproject.toml Makefile
git commit -m bump

Push:
```
git push && git push --tags
```
Release the tagged commit: https://github.com/neovim/tree-sitter-vimdoc/releases/new

tree-sitter-vimdoc's People

Contributors

Stargazers

Watchers

Forkers

otherjl0 justinmk alemuller m4kxjxcv amaanq taekwombo mohsenghahremani dundargoc sffej observeroftime novusnota-forks melo0806a

tree-sitter-vimdoc's Issues

parse error for indent text block

The display here is not right compared to builtin help file.
Maybe the text block should be rendered as code blocks.

issue with angle-brackets in change.txt

reproduce: :help <
How it's actually displayed:

SHIFTING LINES LEFT OR RIGHT                            shift-left-right

                                                        <
{motion}                Shift {motion} lines one 'shiftwidth' leftwards.

                        If the 'shiftwidth' option is set to zero, the amount
                        of indent is calculated at the first non-blank
                        character in the line.
                                                        <<
<                       Shift [count] lines one 'shiftwidth' leftwards.

                                                        v_<
{Visual}[count]<        Shift the highlighted lines [count] 'shiftwidth'

I hope this is the right place to report this...

parse error: options.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/options.html

Context:

	   "number"	display signs in the 'number' column. If the number
			column is not present, then behaves like "auto".

	Note regarding 'orphaned signs': with signcolumn numbers higher than
	1, deleting lines will also remove the associated signs automatically,
	in contrast to the default Vim behavior of keeping and grouping them.
	This is done in order for the signcolumn appearance not appear weird

Umm. What just happened? Where am I?

missed codeblock

In this example from eval.txt, the middle codeblock is not recognized:

Note that in the command: >
	:if "foo"
	:" NOT executed
"foo" is converted to 0, which means FALSE.  If the string starts with a
non-zero number it means TRUE: >
	:if "8foo"
	:" executed
To test for a non-empty string, use empty(): >
	:if !empty("foo")
<

parse error: motion.txt , spurious listitem

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/motion.html

Context:

-  <minus>		[count] lines upward, on the first non-blank
			character |linewise|.

+		or					*+*
CTRL-M		or					*CTRL-M* *<CR>*
<CR>			[count] lines downward, on the first non-blank
			character |linewise|.

parse error: options.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/options.html

Context:

		    Value must be 50 or less.
	maxwid	    Maximum width of the item.  Truncation occurs with a '<'
		    on the left for text items.  Numeric items will be
		    shifted down to maxwid-2 digits followed by '>'number
		    where number is the amount of missing digits, much like
		    an exponential notation.
	item	    A one letter code as described below.

parser gets confused by single bar (|)

Example from autocmd.txt:. The '|' seems to confuse the parser, and changing it to "|" doesn't help. Likely | should be treated as a literal if any whitespace follows it before the closing |.

The special pattern <buffer> or <buffer=N> defines a buffer-local autocommand.
See |autocmd-buflocal|.

Note: The ":autocmd" command can only be followed by another command when the
'|' appears where the pattern is expected.  This works: >
	:augroup mine | au! BufRead | augroup END
But this sees "augroup" as part of the defined command: >
	:augroup mine | au! BufRead * | augroup END
	:augroup mine | au BufRead * set tw=70 | augroup END
Instead you can put the group name into the command: >
	:au! mine BufRead *
	:au mine BufRead * set tw=70
Or use `:execute`: >
	:augroup mine | exe "au! BufRead *" | augroup END
	:augroup mine | exe "au BufRead * set tw=70" | augroup END

parse error: two lines merged as one line?

The rendered html here is not right. The group as an atom and nothing: separates alternatives line are merged as a single long line.

parse error: windows.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/windows.html

Context:

		  a	an active buffer: it is loaded and visible
		  h	a hidden buffer: It is loaded, but currently not
			   displayed in a window |hidden-buffer|
		   -	a buffer with 'modifiable' off
		   =	a readonly buffer
		   R	a terminal buffer with a running job
		   F	a terminal buffer with a finished job

two lines are combined as one: options.txt

Context:

If you want to include a ':' in a set command precede it with a '\'.  The
backslash in front of the ':' will be removed.  Example:

The code block below are combined with the line following it (This sets the 'fillchars' option).

Also in context for option filetype

When a dot appears in the value then this separates two filetype
	names.  Example:

Same issue.

more syntax features: keycodes, special, parameters

In testing this parser for nvim-treesitter (nvim-treesitter/nvim-treesitter#2679), I noticed that some features are not yet supported, especially

Parameters as in {pattern}
Keycodes and chords as in <c-o> and CTRL-O
Command modifiers as in [range] (that's just a bunch of literals, and so could simply be listed as anonymous nodes in the highlight query)
optional arguments #109
modelines? #125

(See https://github.com/nanotee/vimdoc-notes for the most comprehensive specification of vimdoc I've seen so far.)

parse error, single star followed by a number

In https://neovim.io/doc/user/intro.html#mode-switching, the mode-switching table is messed up due to several *1 like item in the table.

Probably should not be parsed as list item, but should be footnote like.

embedded languages in codeblocks

This is a moonshot, but would it be possible (at least in principle) to detect the language of a code_block? In contrast to Markdown, Vimdoc doesn't use language markers, so it's nigh impossible to set up proper injections for them. At least being able to distinguish Lua/Vimscript/Other (the last highlighted as plain string) would be great.

tilde "~" headings are not recognized

The Attributes: ~ subheading in the sample text below is parsed as a normal "line"

line [9, 4] - [10, 0]
  word [9, 4] - [9, 15]
  word [9, 16] - [9, 17]

sample text:

nvim_ui_try_resize({width}, {height})                   *nvim_ui_try_resize()*
    TODO: Documentation

    Attributes: ~
        |RPC| only

Maybe there should be h1, h2, h3 headings?

h1: ======== headings
h2: ---------- headings
h3: Foo ~ headings

parse error: cmdline.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/cmdline.html

Context:

					*c_CTRL-R_CTRL-R* *c_<C-R>_<C-R>*
					*c_CTRL-R_CTRL-O* *c_<C-R>_<C-O>*
CTRL-R CTRL-R {register CTRL-F CTRL-P CTRL-W CTRL-A CTRL-L}
CTRL-R CTRL-O {register CTRL-F CTRL-P CTRL-W CTRL-A CTRL-L}
		Insert register or object under the cursor.  Works like
		|c_CTRL-R| but inserts the text literally.  For example, if

parse error: autocmd.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/autocmd.html

Context:

	~	matches a '~'
	,	separates patterns
	\,	matches a ','
	{ }	like \( \) in a |pattern|
	,	inside { }: like \| in a |pattern|
	\}	literal }
	\{	literal {

Tilde not concealed in column headers

E.g., for :h api-types, the first table has a column heading

Basic types ~

which corresponds to the playground

block [94, 0] - [107, 0]
  line [94, 0] - [95, 0]
    column_heading [94, 0] - [95, 0]
      name: word [94, 0] - [94, 5]
      name: word [94, 6] - [94, 11]

Since ~ does not appear as an (anonymous) node, it cannot be captured and thus concealed as in the legacy syntax.

This is the same issue (with the same likely fix) as in #33.

parse error: pi_spec.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/pi_spec.html

Context:

2. Customizing					*spec-customizing*

The format string				*spec_chglog_format*
-----------------

You can easily customize how your spec file entry will look like.  To do
this just set the variable "spec_chglog_format" in your vimrc file like
this: >

paragraphs (and nested paragraphs?)

Preamble from #12 :

In order for :help to render as something other than fixed-width, hard-wrapped text, (e.g. for HTML format) the parser needs to give some basic hints about structure besides "line". Sometimes (often) the :help source needs to be fixed (because it can't be reasonably parsed), but in other common cases it's probably reasonable for tree-sitter-vimdoc to provide some basic idea of:

paragraph (and "nested" paragraphs? AKA indentation) [this issue]
list items (which may be nested) #12

Example of paragraphs and nested (indented) paragraphs

https://github.com/neovim/neovim/blob/52e4b779e3df1335efc4774e8dfc199d07e24bfd/runtime/doc/eval.txt#L51-L74

By "nested paragraphs" I simply mean blocks of text that have common indentation. In the following example,

we can't parse the Number 123... block but if we can at least recognize that it is indented, then the HTML result can wrap that block in <pre> so that it is readable.
the *octal* line doesn't share indentation so it should be parsed as a normal "line", possibly part of the next paragraph.
it may be reasonable for us to require blank lines between these blocks.

The Number and String types are converted automatically, depending on how they
are used.

Conversion from a Number to a String is by making the ASCII representation of
the Number.  Examples:
	Number 123	-->	String "123" ~
	Number 0	-->	String "0" ~
	Number -1	-->	String "-1" ~
							*octal*
Conversion from a String to a Number is done by converting the first digits to
a number.  Hexadecimal "0xf9", Octal "017" or "0o17", and Binary "0b10"
numbers are recognized.  If the String doesn't start with digits, the result
is zero. Examples:
	String "456"	-->	Number 456 ~
	String "6bar"	-->	Number 6 ~
	String "foo"	-->	Number 0 ~
	String "0xf1"	-->	Number 241 ~
	String "0100"	-->	Number 64 ~
	String "0o100"	-->	Number 64 ~
	String "0b101"	-->	Number 5 ~
	String "-8"	-->	Number -8 ~
	String "+8"	-->	Number 0 ~

To force conversion from String to Number, add zero to it: >
	:echo "0100" + 0
<	64 ~

To avoid a leading zero to cause octal conversion, or for using a different
base, use |str2nr()|.

Hot-links are not recognized inside parenthesis

Text:

(|aaa|) |bbb| ccc |ddd|

Current behavior with tree-sitter:

Behavior without tree-sitter:

list items (and nested list items)

In order for :help to render as something other than fixed-width, hard-wrapped text, (e.g. for HTML format) the parser needs to give some basic hints about structure besides "line". Sometimes (often) the :help source needs to be fixed (because it can't be reasonably parsed), but in other common cases it's probably reasonable for tree-sitter-vimdoc to provide some basic idea of:

paragraph (and "nested" paragraphs? AKA indentation) #13
list items (which may be nested) [this issue]

Example of a basic list

https://github.com/neovim/neovim/blob/52e4b779e3df1335efc4774e8dfc199d07e24bfd/runtime/doc/api.txt#L25-L34

Many clients use the API: user interfaces (GUIs), remote plugins, scripts like
"nvr" (https://github.com/mhinz/neovim-remote).  Even Nvim itself can control
other Nvim instances.  API clients can:

  - Call any API function
  - Listen for events
  - Receive remote calls from Nvim

The RPC API is like a more powerful version of Vim's "clientserver" feature.

Example of a nested list

api.txt has many nested lists. Maybe it's not worth supporting these?

https://github.com/neovim/neovim/blob/52e4b779e3df1335efc4774e8dfc199d07e24bfd/runtime/doc/api.txt#L3257-L3273

                 • callback (function|string) optional: if a string, the name
                   of a Vimscript function to call when this autocommand is
                   triggered. Otherwise, a Lua function which is called when
                   this autocommand is triggered. Cannot be used with
                   {command}. Lua callbacks can return true to delete the
                   autocommand; in addition, they accept a single table
                   argument with the following keys:
                   • id: (number) the autocommand id
                   • event: (string) the name of the event that triggered the
                     autocommand |autocmd-events|
                   • group: (number|nil) the autocommand group id, if it
                     exists
                   • match: (string) the expanded value of |<amatch>|
                   • buf: (number) the expanded value of |<abuf>|
                   • file: (string) the expanded value of |<afile>|
                   • data: (any) arbitrary data passed to
                     |nvim_exec_autocmds()|

Conceal eats adjacent whitespace

With the new grammar and queries, open any help file (say, :h api.txt) and look at option or tag links: the concealed markers will include all adjacent whitespace. E.g.,

API Usage                                               *api-rpc* *RPC* *rpc*

becomes

API Usageapi-rpcRPCrpc

instead of

API Usage                                               api-rpc RPC rpc

(This was not the case with the old parser version.)

Odd highlighting for `local-additions`

Some odd highlighting when looking at help.txt's local-additions section

w/o TreeSitter	w/ TreeSitter

text:

LOCAL ADDITIONS:				*local-additions*
|nvim-treesitter|  Treesitter configurations and abstraction layer for Neovim.
|packer.txt|                      A use-package inspired Neovim plugin manager
|matchit.txt|	Extended "%" matching

notice the h in matchit.txt is concealed

parse error: motion.txt, invalid option: 'a - 'z

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/motion.html

Context:

remembered.  Do not confuse marks with named registers, they are totally
unrelated.

'a - 'z		lowercase marks, valid within one file
'A - 'Z		uppercase marks, also called file marks, valid between files
'0 - '9		numbered marks, set from .shada file

The single quote followed by a digit or character (for those register types) is not handled correctly.

single apostrophe (') incorrectly parsed as option link

Sample text (from api.txt):

Example:

Let's set an extmark at the first row (row=0) and third column (column=2).
|api-indexing| Passing id=0 creates a new mark and returns the id: >

      01 2345678
    0 ex|ample..
<        ^ extmark position
>
    let g:mark_ns = nvim_create_namespace('myplugin')
    let g:mark_id = nvim_buf_set_extmark(0, g:mark_ns, 0, 2, {})
<
We can get the mark by its id: >

The ' char in Let's confuses the parser, and lots of errors follow.

parse error: windows.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/windows.html

Context:


Closing a window
----------------

:q[uit]
:{count}q[uit]						*:count_quit*
CTRL-W q						*CTRL-W_q*

multiple keycodes must be separated by whitespace

Found in https://neovim.io/doc/user/options.html#'wildmenu'

CTRL-N isn't highlighted, but it should be.

Context:

Keys that show the previous/next match, such as <Tab> or
CTRL-P/CTRL-N, cause the highlight to move to the appropriate match.
['wildmode'](https://neovim.io/doc/user/options.html#'wildmode') must specify "full": "longest" and "list" do not start
['wildmenu'](https://neovim.io/doc/user/options.html#'wildmenu') mode. You can check the current mode with [wildmenumode()](https://neovim.io/doc/user/builtin.html#wildmenumode()).

parse error: pi_spec.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/pi_spec.html

Context:

2. Customizing					*spec-customizing*

The format string				*spec_chglog_format*
-----------------

You can easily customize how your spec file entry will look like.  To do
this just set the variable "spec_chglog_format" in your vimrc file like
this: >

Tags with a dot "." beforehand isn't handled correctly

The following doesn't conceal "|": vim.api.|nvim_get_keymap()|

parse error: options.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/options.html

Context:

	to check what is following.  E.g. for Javascript, if a function is
	defined with "func_name = function(args)": >
		^\s*\ze\i\+\s*=\s*function(
<	If the function is defined with "func_name : function() {...": >
	        ^\s*\ze\i\+\s*[:]\s*(*function\s*(
<	When using the ":set" command, you need to double the backslashes!
	To avoid that use `:let` with a single quote string: >

The 3rd code block is not rendered correctly as code block.

Allow options as tags

Currently, the parser only allows a simple word as tag, as in *foo*; option like *'sw'* and argument like {subject} (possibly the only instance) are a parsing error.

parse error: options.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/options.html

Context:

	      Note that there is no '%' before the closing '}'.  The
	      expression cannot contain a '}' character, call a function to
	      work around that.  See |stl-%{| below.
	{% -  This is almost same as { except the result of the expression is
	      re-evaluated as a statusline format string.  Thus if the
	      return value of expr contains % items they will get expanded.
	      The expression can contain the } character, the end of

single-char 'x' is not an option hotlink

Noticed this while working on gen_help_html.lua neovim/neovim#11967 .

This is just for reference; as mentioned in #11 (comment) I'm pretty deep into rewriting tree-sitter-vimdoc. Results are promising so far, but will take time.

Problem

The parser currently thinks that 'x' is a link to an option, but that is not correct. single-char surrounded by single-quotes should be treated as a plain word.

  ["'!'"] = true,
  ["'#'"] = true,
  ["'$'"] = true,
  ["'%'"] = true,
  ["'&'"] = true,
  ["'('"] = true,
  ["')'"] = true,
  ["'+'"] = true,
  ["','"] = true,
  ["','sqlKeyword'"] = true,
  ["','sqlOption'"] = true,
  ["','sqlStatement'"] = true,
  ["'-'"] = true,
  ["'-'Z'"] = true,
  ["'.'"] = true,
  ["'/'"] = true,
  ["'0'"] = true,
  ["'1'"] = true,
  ["'2'"] = true,
  ["'4'"] = true,
  ["'8'"] = true,
  ["'9'"] = true,
  ["':'"] = true,
  ["';'"] = true,
  ["'<'"] = true,
  ["'='"] = true,
  ["'>'"] = true,
  ["'?'"] = true,
  ["'@'"] = true,
  ["'A'"] = true,
  ["'B'"] = true,
  ["'C'"] = true,
  ["'D'"] = true,
  ["'E'"] = true,
  ["'F'"] = true,
  ["'G'"] = true,
  ["'H'"] = true,
  ["'I'"] = true,
  ["'J'"] = true,
  ["'K'"] = true,
  ["'L'"] = true,
  ["'M'"] = true,
  ["'N'"] = true,
  ["'O'"] = true,
  ["'P'"] = true,
  ["'Q'"] = true,
  ["'R'"] = true,
  ["'S'"] = true,
  ["'T'"] = true,
  ["'W'"] = true,
  ["'X'"] = true,
  ["'Y'"] = true,
  ["'Z'"] = true,
  ["'['"] = true,
  ["']'"] = true,
  ["'^'"] = true,
  ["'_'"] = true,
  ["'a'"] = true,
  ["'b'"] = true,
  ["'c'"] = true,
  ["'d'"] = true,
  ["'e'"] = true,
  ["'f'"] = true,
  ["'g'"] = true,
  ["'h'"] = true,
  ["'i'"] = true,
  ["'j'"] = true,
  ["'k'"] = true,
  ["'l'"] = true,
  ["'m'"] = true,
  ["'n'"] = true,
  ["'o'"] = true,
  ["'p'"] = true,
  ["'q'"] = true,
  ["'r'"] = true,
  ["'s'"] = true,
  ["'t'"] = true,
  ["'u'"] = true,
  ["'v'"] = true,
  ["'w'"] = true,
  ["'x'"] = true,
  ["'y'"] = true,
  ["'z'"] = true,
  ["'~'"] = true,
  ["'·'"] = true,
  ["'ö'"] = true,
  ["'─'"] = true,
  ["'│'"] = true,
  ["'├'"] = true,
  ["'┤'"] = true,
  ["'┬'"] = true,
  ["'┴'"] = true,
  ["'┼'"] = true,
  ["'보'"] = true,
  ["'세'"] = true,
  ["'여'"] = true,
  ["'요'"] = true,

parse error: pi_spec.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/pi_spec.html

Context:

function man page.

Where to insert new items			*spec_chglog_prepend*
-------------------------

The plugin will usually insert new %changelog entry items (note that it's
not the entry itself) after the existing ones.  If you set the
spec_chglog_prepend variable >

URL token

Seems like the parser currently doesn't recognize URLs in text. The old makehtml.awk script had primitive support for that, and it's useful for rendering URLs as hyperlinks.

Example text: https://github.com/neovim/neovim/blob/52e4b779e3df1335efc4774e8dfc199d07e24bfd/runtime/doc/api.txt#L20-L31

==============================================================================
API Usage                                               *api-rpc* *RPC* *rpc*

                                                        *msgpack-rpc*
RPC is the typical way to control Nvim programmatically.  Nvim implements the
MessagePack-RPC protocol:
  https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
  https://github.com/msgpack/msgpack/blob/0b8f5ac/spec.md

Many clients use the API: user interfaces (GUIs), remote plugins, scripts like
"nvr" (https://github.com/mhinz/neovim-remote).  Even Nvim itself can control
other Nvim instances.  API clients can:

  - Call any API function
  - Listen for events
  - Receive remote calls from Nvim

parse error: pi_spec.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/pi_spec.html

Context:

it will insert new items before the existing ones.

Inserting release info				*spec_chglog_release_info*
----------------------

If you want, the plugin may automatically insert release information
on each changelog entry.  One advantage of turning this feature on is
that it may control if the release has been updated after the last

Severe slowdown on large files

Description

Neovim lags severely when moving around large files. The lag gets worse the closer you are to the bottom of the file.

To reproduce

:h netrw (4418 lines)
G
Observe slowdown with normal commands or simply holding k

CI: run gen_help_html.lua in pull requests

gen_help_html.lua in the Nvim source is used to generate HTML from vimdoc :help files:

https://github.com/neovim/neovim/blob/6afe5cce7964258f7ee0b25ae08faa551bc76090/scripts/gen_help_html.lua#L766

Currently the full gen() step in gen_help_html.lua is not fully run in Nvim CI because it takes 30-60s. However, 30-60s in the CI for tree-sitter-vimdoc could be acceptable. And it would add a lot of coverage from an actual consumer.

However, it would cause CI here to fail if we rename nodes. That would require a corresponding PR to update gen_help_html.lua.

Some whole paragraphs colored

When having TS enabled with highlight, often there are whole paragraphs highlighted, for me (gruvbox-baby) in yellow. Example from usr_03.txt below:

With TS:

Without TS:

parse error: syntax.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/syntax.html

Context:

	a number instead of a color name.

	Note that for 16 color ansi style terminals (including xterms), the
	numbers in the NR-8 column is used.  Here '*' means 'add 8' so that 
	Blue is 12, DarkGray is 8 etc.

	Note that for some color terminals these names may result in the wrong

nested blocks

(Continuation of #13 . See also #21 )

Problem

#16 added (block), but these don't "nest".

Preamble from #12 :

In order for :help to render as something other than fixed-width, hard-wrapped text, (e.g. for HTML format) the parser needs to give some basic hints about structure besides "line". Sometimes (often) the :help source needs to be fixed (because it can't be reasonably parsed), but in other common cases it's probably reasonable for tree-sitter-vimdoc to provide some basic idea of:

Example of nested (indented) blocks

https://github.com/neovim/neovim/blob/52e4b779e3df1335efc4774e8dfc199d07e24bfd/runtime/doc/eval.txt#L51-L74

"Nested blocks" simply means blocks of text that have common indentation. In the following example,

we can't parse the Number 123... block but if we can at least recognize that it is indented, then the HTML result can wrap that block in <pre> so that it is readable.
the *octal* line doesn't share indentation so it should be parsed as (line) (or possibly introduce a new (tags_heading) node?).
it may be reasonable for us to require blank lines between these blocks.

The Number and String types are converted automatically, depending on how they
are used.

Conversion from a Number to a String is by making the ASCII representation of
the Number.  Examples:
	Number 123	-->	String "123" ~
	Number 0	-->	String "0" ~
	Number -1	-->	String "-1" ~
							*octal*
Conversion from a String to a Number is done by converting the first digits to
a number.  Hexadecimal "0xf9", Octal "017" or "0o17", and Binary "0b10"
numbers are recognized.  If the String doesn't start with digits, the result
is zero. Examples:
	String "456"	-->	Number 456 ~
	String "6bar"	-->	Number 6 ~
	String "foo"	-->	Number 0 ~
	String "0xf1"	-->	Number 241 ~
	String "0100"	-->	Number 64 ~
	String "0o100"	-->	Number 64 ~
	String "0b101"	-->	Number 5 ~
	String "-8"	-->	Number -8 ~
	String "+8"	-->	Number 0 ~

To force conversion from String to Number, add zero to it: >
	:echo "0100" + 0
<	64 ~

To avoid a leading zero to cause octal conversion, or for using a different
base, use |str2nr()|.

Inline code highlighting has issues

There are multiple issues of recognizing inline code blocks. It usually works fine when backticks surround single word consisting from letters. Other than that, not so much.

Text:

`inline code with s`
`inline code with spaces`
`require('')`
`require('mod')`
`{}`

Current behavior with tree-sitter:

Current behavior without tree-sitter:

parse error: motion.txt single quote followed by register type

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/motion.html

Context:

|shada-file-marks|.

Numbered marks '0 to '9 are quite different.  They can not be set directly.
They are only present when using a shada file |shada-file|.  Basically '0
is the location of the cursor when you last exited Vim, '1 the last but one
time, etc.  Use the "r" flag in 'shada' to specify files for which no
Numbered mark should be stored.  See |shada-file-marks|.

Same as #45?

parse error: lsp.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/lsp.html

Context:

    the file system upwards starting from the current directory until either a
    `pyproject.toml` or `setup.py` file is found.

    `workspace_folders` a list of { uri:string, name: string } tables. The
    project root folders used by the language server. If `nil` the property is
    derived from the `root_dir` for convenience.

parse error: pi_spec.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/pi_spec.html

Context:

done so.

Setting a map					*spec-setting-a-map*
-------------

As you should know, you can easily set a map to access any Vim command (or
anything, for that matter).  If you don't like the default map of
<LocalLeader>c, you may just set up your own key.  The following line

query.lua:219: query: invalid node type at position 14 for language help

No matter which help file I open, be it an empty file or the help file for treesitter, I'm always getting this error on load and repeatedly as I move around the file.

Error detected while processing FileType Autocommands for "*":
Error executing lua callback: ...im/0.8.1/share/nvim/runtime/lua/vim/treesitter/query.lua:219: query: invalid node type at position 14 for language help
stack traceback:
        [C]: in function '_ts_parse_query'
        ...im/0.8.1/share/nvim/runtime/lua/vim/treesitter/query.lua:219: in function 'get_query'
        ...1/share/nvim/runtime/lua/vim/treesitter/languagetree.lua:52: in function 'new'
        ...r/neovim/0.8.1/share/nvim/runtime/lua/vim/treesitter.lua:45: in function '_create_parser'
        ...r/neovim/0.8.1/share/nvim/runtime/lua/vim/treesitter.lua:96: in function 'get_parser'
        .../start/nvim-treesitter/lua/nvim-treesitter/highlight.lua:34: in function 'start'
        .../start/nvim-treesitter/lua/nvim-treesitter/highlight.lua:42: in function 'attach'
        ...er/start/nvim-treesitter/lua/nvim-treesitter/configs.lua:505: in function 'attach_module'
        ...er/start/nvim-treesitter/lua/nvim-treesitter/configs.lua:528: in function 'reattach_module'
        ...er/start/nvim-treesitter/lua/nvim-treesitter/configs.lua:131: in function <...er/start/nvim-treesitter/lua/nvim-treesitter/configs.lua:130>

I've removed all of my plugins and configuration and I'm still getting this issue whenever the filetype is help.

$XDG_CONFIG_HOME/nvim/init.vim

parse error: pattern.txt

Found tree-sitter-vimdoc parse error at: https://neovim.io/doc/user/pattern.html#%2Fmagic

Context:

	  ()	   \(\)     \(\)     \(\)	group as an atom
	  |	   \|	    \|	     \|		nothing: separates alternatives
	  \\	   \\	    \\	     \\		literal backslash
	  \{	   {	    {	     {		literal curly brace

{only Vim supports \m, \M, \v and \V}

The line literal curly brace has an alignment issue w.r.t the previous line.

misalignment of boxed text

In this page, text inside the character boxes are all misaligned, either out of the box or sticking out of the enclosing box.

nested list items

(Continuation of #12 )

Problem

#16 added parsing for list items as (line_li), but these don't "nest".

Example of a nested list

api.txt has many nested lists.

https://github.com/neovim/neovim/blob/52e4b779e3df1335efc4774e8dfc199d07e24bfd/runtime/doc/api.txt#L3257-L3273

   • callback (function|string) optional: if a string, the name
     of a Vimscript function to call when this autocommand is
     triggered. Otherwise, a Lua function which is called when
     this autocommand is triggered. Cannot be used with
     {command}. Lua callbacks can return true to delete the
     autocommand; in addition, they accept a single table
     argument with the following keys:
     • id: (number) the autocommand id
     • event: (string) the name of the event that triggered the
       autocommand |autocmd-events|
     • group: (number|nil) the autocommand group id, if it
       exists
     • match: (string) the expanded value of |<amatch>|
     • buf: (number) the expanded value of |<abuf>|
     • file: (string) the expanded value of |<afile>|
     • data: (any) arbitrary data passed to
       |nvim_exec_autocmds()|

neovim / tree-sitter-vimdoc Goto Github PK

tree-sitter-vimdoc's Introduction

tree-sitter-vimdoc

Overview

Known issues

TODO

Release

tree-sitter-vimdoc's People

Contributors

Stargazers

Watchers

Forkers

tree-sitter-vimdoc's Issues

Example of paragraphs and nested (indented) paragraphs

Example of a basic list

Example of a nested list

Problem

Description

To reproduce

Problem

Example of nested (indented) blocks

Problem

Example of a nested list

Recommend Projects

Recommend Topics

Recommend Org