How are people going to implement their own server on top of pygls?

are they going to create plugins similar to django?
are they going to extend an instance, similar to flask?
are they going to subclass?
some other way that I'm not thinking of?

I kinda like the flask api (the simple one, not all the complex blueprint stuff). I am just totally making this up :):

import pygls
server = pgls.LanguageServer()

@server.format_document('xml')
def format_xml(document):
    ...
    return formatted_document

Implement sending notifications to client.

E.G. codify_prototype.send_notification("customNotif", params={})

I tried to reproduce my problem with a simple example and modified the json-example from the repository in the following way:

@json_server.feature(COMPLETION)
def completions(ls, params: CompletionParams = None):
    """Returns completion items."""
    line_index = params.position.line
    character_index = params.position.character
    to_insert = "Label"

    end = Position(line_index, character_index)
    text_edit = TextEdit(Range(end, end), to_insert)

    return CompletionList(True, [
        CompletionItem(
            label=to_insert,
            sort_text="1",
            # insert_text=to_insert,
            # text_edit=text_edit,
        ),
        CompletionItem(
            label="Laaa",
            sort_text="2",
        )
    ])

One would expect the sort_text property to overwrite the alphabetical order in which the completion items are shown. This is the case if the insert_text and text_edit are left commented out or just the insert_text property is used. If a completion is then triggered by typing "la" the expected completions are shown:

Label
Laaa

However, if text_edit is used with or without insert_text the provided sorting is ignored and the completion items are shown in alphabetical order:

Laaa
Label

I can not test this with another client to see is this problem only occurs with VS Code.

Another interesting thing that might be helpful is that VS Code does not highlight the prefix "la" in the completion item label when shown in the drop down menu if text_edit is used. This could be because the label is not the text to be inserted if a text_edit is provided. Maybe pygls or VS Code internally treat these cases differently and show the completion items with text_edit at the end of the drop down list.

It appears that the current LanguageServer API does not provide a way to programmatically access client capabilities. It's important to properly detect when client doesn't support a certain feature, so as to not generate a request that will fail. For example, the client may not support workspace/configuration, and so ls.get_configuration() request fails, which then crashes the client (and we don't always have control over client code).

Would it be possible to add something like ls.client_capabilities() to request them? It would just directly return information cached from initialize request.

Thanks!

• update pygls dependencies (@danixeee)
• update versions (@danixeee)
• update CHANGELOG (@danixeee)
• tag/build release (@danixeee )
• upload to PyPi (@danixeee )
• post-mortem (@renatav , @danixeee)

Current behavior:

Both result and error fields are serialized and returned as part of response object to the client.

Expected behavior:

From json RPC specification:

jsonrpc
A String specifying the version of the JSON-RPC protocol. MUST be exactly "2.0".

result
This member is REQUIRED on success.
This member MUST NOT exist if there was an error invoking the method.
The value of this member is determined by the method invoked on the Server.

error
This member is REQUIRED on error.
This member MUST NOT exist if there was no error triggered during invocation.
The value for this member MUST be an Object as defined in section 5.1.

id
This member is REQUIRED.
It MUST be the same as the value of the id member in the Request Object.
If there was an error in detecting the id in the Request object (e.g. Parse error/Invalid Request), it MUST be Null.

• update pygls dependencies (@danixeee)
• update versions (@danixeee)
• update CHANGELOG (@danixeee)
• tag/build release (@danixeee )
• upload to PyPi (@danixeee )
• post-mortem (@renatav , @danixeee)

An endpoint request handler may throw an exception, e.g.

:: --> anakin textDocument/hover(3): {'position': {'character': 30, 'line': 37}, 'textDocument': {'uri': 'file:///home/raoul/.config/sublime-text-3/Packages/LSP/plugin/core/session_view.py'}}
anakin: ERROR:pygls.protocol:Failed to handle request 3 textDocument/hover Object(position=Object(character=30, line=37), textDocument=Object(uri='file:///home/raoul/.config/sublime-text-3/Packages/LSP/plugin/core/session_view.py'))
:: <<< anakin 3: AttributeError: 'NoneType' object has no attribute 'type'
anakin: Traceback (most recent call last):
anakin: File "/home/raoul/.local/lib/python3.7/site-packages/pygls/protocol.py", line 325, in _handle_request
anakin: self._execute_request(msg_id, handler, params)
anakin: File "/home/raoul/.local/lib/python3.7/site-packages/pygls/protocol.py", line 250, in _execute_request
anakin: self._send_response(msg_id, handler(params))
anakin: File "/home/raoul/.local/lib/python3.7/site-packages/anakinls/server.py", line 498, in hover
anakin: names = fn(params.position.line + 1, params.position.character)
anakin: File "/home/raoul/.local/lib/python3.7/site-packages/jedi/api/helpers.py", line 465, in wrapper
anakin: return func(self, line, column, *args, **kwargs)
anakin: File "/home/raoul/.local/lib/python3.7/site-packages/jedi/api/__init__.py", line 475, in help
anakin: if leaf.type in ('keyword', 'operator', 'error_leaf'):
anakin: AttributeError: 'NoneType' object has no attribute 'type'

If something failed to handle a request, this library should return a ResponseError with error code InternalError, and hopefully some descriptive message. Right now, the stacktrace is dumped as a string, but that's not a valid response.

Some semi-related questions:
Are there any examples or documentation of wrapping/extending pygls to:

1. ... serve LSP over websockets?
2. ... to wrap existing LS implementations?
3. ... use non-traditional filesystems and formats?

Background

We've been looking at bringing LSP to the Jupyter ecosystem for a while now. While the execution message format serves well for executing code, completion, inspection, shared frontend/backend state, and rudimentary code modification, LSP has many additional features and implementations which would be valuable to augment our kernels.

One of the current attempts at [1,2] is jupyterlab-lsp, which presently has a hard dependency on nodejs by way of jsonrpc-ws-proxy, which allows a single websocket port to server many stdio language servers. I've been making the configuration more jupyter-forward with this wip PR, which in turn wraps jsonrpc-ws-proxy with jupyter-server-proxy.

I've run into some issues in making it work simply for the general case, especially around absolute file paths, which aren't necessarily a given due to [3] the Jupyter Contents API, which allows serving "files" from... well, just about anything.

Another thing I want is to be able to have multiple, language-oriented URLs (or some message envelope format) be answered by the same underlying process, e.g. javascript-typescript-langserver.

Happy to help explore this space in any way!

• update pygls dependencies (@danixeee)
• update versions (@danixeee)
• update CHANGELOG (@danixeee)
• tag/build release (@danixeee )
• upload to PyPi (@danixeee )

Related to openlawlibrary/open-law-codify-prototype#3.

It's not always the case that this framework shuts down its process once the exit notification is received. In fact once the parent process (the editor) exits, it might get into a busy spin loop, eating up an entire core in my CPU.

I'm fixing killing subprocesses editor-side, but you might want to make sure your process exits anyway, for other editors.

This will allow setting up file extensions that will be processed by pygls. Other files will be ignored.

Two builds have done this, both are PR builds:

most recent - https://ci.appveyor.com/project/oll-team/pygls/builds/21215549
first failure - https://ci.appveyor.com/project/oll-team/pygls/builds/21215533

Branch build works - https://ci.appveyor.com/project/oll-team/pygls/builds/21215526

Tag build worked last week - https://ci.appveyor.com/project/oll-team/pygls/builds/21180826

The OLL_GIT_USER_NAME environment variable seems to be set properly in the settings UI.

Arguments passed to pygls are instances of type dict. Goal is to deserialize those arguments, so the fields can be accessed with dot notation.

Example:
textDocument['uri'] -> textDocument.uri

Add possibility to register multiple LSP features per one function.

E.G.
@codify_server.feature(
lsp.TEXT_DOCUMENT_DID_OPEN,
lsp.TEXT_DOCUMENT_DID_CHANGE)
def text_doc_did_open(ls, textDocument=None, **_kwargs):
validate_schema(ls, textDocument)

textDocumentSync option can be a number (which pygls uses) or be of the form of TextDocumentSyncOptions.

I am working on a language server to interface with the native LSP client inside of neovim. The neovim client lsp implementation will not send didSave notifications unless "TextDocumentSyncOptions" is sent back. if a number is sent back only open, change, and close notifications are sent even if i define a didSave capability.

Thoughts? I am new and just start started reading the LSP spec. and I might be missing something.

Btw I added this to the aur:
https://aur.archlinux.org/packages/python-pygls/

Update features.py and types.py to support the current LSP specification.

This should probably be:

def __init__(self, value_set: List[SymbolKind]):

https://microsoft.github.io/language-server-protocol/specifications/specification-current/#workspace_symbol

The Problem is that filtering the suggestions in the drop down menu of the client does not work if the CompletionList.isInComplete property is set to True and the CompletionItem.textEdit of an element of the CompletionList is set.

The simplest example for the problem is a small extension to the exmaple server provided in the GitHub repository. It just adds a simple completion feature:

@server.feature(COMPLETION)
def simple_completion(ls: LanguageServer, params: CompletionParams):
    # Equivalent TextEdit to default insertion
    line_number = params.position.line
    start = Position(line_number, params.position.character - 1)
    edit_range = Range(start, start)
    text_edit = TextEdit(edit_range, new_text="Inserted with textEdit")

    return CompletionList(False, [CompletionItem(
                                    label="Label",
                                    filter_text="filter_for",
                                    insert_text="Inserted with insert_text",
                                    # text_edit=text_edit
                                )]
                         )

I verified that text_edit is equivalent to the default insertion of insert_text.

I have tried the 4 combinations of changing the isInComplete property and using either insert_text or text_edit and can confirm that the other 3 combinations work.

Is this intended behavior? It makes it impossible to update CompletionItem.filterText in the recomputation of the completion suggestions.

• Refactor LSP initialize method
• appveyor use python 3.5
• appveyor run flake8 and bandit checks
• Fix python 3.5 compatibility
• Add more tests

https://github.com/openlawlibrary/pygls/blob/master/appveyor.yml only runs tests for python3.5 on windows.

It would be nice to run tests on other python versions, isn't it ?

I started a patch that would run tests on a matrix like this: https://ci.appveyor.com/project/perrinjerome/pygls/builds/29961016 . Patch is still very dirty, there's lots of duplication ( it's perrinjerome@8879078 ), but what are your thoughts on this ? And about using tox ? (tox makes it easier for developers to run test suite on multiple python versions as long as they have these versions installed locally)

Goal is to change current server to asyncio server

The goal is to refactor current implementation of how server capabilities are calculated.
Client capabilities (InitializeRequest) should be considered when creating server capabilities.

https://github.com/openlawlibrary/pygls/blob/master/pygls/types.py#L879

What is the best way to provide "user configuration" when developing a server using pygls? I'm currently developing https://github.com/pappasam/jedi-language-server and am having trouble with this.

As a simple example, I've provided the following trigger characters:

@SERVER.feature(COMPLETION, trigger_characters=[".", "'", '"'])
def lsp_completion(server: LanguageServer, params: CompletionParams):
    """Returns completion items"""

That said, it's possible some users would prefer to use only a "." as a trigger character. I'd like to give those users the ability to change this feature setting in their settings.json. For example:

    "jls": {
      "command": "jedi-language-server",
      "args": [],
      "filetypes": ["python"],
      "initializationOptions": {
        "triggerCharacters": ["."]
      }
    },

How do I:

1. effectively access this configuration option, and
2. dynamically configure my server's features with it?

Thanks in advance for your help!

Currently we support just incremental updates. The goal is to support all synchronization options specified by the protocol:

/**
 * Defines how the host (editor) should sync document changes to the language server.
 */
export namespace TextDocumentSyncKind {
	/**
	 * Documents should not be synced at all.
	 */
	export const None = 0;

	/**
	 * Documents are synced by always sending the full content
	 * of the document.
	 */
	export const Full = 1;

	/**
	 * Documents are synced by sending the full content on open.
	 * After that only incremental updates to the document are
	 * send.
	 */
	export const Incremental = 2;
}

I'm trying to develop with pygls using vim and the vim-lsc Language Server Client plugin (https://github.com/natebosch/vim-lsc) as a client. I get this error in pygls:

ERROR Failed to handle notification textDocument/didSave: Object....
Traceback (most recent call last):
File.... protocol.py line 272, in _get_handler
return self.fm.buildin_features[feature_name]
KeyError: 'textDocument/didSave'

I've commented out the line in vim-lsc that sends this notification so I can carry on working, but I thought you may want to know about it.

Initial discussion started in #78.

Currently, data received by the client is deserialized to dict object which is then converted to namedtuple object. Reason behind this is that we want to access fields with dot notation.
Some clients are not serializing/passing null fields, which throws AttributeError when trying to access them in server methods.

Here are couple of ideas/solutions which need further discussion:

• #78 (review)
• #78 (comment)

This notification is sent from editor to server upon startup.

vscode-json and vscode-eslint servers rely on this notification to be sent during startup, or they won't work.

So we notify the server regardless.

This framework throws an exception when encountering an unhandled notification. I think it should instead log a neutral message. Something like: "unhandled notification workspace/didChangeConfiguration".

The JSON-RPC spec states that clients/servers are free to ignore notifications, so this should not raise a hard exception.

Target release: 12/28/2018

• add ls support for diagnostics (@danixeee)
• correct advanced usage doc (@danixeee)
• small fix in json example (@danixeee)
• "pygls" -> pygls almost everywhere (@danixeee)
• update CHANGELOG (@danixeee)
• tag/build release (@augb)
• upload to PyPi (@augb)

... based off of python-language-server.

The goal is to only have the core functionality (language-agnostic) to implement the LSP. Ideally, other languages can base off of pygls to implement their specific language server (e.g. XML, Python, etc.).

Add API (sync and async) to protocol and server for dynamic (un)registration of client capabilities.

One of the major downsides of pyls is that it craps out at UTF-16 offsets.

I just want to make sure that this library doesn't fail in the same way. Just to be clear, the LSP spec says:

A position inside a document (see Position definition below) is expressed as a zero-based line and character offset. The offsets are based on a UTF-16 string representation. So a string of the form a𐐀b the character offset of the character a is 0, the character offset of 𐐀 is 1 and the character offset of b is 3 since 𐐀 is represented using two code units in UTF-16.

Since python internally works with UTF-32 and indexes strings with UTF-32, you have to be very careful in translating LSP-ranges to pygls ranges. It is not as simple as converting a (row, col) to a character offset.

If you already accounted for this, then you may close this issue.

I'd like to use this to build an LSP for a small language called OpenSCAD, so I started investigating this project (excellent code and docs by the way!). I'm normally a Neovim user, but for sake of better understanding pygls, I installed VS code (in other words, might just be a configuration error) and ran the example JSON language server in debug mode. Unfortunately, I'm quickly met with this error in the debug client running alongside the server:

[Error - 12:11:01 AM] Server initialization failed.
  Message: TypeError: 'NoneType' object is not iterable
  Code: -32602 
[object Object]

Any ideas on how I could get around this? Is this a new error, or is it a common bump in the road?

Thanks in advance.

Hi folks,

When I implement @server.feature(DOCUMENT_SYMBOL), which returns a List[DocumentSymbol], the server crashes with

ERROR:pygls.protocol:Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pygls/protocol.py", line 364, in _send_data
    body = json.dumps(data, default=lambda o: o.__dict__)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.7/site-packages/pygls/protocol.py", line 364, in <lambda>
    body = json.dumps(data, default=lambda o: o.__dict__)
AttributeError: 'mappingproxy' object has no attribute '__dict__'

What needs to be done to fix this error?

Thanks!

• update versions (@danixeee)
• update CHANGELOG (@danixeee)
• tag/build release (@augb)
• upload to PyPi (@augb)
• post-mortem (@augb, @danixeee)

Adding clients other than VS Code, which uses the same jso-extension server will allow us to test pygls for different editors and catch possible bugs like here.

I'm currently experimenting with pygls (great so far!) and while things look promising, I'm struggling with setting up a process which results in a single installable VSCode extension.

Right at this moment, I'm trying to compile the server script with Nuitka, which results in a single executable instead of having to point to an external Python distribution with pygls installed.

Regardless, I was wondering how I can build a single extension that I could install into any VSCode installation (e.g. it should include both client/server).

How would I go about that? I'm not as comfortable with JS/Node as I'm with Python.

• update pygls dependencies (@danixeee)
• update versions (@danixeee)
• update CHANGELOG (@danixeee)
• tag/build release (@danixeee )
• upload to PyPi (@danixeee )
• post-mortem (@augb, @danixeee)

Currently pygls supports two types of connections:

• TCP/IP
• IO (stdin/stdout)

We want to add another type of connection - web-sockets.

Handle workspace/didChangeConfiguration LSP method and update global configuration

Moved here from #93 (comment)

In a nutshell:

• complete the derivation of a JSON Schema for the Language Server Protocol
• Use the schema to generate a new, standalone package, e.g. language-server-protocol
  • with good typing, e.g. dataclasses, typeddict or namedtuple
  • either
    • offer multiple versions of the library, e.g. language-server-protocol==3.14.x (only one per env)
    • include multiple versions of the spec e.g. currently lsp.v3_14 (would allow multiple servers with different versions)
  • offer optionally pluggable serialization (e.g. json, orjson, ujson) and runtime validation (e.g. jsonschema, fastjsonschema)
• use language-server-protocol in pygls
  • remove most of the constants and classes of pygls.types and pygls.protocol

For a language server implementer, the default would look the same:

# setup.py
    install_requires = ["pygls"],

# server.py
import pygls
server = pygls.LanguageServer()

@server.feature(server.lsp.SELECTION_RANGE)
def foo():
    return None  # would fail because SELECTION_RANGE not defined

no validation, stdlib json, use the latest stable LSP

While a fancier implementation might choose:

# setup.py
    install_requires = ["pygls", "orjson", "fast-jsonschema"],

# server.py
import language_server_protocol
import pygls
lsp = language_server_protocol.V3_15(
    json=lsp.io.OrJSON(), 
    validator=lsp.schema.FastJSONSchema()
)

server: LanguageServer[lsp.V3_15] = LanguageServer(lsp=lsp)

@server.feature(lsp.SELECTION_RANGE)
async def bar():
    return await some_untyped_function()  # would fail at runtime if `range` not defined in result

use bleeding edge protocol, hotrod rust-based JSON, and do runtime checking with a code-generating parser

E.g.

types.Completion('defaultdict')

will result in sending all other attributes along with label:

server-reply (id:21) Wed May 13 14:25:56 2020:
(:id 21 :jsonrpc "2.0" :result
     (:isIncomplete :json-false :items
		    [(:label "defaultdict" :kind nil :detail nil :documentation nil :deprecated :json-false :preselect :json-false :sortText nil :filterText nil :insertText nil :insertTextFormat nil :textEdit nil :additionalTextEdits nil :commitCharacters nil :command nil :data nil)]))

https://github.com/openlawlibrary/pygls/blob/master/examples/json-extension/README.md#install-server-dependencies says to "Install pygls following this README" but the README at https://github.com/openlawlibrary/pygls/blob/master/README.md does not have installation instructions.

FailureHandlingKind is specified as

export namespace FailureHandlingKind {

	/**
	 * Applying the workspace change is simply aborted if one of the changes provided
	 * fails. All operations executed before the failing operation stay executed.
	 */
	export const Abort: FailureHandlingKind = 'abort';

	/**
	 * All operations are executed transactionally. That means they either all
	 * succeed or no changes at all are applied to the workspace.
	 */
	export const Transactional: FailureHandlingKind = 'transactional';


	/**
	 * If the workspace edit contains only textual file changes they are executed transactionally.
	 * If resource changes (create, rename or delete file) are part of the change the failure
	 * handling strategy is abort.
	 */
	export const TextOnlyTransactional: FailureHandlingKind = 'textOnlyTransactional';

	/**
	 * The client tries to undo the operations already executed. But there is no
	 * guarantee that this succeeds.
	 */
	export const Undo: FailureHandlingKind = 'undo';
}

but pygls definition is:

I'm thinking of a fix like:

diff --git a/pygls/types.py b/pygls/types.py
index 48b79d1..0642b57 100644
--- a/pygls/types.py
+++ b/pygls/types.py
@@ -574,7 +574,7 @@ class FailureHandlingKind(str, enum.Enum):
     Abort = 'abort'
     Transactional = 'transactional'
     TextOnlyTransactional = 'textOnlyTransactional'
-    FailureHandlingKind = 'undo'
+    Undo = 'undo'
 

Let's address this after #92 to prevent conflict.

I have been using pygls, it's excellent, but I would love that we could use more of python typing features:

• Make pygls itself type-check:
  • Fix type errors reported by type checkers (maybe target only mypy) ( #92 is a start, but more is needed here )
  • Run type checkers as part of automated test suite
• Have a bit more "stricts" type definitions (eg #88 )
• Includes a py.typed for https://www.python.org/dev/peps/pep-0561/ ( #91 )

If that's OK with you, I can send more pull requests for all this.

If I use this patch on json-extension example:

diff --git a/examples/json-extension/server/server.py b/examples/json-extension/server/server.py
index efd6087..70fd3bf 100644
--- a/examples/json-extension/server/server.py
+++ b/examples/json-extension/server/server.py
@@ -90,7 +90,7 @@ def _validate_json(source):
 
 
 @json_server.feature(COMPLETION, trigger_characters=[','])
-def completions(params: CompletionParams = None):
+def completions(params: CompletionParams = None) -> CompletionList:
     """Returns completion items."""
     return CompletionList(False, [
         CompletionItem('"'),
diff --git a/examples/json-extension/server/tests/unit/test_features.py b/examples/json-extension/server/tests/unit/test_features.py
index b2ff762..3ec98d7 100644
--- a/examples/json-extension/server/tests/unit/test_features.py
+++ b/examples/json-extension/server/tests/unit/test_features.py
@@ -50,8 +50,9 @@ def _reset_mocks():
     server.show_message_log.reset_mock()
 
 
-def test_completions():
-    completion_list = completions()
+def test_completions() -> None:
+    completion_list = completions("type error")
+    reveal_type(completion_list)
     labels = [i.label for i in completion_list.items]
 
     assert '"' in labels

I would expect that mypy report type error for wrong "type error" argument and reveal type of completion_list as CompletionList, instead, I have:

$ mypy examples/json-extension/server/tests/unit/test_features.py 
...
examples/json-extension/server/tests/unit/test_features.py:55: note: Revealed type is 'Any'

I would have expected something like:

$ mypy examples/json-extension/server/tests/unit/test_features.py
...
examples/json-extension/server/tests/unit/test_features.py:54: error: Argument 1 to "completions" has incompatible type "str"; expected "Optional[CompletionParams]"
examples/json-extension/server/tests/unit/test_features.py:55: note: Revealed type is 'pygls.types.CompletionList'

We are not able to start json-extension sample with the new python extension for VS Code. I have opened the issue to find the solution.