Tab-completion crashes about ilua HOT 7 OPEN

EDIT: Should probably tag you, since this was initially addressing your comment @rhaberkorn

I'm running into the same problem, except I can't evaluate anything before it crashes.

I was/still am digging through the code, and traced the problem back to breaking changes in twisted, with the latest working version being v22.10.0, 23+ breaks ILua. I never used twisted, nor have I created a Jupyter kernel before, or interacted with its libraries, but I'm currently spending some time trying to make ILua work with twisted 23+

Until then, the only way to make ILua run is to use a version less than 23. Since Twisted may be used by various other Python packages on the system, globally downgrading it just for ILua is not a good idea.

Instead, if you just want to get it working for the moment, here are some overly methodical instructions for people who come across this issue:

# Pick somewhere to put it. I just placed it next to the repo for debugging purposes.
cd ~/Source/third-party/clones/ilua/

# Create and enter the virtual environment.
python -m venv vilua
cd vilua

# Activate the virtual environment by sourcing whichever activation script you nee to source. 
source ./bin/activate.fish

# Make sure it's activated by checking if python and or pip points to the virtual environment, not the system. 
which pip

# Specify any version < 23 for twisted. 22.10.0 is the latest compatible.
pip install twisted==22.10.0 ilua

# Success?
ilua 

Maybe a script to run it from anywhere conveniently:

#!/bin/sh

source ~/Source/third-party/ilua/vilua/bin/activate
ilua
deactivate

I placed it next to ILua

mv vilua ~/.local/bin/vilua
sudo chmod ug+x ~/.local/bin/vilua

This isn't a permanent solution, nor a complete one.

I think the dependency stems from ILua using it to re-implement existing parts of the ipython / jupyter libraries to work with Twisted, in order to have Python 2 async support?

From ilua/ilua/kernelbase.py

"""
Twisted-based Jupyter base kernel implementation
Pretty much a reimplementation of ipykernel.kernelbase,
but with Twisted, which allows async code in python 2,
and provides txzmq.
"""

Since Python2 is deprecated, and this re-write would need to be maintained long-term, I think the best solution would be to drop twisted entirely. It seems to be more trouble than it's worth, having broken ILua with v23 here, as well as unnecessarily broadening its attack surface (see CVE-2023-46137), emphasizing the importance of maintenance. In this case the CVE had nothing to do with the way ILua uses Twisted, but why even have an "in this case" and not just drop it entirely and not be forced to maintain it?

That's the ideal anyway, but the fastest fix would be to resolve the breaking changes from v22 to v23 of twisted, and write the v23 equivalent of the v22 code, which is what I'm investigating. That being said I don't have an awful lot of time, but I felt like I should at least share what I learned so far.

from ilua.

Trying to execute 23:

In [1]: 232023-10-22T01:51:42+0300 [ilua.kernel.ILuaKernel#critical] Uncought exception in message handler
	Traceback (most recent call last):
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
	    result = context.run(gen.send, result)
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 196, in handle_message
	    content = yield self.do_is_complete(**msg['content'])
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2245, in unwindGenerator
	    return _cancellableInlineCallbacks(gen)
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2157, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, gen, status, _copy_context())
	--- <exception caught here> ---
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 196, in handle_message
	    content = yield self.do_is_complete(**msg['content'])
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
	    result = context.run(gen.send, result)
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernel.py", line 184, in do_is_complete
	    result = yield self.proto.sendRequest({"type": "is_complete",
	builtins.AttributeError: 'ILuaKernel' object has no attribute 'proto'
	
/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/jupyter_console/ptshell.py:787: UserWarning: The kernel did not respond to an is_complete_request. Setting `use_kernel_is_complete` to False.
  warn('The kernel did not respond to an is_complete_request. '
In [1]: 23
2023-10-22T01:51:43+0300 [ilua.kernel.ILuaKernel#critical] Uncought exception in message handler
	Traceback (most recent call last):
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
	    result = context.run(gen.send, result)
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 193, in handle_message
	    content = yield self.do_execute(**msg['content'])
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2245, in unwindGenerator
	    return _cancellableInlineCallbacks(gen)
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 2157, in _cancellableInlineCallbacks
	    _inlineCallbacks(None, gen, status, _copy_context())
	--- <exception caught here> ---
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernelbase.py", line 193, in handle_message
	    content = yield self.do_execute(**msg['content'])
	  File "/home/rhaberkorn/working-copies/ilua/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1997, in _inlineCallbacks
	    result = context.run(gen.send, result)
	  File "/home/rhaberkorn/working-copies/ilua/ilua/kernel.py", line 131, in do_execute
	    result = yield self.proto.sendRequest({"type": "execute",
	builtins.AttributeError: 'ILuaKernel' object has no attribute 'proto'

from ilua.

@guysv What's the recommended environment to run this in? Perhaps I can try with Docker. However it won't be an option for me to run the Lua interpreter within Docker as well.

Also, will it be possible to use the web frontend as well?

from ilua.

Also, will it be possible to use the web frontend as well?

Just to explain this. I have a project based on a custom LuaJIT REPL loop and I am planning to add context sensitive auto-completions anyway. That's not really what I need ILua for. What really interests me is a way to cheaply add a Web interface. People nowadays are spoiled you know. In this context, it would also be interesting to find out whether it will be possible to render any markup or graphics.

from ilua.

@PsychedelicShayna Will try that. Thanks! Using virtual environments is IMHO not hacky at all. Library conflicts are almost unavoidable in real life.
Somebody somewhere is not declaring the Twister-dependency on versions < 23 properly.

from ilua.

I can confirm that everything works for me once forcing Twister to the right version.

I can even run my custom Lua interpreter in a console - although, that's not terribly useful as I wrote.

It is indeed possible to run ILua in a notebook (Web UI) as well and it will even work with custom Lua interpreters. First you will have to install jupyter-notebook (into the same environment as everything else of course):

pip3 install notebook

Now just running jupyter notebook will start a web server and when creating a notebook from within the website you can manually choose a Lua kernel. It's probably also possible to make this the default choice, but I haven't yet figured out how that works.

Unfortunately, there does not seem to be a way to pass arguments to selected kernels via the jupyter notebook command line. That would be useful in order to set ILua's --lua-interpreter. If you know how, just tell me. It might be possible to just shadow lua in PATH of course and that might even be the best option if you need to pass any custom arguments to your interpreter as ILua does not support passing anything down to the interpreter so passing custom arguments requires a wrapper script anyway. Otherwise, you might just create a new kernel by copying ilua/env/share/jupyter/kernels/lua/ to some new directory, let's say ilua/env/share/jupyter/kernels/luajit/ and edit its kernel.json file. For instance to create an ILua-based kernel using luajit as the interpreter:

{
        "argv":
        [
                "python",
                "-m",
                "ilua.app",
                "-c",
                "{connection_file}",
                "--lua-interpreter", "luajit"
        ],
        "display_name": "LuaJIT",
        "language": "lua",
        "interrupt_mode": "message"
}

What I haven't found out yet is:

• How to interrupt a busy ILua kernel from a running notebook. At least, the interrupt button does not appear to send SIGINT to the corresponding Lua interpreter process.
• How to display some rich text or graphics in command output. From reading the IPython source code I get the impression that this must be supported by the kernel, ie. ILua in our case.

All of this should really be documented in some official place.

from ilua.

thanks for the heads up for the twisted version. working on a release pinning the version. I definitely can't keep up with latest twisted for this project so virtual envs are the way to go.

from ilua.