Allow comm message to be processed during cell execution about ipykernel HOT 4 OPEN

I think this can be handled by processing messages in a thread while running code in another. What complicates things is that comm messages come on the same queue as execution, but I think we can deal with that if we are careful. I've been designing the threaded IO in my head for a bit, because I think it would solve several problems with flushing stdout as well.

from ipykernel.

@minrk Thanks for the time on this issue. BTW, when we implemented our backend in C#, we execute the cell code in the background:

https://bitbucket.org/ipre/calico/src/master/Source/Calico/ZMQServer.cs?fileviewer=file-view-default#ZMQServer.cs-964

I thought that that was the way it was supposed to work. I don't remember any adverse effects, but I think we made sure that the kernel acted as if it were blocking (didn't accept another cell execution message, for example). I do remember testing parallel cell execution, which worked, if you keep the output cells associated with the execution. Although, that can be a confusing UI. It might be nice to have a magic that would allow cell code to run in background, and not block. No need to write specific code to have it run in a thread if it is already running in a thread.

from ipykernel.

@minrk I'm glad, I found this issue, because I also need to handle comm messages in background. I'm about to prototype a kernel based on IPythonKernel which processes comm messages in another thread. My plan is to use two queues for comm and other messages, which would be polled in different threads. Please share your ideas on the treaded IO, maybe it's helpful.

Also I'd like to create a PR once it passes existing tests, and it would be great to receive feedback on this.

from ipykernel.

I recently encountered the same issue, in the cell we run something like await prompt("x=?") and the front end popup a dialog to get user input, once done, it sends a comm message to the backend. However, because the cell is running, the comm message cannot go through, it ended up with a dead state that we can only send an interrupt execution command to end it.

@AlexTugarev do you have a chance to work on a solution? Two queues for comm and other messages sounds like a good idea.

from ipykernel.