Capture output coming from C and C++ libraries about ipykernel HOT 13 OPEN

Hi, this is a pretty big issue for all Tensorflow users (because print operators run in the C++ internals)
tensorflow/community#14

Is it possible to revisit / reprioritize this?

from ipykernel.

I made this package based on what we learned making the Cling kernel, so right now you can capture any C output in a notebook with:

with wurlitzer.sys_pipes():
    run_c_code()

from ipykernel.

(@dpiparo contacted me by email, and I suggested he open an issue so that the 'nanny process' idea we've been thinking of for a while gets back on our radar)

from ipykernel.

@takluyver , thanks for the clarification.

from ipykernel.

I confirm Wurlitzer works great: thanks for sharing it! It successfully captures all output w/o freezing and is very easy to use. Technically one could think of having it enabled by default hooked on pre and post_execute.

The limitation I see, as Wurlitzer is today, is that the printing of the output can happen only at the end of the execution of the code/cell. In presence of long calculations the user may get the impression that the system is stuck even if the library underneath is trying to print some sort of progress indicator.

Another improvement that one might wish is the ability to properly "interleave" output from C(++) and Python by default.

from ipykernel.

Looking at the code, I think Wurlitzer should be capable of forwarding output 'live', without waiting to the end of the context. It's using a thread to pull from the read end of the pipe and forward it to Python sys.std*.

from ipykernel.

@dpiparo it should print outputs as they arrive, as long as you are using ipykernel >= 4.3.

The only impediment I see to integrating wurlitzer into IPython by default is how reliable it is on different platforms.

from ipykernel.

Since it is making ctypes calls, I could certainly imagine it causing crashes on weird systems. And I just assume it doesn't work on Windows.

from ipykernel.

I'm pretty sure there's no dup2 on Windows, so there's probably not a practical way to do this in process there.

from ipykernel.

On the other hand, if this would get integrated, on linux/mac would add many nice features while windows would stay as it is. I am not sure though how similar cases were treated in the past within the Jupyter project.

from ipykernel.

We have plenty of things that only work on certain platforms (subprocesses behave nicer on non-Windows due to pexpect, and web terminals only work on non-Windows, also due to using pexpect), so I don't see that as a big issue.

The only thing in between me and proposing that we use it on non-Windows is the stability question. I don't know enough to say if/when it won't work, even on posixy systems.

from ipykernel.

I don't think we should integrate that in the Python kernel - most stuff works well enough without it, and now there is an easy solution we can point people to when they do need it. I think the longer term answer to this is to get round to the nanny process we've talked about for a long time, because that lets us capture std streams at the OS level on all platforms without needing ctypes or anything.

Maybe this is something we can discuss at the dev meeting in a couple of weeks.

from ipykernel.

Sure.

from ipykernel.