TurboVnc Server/Session freeze after network interruption about turbovnc HOT 15 CLOSED

I've tried it out for several days now without issues. It looks like it is resolved now. Thanks for looking into this!

from turbovnc.

@brandonbiggs Yes, it's in 3.0.3.

from turbovnc.

The only time I have ever seen anything remotely like this was with a client of mine who was using a stateful firewall that held onto the TurboVNC connection like a dog with a bone for some reason. TurboVNC, in and of itself, should never behave as you describe.

To answer your specific questions:

• There is a "client wait" timer, which kicks in if the TurboVNC server tries to communicate with the viewer but can't do so for a specified period of time, but the default value is 20 seconds. That's definitely not the cause of the problem, and in fact, I suspect that something outside of TurboVNC is interfering with that mechanism.
• tcpkill (usually part of the dsniff package) might be able to do that, but I haven't tried it.
• Session migration isn't possible, because X11 applications are just clients communicating with the X server, and most of the X11 state is on the X server (Xvnc.)

from turbovnc.

It could be stateful firewall related, but after the connection has failed, and since I can still get through to the server if I start another session on the server, it would seem to me that the firewall may not be the issue. When the client wait timer times out and the server drops the connection, does it reuse TCP sequence numbers for new connections? Has this client wait timer been tested to confirm that it works and hasn't regressed in newer versions? Because I don't recall having these same issues on the 2.x versions of TurboVnc, and that was back when I had a much worse network connection.

I've tried tcpkill, but that doesn't really help because that will just kill incoming connections, but it doesn't help communicate to the server that it should stop waiting for the previous client to disconnect. It really does seem like the server gets stuck in an infinite waiting loop of some sort, and during these crashes it sits there using >90% cpu usage (as reported by top) which is higher than it typically is even when under normal operation (typically 20-40% cpu usage).

from turbovnc.

I'll try to reproduce the problem. I assume that you are connecting manually to a specific TurboVNC session, as opposed to using the Session Manager? Which security type are you using? Which Linux distribution?

The only thing that changed between 2.2.x and 3.0.x that might explain this is the adoption of new RFB flow control algorithms from TigerVNC. If you're comfortable getting your hands dirty with the code, here is a patch that reverts to the older algorithms used in TurboVNC 2.2.x. If that eliminates the problem, then at least we know where the issue is, but I still need to be able to reproduce the issue in order to fix it.

0001-Restore-TurboVNC-2.2.x-RFB-flow-control-algorithms.patch

from turbovnc.

I don't have control over the server installation so that will be a bit difficult, but I'll see what I can do.

One more bit of data, when I try to kill using vncserver -kill :1 I get the following message, which does make it seem more likely that it is vncserver that is getting stuck in an infinite loop, and not just some firewall or other piece of external networking:

Xvnc seems to be deadlocked.  Kill the process manually and then re-run
    /opt/TurboVNC/bin/vncserver -kill :1
to clean up the socket files.

The config file looks like everything is commented out, so I assume it's the default security types with simple password. If there is a way to check which security type is used, let me know.

The Linux distro is RedHat 7 Workstation. If there are particular package versions you need, let me know.

from turbovnc.

I actually was able to duplicate the problem, but only when using a Windows client. I'm investigating.

from turbovnc.

That's awesome news!

I do see that there is a -noflowcontrol option for the server, so I might try running with that as well if that disables the new RFB flow control changes.

from turbovnc.

I can't seem to reproduce the problem with the older flow control algorithms, so unfortunately that does seem to be the source of it. :( Since that's not my code, it may take me a while to diagnose.

from turbovnc.

I was able to reproduce the issue with a macOS client as well. When the network connection drops, the TurboVNC Viewer will eventually time out and ask if you want to reconnect, but the TurboVNC Server hangs onto the connection for some reason. That is true regardless of whether the 2.2.x or 3.0.x flow control algorithms are used. It's just that the 3.0.x flow control algorithms apparently have a bug that causes an infinite loop when that happens, which ties up the TurboVNC X server and prevents the viewer from reconnecting. With the 2.2.x algorithms, the viewer can reconnect, but the original connection is still active for some reason. (So really there are two bugs at work here.) Note that, to make this occur reliably, I have to run an application (such as a video player, or vglrun -sp /opt/VirtualGL/bin/glxspheres64 if you have VirtualGL installed) that constantly draws something to the TurboVNC X server. This is because the viewer and the server can only detect network interruptions if they are trying to send or receive something. I'll keep you posted as I learn more about the issue.

from turbovnc.

New data points:

• The issue occurs with or without TLS encryption, i.e. with or without the use of OpenSSL.
• In most cases, when the issue occurs, I observe that the password dialog pops up much more quickly (after a minute or two) than you observed, but sometimes the server hangs for much longer. On a few occasions, it appeared to hang "forever" (20-30 minutes is "forever", because I'm not going to wait that long to find out.) It seems more likely to hang "forever" when connecting from a Windows client.
• The proximate cause of the issue is 94d4718, but that commit fixed an issue reported by another user, so I will have to figure out how to fix this issue without unfixing that one. Again we're dealing with two issues: Issue 1 is the hang, and Issue 2 (which also exists in TurboVNC 2.2.x) is the fact that the server holds onto the connection.
• Enabling TCP keepalive or disabling TCP corking doesn't appear to improve the situation.

from turbovnc.

Yeah, that matches with what I've been seeing as well. I suppose I've come to think if it pops up within 1-5 minutes that it's working fine so I've only really considered it truly frozen if it takes much longer to unfreeze.

from turbovnc.

Should be fixed now in the latest pre-release builds. Please test it.

from turbovnc.

Thanks, I'll try out the pre-release build.

from turbovnc.

Did this make it into the 3.0.3 release or is it planned for a future release?

from turbovnc.