QUIC: Blocking calls are unreliable in multi-threaded usage about openssl HOT 4 OPEN

@nhorman So, this is a nice idea, but won't quite work. There are various reasons, but here's an arbitrary one:

T1 calls some blocking API call, and enters block_until_pred and goes to sleep
  There was no data queued for the network when this happened, so poll() is listening to POLLIN but not POLLOUT
    (since the UDP socket kernel buffers are not full, listening to POLLOUT would make it wake up immediately)

Later T2 calls a non-blocking API call, and the connection is ticked
Data is generated and queued for transmission to the network
The amount of data generated fills the UDP socket kernel buffers, which means some
  data needs to be transmitted in the future when POLLOUT is asserted
 
The data to be transmitted may e.g. be an application message which yields a response which T1 is waiting for
Therefore to make forward progress, T1's internal poll(2) call must return and be re-executed
  so that it is now listening for POLLOUT
Since this can't happen without T2 having some way to notify T1 to wake up, T1 still has circumstances where
  blocking calls will incorrectly hang (at least until an event timeout)

But it's a nice idea and closing in on a decent solution. My take on the fix here (which does still need a notifier) is in #24257.

from openssl.

Couldn't this problem be solved more efficiently with the use of a muted, condition variable, and state variable (the last of which records which streams are readable)? It would save a potentially significant number of syscalls, and as you note, avoid having to create socket resources under the covers

from openssl.

Couldn't this problem be solved more efficiently with the use of a muted, condition variable, and state variable (the last of which records which streams are readable)? It would save a potentially significant number of syscalls, and as you note, avoid having to create socket resources under the covers

The problem is that modern OSes generally don't have syscalls which can block on both a condition variable and a socket at the same time.

This basically means your options for blocking on something are:

• Block on a condition variable and have it woken up by another thread (which means we have to spin up another thread to do background processing, and that thread makes the poll() call)
• Block on some set of sockets (as described above)

So there will always be more resources consumed — the choice is between creating a socketpair/eventfd and creating a thread.

from openssl.

Sorry, Ii wasn't being clear, all I meant to suggest was that you could implement what you needed without creating an additional thread - i.e. a "someone is waiting" heuristic. something like the following pseudo code:

lock_mutex
   if another_thread_is_blocking_on_the_socket
      wait_on_condition_variable
   set another_thread_is_blocking_on_the_socket = true
unlock_mutex
block_on_socket_ready
read_socket_data
lock_mutex
set_another_thread_is_blocking_on_the_socket = false
signal_condition_variable
unlock_mutex
return socket data

Thats pretty simplistic of course, and it doesn't account for cases in which the socket data has to be from a given stream in the quic connection, but you could setup a per stream signal variable, so that whatever thread gets into your critical section only wakes the correct thread (which may be itself, in which case you should signal all waiting threads, so another one can take over the blocking operation. The resource fanout is still linear of course if you do a per stream condition variable, but it might be less egregious than having to pump data through various socket pairs, or spawn a new entire thread.

from openssl.