event network may not be wired yet when session is shut down about flux-core HOT 4 CLOSED

The cmbd issues all overlay bind/connect calls before entering its event loop, however that is insufficient to ensure that messages are not lost.

This note from zmq_connect(3) explains the fundamental issue:

for most transports and socket types the connection is not
performed immediately but as needed by 0MQ. Thus a successful call
to zmq_connect() does not mean that the connection was or could
actually be established. Because of this, for most transports and
socket types the order in which a server socket is bound and a
client socket is connected to it does not matter. The first
exception is when using the inproc:// transport: you must call
zmq_bind() before calling zmq_connect(). The second exception are
ZMQ_PAIR sockets, which do not automatically reconnect to
endpoints.

I think epgm:.// is the the exception alluded to, but for testing with multiple cmbds per node, we are distributing events over ipc://.

One solution is to use the request/reply network to obtain events until the first one arrives on the pub/sub event network. We could sequence events and have a fresh cmbd send a request to its parent for event 0, 1, 2... until it receives an event N on its event overlay, the stop doing that.

from flux-core.

Simple solution: make connect synchronous by using zmonitor_t to monitor for ZMQ_EVENT_CONNECTED.

from flux-core.

As zmonitor seems unreliable and its API is not stable, ZMQ_EVENT_CONNECTED turns out to not be a very good way to fix this.

from flux-core.

Should have been auto-closed when the above was merged.

from flux-core.