Be smarter about initial reconfigure about synapse HOT 3 OPEN

@dcosson Yea this has been somewhat of an annoying bug to us as well. We mitigated it slightly at Yelp by reducing the amount of zookeeper connecting we do and using the state_file so that at the very least as soon as the watcher gets information we can quickly enable the backend over the stat socket, but this just shortens the window of sad.

How are you thinking about implementing this? The watchers all live in separate threads, but perhaps we can communicate back to the main thread a la reconfigure?

from synapse.

Now that I'm looking at it, the zookeeper watchers do setup before returning because they don't use a Thread.new in the start method (they rely on the zk-ruby gem's async callbacks to multithread themselves).

@dcosson I imagine you're using the ec2 tag watcher or some such?

from synapse.

Oops, didn't mean to close, sorry about that.

from synapse.