Forking issues on REE about zookeeper HOT 4 CLOSED

(gdb) thread 3
[Switching to thread 3 (Thread 0x7ff90eea0700 (LWP 32434))]#0 0x00007ff91858f543 in poll () from /lib/libc.so.6

(gdb) bt
#0 0x00007ff91858f543 in poll () from /lib/libc.so.6
#1 0x00007ff9102f9ba5 in do_io (v=0xb51ec00) at src/mt_adaptor.c:299
#2 0x00007ff91829e9ca in start_thread () from /lib/libpthread.so.0
#3 0x00007ff91859bcdd in clone () from /lib/libc.so.6
#4 0x0000000000000000 in ?? ()

(gdb) thread 2
[Switching to thread 2 (Thread 0x7ff90f8a1700 (LWP 32435))]#0 0x00007ff9182a385c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

(gdb) bt
#0 0x00007ff9182a385c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1 0x00007ff9102f999b in do_completion (v=) at src/mt_adaptor.c:330
#2 0x00007ff91829e9ca in start_thread () from /lib/libpthread.so.0
#3 0x00007ff91859bcdd in clone () from /lib/libc.so.6
#4 0x0000000000000000 in ?? ()

(gdb) thread 1
(gdb) bt
#0 0x00007ff9182a003d in pthread_join () from /lib/libpthread.so.0
#1 0x00007ff9102f9868 in adaptor_finish (zh=0x78ea300) at src/mt_adaptor.c:211
#2 0x00007ff9102f2a9c in zookeeper_close (zh=0x78ea300) at src/zookeeper.c:2297
#3 0x00007ff9102e9d61 in destroy_zkrb_instance (ptr=0x30deb70) at zookeeper_c.c:48
#4 0x0000000000432563 in run_final (obj=181981800) at gc.c:2664
#5 0x000000000043258e in finalize_list (p=0xad8d268) at gc.c:1645
#6 0x0000000000432942 in rb_gc_finalize_deferred () at gc.c:2692
#7 0x0000000000420660 in rb_thread_schedule () at eval.c:11336
#8 0x0000000000421fe5 in eval_check_tick () at eval.c:5755
#9 0x000000000041b93c in rb_call0 (klass=13330720, recv=0, id=3031, oid=3031, argc=0, argv=0x7eb1, body=0xcb6678, flags=0) at eval.c:6010
#10 0x000000000041c8b3 in rb_call (klass=13330720, recv=0, mid=3031, argc=0, argv=0x0, scope=0, self=185131760) at eval.c:6308
#11 0x0000000000427bc8 in eval_call (self=185131760, node=0x2b3fae0) at eval.c:3381
#12 0x0000000000419d44 in rb_eval (self=185131760, node=0x2b3fab8) at eval.c:3910
#13 0x000000000041970c in rb_eval (self=185131760, node=0x2b3dee8) at eval.c:3858
#14 0x000000000041c5ab in rb_call0 (klass=45092760, recv=185131760, id=73289, oid=, argc=0, argv=0x7fff3828ba18, body=0x2b38d30, flags=942193176) at eval.c:6211

from zookeeper.

This specific snapshot is from zookeeper version 0.4.4, but after upgrading to the latest we still see the same behavior.

Of note, our "workaround" is to keep the connection object around so that GC doen't get to it.

from zookeeper.

Zookeeper had a lot of work done on it to make this as safe as possible in the MRI 1.8 world. In > 1.0, try this procedure:

z = Zookeeper.new("localhost:2181")

# in parent
z.pause_before_fork_in_parent

fork do
  # in child
  z.reopen
end

# in parent, after fork
z.resume_after_fork_in_parent

This takes care of stopping the event loop and making sure the main thread owns all the locks before forking, then re-opening the connection in the child, and re-starting the event loop in the parent. Getting fork() to work correctly is a huge pain with a multi-threaded lib like zookeeper. The above procedure should preserve in-flight state in the parent (callbacks, events, etc.) and reset state in the child (i.e. open a new connection, as sharing a session between parent and child wouldn't work).

from zookeeper.

in the ZK lib, I integrate with these two modules to handle this automatically: https://github.com/slyphon/zk/blob/master/lib/zk/fork_hook.rb and https://github.com/slyphon/zk/blob/master/lib/zk/install_fork_hooks.rb

from zookeeper.