The next step is attaching a Mesos-slave to the Master, but the slave never successfully registers. I tried using zetcd in zkbridge mode (didn't specify -oracle, don't know if it's relevant?), but couldn't find any error or warning in zetcd logs while doing so, and the slave registered correctly.
I1118 21:29:46.485221 76 main.cpp:263] Build: 2016-08-26 23:00:07 by ubuntu
I1118 21:29:46.485282 76 main.cpp:264] Version: 1.0.1
I1118 21:29:46.485285 76 main.cpp:267] Git tag: 1.0.1
I1118 21:29:46.485287 76 main.cpp:271] Git SHA: 3611eb0b7eea8d144e9b2e840e0ba16f2f659ee3
I1118 21:29:46.486155 76 main.cpp:370] Using 'HierarchicalDRF' allocator
I1118 21:29:46.486438 76 leveldb.cpp:174] Opened db in 227954ns
I1118 21:29:46.486568 76 leveldb.cpp:181] Compacted db in 116695ns
I1118 21:29:46.486580 76 leveldb.cpp:196] Created db iterator in 4884ns
I1118 21:29:46.486585 76 leveldb.cpp:202] Seeked to beginning of db in 370ns
I1118 21:29:46.486589 76 leveldb.cpp:271] Iterated through 0 keys in the db in 193ns
I1118 21:29:46.486613 76 replica.cpp:779] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
I1118 21:29:46.486932 86 log.cpp:107] Attempting to join replica to ZooKeeper group
2016-11-18 21:29:46,486:76(0x7f73e021a700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2016-11-18 21:29:46,486:76(0x7f73e021a700):ZOO_INFO@log_env@730: Client environment:host.name=mesosmaster-master-1
2016-11-18 21:29:46,486:76(0x7f73e021a700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2016-11-18 21:29:46,486:76(0x7f73e021a700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-47-generic
2016-11-18 21:29:46,486:76(0x7f73e021a700):ZOO_INFO@log_env@739: Client environment:os.version=#68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016
2016-11-18 21:29:46,487:76(0x7f73e121c700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2016-11-18 21:29:46,487:76(0x7f73e121c700):ZOO_INFO@log_env@730: Client environment:host.name=mesosmaster-master-1
I1118 21:29:46.487156 88 recover.cpp:451] Starting replica recovery
2016-11-18 21:29:46,487:76(0x7f73e121c700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2016-11-18 21:29:46,487:76(0x7f73e121c700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-47-generic
2016-11-18 21:29:46,487:76(0x7f73e121c700):ZOO_INFO@log_env@739: Client environment:os.version=#68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016
2016-11-18 21:29:46,487:76(0x7f73dea17700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2016-11-18 21:29:46,487:76(0x7f73dea17700):ZOO_INFO@log_env@730: Client environment:host.name=mesosmaster-master-1
2016-11-18 21:29:46,487:76(0x7f73dea17700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2016-11-18 21:29:46,487:76(0x7f73dea17700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-47-generic
2016-11-18 21:29:46,487:76(0x7f73dea17700):ZOO_INFO@log_env@739: Client environment:os.version=#68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016
I1118 21:29:46.487267 76 main.cpp:543] Starting Mesos master
2016-11-18 21:29:46,487:76(0x7f73e1a1d700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2016-11-18 21:29:46,487:76(0x7f73e1a1d700):ZOO_INFO@log_env@730: Client environment:host.name=mesosmaster-master-1
2016-11-18 21:29:46,487:76(0x7f73e1a1d700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2016-11-18 21:29:46,487:76(0x7f73e1a1d700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-47-generic
2016-11-18 21:29:46,487:76(0x7f73e021a700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2016-11-18 21:29:46,487:76(0x7f73e1a1d700):ZOO_INFO@log_env@739: Client environment:os.version=#68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016
2016-11-18 21:29:46,487:76(0x7f73dea17700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2016-11-18 21:29:46,487:76(0x7f73e121c700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2016-11-18 21:29:46,487:76(0x7f73e021a700):ZOO_INFO@log_env@755: Client environment:user.home=/home/mesos
2016-11-18 21:29:46,487:76(0x7f73e021a700):ZOO_INFO@log_env@767: Client environment:user.dir=/etc/service/mesos
2016-11-18 21:29:46,487:76(0x7f73e021a700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zookeeper:2181 sessionTimeout=10000 watcher=0x7f73e9f706d0 sessionId=0 sessionPasswd=<null> context=0x7f73c0001060 flags=0
2016-11-18 21:29:46,487:76(0x7f73e1a1d700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2016-11-18 21:29:46,487:76(0x7f73e121c700):ZOO_INFO@log_env@755: Client environment:user.home=/home/mesos
2016-11-18 21:29:46,487:76(0x7f73dea17700):ZOO_INFO@log_env@755: Client environment:user.home=/home/mesos
2016-11-18 21:29:46,487:76(0x7f73e121c700):ZOO_INFO@log_env@767: Client environment:user.dir=/etc/service/mesos
2016-11-18 21:29:46,487:76(0x7f73e121c700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zookeeper:2181 sessionTimeout=10000 watcher=0x7f73e9f706d0 sessionId=0 sessionPasswd=<null> context=0x7f73bc000930 flags=0
2016-11-18 21:29:46,487:76(0x7f73dea17700):ZOO_INFO@log_env@767: Client environment:user.dir=/etc/service/mesos
2016-11-18 21:29:46,487:76(0x7f73dea17700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zookeeper:2181 sessionTimeout=10000 watcher=0x7f73e9f706d0 sessionId=0 sessionPasswd=<null> context=0x7f73c8001960 flags=0
2016-11-18 21:29:46,487:76(0x7f73e1a1d700):ZOO_INFO@log_env@755: Client environment:user.home=/home/mesos
2016-11-18 21:29:46,487:76(0x7f73e1a1d700):ZOO_INFO@log_env@767: Client environment:user.dir=/etc/service/mesos
2016-11-18 21:29:46,487:76(0x7f73e1a1d700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zookeeper:2181 sessionTimeout=10000 watcher=0x7f73e9f706d0 sessionId=0 sessionPasswd=<null> context=0x7f73d0001740 flags=0
I1118 21:29:46.488404 76 master.cpp:375] Master 9f8b1b5c-c8d4-4e4b-bbd0-c03213ab4385 (mesosmaster-master-1) started on 172.16.102.133:5050
I1118 21:29:46.488423 76 master.cpp:377] Flags at startup: --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="false" --authenticate_frameworks="false" --authenticate_http_frameworks="false" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticators="crammd5" --authorizers="local" --framework_sorter="drf" --help="false" --hostname="mesosmaster-master-1" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --port="5050" --quiet="false" --quorum="2" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="20secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/share/mesos/webui" --work_dir="/data" --zk="zk://zookeeper:2181/mesos" --zk_session_timeout="10secs"
I1118 21:29:46.488560 76 master.cpp:429] Master allowing unauthenticated frameworks to register
I1118 21:29:46.488576 76 master.cpp:443] Master allowing unauthenticated agents to register
I1118 21:29:46.488590 76 master.cpp:457] Master allowing HTTP frameworks to register without authentication
I1118 21:29:46.488608 76 master.cpp:499] Using default 'crammd5' authenticator
W1118 21:29:46.488617 76 authenticator.cpp:512] No credentials provided, authentication requests will be refused
I1118 21:29:46.488629 76 authenticator.cpp:519] Initializing server SASL
I1118 21:29:46.491000 92 recover.cpp:477] Replica is in EMPTY status
2016-11-18 21:29:46,491:76(0x7f73b6423700):ZOO_INFO@check_events@1728: initiated connection to server [172.16.88.196:2181]
2016-11-18 21:29:46,491:76(0x7f73b6c24700):ZOO_INFO@check_events@1728: initiated connection to server [172.16.88.196:2181]
2016-11-18 21:29:46,491:76(0x7f73affff700):ZOO_INFO@check_events@1728: initiated connection to server [172.16.52.4:2181]
2016-11-18 21:29:46,491:76(0x7f73b4c20700):ZOO_INFO@check_events@1728: initiated connection to server [172.16.52.4:2181]
I1118 21:29:46.493103 90 contender.cpp:152] Joining the ZK group
2016-11-18 21:29:46,493:76(0x7f73b6c24700):ZOO_INFO@check_events@1775: session establishment complete on server [172.16.88.196:2181], sessionId=0x4cfa587954761aab, negotiated timeout=10000
2016-11-18 21:29:46,493:76(0x7f73b6423700):ZOO_INFO@check_events@1775: session establishment complete on server [172.16.88.196:2181], sessionId=0x4cfa587954761aad, negotiated timeout=10000
I1118 21:29:46.493445 90 group.cpp:349] Group process (group(3)@172.16.102.133:5050) connected to ZooKeeper
I1118 21:29:46.493461 90 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
2016-11-18 21:29:46,493:76(0x7f73affff700):ZOO_INFO@check_events@1775: session establishment complete on server [172.16.52.4:2181], sessionId=0x6bf85879547643ef, negotiated timeout=10000
2016-11-18 21:29:46,493:76(0x7f73b4c20700):ZOO_INFO@check_events@1775: session establishment complete on server [172.16.52.4:2181], sessionId=0x6bf85879547643f1, negotiated timeout=10000
I1118 21:29:46.493489 90 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
I1118 21:29:46.493490 91 group.cpp:349] Group process (group(4)@172.16.102.133:5050) connected to ZooKeeper
I1118 21:29:46.493501 91 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1118 21:29:46.493505 91 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
I1118 21:29:46.493578 89 group.cpp:349] Group process (group(1)@172.16.102.133:5050) connected to ZooKeeper
I1118 21:29:46.493587 88 group.cpp:349] Group process (group(2)@172.16.102.133:5050) connected to ZooKeeper
I1118 21:29:46.493743 89 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1118 21:29:46.493754 89 group.cpp:427] Trying to create path '/mesos/log_replicas' in ZooKeeper
I1118 21:29:46.493784 88 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (1, 0, 0)
I1118 21:29:46.493793 88 group.cpp:427] Trying to create path '/mesos/log_replicas' in ZooKeeper
I1118 21:29:46.497033 91 detector.cpp:152] Detected a new leader: (id='1')
I1118 21:29:46.497112 92 group.cpp:706] Trying to get '/mesos/json.info_0000000001' in ZooKeeper
I1118 21:29:46.499739 86 zookeeper.cpp:259] A new leading master ([email protected]:5050) is detected
I1118 21:29:46.499851 86 master.cpp:1847] The newly elected leader is [email protected]:5050 with id 01d0998e-8d9b-4791-af19-227693b85d0c
I1118 21:29:46.504482 93 contender.cpp:268] New candidate (id='4') has entered the contest for leadership
E1118 21:29:48.699439 94 process.cpp:2022] Failed to shutdown socket with fd 23: Transport endpoint is not connected
I1118 21:29:55.822985 90 detector.cpp:152] Detected a new leader: (id='4')
I1118 21:29:55.823264 89 group.cpp:706] Trying to get '/mesos/json.info_0000000004' in ZooKeeper
I1118 21:29:55.830425 91 zookeeper.cpp:259] A new leading master ([email protected]:5050) is detected
I1118 21:29:55.830564 89 master.cpp:1847] The newly elected leader is [email protected]:5050 with id 9f8b1b5c-c8d4-4e4b-bbd0-c03213ab4385
I1118 21:29:55.830605 89 master.cpp:1860] Elected as the leading master!
I1118 21:29:55.830620 89 master.cpp:1547] Recovering from registrar
I1118 21:29:55.830691 91 registrar.cpp:332] Recovering registrar
E1118 21:29:56.479151 94 process.cpp:2022] Failed to shutdown socket with fd 25: Transport endpoint is not connected
I1118 21:29:56.492329 94 recover.cpp:110] Unable to finish the recover protocol in 10secs, retrying
E1118 21:30:06.721837 94 process.cpp:2022] Failed to shutdown socket with fd 25: Transport endpoint is not connected
E1118 21:30:26.533833 94 process.cpp:2022] Failed to shutdown socket with fd 25: Transport endpoint is not connected
E1118 21:30:34.553061 94 process.cpp:2022] Failed to shutdown socket with fd 25: Transport endpoint is not connected
F1118 21:30:55.832010 93 master.cpp:1536] Recovery failed: Failed to recover registrar: Failed to perform fetch within 1mins
*** Check failure stack trace: ***
@ 0x7f73ea4ea39d google::LogMessage::Fail()
@ 0x7f73ea4ec1cd google::LogMessage::SendToLog()
@ 0x7f73ea4e9f8c google::LogMessage::Flush()
@ 0x7f73ea4ecac9 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f73e9ad8f7c mesos::internal::master::fail()
@ 0x7f73e9b209c0 _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvS1_S1_EPKcSt12_PlaceholderILi1EEEEvEERKS6_OT_NS6_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_
@ 0x42a426 process::Future<>::fail()
@ 0x7f73e9b4a615 process::internal::thenf<>()
@ 0x7f73e9ba7a56 _ZN7process8internal3runISt8functionIFvRKNS_6FutureIN5mesos8internal8RegistryEEEEEJRS7_EEEvRKSt6vectorIT_SaISE_EEDpOT0_
@ 0x7f73e9bba9cf process::Future<>::fail()
@ 0x7f73e99056c6 process::internal::run<>()
@ 0x7f73e9bba9bc process::Future<>::fail()
@ 0x7f73e9b9d854 mesos::internal::master::RegistrarProcess::_recover()
@ 0x7f73ea479cf1 process::ProcessManager::resume()
@ 0x7f73ea479ff7 _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
@ 0x7f73e8b1da60 (unknown)
@ 0x7f73e833a184 start_thread
@ 0x7f73e806737d (unknown)
ARNING: Logging before InitGoogleLogging() is written to STDERR
I1118 21:45:16.571516 49 main.cpp:243] Build: 2016-08-26 23:00:07 by ubuntu
I1118 21:45:16.571648 49 main.cpp:244] Version: 1.0.1
I1118 21:45:16.571655 49 main.cpp:247] Git tag: 1.0.1
I1118 21:45:16.571660 49 main.cpp:251] Git SHA: 3611eb0b7eea8d144e9b2e840e0ba16f2f659ee3
I1118 21:45:16.681454 49 containerizer.cpp:196] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni
I1118 21:45:16.693532 49 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
2016-11-18 21:45:16,696:49(0x7f04a697c700):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2016-11-18 21:45:16,696:49(0x7f04a697c700):ZOO_INFO@log_env@730: Client environment:host.name=mesosslave-worker-1
2016-11-18 21:45:16,696:49(0x7f04a697c700):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2016-11-18 21:45:16,696:49(0x7f04a697c700):ZOO_INFO@log_env@738: Client environment:os.arch=4.4.0-47-generic
2016-11-18 21:45:16,696:49(0x7f04a697c700):ZOO_INFO@log_env@739: Client environment:os.version=#68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016
2016-11-18 21:45:16,696:49(0x7f04a697c700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
I1118 21:45:16.697005 49 main.cpp:434] Starting Mesos agent
2016-11-18 21:45:16,697:49(0x7f04a697c700):ZOO_INFO@log_env@755: Client environment:user.home=/root
2016-11-18 21:45:16,697:49(0x7f04a697c700):ZOO_INFO@log_env@767: Client environment:user.dir=/etc/service/mesos
2016-11-18 21:45:16,697:49(0x7f04a697c700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=zookeeper:2181 sessionTimeout=10000 watcher=0x7f04b16d46d0 sessionId=0 sessionPasswd=<null> context=0x7f0490001090 flags=0
I1118 21:45:16.698128 62 slave.cpp:198] Agent started on 1)@172.16.169.4:5051
I1118 21:45:16.698164 62 slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://" --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="docker,mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="10secs" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="5mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname="mesosslave-worker-1" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="false" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --master="zk://zookeeper:2181/mesos" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/mnt/poc/cluster-data/mesosslave-worker"
I1118 21:45:16.699134 62 slave.cpp:519] Agent resources: cpus(*):8; mem(*):63313; disk(*):27048; ports(*):[31000-32000]
I1118 21:45:16.699211 62 slave.cpp:527] Agent attributes: [ ]
I1118 21:45:16.699220 62 slave.cpp:532] Agent hostname: mesosslave-worker-1
I1118 21:45:16.706522 58 state.cpp:57] Recovering state from '/mnt/poc/cluster-data/mesosslave-worker/meta'
I1118 21:45:16.706658 58 state.cpp:697] No checkpointed resources found at '/mnt/poc/cluster-data/mesosslave-worker/meta/resources/resources.info'
I1118 21:45:16.706874 58 state.cpp:100] Failed to find the latest agent from '/mnt/poc/cluster-data/mesosslave-worker/meta'
I1118 21:45:16.707129 58 status_update_manager.cpp:200] Recovering status update manager
I1118 21:45:16.707309 59 docker.cpp:775] Recovering Docker containers
I1118 21:45:16.707326 58 containerizer.cpp:522] Recovering containerizer
2016-11-18 21:45:16,711:49(0x7f049fdf9700):ZOO_INFO@check_events@1728: initiated connection to server [172.16.52.4:2181]
I1118 21:45:16.711513 60 provisioner.cpp:253] Provisioner recovery complete
2016-11-18 21:45:16,717:49(0x7f049fdf9700):ZOO_INFO@check_events@1775: session establishment complete on server [172.16.52.4:2181], sessionId=0x6bf85879547673af, negotiated timeout=10000
I1118 21:45:16.717540 56 group.cpp:349] Group process (group(1)@172.16.169.4:5051) connected to ZooKeeper
I1118 21:45:16.717618 56 group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I1118 21:45:16.717635 56 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
I1118 21:45:16.726135 56 detector.cpp:152] Detected a new leader: (id='8')
I1118 21:45:16.726408 59 group.cpp:706] Trying to get '/mesos/json.info_0000000008' in ZooKeeper
I1118 21:45:16.731709 55 zookeeper.cpp:259] A new leading master ([email protected]:5050) is detected
I1118 21:45:16.782778 62 slave.cpp:4782] Finished recovery
I1118 21:45:16.783553 60 status_update_manager.cpp:174] Pausing sending status updates
I1118 21:45:16.783586 57 slave.cpp:895] New master detected at [email protected]:5050
I1118 21:45:16.783640 57 slave.cpp:916] No credentials provided. Attempting to register without authentication
I1118 21:45:16.783668 57 slave.cpp:927] Detecting new master
I1118 21:46:16.700013 57 slave.cpp:4591] Current disk usage 51.11%. Max allowed age: 2.722231364545590days
I1118 21:46:54.897162 62 slave.cpp:3732] [email protected]:5050 exited
W1118 21:46:54.897258 62 slave.cpp:3737] Master disconnected! Waiting for a new master to be elected
I1118 21:47:04.767259 61 detector.cpp:152] Detected a new leader: (id='9')
I1118 21:47:04.767319 61 group.cpp:706] Trying to get '/mesos/json.info_0000000009' in ZooKeeper
I1118 21:47:04.768859 60 zookeeper.cpp:259] A new leading master ([email protected]:5050) is detected
I1118 21:47:04.768920 60 slave.cpp:895] New master detected at [email protected]:5050
I1118 21:47:04.768929 60 slave.cpp:916] No credentials provided. Attempting to register without authentication
I1118 21:47:04.768939 60 slave.cpp:927] Detecting new master
I1118 21:47:04.768957 60 status_update_manager.cpp:174] Pausing sending status updates
I1118 21:47:16.701213 60 slave.cpp:4591] Current disk usage 51.13%. Max allowed age: 2.721178202893114days
I1118 21:49:13.493881 42 conn.go:133] conn.Send(xid=1479504503, zxid=142, &{Children:[json.info_0000000010 json.info_0000000011 log_replicas log_replicas0000000010 log_replicas0000000011]})
I1118 21:49:13.494000 42 zketcd.go:425] GetChildren(1479505765) = (zxid=143, resp={Children:[]})
I1118 21:49:13.494552 42 conn.go:133] conn.Send(xid=1479505765, zxid=142, &{Children:[]})
I1118 21:49:13.660479 42 server.go:110] zkreq={xid:4738 req:*zetcd.GetChildren2Request:&{Path:/marathon/leader-curator Watch:false}}
I1118 21:49:13.660520 42 zklog.go:78] GetChildren2(4738,{Path:/marathon/leader-curator Watch:false})
I1118 21:49:13.660859 42 zketcd.go:165] GetChildren2(143) = (zxid=4738, resp={Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:13.661333 42 conn.go:133] conn.Send(xid=4738, zxid=142, &{Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:13.661884 42 server.go:110] zkreq={xid:4739 req:*zetcd.GetDataRequest:&{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false}}
I1118 21:49:13.661907 42 zklog.go:43] GetData(4739,{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false})
I1118 21:49:13.662279 42 zketcd.go:312] GetData(4739) = (zxid=143, resp={Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:13.662767 42 conn.go:133] conn.Send(xid=4739, zxid=142, &{Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:14.175323 42 server.go:110] zkreq={xid:4740 req:*zetcd.GetChildren2Request:&{Path:/marathon/leader-curator Watch:false}}
I1118 21:49:14.175350 42 zklog.go:78] GetChildren2(4740,{Path:/marathon/leader-curator Watch:false})
I1118 21:49:14.175721 42 zketcd.go:165] GetChildren2(143) = (zxid=4740, resp={Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:14.176448 42 conn.go:133] conn.Send(xid=4740, zxid=142, &{Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:14.177547 42 server.go:110] zkreq={xid:4741 req:*zetcd.GetDataRequest:&{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false}}
I1118 21:49:14.177575 42 zklog.go:43] GetData(4741,{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false})
I1118 21:49:14.178028 42 zketcd.go:312] GetData(4741) = (zxid=143, resp={Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:14.178265 42 conn.go:133] conn.Send(xid=4741, zxid=142, &{Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:14.700613 42 server.go:110] zkreq={xid:4742 req:*zetcd.GetChildren2Request:&{Path:/marathon/leader-curator Watch:false}}
I1118 21:49:14.700677 42 zklog.go:78] GetChildren2(4742,{Path:/marathon/leader-curator Watch:false})
I1118 21:49:14.701415 42 zketcd.go:165] GetChildren2(143) = (zxid=4742, resp={Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:14.703378 42 conn.go:133] conn.Send(xid=4742, zxid=142, &{Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:14.703934 42 server.go:110] zkreq={xid:4743 req:*zetcd.GetDataRequest:&{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false}}
I1118 21:49:14.703965 42 zklog.go:43] GetData(4743,{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false})
I1118 21:49:14.704545 42 zketcd.go:312] GetData(4743) = (zxid=143, resp={Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:14.705244 42 conn.go:133] conn.Send(xid=4743, zxid=142, &{Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:15.230430 42 server.go:110] zkreq={xid:4744 req:*zetcd.GetChildren2Request:&{Path:/marathon/leader-curator Watch:false}}
I1118 21:49:15.230481 42 zklog.go:78] GetChildren2(4744,{Path:/marathon/leader-curator Watch:false})
I1118 21:49:15.231662 42 zketcd.go:165] GetChildren2(143) = (zxid=4744, resp={Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:15.232142 42 conn.go:133] conn.Send(xid=4744, zxid=142, &{Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:15.232605 42 server.go:110] zkreq={xid:4745 req:*zetcd.GetDataRequest:&{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false}}
I1118 21:49:15.232646 42 zklog.go:43] GetData(4745,{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false})
I1118 21:49:15.233181 42 zketcd.go:312] GetData(4745) = (zxid=143, resp={Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:15.234508 42 conn.go:133] conn.Send(xid=4745, zxid=142, &{Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:15.766557 42 server.go:110] zkreq={xid:4746 req:*zetcd.GetChildren2Request:&{Path:/marathon/leader-curator Watch:false}}
I1118 21:49:15.766609 42 zklog.go:78] GetChildren2(4746,{Path:/marathon/leader-curator Watch:false})
I1118 21:49:15.767609 42 zketcd.go:165] GetChildren2(143) = (zxid=4746, resp={Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:15.769301 42 conn.go:133] conn.Send(xid=4746, zxid=142, &{Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:15.770105 42 server.go:110] zkreq={xid:4747 req:*zetcd.GetDataRequest:&{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false}}
I1118 21:49:15.770147 42 zklog.go:43] GetData(4747,{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false})
I1118 21:49:15.771021 42 zketcd.go:312] GetData(4747) = (zxid=143, resp={Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:15.771336 42 conn.go:133] conn.Send(xid=4747, zxid=142, &{Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:16.272710 42 server.go:110] zkreq={xid:-2 req:*zetcd.PingRequest:&{}}
I1118 21:49:16.272750 42 zklog.go:73] Ping(-2,{})
I1118 21:49:16.273108 42 conn.go:133] conn.Send(xid=-2, zxid=142, &{})
I1118 21:49:16.287996 42 server.go:110] zkreq={xid:4748 req:*zetcd.GetChildren2Request:&{Path:/marathon/leader-curator Watch:false}}
I1118 21:49:16.288026 42 zklog.go:78] GetChildren2(4748,{Path:/marathon/leader-curator Watch:false})
I1118 21:49:16.288454 42 zketcd.go:165] GetChildren2(143) = (zxid=4748, resp={Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:16.289076 42 conn.go:133] conn.Send(xid=4748, zxid=142, &{Children:[_c_934bdef9-90ff-42d3-a10a-4bd4f11d2913-latch-0000000005 _c_9640b680-f8fc-4e08-bf10-4465125f95c0-latch-0000000002 _c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 _c_d4d75d06-4b20-42ff-b537-876c499e5994-latch-0000000004] Stat:{Czxid:28 Mzxid:43 Ctime:1479504464966800 Mtime:1479504482688506 Version:2 Cversion:8 Aversion:0 EphemeralOwner:0 DataLength:0 NumChildren:4 Pzxid:46}})
I1118 21:49:16.289585 42 server.go:110] zkreq={xid:4749 req:*zetcd.GetDataRequest:&{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false}}
I1118 21:49:16.289605 42 zklog.go:43] GetData(4749,{Path:/marathon/leader-curator/_c_b01520e1-6cad-48de-81de-23e873dc0fc8-latch-0000000000 Watch:false})
I1118 21:49:16.289953 42 zketcd.go:312] GetData(4749) = (zxid=143, resp={Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:16.290137 42 conn.go:133] conn.Send(xid=4749, zxid=142, &{Data:[109 97 114 97 116 104 111 110 109 97 115 116 101 114 45 109 97 115 116 101 114 45 51 58 56 48 56 48] Stat:{Czxid:29 Mzxid:29 Ctime:1479504464993953 Mtime:1479504464993953 Version:0 Cversion:0 Aversion:0 EphemeralOwner:7780065634413853912 DataLength:28 NumChildren:0 Pzxid:29}})
I1118 21:49:16.811115 42 server.go:110] zkreq={xid:4750 req:*zetcd.GetChildren2Request:&{Path:/marathon/leader-curator Watch:false}}
I1118 21:49:16.811164 42 zklog.go:78] GetChildren2(4750,{Path:/marathon/leader-curator Watch:false})
...