Comments (5)
Apologizes for posting the note above too quickly, it seems that I did not have etcd configured properly. After fixing that I can confirm it is working as expected on both Mesos 1.2.0
and 1.3.0
.
from zetcd.
@wegel @heyitsanthony @xiang90 @wrouesnel ^^
from zetcd.
@haoyixin could you set the mesos log level to debug and post a sample of the init and the timeout/expiry? Also are you using the current master branch head?
from zetcd.
I was able to reproduce similar timeouts as @haoyixin on Mesos 1.2.0 and 1.3.0.
Any thoughts?
zetcd:
zetcd -zkaddr localhost:2181 -endpoints localhost:2379 -v 9 -logtostderr
I0711 22:57:26.068852 12676 server.go:128] accepted remote connection "127.0.0.1:32894"
I0711 22:57:26.069031 12676 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
I0711 22:57:26.072130 12676 server.go:128] accepted remote connection "127.0.0.1:32896"
I0711 22:57:26.072194 12676 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
I0711 22:57:26.072490 12676 server.go:128] accepted remote connection "127.0.0.1:32898"
I0711 22:57:26.072706 12676 server.go:128] accepted remote connection "127.0.0.1:32900"
I0711 22:57:26.072746 12676 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
I0711 22:57:26.073050 12676 authconn.go:53] auth(&{ProtocolVersion:0 LastZxidSeen:0 TimeOut:10000 SessionID:0 Passwd:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]})
mesos-master
Launched with a container image I maintain.
docker run --rm -ti -e MESOS_LOGGING_LEVEL=INFO --net host quay.io/vektorcloud/mesos mesos-master
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0712 01:57:26.044873 7 main.cpp:278] Build: 2017-04-20 04:43:12 by
I0712 01:57:26.044936 7 main.cpp:279] Version: 1.2.0
I0712 01:57:26.045655 7 logging.cpp:194] INFO level logging started!
I0712 01:57:26.045845 7 main.cpp:385] Using 'HierarchicalDRF' allocator
I0712 01:57:26.064152 7 replica.cpp:778] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
2017-07-12 01:57:26,065:7(0x7f3920d21ab0):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2017-07-12 01:57:26,065:7(0x7f3920d21ab0):ZOO_INFO@log_env@730: Client environment:host.name=
2017-07-12 01:57:26,065:7(0x7f3920d21ab0):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2017-07-12 01:57:26,065:7(0x7f3920d21ab0):ZOO_INFO@log_env@738: Client environment:os.arch=4.11.9-1-ARCH
2017-07-12 01:57:26,065:7(0x7f3920d21ab0):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP PREEMPT Wed Jul 5 18:23:08 CEST 2017
2017-07-12 01:57:26,065:7(0x7f3920d21ab0):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2017-07-12 01:57:26,066:7(0x7f3920d21ab0):ZOO_INFO@log_env@755: Client environment:user.home=/root
2017-07-12 01:57:26,066:7(0x7f3920d21ab0):ZOO_INFO@log_env@767: Client environment:user.dir=/
2017-07-12 01:57:26,066:7(0x7f3920d21ab0):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=localhost:2181 sessionTimeout=10000 watcher=0x7f391facd680 sessionId=0 sessionPasswd=<null> context=0x55f19c452678 flags=0
2017-07-12 01:57:26,066:7(0x7f3920cb3ab0):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2017-07-12 01:57:26,066:7(0x7f3920cb3ab0):ZOO_INFO@log_env@730: Client environment:host.name=
2017-07-12 01:57:26,066:7(0x7f3920cb3ab0):ZOO_INFO@log_env@737: Client environment:os.name=Linux
I0712 01:57:26.066527 12 log.cpp:107] Attempting to join replica to ZooKeeper group
2017-07-12 01:57:26,067:7(0x7f3920cb3ab0):ZOO_INFO@log_env@738: Client environment:os.arch=4.11.9-1-ARCH
I0712 01:57:26.067672 13 recover.cpp:451] Starting replica recovery
2017-07-12 01:57:26,068:7(0x7f3920c4bab0):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
2017-07-12 01:57:26,068:7(0x7f3920cb3ab0):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP PREEMPT Wed Jul 5 18:23:08 CEST 2017
2017-07-12 01:57:26,069:7(0x7f3920cb3ab0):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2017-07-12 01:57:26,069:7(0x7f3920cb3ab0):ZOO_INFO@log_env@755: Client environment:user.home=/root
2017-07-12 01:57:26,069:7(0x7f3920cb3ab0):ZOO_INFO@log_env@767: Client environment:user.dir=/
2017-07-12 01:57:26,069:7(0x7f3920cb3ab0):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=localhost:2181 sessionTimeout=10000 watcher=0x7f391facd680 sessionId=0 sessionPasswd=<null> context=0x55f19c463a78 flags=0
I0712 01:57:26.069798 13 recover.cpp:477] Replica is in EMPTY status
2017-07-12 01:57:26,070:7(0x7f3920d37ab0):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2017-07-12 01:57:26,070:7(0x7f3920d37ab0):ZOO_INFO@log_env@730: Client environment:host.name=
2017-07-12 01:57:26,070:7(0x7f3920d37ab0):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2017-07-12 01:57:26,070:7(0x7f3920d37ab0):ZOO_INFO@log_env@738: Client environment:os.arch=4.11.9-1-ARCH
2017-07-12 01:57:26,070:7(0x7f3920d37ab0):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP PREEMPT Wed Jul 5 18:23:08 CEST 2017
2017-07-12 01:57:26,070:7(0x7f3920d37ab0):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2017-07-12 01:57:26,070:7(0x7f3920d37ab0):ZOO_INFO@log_env@755: Client environment:user.home=/root
2017-07-12 01:57:26,070:7(0x7f3920d37ab0):ZOO_INFO@log_env@767: Client environment:user.dir=/
2017-07-12 01:57:26,070:7(0x7f3920d37ab0):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=localhost:2181 sessionTimeout=10000 watcher=0x7f391facd680 sessionId=0 sessionPasswd=<null> context=0x55f19c473c58 flags=0
2017-07-12 01:57:26,070:7(0x7f3920cf5ab0):ZOO_INFO@log_env@726: Client environment:zookeeper.version=zookeeper C client 3.4.8
2017-07-12 01:57:26,070:7(0x7f3920cf5ab0):ZOO_INFO@log_env@730: Client environment:host.name=
2017-07-12 01:57:26,070:7(0x7f3920cf5ab0):ZOO_INFO@log_env@737: Client environment:os.name=Linux
2017-07-12 01:57:26,070:7(0x7f3920cf5ab0):ZOO_INFO@log_env@738: Client environment:os.arch=4.11.9-1-ARCH
2017-07-12 01:57:26,070:7(0x7f3920cf5ab0):ZOO_INFO@log_env@739: Client environment:os.version=#1 SMP PREEMPT Wed Jul 5 18:23:08 CEST 2017
2017-07-12 01:57:26,070:7(0x7f3920cf5ab0):ZOO_INFO@log_env@747: Client environment:user.name=(null)
2017-07-12 01:57:26,070:7(0x7f3920cf5ab0):ZOO_INFO@log_env@755: Client environment:user.home=/root
2017-07-12 01:57:26,070:7(0x7f3920cf5ab0):ZOO_INFO@log_env@767: Client environment:user.dir=/
2017-07-12 01:57:26,070:7(0x7f3920cf5ab0):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=localhost:2181 sessionTimeout=10000 watcher=0x7f391facd680 sessionId=0 sessionPasswd=<null> context=0x55f19c482ad8 flags=0
I0712 01:57:26.071519 16 replica.cpp:675] Replica in EMPTY status received a broadcasted recover request from __req_res__(1)@127.0.0.1:5050
I0712 01:57:26.071813 12 recover.cpp:197] Received a recover response from a replica in EMPTY status
2017-07-12 01:57:26,071:7(0x7f3920bf3ab0):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
I0712 01:57:26.072665 16 recover.cpp:568] Updating replica status to STARTING
2017-07-12 01:57:26,072:7(0x7f3920bc7ab0):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
2017-07-12 01:57:26,073:7(0x7f3920c1fab0):ZOO_INFO@check_events@1728: initiated connection to server [127.0.0.1:2181]
I0712 01:57:26.075845 7 master.cpp:383] Master 4af6b429-1272-445f-a651-8ce52f8dcf89 (localhost.localdomain) started on 127.0.0.1:5050
I0712 01:57:26.075912 7 master.cpp:385] Flags at startup: --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="false" --authenticate_frameworks="false" --authenticate_http_frameworks="false" --authenticate_http_readonly="false" --authenticate_http_readwrite="false" --authenticators="crammd5" --authorizers="local" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --log_dir="/var/run/mesos/log" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --max_unreachable_tasks_per_framework="1000" --quiet="false" --quorum="1" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="20secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/usr/local/share/mesos/webui" --work_dir="/var/run/mesos" --zk_session_timeout="10secs"
W0712 01:57:26.076223 7 master.cpp:388]
**************************************************
Master bound to loopback interface! Cannot communicate with remote schedulers or agents. You might want to set '--ip' flag to a routable IP address.
**************************************************
I0712 01:57:26.076459 7 master.cpp:437] Master allowing unauthenticated frameworks to register
I0712 01:57:26.076480 7 master.cpp:451] Master allowing unauthenticated agents to register
I0712 01:57:26.076498 7 master.cpp:465] Master allowing HTTP frameworks to register without authentication
I0712 01:57:26.076517 7 master.cpp:507] Using default 'crammd5' authenticator
W0712 01:57:26.076536 7 authenticator.cpp:512] No credentials provided, authentication requests will be refused
I0712 01:57:26.076560 7 authenticator.cpp:519] Initializing server SASL
I0712 01:57:26.078598 15 replica.cpp:322] Persisted replica status to STARTING
I0712 01:57:26.078807 17 recover.cpp:477] Replica is in STARTING status
I0712 01:57:26.079367 17 replica.cpp:675] Replica in STARTING status received a broadcasted recover request from __req_res__(2)@127.0.0.1:5050
I0712 01:57:26.079587 11 recover.cpp:197] Received a recover response from a replica in STARTING status
I0712 01:57:26.079692 15 recover.cpp:568] Updating replica status to VOTING
I0712 01:57:26.081921 16 replica.cpp:322] Persisted replica status to VOTING
I0712 01:57:26.082012 16 recover.cpp:582] Successfully joined the Paxos group
I0712 01:57:26.085460 14 master.cpp:2058] Successfully attached file '/var/run/mesos/log/mesos-master.INFO'
I0712 01:57:26.085683 14 contender.cpp:152] Joining the ZK group
2017-07-12 01:57:32,737:7(0x7f3920c4bab0):ZOO_ERROR@handle_socket_error_msg@1666: Socket [127.0.0.1:2181] zk retcode=-7, errno=110(Operation timed out): connection to 127.0.0.1:2181 timed out (exceeded timeout by 2ms)
2017-07-12 01:57:32,737:7(0x7f3920c4bab0):ZOO_ERROR@handle_socket_error_msg@1722: Socket [::1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
2017-07-12 01:57:32,738:7(0x7f3920c1fab0):ZOO_ERROR@handle_socket_error_msg@1666: Socket [127.0.0.1:2181] zk retcode=-7, errno=110(Operation timed out): connection to 127.0.0.1:2181 timed out (exceeded timeout by 0ms)
2017-07-12 01:57:32,738:7(0x7f3920c1fab0):ZOO_ERROR@handle_socket_error_msg@1722: Socket [::1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
2017-07-12 01:57:32,739:7(0x7f3920bf3ab0):ZOO_ERROR@handle_socket_error_msg@1666: Socket [127.0.0.1:2181] zk retcode=-7, errno=110(Operation timed out): connection to 127.0.0.1:2181 timed out (exceeded timeout by 1ms)
2017-07-12 01:57:32,739:7(0x7f3920bc7ab0):ZOO_ERROR@handle_socket_error_msg@1666: Socket [127.0.0.1:2181] zk retcode=-7, errno=110(Operation timed out): connection to 127.0.0.1:2181 timed out (exceeded timeout by 0ms)
2017-07-12 01:57:32,739:7(0x7f3920bf3ab0):ZOO_ERROR@handle_socket_error_msg@1722: Socket [::1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
2017-07-12 01:57:32,739:7(0x7f3920bc7ab0):ZOO_ERROR@handle_socket_error_msg@1722: Socket [::1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
from zetcd.
@haoyixin this is tested in #76; mesos boots OK. Possibly an etcd misconfiguration. Please reopen with more information about the etcd/zetcd cluster configuration (e.g., etcd server logs, zetcd logs, command line flags) if the problem persists. Thanks!
from zetcd.
Related Issues (20)
- [Feature] sd_notify for systemd notification
- [Question/Feature] Etcd base key for zk schema
- 4 letter word commands leaking memory
- zookeeper exception, KeeperErrorCode = apiError
- GetChildren doesn't get children HOT 1
- Why separate keys for each metadata parameter?
- zetcd ha
- Kafka Unable to connect to zookeeper server
- How to build the project source code after change it?
- when zetcd should release v1.0.0? and is it have any tool to migrate data from zk to etcd?
- Need a new release to get rid of known vulnerabilities in the latest image HOT 1
- Performance benchmarking
- go get error:gcc_darwin_amd64.c:62:15: error: unknown token in expression HOT 1
- Proposal to mark etcd-io/zetc as public archive HOT 7
- etcd3 STM client does not Support Range fetch over keys? HOT 1
- zetcd performance HOT 3
- New version/future plans? HOT 2
- Memory/leak issue HOT 9
- ZK MultiOp provided by zetcd not working with Kafka 2.1.0 HOT 1
- Mtime and Ctime off by a factor of 1000 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zetcd.