Comments (16)
miguno
commented on August 26, 2024
Do you have any further information? Could you set the log level to DEBUG (log4j.properties and, if needed, also logback-test.xml) and re-run the tests?
from kafka-storm-starter.
mtrienis
commented on August 26, 2024
Hi, I have the same problem, and I would really appreciate some help as I am fairly new to Scala
The debug output is below
[2014-08-26 21:13:51,185] DEBUG Delivering event #1 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-26 21:13:51,189] DEBUG Delivering event #2 ZkEvent[Data of /controller_epoch changed sent to kafka.controller.ControllerEpochListener@19cc4c06] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-26 21:13:51,197] DEBUG Delivering event #2 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-26 21:13:51,197] DEBUG Delivering event #3 ZkEvent[Children of /brokers/ids changed sent to kafka.controller.ReplicaStateMachine$BrokerChangeListener@414e786e] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-26 21:13:51,246] DEBUG Creating new ZookKeeper instance to connect to 127.0.0.1:39094. (org.I0Itec.zkclient.ZkConnection)
[2014-08-26 21:13:51,247] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-08-26 21:13:51,251] DEBUG Awaiting connection to Zookeeper server (org.I0Itec.zkclient.ZkClient)
[2014-08-26 21:13:51,251] DEBUG Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-26 21:13:51,254] DEBUG Received event: WatchedEvent state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
[2014-08-26 21:13:51,254] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-08-26 21:13:51,254] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-26 21:13:51,255] DEBUG State is SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-26 21:13:51,415] DEBUG Delivering event #3 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-26 21:13:51,415] DEBUG Delivering event #4 ZkEvent[Children of /brokers/topics changed sent to kafka.controller.PartitionStateMachine$TopicChangeListener@15c0f238] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-26 21:13:51,458] DEBUG Subscribed data changes for /brokers/topics/testing (org.I0Itec.zkclient.ZkClient)
[2014-08-26 21:13:51,497] DEBUG Subscribed data changes for /brokers/topics/testing (org.I0Itec.zkclient.ZkClient)
[2014-08-26 21:13:51,717] DEBUG Delivering event #4 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-26 21:13:53,410] ERROR Closing socket for /127.0.0.1 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at kafka.utils.Utils$.read(Utils.scala:375)
at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Processor.read(SocketServer.scala:347)
at kafka.network.Processor.run(SocketServer.scala:245)
at java.lang.Thread.run(Thread.java:744)
[2014-08-26 21:13:53,467] DEBUG Closing ZkClient... (org.I0Itec.zkclient.ZkClient)
[2014-08-26 21:13:53,467] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-08-26 21:13:53,467] DEBUG Closing ZooKeeper connected to 127.0.0.1:39094 (org.I0Itec.zkclient.ZkConnection)
[2014-08-26 21:13:53,471] DEBUG Received event: WatchedEve
from kafka-storm-starter.
miguno
commented on August 26, 2024
@trienism Thanks for the logging output. However there's nothing that jumps at me here to pinpoint the problem. The single ERROR in the output is actually ok, as it denotes that the Kafka client disconnected (which is an expected event during the tests):
[2014-08-26 21:13:53,410] ERROR Closing socket for /127.0.0.1 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
The remaining output looks ok either.
from kafka-storm-starter.
miguno
commented on August 26, 2024
@trienism Are you also working on Debian / Ubuntu boxes like @faustroll ?
from kafka-storm-starter.
mtrienis
commented on August 26, 2024
@miguno Thanks for your quick reply. I am using a VM with CentOS release 6.4 (Final) installed. There a couple other errors as well, I'll post them later today.
from kafka-storm-starter.
miguno
commented on August 26, 2024
FYI: I successfully tested kafka-storm-starter via ./sbt test on a VM running Ubuntu 14.04 LTS for both Open JDK 6 (1.6.0_32) and Open JDK 7 (1.7.0_65), with multiple test runs each. No failures whatsoever.
from kafka-storm-starter.
miguno
commented on August 26, 2024
Addendum: I can confirm this bug with Open JDK 7 and Oracle JDK 7 on Ubuntu 14.04 once I make the following change to build.sbt and then run ./sbt clean test:
$ git diff
diff --git a/build.sbt b/build.sbt
index 567af83..17e7dcf 100644
--- a/build.sbt
+++ b/build.sbt
@@ -73,8 +73,8 @@ javaOptions ++= Seq(
"-Djava.net.preferIPv4Stack=true")
javacOptions in Compile ++= Seq(
- "-source", "1.6",
- "-target", "1.6",
+ "-source", "1.7",
+ "-target", "1.7",
"-Xlint:unchecked",
"-Xlint:deprecation")Interestingly I do not see this error on e.g. CentOS 6 under the same conditions.
from kafka-storm-starter.
mtrienis
commented on August 26, 2024
Here is my Java version
[root@vm quickstart]# java -version
java version "1.6.0_24"
OpenJDK Runtime Environment (IcedTea6 1.11.14) (rhel-1.65.1.11.14.el6_4-x86_64)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
And the output from running unit tests
[root@vm kafka-storm-starter]# ./sbt clean test
[info] Loading project definition from /opt/kafka-storm-starter/project
[info] Updating {file:/opt/kafka-storm-starter/project/}kafka-storm-starter-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Set current project to kafka-storm-starter (in build file:/opt/kafka-storm-starter/)
[success] Total time: 0 s, completed 27-Aug-2014 3:57:02 PM
[info] Updating {file:/opt/kafka-storm-starter/}kafka-storm-starter...
[info] Resolving org.scala-lang#scala-library;2.10.3 ...
[info] Done updating.
[info] Avro compiler using stringType=String
[info] Compiling Avro schema /opt/kafka-storm-starter/src/main/avro/twitter.avsc
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[info] Compiling 10 Scala sources and 1 Java source to /opt/kafka-storm-starter/target/scala-2.10/classes...
[info] Compiling 10 Scala sources to /opt/kafka-storm-starter/target/scala-2.10/test-classes...
[info] KafkaSpec:
[info] Kafka
[info] - should synchronously send and receive a Tweet in Avro format
[info] + Given a ZooKeeper instance
[info] + And a Kafka broker instance
[info] + And some tweets
[info] + And a single-threaded Kafka consumer group
[info] + When I start a synchronous Kafka producer that sends the tweets in Avro binary format
[info] + Then the consumer app should receive the tweets
[info] - should asynchronously send and receive a Tweet in Avro format
[info] + Given a ZooKeeper instance
[info] + And a Kafka broker instance
[info] + And some tweets
[info] + And a single-threaded Kafka consumer group
[info] + When I start an asynchronous Kafka producer that sends the tweets in Avro binary format
[info] + Then the consumer app should receive the tweets
[info] StormSpec:
[info] Storm
[info] - should start a local cluster
[info] + Given no cluster
[info] + When I start a LocalCluster instance
[info] + Then the local cluster should start properly
[info] - should run a basic topology
[info] + Given a local cluster
[info] + And a wordcount topology
[info] + And the input words alice, bob, joe, alice
[info] + When I submit the topology
[info] + Then the topology should properly count the words
[info] KafkaStormSpec:
[info] Feature: AvroDecoderBolt[T]
[info] Scenario: User creates a Storm topology that uses AvroDecoderBolt *** FAILED ***
[info] SynchronizedQueue() was not equal to List({"username": "ANY_USER_1", "text": "ANY_TEXT_1", "timestamp": 1409169521}, {"username": "ANY_USER_2", "text": "ANY_TEXT_2", "timestamp": 0}, {"username": "ANY_USER_3", "text": "ANY_TEXT_3", "timestamp": 1234}) (KafkaStormSpec.scala:329)
[info] Given a ZooKeeper instance
[info] And a Kafka broker instance
[info] And a Storm topology that uses AvroDecoderBolt and that reads tweets from topic testing-input and writes them as-is to topic testing-output
[info] And some tweets
[info] And a synchronous Kafka producer app that writes to the topic testing-input
[info] And a single-threaded Kafka consumer app that reads from topic testing-output and Avro-decodes the incoming data
[info] And a Storm topology configuration that registers an Avro Kryo decorator for Tweet
[info] When I run the Storm topology
[info] And I Avro-encode the tweets and use the Kafka producer app to sent them to Kafka
[info] Then the Kafka consumer app should receive the original tweets from the Storm topology
[info] Feature: AvroScheme[T] for Kafka spout
And then running with debug reporting....
[info] KafkaStormSpec:
[info] Feature: AvroDecoderBolt[T]
[2014-08-27 16:10:11,577] DEBUG Creating new ZookKeeper instance to connect to 127.0.0.1:58629. (org.I0Itec.zkclient.ZkConnection)
[2014-08-27 16:10:11,577] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,586] DEBUG Awaiting connection to Zookeeper server (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,586] DEBUG Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,601] DEBUG Received event: WatchedEvent state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,601] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,601] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,601] DEBUG State is SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,693] DEBUG Subscribed data changes for /controller_epoch (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,736] DEBUG Subscribed data changes for /controller (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,738] DEBUG Received event: WatchedEvent state:SyncConnected type:NodeCreated path:/controller (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,739] DEBUG New event: ZkEvent[Data of /controller changed sent to kafka.server.ZookeeperLeaderElector$LeaderChangeListener@375dff59] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,739] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,739] DEBUG Delivering event #5 ZkEvent[Data of /controller changed sent to kafka.server.ZookeeperLeaderElector$LeaderChangeListener@375dff59] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,742] DEBUG Received event: WatchedEvent state:SyncConnected type:NodeCreated path:/controller_epoch (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,742] DEBUG New event: ZkEvent[Data of /controller_epoch changed sent to kafka.controller.ControllerEpochListener@7c274874] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,742] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,743] DEBUG Subscribed data changes for /admin/reassign_partitions (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,745] DEBUG Subscribed data changes for /admin/preferred_replica_election (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,761] DEBUG Delivering event #5 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,761] DEBUG Delivering event #6 ZkEvent[Data of /controller_epoch changed sent to kafka.controller.ControllerEpochListener@7c274874] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,767] DEBUG Received event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/brokers/ids (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,767] DEBUG New event: ZkEvent[Children of /brokers/ids changed sent to kafka.controller.ReplicaStateMachine$BrokerChangeListener@7705f759] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,767] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,770] DEBUG Delivering event #6 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,770] DEBUG Delivering event #7 ZkEvent[Children of /brokers/ids changed sent to kafka.controller.ReplicaStateMachine$BrokerChangeListener@7705f759] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,779] DEBUG Creating new ZookKeeper instance to connect to 127.0.0.1:58629. (org.I0Itec.zkclient.ZkConnection)
[2014-08-27 16:10:11,779] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,790] DEBUG Awaiting connection to Zookeeper server (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,790] DEBUG Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,790] DEBUG Received event: WatchedEvent state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,790] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,790] DEBUG State is SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,800] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,804] DEBUG Received event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/brokers/topics (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,804] DEBUG New event: ZkEvent[Children of /brokers/topics changed sent to kafka.controller.PartitionStateMachine$TopicChangeListener@1ae40317] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,804] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,836] DEBUG Delivering event #7 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,836] DEBUG Delivering event #8 ZkEvent[Children of /brokers/topics changed sent to kafka.controller.PartitionStateMachine$TopicChangeListener@1ae40317] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:11,871] DEBUG Subscribed data changes for /brokers/topics/testing-input (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,872] DEBUG Subscribed data changes for /brokers/topics/testing-output (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:11,951] DEBUG Delivering event #8 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:12,004] DEBUG Creating new ZookKeeper instance to connect to 127.0.0.1:58629. (org.I0Itec.zkclient.ZkConnection)
[2014-08-27 16:10:12,005] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:12,013] DEBUG Awaiting connection to Zookeeper server (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:12,013] DEBUG Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:12,013] DEBUG Received event: WatchedEvent state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:12,013] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:12,013] DEBUG State is SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:12,014] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:12,037] DEBUG Subscribed data changes for /brokers/topics/testing-output (org.I0Itec.zkclient.ZkClient)
[info] Scenario: User creates a Storm topology that uses AvroDecoderBolt *** FAILED ***
[info] SynchronizedQueue() was not equal to List({"username": "ANY_USER_1", "text": "ANY_TEXT_1", "timestamp": 1409170170}, {"username": "ANY_USER_2", "text": "ANY_TEXT_2", "timestamp": 0}, {"username": "ANY_USER_3", "text": "ANY_TEXT_3", "timestamp": 1234}) (KafkaStormSpec.scala:329)
[info] Given a ZooKeeper instance
[info] And a Kafka broker instance
[info] And a Storm topology that uses AvroDecoderBolt and that reads tweets from topic testing-input and writes them as-is to topic testing-output
[info] And some tweets
[info] And a synchronous Kafka producer app that writes to the topic testing-input
[info] And a single-threaded Kafka consumer app that reads from topic testing-output and Avro-decodes the incoming data
[info] And a Storm topology configuration that registers an Avro Kryo decorator for Tweet
[info] When I run the Storm topology
[info] And I Avro-encode the tweets and use the Kafka producer app to sent them to Kafka
[info] Then the Kafka consumer app should receive the original tweets from the Storm topology
[2014-08-27 16:10:26,321] DEBUG Closing ZkClient... (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:26,321] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:26,321] DEBUG Closing ZooKeeper connected to 127.0.0.1:58629 (org.I0Itec.zkclient.ZkConnection)
[2014-08-27 16:10:26,331] DEBUG Received event: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/brokers/ids (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:26,331] DEBUG ignoring event '{NodeChildrenChanged | /brokers/ids}' since shutdown triggered (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:26,332] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:26,332] DEBUG Received event: WatchedEvent state:SyncConnected type:NodeDeleted path:/controller (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:26,332] DEBUG ignoring event '{NodeDeleted | /controller}' since shutdown triggered (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:26,332] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:26,332] DEBUG Closing ZkClient...done (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:26,334] DEBUG Closing ZkClient... (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:10:26,334] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:10:26,334] DEBUG Closing ZooKeeper connected to 127.0.0.1:58629 (org.I0Itec.zkclient.ZkConnection)
[2014-08-27 16:10:26,336] DEBUG Closing ZkClient...done (org.I0Itec.zkclient.ZkClient)
[info] Feature: AvroScheme[T] for Kafka spout
I am going to try and build on a new Centos VM and see if that helps
from kafka-storm-starter.
miguno
commented on August 26, 2024
@faustroll : Have you run your tests in a VM or on bare metal machines?
from kafka-storm-starter.
miguno
commented on August 26, 2024
Could you try to disable IPv6 in your VM, reboot, and then re-run the tests? In my testing this solved the issue for an Ubuntu 14.04 VM w/ Java 7.
You can disable IPv6 via sysctl:
# Add the lines below to /etc/sysctl.conf (tested on Ubuntu 14.04)
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
Reboot the VM (don't just do sysctl -p) and then re-run the tests.
from kafka-storm-starter.
mtrienis
commented on August 26, 2024
Disabling firewall
/etc/init.d/ip6tables stop
chkconfig ip6tables off
rebooted
[root@vm opt]# /etc/init.d/ip6tables status
ip6tables: Firewall is not running.
same failure
[2014-08-27 16:49:27,472] DEBUG New event: ZkEvent[Children of /brokers/topics changed sent to kafka.controller.PartitionStateMachine$TopicChangeListener@1116ce6e] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:49:27,472] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,525] DEBUG Delivering event #7 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:49:27,525] DEBUG Delivering event #8 ZkEvent[Children of /brokers/topics changed sent to kafka.controller.PartitionStateMachine$TopicChangeListener@1116ce6e] (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:49:27,584] DEBUG Subscribed data changes for /brokers/topics/testing-input (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,585] DEBUG Subscribed data changes for /brokers/topics/testing-output (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,698] DEBUG Delivering event #8 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:49:27,787] DEBUG Creating new ZookKeeper instance to connect to 127.0.0.1:41114. (org.I0Itec.zkclient.ZkConnection)
[2014-08-27 16:49:27,788] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:49:27,793] DEBUG Awaiting connection to Zookeeper server (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,793] DEBUG Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,793] DEBUG Received event: WatchedEvent state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,795] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,795] DEBUG State is SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,796] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,829] DEBUG Subscribed data changes for /brokers/topics/testing-output (org.I0Itec.zkclient.ZkClient)
[info] Scenario: User creates a Storm topology that uses AvroDecoderBolt *** FAILED ***
[info] SynchronizedQueue() was not equal to List({"username": "ANY_USER_1", "text": "ANY_TEXT_1", "timestamp": 1409172522}, {"username": "ANY_USER_2", "text": "ANY_TEXT_2", "timestamp": 0}, {"username": "ANY_USER_3", "text": "ANY_TEXT_3", "timestamp": 1234}) (KafkaStormSpec.scala:329)
[info] Given a ZooKeeper instance
[info] And a Kafka broker instance
[info] And a Storm topology that uses AvroDecoderBolt and that reads tweets from topic testing-input and writes them as-is to topic testing-output
[info] And some tweets
[info] And a synchronous Kafka producer app that writes to the topic testing-input
[info] And a single-threaded Kafka consumer app that reads from topic testing-output and Avro-decodes the incoming data
[info] And a Storm topology configuration that registers an Avro Kryo decorator for Tweet
[info] When I run the Storm topology
[info] And I Avro-encode the tweets and use the Kafka producer app to sent them to Kafka
[info] Then the Kafka consumer app should receive the original tweets from the Storm topology
from kafka-storm-starter.
miguno
commented on August 26, 2024
Note that you disabled some firewall functionality -- but you did not disable IPv6 networking. See my instructions above, or e.g. http://wiki.centos.org/FAQ/CentOS6#head-d47139912868bcb9d754441ecb6a8a10d41781df for CentOS.
On 27.08.2014, at 22:50, trienism [email protected] wrote:
Disabling firewall
/etc/init.d/ip6tables stop
chkconfig ip6tables off
rebooted[root@vm opt]# /etc/init.d/ip6tables status
ip6tables: Firewall is not running.
same failure[2014-08-27 16:49:27,472] DEBUG New event: ZkEventChildren of /brokers/topics changed sent to kafka.controller.PartitionStateMachine$TopicChangeListener@1116ce6e
[2014-08-27 16:49:27,472] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,525] DEBUG Delivering event #7 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:49:27,525] DEBUG Delivering event #8 ZkEventChildren of /brokers/topics changed sent to kafka.controller.PartitionStateMachine$TopicChangeListener@1116ce6e
[2014-08-27 16:49:27,584] DEBUG Subscribed data changes for /brokers/topics/testing-input (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,585] DEBUG Subscribed data changes for /brokers/topics/testing-output (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,698] DEBUG Delivering event #8 done (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:49:27,787] DEBUG Creating new ZookKeeper instance to connect to 127.0.0.1:41114. (org.I0Itec.zkclient.ZkConnection)
[2014-08-27 16:49:27,788] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2014-08-27 16:49:27,793] DEBUG Awaiting connection to Zookeeper server (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,793] DEBUG Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,793] DEBUG Received event: WatchedEvent state:SyncConnected type:None path:null (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,795] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,795] DEBUG State is SyncConnected (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,796] DEBUG Leaving process event (org.I0Itec.zkclient.ZkClient)
[2014-08-27 16:49:27,829] DEBUG Subscribed data changes for /brokers/topics/testing-output (org.I0Itec.zkclient.ZkClient)
[info] Scenario: User creates a Storm topology that uses AvroDecoderBolt *** FAILED ***
[info] SynchronizedQueue() was not equal to List({"username": "ANY_USER_1", "text": "ANY_TEXT_1", "timestamp": 1409172522}, {"username": "ANY_USER_2", "text": "ANY_TEXT_2", "timestamp": 0}, {"username": "ANY_USER_3", "text": "ANY_TEXT_3", "timestamp": 1234}) (KafkaStormSpec.scala:329)
[info] Given a ZooKeeper instance
[info] And a Kafka broker instance
[info] And a Storm topology that uses AvroDecoderBolt and that reads tweets from topic testing-input and writes them as-is to topic testing-output
[info] And some tweets
[info] And a synchronous Kafka producer app that writes to the topic testing-input
[info] And a single-threaded Kafka consumer app that reads from topic testing-output and Avro-decodes the incoming data
[info] And a Storm topology configuration that registers an Avro Kryo decorator for Tweet
[info] When I run the Storm topology
[info] And I Avro-encode the tweets and use the Kafka producer app to sent them to Kafka
[info] Then the Kafka consumer app should receive the original tweets from the Storm topology
—
Reply to this email directly or view it on GitHub.
from kafka-storm-starter.
mtrienis
commented on August 26, 2024
Apologies for breathing life back into this thread. However, I've now disabled IPv6 networking
[root@vm ]# lsmod|grep ipv6
[root@vm ]#
After running the tests again
[root@vm kafka-storm-starter]# ./sbt clean test
And now I seem to have a new error.
[info] Storm
[info] - should start a local cluster
[info] + Given no cluster
[info] + When I start a LocalCluster instance
[info] + Then the local cluster should start properly
2014-09-04 16:43:09 b.s.testing4j [ERROR] Error in cluster
java.lang.AssertionError: Test timed out (5000ms)
at backtype.storm.testing$wait_until_cluster_waiting.invoke(testing.clj:206) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
As well as the same failure
[info] KafkaStormSpec:
[info] Feature: AvroDecoderBolt[T]
[info] Scenario: User creates a Storm topology that uses AvroDecoderBolt *** FAILED ***
[info] SynchronizedQueue() was not equal to List({"username": "ANY_USER_1", "text": "ANY_TEXT_1", "timestamp": 1409863343}, {"username": "ANY_USER_2", "text": "ANY_TEXT_2", "timestamp": 0}, {"username": "ANY_USER_3", "text": "ANY_TEXT_3", "timestamp": 1234}) (KafkaStormSpec.scala:329)
from kafka-storm-starter.
miguno
commented on August 26, 2024
Can you provide the output of a) /sbin/ifconfig -a and b) /sbin/sysctl -a|grep ipv6 on the machine/VM you are running ./sbt clean test?
from kafka-storm-starter.
miguno
commented on August 26, 2024
@trienism Also, what are your host machine specs as well as the specs for your VM? It could be that the VM is actually so slow that the (unfortunately hardcoded = not configurable) 5s timeout in Storm is being triggered.
014-09-04 16:43:09 b.s.testing4j [ERROR] Error in cluster
java.lang.AssertionError: Test timed out (5000ms)
at backtype.storm.testing$wait_until_cluster_waiting.invoke(testing.clj:206) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
See above -- this 5 second test timeout is hardcoded in Storm's testing API.
from kafka-storm-starter.
faustroll
commented on August 26, 2024
Yes. I think that it is a timeout issue.
I'll try running the tests on Ubuntu and bare metal.
timeout will be configurable in the next release of Storm.
On Fri, Sep 05, 2014 at 08:45:01AM -0700, Michael G. Noll wrote:
@trienism Also, what are your host machine specs as well as the specs for your VM? It could be that the VM is actually so slow that the (unfortunately hardcoded = not configurable) 5s timeout in Storm is being triggered.
014-09-04 16:43:09 b.s.testing4j [ERROR] Error in cluster java.lang.AssertionError: Test timed out (5000ms) at backtype.storm.testing$wait_until_cluster_waiting.invoke(testing.clj:206) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]See above -- this 5 second test timeout is hardcoded in Storm's testing API.
Reply to this email directly or view it on GitHub:
#1 (comment)
from kafka-storm-starter.
Related Issues (12)
- Failed to execute goal on project streams-examples HOT 1
- Failed to run "sbt test" HOT 3
- java.lang.UnsatisfiedLinkError
- org.eclipse.aether.resolution.ArtifactResolutionException:
- Weird NoRouteToHostException trying to run the tests HOT 2
- KafkaSparkStreamingSpec testsuite does not pass. HOT 2
- Storm UI showing improper values for Kafka Trident HOT 6
- KafkaSparkStreamingSpec.scala: NotSerializableException: org.apache.commons.pool2.impl.GenericObjectPool HOT 4
- we have setup all the tunning params for kafka and strom still getting very high latency HOT 1
- Test suit doesn't pass HOT 6
- User creates a Storm topology that uses AvroScheme in Kafka spout *** FAILED *** HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-storm-starter.