Broker throws SchemaException: Error reading field 'topics' about kafka HOT 8 CLOSED

Kafka and no-kafka versions?

from kafka.

ches/kafka seems to install Kafka 0.8.1.1
no-kafka uses V2 group membership which is only available in Kafka 0.9.

Please confirm Kafka version

from kafka.

My bad, meant to install greytip/kafka (https://hub.docker.com/r/greytip/kafka/) but in the description it's still referencing ches/kafka. I'm running now the correct version of Kafka, that is 0.9, and [email protected], the above error is gone, but I'm getting this:

Error: Failed to join the group
    at tryJoinGroup (/home/vagrant/altorepos/new-kafka/node_modules/no-kafka/lib/group_consumer.js:76:19)
    at /home/vagrant/altorepos/new-kafka/node_modules/no-kafka/lib/group_consumer.js:82:24
    at Timer.listOnTimeout [as ontimeout] (timers.js:112:15)
From previous event:
    at /home/vagrant/altorepos/new-kafka/node_modules/no-kafka/lib/group_consumer.js:81:40

from kafka.

This error is only triggered when Kafka server returns 'GroupCoordinatorNotAvailable'. Are you running Kafka 0.9 with the same Zookeeper instance that was used by Kafka 0.8?

from kafka.

I used the same Zookeeper instance, but a different new topic name test-2. I've just tried with a fresh instance of both Zookeeper & Kafka and it worked fine. But two questions:

1. Why was Kafka returning GroupCoordinatorNotAvailable in my second test using a new topic?
2. How would you go on migrating from 0.8.x to 0.9? My idea was to (somehow) do a rolling upgrade of Kafka, keep the same Zookeeper instance, and probably rename the consumer group ids.

from kafka.

1. Kafka 0.9 uses incompatible with 0.8 Zookeeper structure
2. Really not sure how to migrate, this should probably be covered by Kafka docs, or maybe you can find answers in Kafka mailing list

In order to use the same Zookeeper instance for both 0.8 and 0.9 running at the same time you should run Kafka 0.9 with a different Zookeeper root:

config/server.properties:

zookeeper.connect=localhost:2181/kafka0.9

In this case all Kafka cli operations with 0.9 should also specify this root:

kafka-topics.sh --zookeeper 127.0.0.1:2181/kafka0.9 --create --topic kafka-test-topic --partitions 3 --replication-factor 1

from kafka.

Thanks @oleksiyk!

from kafka.

good job

from kafka.