Topic discovery about kafka-python HOT 3 CLOSED

By continuous discovery, do you imply being notified (asynchronously) whenever the list of brokers of topic metadata gets updated? Did you ask your question on the kafka user mailing list?

Otherwise, the library has the topic information but it is somewhat hidden in the KafkaClient. The list of brokers and topic metadata (topic_partitions) is loaded by the client as construction (see load_metadata_for_topics). You can access them using the client instance.

IMO, the best option at the moment if you want to be notified on the spot upon brokers and topic metadata changes is still to use a connection to zookeeper with watches (see kazoo).

from kafka-python.

@roycamp, did you have a chance to try my suggestions, do you have any feedback on that?

from kafka-python.

To get all the topics in a cluster, you should send a metadata request to one of the brokers and have your application logic manage finding "new" topics:

from kafka.conn import KafkaConnection
from kafka.protocol import KafkaProtocol

# some config settings
host = 'srv-130-12'
port = 9092

client_id = "topic_checker" # any str will do here
request_id = 0 # any int will do here
correlation_id = 0 # any int will do here

conn = KafkaConnection(host, port)
request = KafkaProtocol.encode_metadata_request(client_id, correlation_id)
conn.send(request_id, request)
response = conn.recv(request_id)
(brokers, topics) = KafkaProtocol.decode_metadata_response(response)
print topics.keys()

from kafka-python.