Giter VIP home page Giter VIP logo

docker-kafka's Introduction

Kafka in Docker

This repository provides everything you need to run Kafka in Docker.

For convenience also contains a packaged proxy that can be used to get data from a legacy Kafka 7 cluster into a dockerized Kafka 8.

Why?

The main hurdle of running Kafka in Docker is that it depends on Zookeeper. Compared to other Kafka docker images, this one runs both Zookeeper and Kafka in the same container. This means:

  • No dependency on an external Zookeeper host, or linking to another container
  • Zookeeper and Kafka are configured to work together out of the box

Run

docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka
export KAFKA=`docker-machine ip \`docker-machine active\``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test
export ZOOKEEPER=`docker-machine ip \`docker-machine active\``:2181
kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test

Running the proxy

Take the same parameters as the spotify/kafka image with some new ones:

  • CONSUMER_THREADS - the number of threads to consume the source kafka 7 with
  • TOPICS - whitelist of topics to mirror
  • ZK_CONNECT - the zookeeper connect string of the source kafka 7
  • GROUP_ID - the group.id to use when consuming from kafka 7
docker run -p 2181:2181 -p 9092:9092 \
    --env ADVERTISED_HOST=`boot2docker ip` \
    --env ADVERTISED_PORT=9092 \
    --env CONSUMER_THREADS=1 \
    --env TOPICS=my-topic,some-other-topic \
    --env ZK_CONNECT=kafka7zookeeper:2181/root/path \
    --env GROUP_ID=mymirror \
    spotify/kafkaproxy

In the box

  • spotify/kafka

    The docker image with both Kafka and Zookeeper. Built from the kafka directory.

  • spotify/kafkaproxy

    The docker image with Kafka, Zookeeper and a Kafka 7 proxy that can be configured with a set of topics to mirror.

Public Builds

https://registry.hub.docker.com/u/spotify/kafka/

https://registry.hub.docker.com/u/spotify/kafkaproxy/

Build from Source

docker build -t spotify/kafka kafka/
docker build -t spotify/kafkaproxy kafkaproxy/

Todo

  • Not particularily optimzed for startup time.
  • Better docs

docker-kafka's People

Contributors

apiri avatar eugenemi avatar jonaslindmark avatar mattnworb avatar mbruggmann avatar rohansingh avatar sherzberg avatar yutaono avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

docker-kafka's Issues

You run Kafka and Zookeeper with root permissions... Intended?

In general, it is not best practice to leave the user as root for Docker containers, since you buy yourself several security issues with that. I hope you are not running it like this in your production servers :-D

The solution is to set up a user in the Dockerfile and give it just the necessary permissions and then run Kafka/Zookeeper with these limited permissions.

Stability of container on high loads

Great tool for dev!

However I see issues in stability for the container if I run for a long time. Am using this out of the box with no custom configuration.

Is there any way to add more verbose logging easily? The consumers seem to go into a loop trying to get the leader.

Docker-Kafka to get started with topic and produce and consume

Hi,
I have used repository image of kafka from docker hub( docker pull spotify/kafka) and run through Docker tool box(Kitematic) and now kafka is running with the existing setup.No changes I did . My requirement is I need to create a topic and produce the topic and consume the topic . How we can proceed from docker tool box or CLI .
Please advise I am new for docker but I am familiar about normal setup of kafka without docker .

Please archive this repo

Dear @mattnworb & @mbruggmann could it make sense to archive this repo to make it more clear that it is no longer maintained and does not work with current versions of Kafka (clients).

See also: #102

Thanks a lot for your consideration

Cannot publish

I´m trying to run Kafka and Zookepper using Docker following the steps defined here https://github.com/spotify/docker-kafka
But after run the docker command just as describe I don't understand for what for the export are, and I´m not able to publish anything pointing to localhost:9092` and I receive a timeout response.

This is the code of the publisher that I´m using https://github.com/politrons/Kafka/blob/master/src/main/scala/Producer.scala

Any idea if I´m doing something wrong?

Has anyone been able to start a topic using Docker Compose?

I attempted to put

kafka:
    image: spotify/kafka
    ports:
      - "9092:9092"
      - "2181:2181"
    hostname: kafka
    expose:
      - "9092"
      - "2181"
    environment:
      TOPICS: test-topic

But this failed to work, is there something that I'm missing? (I tried with quotes around the topic as well)

Also, if anyone has had any luck using Spring to connect to a topic, I would greatly appreciate some help connecting to the topics.

Support for clustering Kafka via new Docker swarm mode?

Are there plans to support clustering Kafka via Docker swarm mode, which is new in Docker v1.12.0? With the new swarm you can define services and automatically scale them up or down. Plus docker handles load balancing automatically between instances of the service (default round robin).

docker service create --replicas 1 --name kafka spotify/kafka
docker service scale kafka=3

Intro: https://docs.docker.com/engine/swarm/
Concepts: https://docs.docker.com/engine/swarm/key-concepts/

support for creating topics

It would be very nice if the container (and so the start up script) would support the creation of topics like the wurstmeister/kafka container does, i.e. "--env KAFKA_CREATE_TOPICS=eventsourcing-traffic-store:1:1".

LeaderNotAvailableException

Hi,

I'm trying to get your Kafka container running for testing purposes, but I always run into the issue that I get an LeaderNotAvailableException if I follow the run steps as described on your Github page.

bin  ./kafka-console-producer.sh --broker-list $KAFKA --topic test
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Message
[2015-05-07 11:30:13,965] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-05-07 11:30:14,027] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-05-07 11:30:14,027] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
[2015-05-07 11:30:14,154] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-05-07 11:30:14,166] WARN Error while fetching metadata [{TopicMetadata for topic test -> 
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)
[2015-05-07 11:30:14,166] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)

Unable to change port mappings

Hi,
I seem to be unable to use different port mappings for this image in order to run different environments on the same machine. In conjunction with spring-kafka:2.1.4.RELEASE i get the following error:

2018-04-10 19:57:38.191 WARN 13318 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=notifications] Error while fetching metadata with correlation id 4 : {notifications.topic=LEADER_NOT_AVAILABLE}
2018-04-10 19:57:38.208 INFO 13318 --- [ Test worker] o.a.kafka.common.utils.AppInfoParser : Kafka version : 1.0.0
2018-04-10 19:57:38.208 INFO 13318 --- [ Test worker] o.a.kafka.common.utils.AppInfoParser : Kafka commitId : aaa7af6d4a11b29d
2018-04-10 19:57:38.228 WARN 13318 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Error while fetching metadata with correlation id 2 : {notifications.topic=LEADER_NOT_AVAILABLE}
2018-04-10 19:57:38.299 WARN 13318 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=notifications] Error while fetching metadata with correlation id 5 : {notifications.topic=LEADER_NOT_AVAILABLE}
2018-04-10 19:57:38.337 WARN 13318 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Error while fetching metadata with correlation id 3 : {notifications.topic=LEADER_NOT_AVAILABLE}
2018-04-10 19:57:38.406 WARN 13318 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=notifications] Error while fetching metadata with correlation id 6 : {notifications.topic=LEADER_NOT_AVAILABLE}
2018-04-10 19:57:38.445 WARN 13318 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Error while fetching metadata with correlation id 4 : {notifications.topic=LEADER_NOT_AVAILABLE}
...
and so on

docker-compose:
kafka:
container_name: integration_kafka
image: spotify/kafka:latest
ports:
- "9093:9092"
- "2182:2181"
environment:
ADVERTISED_HOST: 127.0.0.1
ADVERTISED_PORT: 9093
TOPICS: notifications.topic
GROUP_ID: notifications

bootstrap.yml:
spring:
kafka:
consumer:
auto-offset-reset: earliest
group-id: notifications
bootstrap-servers:
- 127.0.0.1:9093

Only if i run the container with the default mappings 9092:9092, 2181:2181 and the bootstrap servers set to 127.0.0.1:9092 it seems to work as intended. I'd be very thankful if you could point out my mistake... or care to correct yours ;-)

Using this docker in production

I am new to kafka and zoo keeper. If I run 3 instances of this docker, will they be related through a network interface. Basically is it equivalent to a cluster of kafka on top of a cluster of zoo keeper.

Tag by kafka version

The recent update to 0.8.2.1 broke some scripts because the bin directory moved. Would be helpful to tag the container version and ensure the install location is consistent for a given tag

Noob: I have problems get it running

Hi thanks for sharing this piece of work!
I have a question:

export KAFKA=docker-machine ip \docker-machine active``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test

Where do you execute those scripts? Outside of the container there is no such shell script. But how to execute it?
Thanks

Kafka cluster

Hi, does this image support Kafka cluster? It means that running multiple broker at the same cluster. I do not see any place where can set the configuration for it.
Thanks.

Docker build no longer works

I tried

git clone https://github.com/spotify/docker-kafka
cd docker-kafka/
cd kafka
docker build -t kafka:0.10.1.0 .

and I get the following error

Get:6 http://deb.debian.org jessie/main amd64 Packages [9098 kB]
Err http://deb.debian.org jessie-backports/main amd64 Packages
  
Err http://deb.debian.org jessie-backports/main amd64 Packages
  
Err http://deb.debian.org jessie-backports/main amd64 Packages
  
Err http://deb.debian.org jessie-backports/main amd64 Packages
  
Err http://deb.debian.org jessie-backports/main amd64 Packages
  404  Not Found
Fetched 10.1 MB in 17s (583 kB/s)
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/InRelease  Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)

W: Failed to fetch http://deb.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages  404  Not Found

E: Some index files failed to download. They have been ignored, or old ones used instead.
The command '/bin/sh -c apt-get update &&     apt-get install -y zookeeper wget supervisor dnsutils &&     rm -rf /var/lib/apt/lists/* &&     apt-get clean &&     wget -q http://apache.mirrors.spacedump.net/kafka/"$KAFKA_VERSION"/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -O /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz &&     tar xfz /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -C /opt &&     rm /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz' returned a non-zero code: 100

Restart of docker does not correctly restart kafka

Hi,
For reasons I don't yet understand, when I do a docker restart container_id, I can no longer send messages to the topics previously created (the kafka client gets the following: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.).

If I connect to the running container and just stop the kafka process directly, supervisord successfully restarts it and I don't have any issues, so I'm assuming this probably is docker-specific.

expose VOLUME

Not being that experienced with kafka it would be nice to have a VOLUME exposed for easily setting up a data container.

Run kafka with an external volume

Hi

I'm newbie to kafka.

I run this docker, and in almost three hours my docker lvs get to 100% making it unavailable

How can I change this docker to an external volume to not exhaust my docker?

Kafka example not working (with 0.10)

For me the example with kafka-console-producer.sh and kafka-console-consumer.sh does not seem to work. No output from producer and no output from consumer. And when I close the consumer it tells me Processed a total of 0 messages.

# variables for all terminals
export IP=$(ifconfig eth0 | grep "inet addr:" | cut -d ":" -f2 | cut -d " " -f1)
export KAFKA=$IP:9092
export ZOOKEEPER=$IP:2181

# terminal 1
docker run \
  --name kafka \
  --restart=always \
  -p 2181:2181 \
  -p 9092:9092 \
  --env ADVERTISED_HOST=$IP\
  --env ADVERTISED_PORT=9092 \
  spotify/kafka

# terminal 2
docker run -it --rm spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list $KAFKA --topic test

# terminal 3
docker run -it --rm spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test

kafka entered FATAL state on an AWS Docker machine

I'm trying to run the kafka container on a AWS docker machine, when the container starts it successfully starts zookeeper but fails to start kafka. Here is the console output:

> docker-machine create --driver amazonec2 --amazonec2-access-key $AWS_ACCESS_KEY_ID --amazonec2-secret-key $AWS_SECRET_ACCESS_KEY --amazonec2-vpc-id $VPC_ID --amazonec2-region $REGION aws01
> eval $(docker-machine env aws01)
> docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka
/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '
2015-11-20 10:47:03,112 CRIT Supervisor running as root (no user in config file)
2015-11-20 10:47:03,112 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2015-11-20 10:47:03,112 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2015-11-20 10:47:03,121 INFO RPC interface 'supervisor' initialized
2015-11-20 10:47:03,121 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-11-20 10:47:03,122 INFO supervisord started with pid 1
2015-11-20 10:47:04,124 INFO spawned: 'zookeeper' with pid 8
2015-11-20 10:47:04,125 INFO spawned: 'kafka' with pid 9
2015-11-20 10:47:04,171 INFO exited: kafka (exit status 1; not expected)
2015-11-20 10:47:05,172 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-11-20 10:47:05,173 INFO spawned: 'kafka' with pid 37
2015-11-20 10:47:05,196 INFO exited: kafka (exit status 1; not expected)
2015-11-20 10:47:07,200 INFO spawned: 'kafka' with pid 45
2015-11-20 10:47:07,224 INFO exited: kafka (exit status 1; not expected)
2015-11-20 10:47:10,228 INFO spawned: 'kafka' with pid 53
2015-11-20 10:47:10,251 INFO exited: kafka (exit status 1; not expected)
2015-11-20 10:47:11,252 INFO gave up: kafka entered FATAL state, too many start retries too quickly

Any idea what is causing this and how to fix it?

Docker image tags based on kafka version

It would be awesome if the docker image had more tags than just "latest". How difficult would it be for there to be different tags for different versions? I'm working with some other systems to do not yet support kafka 0.10.

Increase max client connections

Hi,

I am new to the Kafka docker, so I want to increase the maximum client connection in the Kafka image to be unlimited
that need to set the zookeeper configuration(/etc/zookeeper/conf/zoo.cfg) by adding this statement(maxClientCnxns=0), can you guy give me a suggestion on this, how can I do that

Thanks,

kafka.common.LeaderNotAvailableException

I get a LeaderNotAvailableException when I use the console producer from the docker host. I'm on Fedora and start the container like this:
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=127.0.0.1 --env ADVERTISED_PORT=9092 spotify/kafka

Then I try to write to it:
./kafka-console-producer.sh --broker-list localhost:9092 --topic test

resulting in:
WARN Error while fetching metadata [{TopicMetadata for topic test -> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2016-06-01 09:57:20,149] WARN Error while fetching metadata [{TopicMetadata for topic test -> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2016-06-01 09:57:20,149] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)

What am I doing wrong?

Running multiple kafka brokers belonging to the same cluster

Hi,

I was able to use the spotify/kafka image and run a kafka/zookeeper container. When i scale it using marathon, it creates additional containers each with kafka and zookeeper. How do make the additional containers part of the same cluster?

Thanks!
Sonali

Failed to start kafka 'exit status 127'

Kafka failed to start with this error message:

2017-08-16 18:21:57,172 CRIT Supervisor running as root (no user in config file)
2017-08-16 18:21:57,173 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2017-08-16 18:21:57,173 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2017-08-16 18:21:57,181 INFO RPC interface 'supervisor' initialized
2017-08-16 18:21:57,181 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2017-08-16 18:21:57,181 INFO supervisord started with pid 1
2017-08-16 18:21:58,185 INFO spawned: 'zookeeper' with pid 7
2017-08-16 18:21:58,187 INFO spawned: 'kafka' with pid 8
2017-08-16 18:21:58,201 INFO exited: kafka (exit status 127; not expected)
2017-08-16 18:21:59,208 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-08-16 18:21:59,210 INFO spawned: 'kafka' with pid 28
2017-08-16 18:21:59,214 INFO exited: kafka (exit status 127; not expected)
2017-08-16 18:22:01,218 INFO spawned: 'kafka' with pid 29
2017-08-16 18:22:01,225 INFO exited: kafka (exit status 127; not expected)
2017-08-16 18:22:04,231 INFO spawned: 'kafka' with pid 30
2017-08-16 18:22:04,238 INFO exited: kafka (exit status 127; not expected)
2017-08-16 18:22:05,239 INFO gave up: kafka entered FATAL state, too many start retries too quickly

Does anyone encountered that, how can I solve this problem?

cant build docker image

Cannot build docker image.

  1. there are errors during apt-get because of invalid repositories.
  2. broken url for downloading kafka sources

using dynamic port allocation on docker host

I'd like to use dynamic port binding (aka docker run -P ...).
Is there a way to get this working?
I guess the dynamically allocated port must be advertised in zk?
Any other things must be done for that?

THanks!

Better docs for this image ?

By running it in docker cloud, or my localhost virtual machine.,
the logs show both zookeeper and kafka server running correctly, but I can not create topic by using
"kafka-console-producer.sh --broker-list $KAFKA --topic test"

since in the default bin file, it has no kafka command there.

Also, how we set up when we wanna multiple zookeeper instances running ?

Docker image build failed !

Hi,
I'm trying to build docker image from source but it failed as following

Sending build context to Docker daemon 9.216 kB
Step 1 : FROM java:openjdk-8-jre
---> 0a290b7fd4c4
Step 2 : ENV DEBIAN_FRONTEND noninteractive
---> Using cache
---> c9c7a60507de
Step 3 : ENV SCALA_VERSION 2.11
---> Using cache
---> 6b1df17b1b9b
Step 4 : ENV KAFKA_VERSION 0.10.0.1
---> Using cache
---> c28d56fc3f16
Step 5 : ENV KAFKA_HOME /opt/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION"
---> Using cache
---> 69728c9aac29
Step 6 : RUN apt-get update && apt-get install -y zookeeper wget supervisor dnsutils && rm -rf /var/lib/apt/lists/* && apt-get clean && wget -q http://apache.mirrors.spacedump.net/kafka/"$KAFKA_VERSION"/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -O /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz && tar xfz /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -C /opt && rm /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz
---> Running in 159de5710043

Err http://security.debian.org jessie/updates InRelease

Err http://security.debian.org jessie/updates Release.gpg
Could not resolve 'security.debian.org'
Err http://httpredir.debian.org jessie InRelease

Err http://httpredir.debian.org jessie-updates InRelease

Err http://httpredir.debian.org jessie-backports InRelease

Err http://httpredir.debian.org jessie Release.gpg
Could not resolve 'httpredir.debian.org'
Err http://httpredir.debian.org jessie-updates Release.gpg
Could not resolve 'httpredir.debian.org'
Err http://httpredir.debian.org jessie-backports Release.gpg
Could not resolve 'httpredir.debian.org'
Reading package lists...
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie/InRelease

W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-updates/InRelease

W: Failed to fetch http://security.debian.org/dists/jessie/updates/InRelease

W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-backports/InRelease

W: Failed to fetch http://security.debian.org/dists/jessie/updates/Release.gpg Could not resolve 'security.debian.org'

W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie/Release.gpg Could not resolve 'httpredir.debian.org'

W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-updates/Release.gpg Could not resolve 'httpredir.debian.org'

W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-backports/Release.gpg Could not resolve 'httpredir.debian.org'

W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package zookeeper
E: Unable to locate package supervisor
E: Unable to locate package dnsutils
The command '/bin/sh -c apt-get update && apt-get install -y zookeeper wget supervisor dnsutils && rm -rf /var/lib/apt/lists/* && apt-get clean && wget -q http://apache.mirrors.spacedump.net/kafka/"$KAFKA_VERSION"/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -O /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz && tar xfz /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -C /opt && rm /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz' returned a non-zero code: 100

Something wrong with getting packages ?

Thanks,
Kevin.

Failed to collate messages by topic (LeaderNotAvailableException)

First off let me say thank you for this great image!

A few days ago I was running the image and it was working well.

Lately, though, I've been trying to follow the most basic steps (as listed in the Run section at dockerhub) and it's been erroring with

root@2995acf13535:/opt/kafka_2.11-0.8.2.1/bin# ./kafka-console-producer.sh --broker-list localhost:9092 --topic test
[2016-08-25 12:06:33,784] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
Hi There
[2016-08-25 12:06:36,183] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2016-08-25 12:06:36,214] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2016-08-25 12:06:36,217] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)

Just to verify that I'm not doing anything ultra dumb, in linux I'm,

  • Running the image (using localhost because I'm in linux): docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=localhost --env ADVERTISED_PORT=9092 spotify/kafka
  • Exec-ing into the image: docker exec -it stupefied_wing bash
  • Navigating to the kafka tools: cd /opt/kafka_2.11-0.8.2.1/bin/
  • Starting the console producer: ./kafka-console-producer.sh --broker-list localhost:9092 --topic test
    • I get a warning here [2016-08-25 12:12:35,959] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
  • Type in some message... and see above :)

Thanks again, any help would be greatly appreciated.

Minor update to README

Hey guys,
super-thanks for creating this, saved me a TON of time!

I just noticed that the instructions in the README may be slightly improved, by using the following (I was having trouble running this on a VM where I hadn't installed Kafka, so wanted to run both consumer/producer inside two separate containers, still using this image - but avoid using the "hack" of using localhost everywhere).

I also believe that Kafka will have trouble if the ADVERTISED_HOST does not reflect the IP of the container.

Anyways, this is how I ran it - please let me know if you would like me to update the README and I'd be happy to submit a PR.

# Launch the "main" Kafka container, this runs the server;
# it also changes the `hostname` to "kafka" and sets an appropriate entry to /etc/hosts:
$ docker run --rm --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 \
    --name kafka -h kafka spotify/kafka 

To launch the Producer (this used to fail with a kafka.common.LeaderNotAvailableException when posting to a topic):

$ docker run --rm -it --name producer --link kafka spotify/kafka \
     /opt/kafka_2.11-0.8.2.1/bin/kafka-console-producer.sh --broker kafka:9092 --topic test

similarly, the consumer can be run with:

$ docker run --rm -it --name consumer --link kafka spotify/kafka \
    /opt/kafka_2.11-0.8.2.1/bin/kafka-console-consumer.sh --zookeeper kafka:2181 --topic test

Also, I would suggest adding the /opt/kafka_xxx directory to the container's PATH in the Dockerfile (again, happy to send a PR, if you guys want me to).

Hope this helps!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.