Failing Dockerized Kafka on MacOS - macos

Question

I’m trying (and failing) to run a dockerized kafka on my mac machine (MacOS Sierra (10.12.2)). I have Docker for Mac version 17.03.1-ce, build c6d412e. These are the images I’m using.
https://hub.docker.com/r/confluentinc/cp-zookeeper/
https://hub.docker.com/r/confluentinc/cp-kafka/
And I’m following the advice in the official quickstart guide, running Zookeeper and Kafka with the following commands.
docker run -d \
--net=host \
--name=zookeeper \
-e ZOOKEEPER_CLIENT_PORT=32181 \
confluentinc/cp-zookeeper:3.2.1
docker run -d \
--net=host \
--name=kafka \
-e KAFKA_ZOOKEEPER_CONNECT=localhost:32181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:29092 \
confluentinc/cp-kafka:3.2.1
MAIN This otherwise doesn’t work with docker-compose. Again it's on my mac, version 1.11.2, build dfed245.
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:3.2.2
environment:
- ZOOKEEPER_CLIENT_PORT=2181
ports:
- 2181:2181
kafka:
image: confluentinc/cp-kafka:3.2.2
environment:
- KAFKA_BROKER_ID=0
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
- KAFKA_AUTO_CREATE_TOPICS_ENABLE=true
ports:
- 9092:9092
- 8082:8082
depends_on:
- zookeeper
The services will start, and I can even create a topic with these commands.
kafka-topics --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic test
kafka-topics --list --zookeeper zookeeper:2181
When I try to produce with this command, zookeeper remains silent.
kafka-console-producer --broker-list kafka:9092 --topic test
message-one
message-two
When I try to consume with this command:
kafka-console-consumer --bootstrap-server zookeeper:2181 --topic test --from-beginning
.. zookeeper continuously spits out this error:
...
zookeeper_1 | [2017-06-28 00:55:07,222] INFO Accepted socket connection from /172.20.0.3:52124 (org.apache.zookeeper.server.NIOServerCnxnFactory)
zookeeper_1 | [2017-06-28 00:55:07,222] WARN Exception causing close of session 0x0 due to java.io.EOFException (org.apache.zookeeper.server.NIOServerCnxn)
zookeeper_1 | [2017-06-28 00:55:07,223] INFO Closed socket connection for client /172.20.0.3:52124 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
...

Answer 1

You have specified the wrong host and port for the new console consumer. Try console-consumer --bootstrap-server kafka:9092 --topic test --from-beginning
Also if you are running these commands from outside docker (I.e. on the native macOS host) then edit your /etc/hosts file to add kafka and zookeeper as aliases for localhost
You might also want to declare and mount an external volume for the zookeeper and kafka logs so your data won't be lost if you destroy the docker images and upgrade to a newer version.
Confluent has a full QuickStart documented for these images here http://docs.confluent.io/current/cp-docker-images/docs/quickstart.html