How to run Kafka with Docker on OS X? - macos

Question

I tried to run Kafka without Docker a few days ago, again, I failed. It's very hard to setup. So, I decided to use Docker, then I failed again. What could be the problem?
I ran these commands:
git clone https://github.com/confluentinc/cp-docker-images
cd cp-docker-images/examples/kafka-single-node
docker-compose up
Btw, this is my:
Docker version: 17.03.1-ce
Docker compose version: 1.11.2, build dfed245
I downloaded the new Docker for OS X here.
SIDE NOTE: New Docker for OS X uses a different architecture than the previous ones and so, solves many previous problems in OS X with Docker. So, most of the issues talked in Confluent Docker documentation will be easier to address, I believe.
Error Log:
kafka_1 | [2017-05-08 21:00:37,382] WARN [Controller-1-to-broker-1-send-thread], Controller 1 epoch 1 fails to send request (type: UpdateMetadataRequest=, controllerId=1, controllerEpoch=1, partitionStates={}, liveBrokers=(id=1, endPoints=(host=localhost, port=29092, listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT), rack=null)) to broker localhost:29092 (id: 1 rack: null). Reconnecting to broker. (kafka.controller.RequestSendThread)
kafka_1 | java.io.IOException: Connection to 1 was disconnected before the response was read
kafka_1 | at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:114)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:112)
kafka_1 | at scala.Option.foreach(Option.scala:257)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:112)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:108)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:136)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:142)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:108)
kafka_1 | at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:196)
kafka_1 | at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:188)
kafka_1 | at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
kafka_1 | [2017-05-08 21:00:37,484] INFO [Controller-1-to-broker-1-send-thread], Controller 1 connected to localhost:29092 (id: 1 rack: null) for sending state change requests (kafka.controller.RequestSendThread)
kafka_1 | [2017-05-08 21:00:37,486] ERROR Processor got uncaught exception. (kafka.network.Processor)
kafka_1 | java.lang.NoClassDefFoundError: Could not initialize class kafka.network.RequestChannel$
kafka_1 | at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:111)
kafka_1 | at kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:513)
kafka_1 | at kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:505)
kafka_1 | at scala.collection.Iterator$class.foreach(Iterator.scala:893)
kafka_1 | at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
kafka_1 | at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
kafka_1 | at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
kafka_1 | at kafka.network.Processor.processCompletedReceives(SocketServer.scala:505)
kafka_1 | at kafka.network.Processor.run(SocketServer.scala:433)
kafka_1 | at java.lang.Thread.run(Thread.java:745)
kafka_1 | [2017-05-08 21:01:07,516] WARN [Controller-1-to-broker-1-send-thread], Controller 1 epoch 1 fails to send request (type: UpdateMetadataRequest=, controllerId=1, controllerEpoch=1, partitionStates={}, liveBrokers=(id=1, endPoints=(host=localhost, port=29092, listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT), rack=null)) to broker localhost:29092 (id: 1 rack: null). Reconnecting to broker. (kafka.controller.RequestSendThread)
kafka_1 | java.io.IOException: Connection to 1 was disconnected before the response was read
kafka_1 | at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:114)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:112)
kafka_1 | at scala.Option.foreach(Option.scala:257)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:112)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:108)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:136)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:142)
kafka_1 | at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:108)
kafka_1 | at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:196)
kafka_1 | at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:188)
kafka_1 | at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
kafka_1 | [2017-05-08 21:01:07,618] INFO [Controller-1-to-broker-1-send-thread], Controller 1 connected to localhost:29092 (id: 1 rack: null) for sending state change requests (kafka.controller.RequestSendThread)
kafka_1 | [2017-05-08 21:01:07,619] ERROR Processor got uncaught exception. (kafka.network.Processor)
kafka_1 | java.lang.NoClassDefFoundError: Could not initialize class kafka.network.RequestChannel$
kafka_1 | at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:111)
kafka_1 | at kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:513)
kafka_1 | at kafka.network.Processor$$anonfun$processCompletedReceives$1.apply(SocketServer.scala:505)
kafka_1 | at scala.collection.Iterator$class.foreach(Iterator.scala:893)
kafka_1 | at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
kafka_1 | at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
kafka_1 | at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
kafka_1 | at kafka.network.Processor.processCompletedReceives(SocketServer.scala:505)
kafka_1 | at kafka.network.Processor.run(SocketServer.scala:433)
kafka_1 | at java.lang.Thread.run(Thread.java:745)

Answer 1

Docker recently changed its name to Moby and that was causing my issues. I needed to add moby as a localhost in my docker-compose config.
By adding extra_hosts parameter that resolves moby as localhost will solve this problem:
extra_hosts:
- "moby:127.0.0.1"
The fixing pull-request about this error is here.
To get this fix, type: git pull origin pull/258/head.
If you also want to read about the root cause of this, click here.