cant connect to kafka from external machine - spring

Question

I´m starting with Apache Kafka and i´m facing problems when i try to conect from an external machine.
With this configuration bellow, all works fine if the application and the docker are running at the same machine.
but when i put the application in machine A and docker at machine B, the application cant connect.
My spring Kafka #Configuration have this line to #Bean consumerFactory and producerFactory (imagine my machine with docker ip = 10.10.10.10)
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "10.10.10.10:9092");
And my docker file is this:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper:3.4.6
ports:
- 2181:2181
kafka:
image: wurstmeister/kafka:0.10.1.1
environment:
KAFKA_ADVERTISED_HOST_NAME: 0.0.0.0
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_CREATE_TOPICS: "topic-jhipster:1:1,PROCESS_ORDER:1:1, PROCESS_CHANNEL:1:1"
JMX_PORT: 9999
KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=9999"
ports:
- 9092:9092
- 9999:9999
kafka-manager:
image: sheepkiller/kafka-manager
ports:
- 9000:9000
links:
- zookeeper
environment:
ZK_HOSTS: zookeeper:2181
i get this error:
org.springframework.kafka.core.KafkaProducerException: Failed to send;
nested exception is org.apache.kafka.common.errors.TimeoutException:
Expiring 1 record(s) for
Edit, add some information..
I think its any configuration about the zookeeper i´m missing .. because if i have only the zookeeper started at my machine A .. and the kafka in machine B.. that works.. i only don´t know how :(

Answer 1

Try setting Listeners,
eg: listeners = PLAINTEXT://your.host.name:9092
Assuming you can telnet between the machines on kafka port.

Answer 2

Set advertised.listeners to the hostname or ip of the docker container's host machine.
environment:
KAFKA_ADVERTISED_LISTENERS: "10.10.10.10:9092"
What is happening is that the client connects to the bootstrap server, does a meta-data request to discover which Kafka broker in the cluster to connect to for a particular topic partition, and gets the advertised host name back as a response (in your case 0.0.0.0) which will only work if everything is on the same machine.
You need to advertise an ip or hostname that will work from the remote machine so not localhost, 127.0.0.1, or 0.0.0.0. Also not the private internal IP or hostname of the docker container. It has to be the external/public IP or hostname.
Also advertised.host.name and advertised.port are deprecated parameters in Kafka 0.10.x so even if you use them (they work for backward compatibility), you also need to set the advertised.host.name to be something that the producer can resolve and connect to. I suggest using a fully qualified host name or IP of the docker host machine (I.e. 10.10.10.10).
From http://kafka.apache.org/0101/documentation.html#brokerconfigs
DEPRECATED: only used when advertised.listeners or listeners are
not set. Use advertised.listeners instead. Hostname to publish to
ZooKeeper for clients to use. In IaaS environments, this may need to
be different from the interface to which the broker binds. If this is
not set, it will use the value for host.name if configured.
Otherwise it will use the value returned from
java.net.InetAddress.getCanonicalHostName().

Answer 3

Some mix of #Krishas and #Hans Jespersen
Here is the code of my docker yml:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper:3.4.6
ports:
- 2181:2181
kafka:
image: wurstmeister/kafka:0.10.1.1
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.10.10.10:9092
KAFKA_ADVERTISED_HOST_NAME: 10.10.10.10
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
That needs the "PLAINTEXT:// prefix !
And config the "host_name" + "port", or "listeners"
The next step is decovery how i will configure another nodes

Answer 4

You need to specify java.rmi.server.hostname=0.0.0.0