I was able to get kafka to work fine when I spun it up on my local machine. But when I try to get it to work on an AWS instance nothing seems to work right. I tried spinning up my own server and doing just like I did locally spinning up zookeeper and kafka like so
curl http://apache.spinellicreations.com/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz | tar -xzf
cd kafka_2.11-0.10.0.0
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
I also tried using the AMI from bitami which seems to be an all in one AMI. Creating the topic seems to work fine. But when I try to run the console producer I get an error
SEASPAULSON-MAC:kafka_2.11-0.10.0.0 spaulson$ bin/kafka-console-producer.sh --broker-list ec2-54-186-31-109.us-west-2.compute.amazonaws.com:9092 --topic test
blah
[2016-10-20 12:13:23,395] ERROR Error when sending message to topic test with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for test-0
I also get an error when I try to start up a console consumer that repeats over and over.
bin/kafka-console-consumer.sh --zookeeper ec2-54-186-31-109.us-west-2.compute.amazonaws.com:2181 --topic test --from-beginning
[2016-10-19 18:26:47,175] WARN Fetching topic metadata with correlation id 152 for topics [Set(test)] from broker [BrokerEndPoint(0,ip-172-31-52-58.ec2.internal,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:80)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
I feel like these kinds of operations should be trivial but it's proving very challenging. I'm having trouble finding documentation on how to diagnose issues and figure out what's going wrong. The best I found is this command
KAFKA_HOME/bin/kafka-topics.sh --describe --topic test --zookeeper ec2-54-186-31-109.us-west-2.compute.amazonaws.com:2181
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Does the Leader: 0 indicate something went wrong? But what?
For AWS or any other IaaS machines, you should set "advertised.listeners" for the clients. Here is what this options means in the Kafka doc:
Listeners to publish to ZooKeeper for clients to use, if different than the listeners above. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value for listeners will be used.
Related
I followed the how to install guide of Spring Cloud Data Flow to install the application on Azure Kubernetes Cluster with kubectl. I use Kafka as a message broker and I created a simple stream, time | log.
The stream cannot be deployed, I enclose the logs which I can't fully understand.
PS kubectl get pods
NAME READY STATUS RESTARTS AGE
grafana-7d7d77d54-m59dx 1/1 Running 0 5h36m
kafka-broker-64bfd5d6b5-9c7ld 1/1 Running 0 25m
kafka-zk-768b548468-mhrrn 1/1 Running 0 145m
mysql-9dbdc88c6-xz4hh 1/1 Running 0 21h
prometheus-64b45b746-zs7z4 1/1 Running 0 5h37m
prometheus-proxy-6764bf4968-4xjz5 1/1 Running 0 28m
scdf-server-7f864c96b7-s8cmm 1/1 Running 0 62m
skipper-7fbd7f47cd-b92v4 1/1 Running 0 6h13m
test-stream-log-v9-ffcd9d55f-8p96j 0/1 Running 13 68m
test-stream-time-v9-6c47699d94-pfzkr 0/1 Running 13 68m
Time app log. https://pastebin.com/JyS8azVk
Log app log. https://pastebin.com/pCe1NqSn
Kafka log. https://pastebin.com/Dj5KfVsQ
From the attached logs in time-source; specifically, this:
2019-12-19 21:15:23.963 ERROR 1 --- [ main] o.s.cloud.stream.binding.BindingService : Failed to create producer binding; retrying in 30 seconds
org.springframework.cloud.stream.provisioning.ProvisioningException: Provisioning exception; nested exception is java.util.concurrent.TimeoutException
at org.springframework.cloud.stream.binder.kafka.provisioning.KafkaTopicProvisioner.createTopic(KafkaTopicProvisioner.java:290) ~[spring-cloud-stream-binder-kafka-core-2.1.4.RELEASE
This is indicative that the Spring Cloud Stream binder provisioner is unable to create the desired topic for the producer (i.e. time-source).
Based on your kubectl get pods output, however, it appears your Kafka and ZK were deployed roughly recently as opposed to Skipper, which was deployed >6hrs ago.
It is likely that you may have deployed components in the wrong order or that you may have reprovisioned Kafka, but that IP/Host/Port changes aren't yet reflecting in Skipper's deployment. The reason being, Skipper keeps track of Kafka credentials in its config-map, so all the stream applications that it deploys (via SCDF) will automatically receive the credentials at deployment time.
I'd guess the credentials that the applications received might have changed when you reprovisioned Kafka/ZK — you could compare that. I'd suggest bouncing Skipper deployment so it can receive the latest in its configmap or clean-slate it all, and follow the deployment order described in the docs from the beginning.
I have configured kafka_2.11-2.3.0 and apache-zookeeper-3.5.5-bin on Windows 10. But while running the topic creation command I am getting the below error:
C:\kafka_2.11-2.3.0>.\bin\windows\kafka-topics.bat --create --bootstrap-server 127.0.0.1:2181 --partitions 1 --replication-factor 1 --topic testD1
Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[2019-10-14 16:42:40,603] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:178)
at kafka.admin.TopicCommand$TopicService$class.createTopic(TopicCommand.scala:149)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:172)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:60)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
Read somewhere in stackoverflow to add listeners=PLAINTEXT://127.0.0.1:9092 in the server.properties file but that didn't work out as expected.
Zookeeper runs on 2181, not Kafka (the bootstrap server)
By default, Kafka runs on port 9092 as below
kafka-topics --bootstrap-server 127.0.0.1:9092 --topic first_topic --create --partitions 3 --replication-factor 1
I've struggled with the same issue on linux. The recommended way to create topics is still via the broker, you shouldn't need to connect directly to zookeeper.
It turned out to be that the shell scripts need a little more configuration when connecting to a TLS endpoint:
Copy the certs linked to by your jdk to a temporary location:
cp /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts /tmp/kafka.client.truststore.jks
Make a properties file (e.g. client.properties)
security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks
Then try running the script again, while passing the option --command-config with your properties file e.g.:
./kafka-topics.sh --bootstrap-server <server>:<port> --list --command-config client.properties
Note that the option is not consistent between the different scripts, for the console consumer / producer you'll need:
--consumer.config and --producer.config
Replacing bootstrap-server with zookeeper fixed the issue.
For version 2.* you have to create the topic using zookeper with the default port 2181 as a parameter.
For the version 3.* the zookeeper is not any more a parameter, you should use --bootstrap-server using localhost or the IP adresse of the server and the default port 9092.
Documentation
Check your broker after you get this error, Kafka broker will give you the correct IP Address in the console which is running in a different terminal.
In my case I replaced 127.0.0.1:2181 with 192.168.0.21:9092 and I was able to create a new topic successfully.
Note: use bootstrap instead of zookeeper.
For mac, It worked for me as follow when I use the bootstrap server with the kafka server port. Initially, it failed when I tried with a zookeeper.
bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic firsttopic --create --partitions 3 --replication-factor 1
I faced the same issue, but everything was working fine before.
So I changed the data directory in both the properties files: zookeeper.properties and server.properties and it started working fine again.
It could be because I didn't shut down the broker and zookeeper properly before.
I have written a kafka consumer and producer that worked fine until today.
This morning, when I started zookeeper and kafka, my consumer was not able to read messages, and I found this in the zookeeper logs:
INFO Got user-level KeeperException when processing sessionid:0x151c41e62e10000
type:create cxid:0x2a zxid:0x1e txntype:-1 reqpath:n/a
Error Path:/brokers/ids
Error:KeeperErrorCode = NodeExists for /brokers/ids
(org.apache.zookeeper.server.PrepRequestProcessor)
Look for log.dirs in your server.properties file and delete all the Kafka and zookeeper logs from there and try restarting zookeeper and Kafka respectively. I was facing the same issue and doing this resolved it.
According to Confluent at https://groups.google.com/forum/#!topic/confluent-platform/h0gEik_Ii1E on 2016/10/08
Those are not errors, you can see the log level is INFO. It is simply
logging that Kafka tried to create a node that already exists. Totally
normal behavior for Kafka and nothing to worry about.
Is there an actual problem related to the message or is everything working correctly?
go to Kafka root directory and look for the logs file. and clear all logs. For instance:
say your kafka is installed in the downloads folder:
cd ~/Downloads/kafka_2.13-2.6.0
rm -rf logs
It will resolve the issue.
I have searched it all over and couldn't find the error.
I have checked This Stackoverflow Issue but it is not the problem with me
I have started a zookeeper server
Command to start server was
bin/zookeeper-server-start.sh config/zookeeper.properties
Then I SSH into VM by using Putty and started kafka server using
$ bin/kafka-server-start.sh config/server.properties
Then I created Kafka Topic and when I list the topic, it appears.
Then I opened another putty and started kafka-console-producer.sh and typed any message (even enter) and get this long repetitive exception.
Configuration files for zookeeper.properties, server.properties, kafka-producer.properties are as following (respectively)
The version of Kafka i am running is 8.2.2. something as I saw it in kafka/libs folder.
P.S. I get no messages in consumer.
Can any body figure out the problem?
The tutorial I was following was [This][9]
8http://%60http://www.bogotobogo.com/Hadoop/BigData_hadoop_Zookeeper_Kafka_single_node_single_broker_cluster.php%60
On the hortonworks sandbox have a look at the server configuration:
$ less /etc/kafka/conf/server.properties
In my case it said
...
listeners=PLAINTEXT://sandbox.hortonworks.com:6667
...
This means you have to use the following command to successfully connect with the console-producer
$ cd /usr/hdp/current/kafka-broker
$ bin/kafka-console-producer.sh --topic test --broker-list sandbox.hortonworks.com:6667
It won't work, if you use --broker-list 127.0.0.1:6667 or --broker-list localhost:6667 . See also http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/configure_kafka.html
To consume the messages use
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
As you mentioned in your question that you are using HDP 2.3 and for that when you are running Console-Producer
You need to provide sandbox.hortonworks.com:6667 in Broker-list.
Please use the same while running Console-Consumer.
Please let me know in case still you face any issue.
Within Kafka internally there is a conversation that goes on between both producers and consumers (clients) and the broker (server). During those conversations clients often ask the server for the address of a server broker that's managing a particular partition. The answer is always a fully-qualified host name. Without going into specifics if you ever refer to a broker with an address that is not that broker's fully-qualified host name there are situations when the Kafka implementation runs into trouble.
Another mistake that's easy to make, especially with the Sandbox, is referring to a broker by an address that's not defined to the DNS. That's why every node on the cluster has to be able to address every other node in the cluster by fully-qualified host name. It's also why, when accessing the sandbox from another virtual image running on the same machine you have to add sandbox.hortonworks.com to the image's hosts file.
After multiple re-tries and changing the bat files, I have finally succeeded in running Kafka and Zookeeper in Windows 7. This is a 32-bit machine running Java 7. Unfortunately though, I am not able to create a topic. I tried using the tutorial here: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
I executed the command:
C:\Cambria\kafka_2.8.0-0.8.0>bin\kafka-create-topic.bat --zookeeper localhost:2181 --replica 1 --partition 1 --topic topic
and end up with the following error:
Exception in thread "main" joptsimple.UnrecognizedOptionException: 'û' is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:89)
at joptsimple.OptionParser.validateOptionCharacters(OptionParser.java:586)
at joptsimple.OptionParser.handleShortOptionCluster(OptionParser.java:511)
at joptsimple.OptionParser.handleShortOptionToken(OptionParser.java:506)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:59)
at joptsimple.OptionParser.parse(OptionParser.java:433)
at kafka.admin.CreateTopicCommand$.main(CreateTopicCommand.scala:56)
at kafka.admin.CreateTopicCommand.main(CreateTopicCommand.scala)
All the classpath entries are proper and the requisite JAR files are also present. Am not sure what the issue is. Can someone help me on this?
P.S: I tried this with Cygwin and am encountering the Class Not Found error although the classpath is being properly set.
For people who just started to play with kafka and encountered this problem (but on Unix machine), this is due to the outdated tutorial you are following.
In the latest version, you have to use
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
to create topic.
run bin\kafka-create-topic.bat to see available options
then for example you will see broker-list and all required parameters.