After multiple re-tries and changing the bat files, I have finally succeeded in running Kafka and Zookeeper in Windows 7. This is a 32-bit machine running Java 7. Unfortunately though, I am not able to create a topic. I tried using the tutorial here: http://janschulte.wordpress.com/2013/10/13/apache-kafka-0-8-on-windows/
I executed the command:
C:\Cambria\kafka_2.8.0-0.8.0>bin\kafka-create-topic.bat --zookeeper localhost:2181 --replica 1 --partition 1 --topic topic
and end up with the following error:
Exception in thread "main" joptsimple.UnrecognizedOptionException: 'รป' is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:89)
at joptsimple.OptionParser.validateOptionCharacters(OptionParser.java:586)
at joptsimple.OptionParser.handleShortOptionCluster(OptionParser.java:511)
at joptsimple.OptionParser.handleShortOptionToken(OptionParser.java:506)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:59)
at joptsimple.OptionParser.parse(OptionParser.java:433)
at kafka.admin.CreateTopicCommand$.main(CreateTopicCommand.scala:56)
at kafka.admin.CreateTopicCommand.main(CreateTopicCommand.scala)
All the classpath entries are proper and the requisite JAR files are also present. Am not sure what the issue is. Can someone help me on this?
P.S: I tried this with Cygwin and am encountering the Class Not Found error although the classpath is being properly set.
For people who just started to play with kafka and encountered this problem (but on Unix machine), this is due to the outdated tutorial you are following.
In the latest version, you have to use
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
to create topic.
run bin\kafka-create-topic.bat to see available options
then for example you will see broker-list and all required parameters.
Related
I've been trying to run both zookeeper and kafka 2.13 on my local windows machine. I have modified the server properties to point to c:/kafka/kafka-logs and zookeeper data to point to c:/kafka/zookeeper-data.
The zookeeper starts without any issues but when I attempt to start kafka with
.\bin\windows\kafka-server-start.bat .\config\server.properties
I get the below error saying AccessDeniedException. I have already tried the below:
Deleting the kafka-logs and zookeeper-data folders and running both zookeeper and kafka again - I still run into the error if I do that
Creating the kafka-logs folder before running kafka - I still get the access denied exception
Running the command prompt as administrator before typing the commands - does not work
Could anyone give some suggestions on how to fix this?
Thank you
what is the kafka version you are using?.
I has this issue in 3.0.0 . I downgraded to 2.8.1 and the issue is resolved.
I think it is something related to kafka.
I reproduced the same issue with Kafka 3.0. Downgrading to 2.8.1 will help.
I want to see all the topic which I created in the Kafka. I used this command to get all the topics but it doesn't work.
bin/kafka-topics.sh --list --zookeeper localhost:2181
I resolved the problem. I did run it with kafka-topics.bat file in Windows directory inside of the bin folder of Kafka.
kafka-topics.bat --list --zookeeper localhost:2181
I have configured kafka_2.11-2.3.0 and apache-zookeeper-3.5.5-bin on Windows 10. But while running the topic creation command I am getting the below error:
C:\kafka_2.11-2.3.0>.\bin\windows\kafka-topics.bat --create --bootstrap-server 127.0.0.1:2181 --partitions 1 --replication-factor 1 --topic testD1
Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[2019-10-14 16:42:40,603] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:178)
at kafka.admin.TopicCommand$TopicService$class.createTopic(TopicCommand.scala:149)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:172)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:60)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
Read somewhere in stackoverflow to add listeners=PLAINTEXT://127.0.0.1:9092 in the server.properties file but that didn't work out as expected.
Zookeeper runs on 2181, not Kafka (the bootstrap server)
By default, Kafka runs on port 9092 as below
kafka-topics --bootstrap-server 127.0.0.1:9092 --topic first_topic --create --partitions 3 --replication-factor 1
I've struggled with the same issue on linux. The recommended way to create topics is still via the broker, you shouldn't need to connect directly to zookeeper.
It turned out to be that the shell scripts need a little more configuration when connecting to a TLS endpoint:
Copy the certs linked to by your jdk to a temporary location:
cp /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts /tmp/kafka.client.truststore.jks
Make a properties file (e.g. client.properties)
security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks
Then try running the script again, while passing the option --command-config with your properties file e.g.:
./kafka-topics.sh --bootstrap-server <server>:<port> --list --command-config client.properties
Note that the option is not consistent between the different scripts, for the console consumer / producer you'll need:
--consumer.config and --producer.config
Replacing bootstrap-server with zookeeper fixed the issue.
For version 2.* you have to create the topic using zookeper with the default port 2181 as a parameter.
For the version 3.* the zookeeper is not any more a parameter, you should use --bootstrap-server using localhost or the IP adresse of the server and the default port 9092.
Documentation
Check your broker after you get this error, Kafka broker will give you the correct IP Address in the console which is running in a different terminal.
In my case I replaced 127.0.0.1:2181 with 192.168.0.21:9092 and I was able to create a new topic successfully.
Note: use bootstrap instead of zookeeper.
For mac, It worked for me as follow when I use the bootstrap server with the kafka server port. Initially, it failed when I tried with a zookeeper.
bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic firsttopic --create --partitions 3 --replication-factor 1
I faced the same issue, but everything was working fine before.
So I changed the data directory in both the properties files: zookeeper.properties and server.properties and it started working fine again.
It could be because I didn't shut down the broker and zookeeper properly before.
I was able to get kafka to work fine when I spun it up on my local machine. But when I try to get it to work on an AWS instance nothing seems to work right. I tried spinning up my own server and doing just like I did locally spinning up zookeeper and kafka like so
curl http://apache.spinellicreations.com/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz | tar -xzf
cd kafka_2.11-0.10.0.0
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
I also tried using the AMI from bitami which seems to be an all in one AMI. Creating the topic seems to work fine. But when I try to run the console producer I get an error
SEASPAULSON-MAC:kafka_2.11-0.10.0.0 spaulson$ bin/kafka-console-producer.sh --broker-list ec2-54-186-31-109.us-west-2.compute.amazonaws.com:9092 --topic test
blah
[2016-10-20 12:13:23,395] ERROR Error when sending message to topic test with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for test-0
I also get an error when I try to start up a console consumer that repeats over and over.
bin/kafka-console-consumer.sh --zookeeper ec2-54-186-31-109.us-west-2.compute.amazonaws.com:2181 --topic test --from-beginning
[2016-10-19 18:26:47,175] WARN Fetching topic metadata with correlation id 152 for topics [Set(test)] from broker [BrokerEndPoint(0,ip-172-31-52-58.ec2.internal,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:80)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
I feel like these kinds of operations should be trivial but it's proving very challenging. I'm having trouble finding documentation on how to diagnose issues and figure out what's going wrong. The best I found is this command
KAFKA_HOME/bin/kafka-topics.sh --describe --topic test --zookeeper ec2-54-186-31-109.us-west-2.compute.amazonaws.com:2181
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Does the Leader: 0 indicate something went wrong? But what?
For AWS or any other IaaS machines, you should set "advertised.listeners" for the clients. Here is what this options means in the Kafka doc:
Listeners to publish to ZooKeeper for clients to use, if different than the listeners above. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value for listeners will be used.
I have searched it all over and couldn't find the error.
I have checked This Stackoverflow Issue but it is not the problem with me
I have started a zookeeper server
Command to start server was
bin/zookeeper-server-start.sh config/zookeeper.properties
Then I SSH into VM by using Putty and started kafka server using
$ bin/kafka-server-start.sh config/server.properties
Then I created Kafka Topic and when I list the topic, it appears.
Then I opened another putty and started kafka-console-producer.sh and typed any message (even enter) and get this long repetitive exception.
Configuration files for zookeeper.properties, server.properties, kafka-producer.properties are as following (respectively)
The version of Kafka i am running is 8.2.2. something as I saw it in kafka/libs folder.
P.S. I get no messages in consumer.
Can any body figure out the problem?
The tutorial I was following was [This][9]
8http://%60http://www.bogotobogo.com/Hadoop/BigData_hadoop_Zookeeper_Kafka_single_node_single_broker_cluster.php%60
On the hortonworks sandbox have a look at the server configuration:
$ less /etc/kafka/conf/server.properties
In my case it said
...
listeners=PLAINTEXT://sandbox.hortonworks.com:6667
...
This means you have to use the following command to successfully connect with the console-producer
$ cd /usr/hdp/current/kafka-broker
$ bin/kafka-console-producer.sh --topic test --broker-list sandbox.hortonworks.com:6667
It won't work, if you use --broker-list 127.0.0.1:6667 or --broker-list localhost:6667 . See also http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/configure_kafka.html
To consume the messages use
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
As you mentioned in your question that you are using HDP 2.3 and for that when you are running Console-Producer
You need to provide sandbox.hortonworks.com:6667 in Broker-list.
Please use the same while running Console-Consumer.
Please let me know in case still you face any issue.
Within Kafka internally there is a conversation that goes on between both producers and consumers (clients) and the broker (server). During those conversations clients often ask the server for the address of a server broker that's managing a particular partition. The answer is always a fully-qualified host name. Without going into specifics if you ever refer to a broker with an address that is not that broker's fully-qualified host name there are situations when the Kafka implementation runs into trouble.
Another mistake that's easy to make, especially with the Sandbox, is referring to a broker by an address that's not defined to the DNS. That's why every node on the cluster has to be able to address every other node in the cluster by fully-qualified host name. It's also why, when accessing the sandbox from another virtual image running on the same machine you have to add sandbox.hortonworks.com to the image's hosts file.