I am trying to connect to kafka through spring xd kafka source.
I need to configure the Kerberos and TLS.
My Requirement is to enable Kerberos and also give client.properties like below to consumer.
ssl.keystore.location=/var/private/ssl/sdp.cloud.keystore.jks
ssl.keystore.password=abc123
ssl.key.password=abc123
ssl.truststore.location=/var/private/ssl/sdp.cloud.keystore.jks
ssl.truststore.password=abc123
ssl.enabled.protocols=TLSv1.2
ssl.keystore.type=JKS
ssl.truststore.type=JKS
security.protocol=SASL_SSL
sasl.kerberos.service.name=kafka
group.id=testgroup
The sample provided by kafka is like below:
kafka-console-consumer --new-consumer --bootstrap-server servers --topic test --from-beginning --consumer.config /etc/kafka/client.properties
--Updated
How can i give above mentioned properties to kafka source module?
Related
I want to see all the topic which I created in the Kafka. I used this command to get all the topics but it doesn't work.
bin/kafka-topics.sh --list --zookeeper localhost:2181
I resolved the problem. I did run it with kafka-topics.bat file in Windows directory inside of the bin folder of Kafka.
kafka-topics.bat --list --zookeeper localhost:2181
I have configured kafka_2.11-2.3.0 and apache-zookeeper-3.5.5-bin on Windows 10. But while running the topic creation command I am getting the below error:
C:\kafka_2.11-2.3.0>.\bin\windows\kafka-topics.bat --create --bootstrap-server 127.0.0.1:2181 --partitions 1 --replication-factor 1 --topic testD1
Error while executing topic command : org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[2019-10-14 16:42:40,603] ERROR java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:178)
at kafka.admin.TopicCommand$TopicService$class.createTopic(TopicCommand.scala:149)
at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:172)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:60)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
Read somewhere in stackoverflow to add listeners=PLAINTEXT://127.0.0.1:9092 in the server.properties file but that didn't work out as expected.
Zookeeper runs on 2181, not Kafka (the bootstrap server)
By default, Kafka runs on port 9092 as below
kafka-topics --bootstrap-server 127.0.0.1:9092 --topic first_topic --create --partitions 3 --replication-factor 1
I've struggled with the same issue on linux. The recommended way to create topics is still via the broker, you shouldn't need to connect directly to zookeeper.
It turned out to be that the shell scripts need a little more configuration when connecting to a TLS endpoint:
Copy the certs linked to by your jdk to a temporary location:
cp /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts /tmp/kafka.client.truststore.jks
Make a properties file (e.g. client.properties)
security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks
Then try running the script again, while passing the option --command-config with your properties file e.g.:
./kafka-topics.sh --bootstrap-server <server>:<port> --list --command-config client.properties
Note that the option is not consistent between the different scripts, for the console consumer / producer you'll need:
--consumer.config and --producer.config
Replacing bootstrap-server with zookeeper fixed the issue.
For version 2.* you have to create the topic using zookeper with the default port 2181 as a parameter.
For the version 3.* the zookeeper is not any more a parameter, you should use --bootstrap-server using localhost or the IP adresse of the server and the default port 9092.
Documentation
Check your broker after you get this error, Kafka broker will give you the correct IP Address in the console which is running in a different terminal.
In my case I replaced 127.0.0.1:2181 with 192.168.0.21:9092 and I was able to create a new topic successfully.
Note: use bootstrap instead of zookeeper.
For mac, It worked for me as follow when I use the bootstrap server with the kafka server port. Initially, it failed when I tried with a zookeeper.
bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic firsttopic --create --partitions 3 --replication-factor 1
I faced the same issue, but everything was working fine before.
So I changed the data directory in both the properties files: zookeeper.properties and server.properties and it started working fine again.
It could be because I didn't shut down the broker and zookeeper properly before.
I was able to get kafka to work fine when I spun it up on my local machine. But when I try to get it to work on an AWS instance nothing seems to work right. I tried spinning up my own server and doing just like I did locally spinning up zookeeper and kafka like so
curl http://apache.spinellicreations.com/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz | tar -xzf
cd kafka_2.11-0.10.0.0
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
I also tried using the AMI from bitami which seems to be an all in one AMI. Creating the topic seems to work fine. But when I try to run the console producer I get an error
SEASPAULSON-MAC:kafka_2.11-0.10.0.0 spaulson$ bin/kafka-console-producer.sh --broker-list ec2-54-186-31-109.us-west-2.compute.amazonaws.com:9092 --topic test
blah
[2016-10-20 12:13:23,395] ERROR Error when sending message to topic test with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for test-0
I also get an error when I try to start up a console consumer that repeats over and over.
bin/kafka-console-consumer.sh --zookeeper ec2-54-186-31-109.us-west-2.compute.amazonaws.com:2181 --topic test --from-beginning
[2016-10-19 18:26:47,175] WARN Fetching topic metadata with correlation id 152 for topics [Set(test)] from broker [BrokerEndPoint(0,ip-172-31-52-58.ec2.internal,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:80)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
I feel like these kinds of operations should be trivial but it's proving very challenging. I'm having trouble finding documentation on how to diagnose issues and figure out what's going wrong. The best I found is this command
KAFKA_HOME/bin/kafka-topics.sh --describe --topic test --zookeeper ec2-54-186-31-109.us-west-2.compute.amazonaws.com:2181
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Does the Leader: 0 indicate something went wrong? But what?
For AWS or any other IaaS machines, you should set "advertised.listeners" for the clients. Here is what this options means in the Kafka doc:
Listeners to publish to ZooKeeper for clients to use, if different than the listeners above. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value for listeners will be used.
I'm trying to send data from kafka (eventually we'll use kafka running on a different instance) to hdfs. I think flume or some sort of ingestion protocol is necessary to get data into hdfs. So we're using cloudera's flume service and hdfs.
This is my flume-conf file. The other conf file is empty
tier1.sources=source1
tier1.channels=channel1
tier1.sinks=sink1
tier1.sources.source1.type=org.apache.flume.source.kafka.KafkaSource
tier1.sources.source1.zookeeperConnect=localhost:2181
tier1.sources.source1.topic=test
tier1.sources.source1.groupId=flume
tier1.sources.source1.channels=channel1
tier1.sources.source1.interceptors=i1
tier1.sources.source1.interceptors.i1.type=timestamp
tier1.sources.source1.kafka.consumer.timeout.ms=100
tier1.channels.channel1.type=memory
tier1.channels.channel1.capacity=10000
tier1.channels.channel1.transactionCapacity=1000
tier1.sinks.sink1.type=hdfs
tier1.sinks.sink1.hdfs.path=/tmp/kafka/test/data
tier1.sinks.sink1.hdfs.rollInterval=5
tier1.sinks.sink1.hdfs.rollSize=0
tier1.sinks.sink1.hdfs.rollCount=0
tier1.sinks.sink1.hdfs.fileType=DataStream
When I start a kafka consumer it can get messages from a kafka producer just fine on localhost:2181. But I don't see any errors from the flume agent and nothing gets put into hdfs. I also can't find any log files.
This is how I start the agent.
flume-ng agent --conf /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/flume-ng/conf --conf-file flume-conf --name agent1 -Dflume.root.logger=DEBUG,INFO,console
Help please?
Fixed it.
Have to change
--name agent1
to --name tier1
I have searched it all over and couldn't find the error.
I have checked This Stackoverflow Issue but it is not the problem with me
I have started a zookeeper server
Command to start server was
bin/zookeeper-server-start.sh config/zookeeper.properties
Then I SSH into VM by using Putty and started kafka server using
$ bin/kafka-server-start.sh config/server.properties
Then I created Kafka Topic and when I list the topic, it appears.
Then I opened another putty and started kafka-console-producer.sh and typed any message (even enter) and get this long repetitive exception.
Configuration files for zookeeper.properties, server.properties, kafka-producer.properties are as following (respectively)
The version of Kafka i am running is 8.2.2. something as I saw it in kafka/libs folder.
P.S. I get no messages in consumer.
Can any body figure out the problem?
The tutorial I was following was [This][9]
8http://%60http://www.bogotobogo.com/Hadoop/BigData_hadoop_Zookeeper_Kafka_single_node_single_broker_cluster.php%60
On the hortonworks sandbox have a look at the server configuration:
$ less /etc/kafka/conf/server.properties
In my case it said
...
listeners=PLAINTEXT://sandbox.hortonworks.com:6667
...
This means you have to use the following command to successfully connect with the console-producer
$ cd /usr/hdp/current/kafka-broker
$ bin/kafka-console-producer.sh --topic test --broker-list sandbox.hortonworks.com:6667
It won't work, if you use --broker-list 127.0.0.1:6667 or --broker-list localhost:6667 . See also http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/configure_kafka.html
To consume the messages use
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
As you mentioned in your question that you are using HDP 2.3 and for that when you are running Console-Producer
You need to provide sandbox.hortonworks.com:6667 in Broker-list.
Please use the same while running Console-Consumer.
Please let me know in case still you face any issue.
Within Kafka internally there is a conversation that goes on between both producers and consumers (clients) and the broker (server). During those conversations clients often ask the server for the address of a server broker that's managing a particular partition. The answer is always a fully-qualified host name. Without going into specifics if you ever refer to a broker with an address that is not that broker's fully-qualified host name there are situations when the Kafka implementation runs into trouble.
Another mistake that's easy to make, especially with the Sandbox, is referring to a broker by an address that's not defined to the DNS. That's why every node on the cluster has to be able to address every other node in the cluster by fully-qualified host name. It's also why, when accessing the sandbox from another virtual image running on the same machine you have to add sandbox.hortonworks.com to the image's hosts file.