I am using the PublishKafka processor in nifi to publish JSON data in Kafka. I want to set Kafka Server and topic name details from attribute values. I am not able to do this. It's giving the below error
PublishKafka_2_0[id=2cb333e9-5fd6-1254-a10c-d1084b230d03] Processing
halted: yielding [1 sec]: org.apache.kafka.common.KafkaException:
Failed to construct kafka producer
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers .
Only hardcoded values of the Kafka server and the topic name is working.
Am I missing something? Any help would be much appreciated!
I'm trying to use the Confluent InfluxDB Sink Connector to get data from a kafka topic into my InfluxDB.
Firstly, I transmit data to kafka topic from a log file by using nifi, and it works well. The kafka topic get the data, like below:
{
"topic": "testDB5",
"key": null,
"value": {
"timestamp": "2019-03-20 01:24:29,461",
"measurement": "INFO",
"thread": "NiFi Web Server-795",
"class": "org.apache.nifi.web.filter.RequestLogger",
"message": "Attempting request for (anonymous)
},
"partition": 0,
"offset": 0
}
Then, I create InfluxDB sink connector through the Kafka Connect UI , and I get the following exception:
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:587)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:323)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:226)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:194)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at io.confluent.influxdb.InfluxDBSinkTask.put(InfluxDBSinkTask.java:140)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:565)
... 10 more
But if I manually input data to another topic testDB1 by using
./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic testDB1 --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"measurement","type":"string"},{"name":"timestamp","type":"string"}]}'
It works, my influxDB can get the data.
Here is the connect configuration:
connector.class=io.confluent.influxdb.InfluxDBSinkConnector
influxdb.url=http://myurl
tasks.max=1
topics=testDB5
the configuration of connecting topic testDB1 is the same except topics name.
Is there any problems in nifi ? But it can transmit data to topic well.
When you use Avro with Kafka Connect, the Avro deserialiser expects the data to have been serialised using the Avro serialiser. This is what the kafak-avro-console-producer uses, which is why your pipeline works when you use that.
This article gives a good background to Avro and the Schema Registry. See also Kafka Connect Deep Dive – Converters and Serialization Explained.
I'm not familiar with Nifi, but looking at the documentation it seems that AvroRecordSetWriter has the option to use Confluent Schema Registry. At a guess you'll also want to set Schema Write Strategy to Confluent Schema Registry Reference.
Once you can consume data from your topic with kafka-avro-console-consumer then you know that it is correctly serialised and will work with your Kafka Connect sink.
I found the reason. It's because in Nifi, I used PublishKafka_0_10 to publish the data to Kafka topic, but its version is to low!
When I make a query in ksql, it says that
Input record ConsumerRecord(..data..) has invalid (negative) timestamp.
Possibly because a pre-0.10 producer client was used to write this record to Kafka without embedding a timestamp,
or because the input topic was created before upgrading the Kafka cluster to 0.10+. Use a different TimestampExtractor to process this data.
So, I change it to PublishKafka_1_0 , and start again, and it works! My influxDB can get the data. I'm speechless.
And thanks Robin Moffatt for the reply, its very helpful to me.
I was trying to implement Windows Service Bus 1.1 with AMQP for my project,
I have installed Windows Service Bus 1.1 and created a Queue with the name test using Service Bus Explorer,
I am testing the sample Service bus functionality with the Java program given in the below link,
https://msdn.microsoft.com/en-us/library/dn574799.aspx
In the example given above link i have removed if else code starting with if(type.compareTo("onprem") == 0) and hard coded my connection string.
And creating the producer as below
Destination que = (Destination) context.lookup("QUEUE"); // reading form properties file
producer = session.createProducer(que);
My connection string and Queue name looks like the below one
connectionstring = amqps://username:pwd#machinename/SampleNameSpace
QueueName = SampleNameSpace/test
When i run the Java program i am getting the ConcurrentTimeOutException when creating the producer.
I am pretty much new to the Service Bus thing and i tried to find out some solution online but it wasn't successful.
Below is my console log
Initial setup
Creating context
Creating connection factory
Creating connection
Creating session
Creating queue
Creating producer
Exception creating producer
javax.jms.JMSException: java.util.concurrent.TimeoutException
at org.apache.qpid.amqp_1_0.jms.impl.MessageProducerImpl.<init>(MessageProducerImpl.java:98)
at org.apache.qpid.amqp_1_0.jms.impl.SessionImpl.createProducer(SessionImpl.java:390)
at org.apache.qpid.amqp_1_0.jms.impl.SessionImpl.createProducer(SessionImpl.java:59)
at prerna.jms.test.SBTest.main(SBTest.java:71)
Caused by: org.apache.qpid.amqp_1_0.client.Sender$SenderCreationException: java.util.concurrent.TimeoutException
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:178)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:119)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:112)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:98)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:84)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:78)
at org.apache.qpid.amqp_1_0.client.Session$1.<init>(Session.java:90)
at org.apache.qpid.amqp_1_0.client.Session.createSender(Session.java:89)
at org.apache.qpid.amqp_1_0.jms.impl.MessageProducerImpl.<init>(MessageProducerImpl.java:86)
... 3 more
Caused by: java.util.concurrent.TimeoutException
at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.waitUntil(ConnectionEndpoint.java:1232)
at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.waitUntil(ConnectionEndpoint.java:1214)
at org.apache.qpid.amqp_1_0.transport.SessionEndpoint.waitUntil(SessionEndpoint.java:681)
at org.apache.qpid.amqp_1_0.transport.LinkEndpoint.waitUntil(LinkEndpoint.java:355)
at org.apache.qpid.amqp_1_0.client.Sender.<init>(Sender.java:167)
... 11 more
Any help will be greatly appreciated.
You appear to be using the legacy AMQP 1.0 JMS client from the Qpid project which is no longer supported. The first thing I'd suggest is switching to the newer AMQP 1.0 JMS client which you can find on the Qpid site.
Here's a link to the Qpid JMS client examples mirrored on Github, and here's what your maven dep should look like.
<dependency>
<groupId>org.apache.qpid</groupId>
<artifactId>qpid-jms-client</artifactId>
<version>0.23.0</version>
</dependency>
I'm trying to get the messages using a GetKafka processor but I'm not able to get it. I tested consuming messages using Kafka command line consumer and it works. I was also able to use PutKafka processor successfully to put the messages in the topic. Attached my settings where I set the Zookeeper Connection String and Topic Name. When I run the flow, I don't see any errors in the processors.
GetKafka Processor
I see an exception in nifi-app.log:
2016-08-03 09:34:33,722 WARN [70e1df87-6097-4ed0-9a40-7e36f9be6921_mydomain.com-1470231250839-1fbd0cfe-leader-finder-thread] kafka.client.ClientUtils$ Fetching topic metadata with correlation id 0 for topics [Set(test)] from broker [id:0,host:DataSlave1.CSE-RD.com,
port:9092] failed
java.nio.channels.ClosedByInterruptException: null
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[na:1.8.0_101]
at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:957) ~[na:1.8.0_101]
Kafka (0.8): 2.10-0.8.2.1, Nifi: 0.7.0
Am I missing anything? Thanks.
The exception went away after a restart. GetKafka is now able to get the messages as they are sent in the producer. It was not receiving the previous messages in the topic (something equivalent to --from-beginning in kafka console consumer). I don't see a setting for that in the processor.
We are using elasticsearch 0.90.0 and java version "1.7.0_25".
We migrate data from oracle DB to Hadoop through a executable jar kept at DB server.after 15-20 mins of successful running, we get following exception
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:343)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:95)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:51)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
at org.elasticsearch.transport.netty.NettyTransport.doStart(NettyTransport.java:240)
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:90)
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:179)
at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:119)
No exception is caught either in namenode/datanode or elasticsearch logs. This error is caught at DB Server but I think it is related to elastic search.
Looks like you're creating too many Netty clients is my guess which is in turn eating up all your threads. Perhaps wrap your Netty client pool with a service that inject? See Helter Skelter's comment on this answer https://stackoverflow.com/a/5253186/266531