I am using the PublishKafka processor in nifi to publish JSON data in Kafka. I want to set Kafka Server and topic name details from attribute values. I am not able to do this. It's giving the below error
PublishKafka_2_0[id=2cb333e9-5fd6-1254-a10c-d1084b230d03] Processing
halted: yielding [1 sec]: org.apache.kafka.common.KafkaException:
Failed to construct kafka producer
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers .
Only hardcoded values of the Kafka server and the topic name is working.
Am I missing something? Any help would be much appreciated!
Related
I'm trying to use the Confluent InfluxDB Sink Connector to get data from a kafka topic into my InfluxDB.
Firstly, I transmit data to kafka topic from a log file by using nifi, and it works well. The kafka topic get the data, like below:
{
"topic": "testDB5",
"key": null,
"value": {
"timestamp": "2019-03-20 01:24:29,461",
"measurement": "INFO",
"thread": "NiFi Web Server-795",
"class": "org.apache.nifi.web.filter.RequestLogger",
"message": "Attempting request for (anonymous)
},
"partition": 0,
"offset": 0
}
Then, I create InfluxDB sink connector through the Kafka Connect UI , and I get the following exception:
org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:587)
at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:323)
at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:226)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:194)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at io.confluent.influxdb.InfluxDBSinkTask.put(InfluxDBSinkTask.java:140)
at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:565)
... 10 more
But if I manually input data to another topic testDB1 by using
./bin/kafka-avro-console-producer --broker-list localhost:9092 --topic testDB1 --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"measurement","type":"string"},{"name":"timestamp","type":"string"}]}'
It works, my influxDB can get the data.
Here is the connect configuration:
connector.class=io.confluent.influxdb.InfluxDBSinkConnector
influxdb.url=http://myurl
tasks.max=1
topics=testDB5
the configuration of connecting topic testDB1 is the same except topics name.
Is there any problems in nifi ? But it can transmit data to topic well.
When you use Avro with Kafka Connect, the Avro deserialiser expects the data to have been serialised using the Avro serialiser. This is what the kafak-avro-console-producer uses, which is why your pipeline works when you use that.
This article gives a good background to Avro and the Schema Registry. See also Kafka Connect Deep Dive – Converters and Serialization Explained.
I'm not familiar with Nifi, but looking at the documentation it seems that AvroRecordSetWriter has the option to use Confluent Schema Registry. At a guess you'll also want to set Schema Write Strategy to Confluent Schema Registry Reference.
Once you can consume data from your topic with kafka-avro-console-consumer then you know that it is correctly serialised and will work with your Kafka Connect sink.
I found the reason. It's because in Nifi, I used PublishKafka_0_10 to publish the data to Kafka topic, but its version is to low!
When I make a query in ksql, it says that
Input record ConsumerRecord(..data..) has invalid (negative) timestamp.
Possibly because a pre-0.10 producer client was used to write this record to Kafka without embedding a timestamp,
or because the input topic was created before upgrading the Kafka cluster to 0.10+. Use a different TimestampExtractor to process this data.
So, I change it to PublishKafka_1_0 , and start again, and it works! My influxDB can get the data. I'm speechless.
And thanks Robin Moffatt for the reply, its very helpful to me.
I need to index documents from an ActiveMQ broker into Elasticsearch. I am using logstash for that purpose, with the JMS input plugin (that is build upon jriver-jms). As I need to ensure that every document stored in a topic of the ActiveMQ broker gets into Elasticsearch, I want to establish a durable subscription with logstash. How can I do that ? There is no parameter such as "durable" in the JMS input plugin configuration.
You can add jms.clientID parameter (http://activemq.apache.org/connection-configuration-uri.html) into connection URL
I'm trying to get the messages using a GetKafka processor but I'm not able to get it. I tested consuming messages using Kafka command line consumer and it works. I was also able to use PutKafka processor successfully to put the messages in the topic. Attached my settings where I set the Zookeeper Connection String and Topic Name. When I run the flow, I don't see any errors in the processors.
GetKafka Processor
I see an exception in nifi-app.log:
2016-08-03 09:34:33,722 WARN [70e1df87-6097-4ed0-9a40-7e36f9be6921_mydomain.com-1470231250839-1fbd0cfe-leader-finder-thread] kafka.client.ClientUtils$ Fetching topic metadata with correlation id 0 for topics [Set(test)] from broker [id:0,host:DataSlave1.CSE-RD.com,
port:9092] failed
java.nio.channels.ClosedByInterruptException: null
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) ~[na:1.8.0_101]
at sun.nio.ch.SocketChannelImpl.poll(SocketChannelImpl.java:957) ~[na:1.8.0_101]
Kafka (0.8): 2.10-0.8.2.1, Nifi: 0.7.0
Am I missing anything? Thanks.
The exception went away after a restart. GetKafka is now able to get the messages as they are sent in the producer. It was not receiving the previous messages in the topic (something equivalent to --from-beginning in kafka console consumer). I don't see a setting for that in the processor.
Reg: JMeter Point-to-Point
Could you please tell how to get the QueueConnnectionFaction
And ContextConnectionFactory
Can anyone help me in configuring JNDI properties in JMS POINT-TO-POINT of JMETER
ERROR - jmeter.protocol.jms.sampler.JMSSampler: progress/message/jclient/TopicConnectionFactory java.lang.NoClassDefFoundError: progress/message/jclient/TopicConnectionFactory
Please provide some solution.
It seems you didn't put a dependency jar from your jms provider, see:
java.lang.NoClassDefFoundError: progress/message/jclient/TopicConnectionFactory
The QueueConnnectionFactory is the container of the queues that is created before the queues. I used to use the Hermes JMS application to get this information but you could get it from WebLogic too.
For example, the next image shows my configuration, using the information of binding for the QueueConnectionFactory field, and using the information of initialContextFactory for the InitialContextFactory field, and do not forget to use the providerURL in the ProviderURL field.
In this example I am not getting the response. For this reason, I am only putting the name of my requester queue.
Ensure that the information is correct. In the JMS Properties you could use Headers for the request.
Quarkus (1.5.0.Final) as native executable works fine with Kafka Streams and Avro Schema Registry.
But in case of a Kafka streams consuming a topic with Avro Serdes, if a new event is added, there is an exception:
The kafka-streams-avro-serde library try to reach (via REST API) the Schema Registry with the schema added.
This exception below occured: (This works fine in Qaurkus + JVM)
Caused by: org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"siteconf","namespace":"test","fields":[{"name":"id","type":["null","string"],"default":null},{"name":"site","type":["null","string"],"default":null},{"name":"configuration","type":["null","string"],"default":null}],"connect.name":"siteconf"}
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Internal Server Error; error code: 500
I don't know how to workaround this problem.
It's very annoying because I think it's the only one problem I've detected in Kafka Streams with Schema Registry.
And I was interrested to adaopt Quarkus instead of Spring Boot/Cloud