I am using confluent Kafka Go for my project. When writing tests, because of the asynchronous nature of Kafka when creating a topic, I might be have errors (error code 3: UNKNOWN_TOPIC_OR_PARTITION) when create the topic then get back immediately.
As I understood, if I can query directly on the controller, I can always get the lastest meta data. So my question is: How can I get Kafka controller's IP or ID when using Confluent Kafka Go.
Related
I have installed the docker version of IBM MQ based on the following link
https://developer.ibm.com/tutorials/mq-connect-app-queue-manager-containers/
Then I created new topic with the following specs:
Name: PROD.TEST
Topic string: dev/test/
Then from C# client I am using dev/test/ to create subscriber to the created topic:
destination = sessionWMQ.CreateTopic(env.Conn.topic_name); subscriber
= sessionWMQ.CreateConsumer(destination);
For some reason if the Topic name doesn't start with DEV. the second line throws the following exception:
XMSException caught: IBM.XMS.IllegalStateException: Failed to
subscribe to topic dev/test/ using MQSUB. There may have been a
problem creating the subscription due to it being used by another
message consumer. Make sure any message consumers using this
subscription are closed before trying to create a new subscription
under the same name.
Linked Exception : CompCode: 2, Reason: 2035
To get you started quickly, container image of MQ's developer edition pre-authorises a user called "app" to be able to connect to the queue manager and access a set of predefined queues and topics. These are the DEV.* queues and the "dev/" branch of the topic tree through the DEV.BASE.TOPIC definition. This is explained here
You can then build on this by adding queues and topics and granting access to these as you require.
To do this with MQ's CLI (runmqsc) you would use the SET AUTHREC command. Or to use the web interface you would click on the configuration of the new topic and select the security tab. You'll need to grant publish or subscribe authority depending on what the application wants to do.
Obviously, this just gets you going, as you move forward you'll want to review the security requirements and decide how to configure MQ to provide this.
I am trying to bulk an Streaming queue kafka to clickhouse following the steps of official web page https://clickhouse.yandex/docs/en/table_engines/kafka.html and there is no way to make that it's run ok.
I've checked kafka configuration and it's ok because I've created a feeder for this queue and I've added to clickhouse configuration the zookeeper's host and port.
For example, the sentence from eclipse is :
System.out.println(ck.connection.createStatement().execute("CREATE TABLE IF NOT EXISTS test.clickhouseMd5 ("st1 String," + "st2 String," + "st3 String) ENGINE = Kafka('node2:2181', 'TestTopic', 'testConsumerGroup', 'JSONEachRow')"));
The result of System.out.println() is always false and there isn't exceptions.
Any ideas?
Thanks,
kind regards.
could you try run your query via commandline clickhouse-client
on clickhouse node?
cliclhouse-client -c "CREATE TABLE IF NOT EXISTS test.clickhouseMd5 (st1 String,st2 String, st3 String) ENGINE = Kafka('node2:2181', 'TestTopic', 'testConsumerGroup', 'JSONEachRow')"
You use port 2181 which is a default port for zookeeper.
But according to documentation you mentioned (https://clickhouse.yandex/docs/en/table_engines/kafka.html) you should specify in the first argument a comma-separated list of brokers (localhost:9092).
Also note that it may not work with old Kafka version. For example with 0.9.0.1. With this version CREATE TABLE command returns OK. But in kafka logs I have error like 'ERROR Processor got uncaught exception. (kafka.network.Processor) java.lang.ArrayIndexOutOfBoundsException: 18'.
But with the latest Kafka 0.11.0.2 it works fine for me.
I'm looking for some advice on setting up a Spring Data Flow stream for a specific use case.
My use case:
I have 2 RDBMS and I need to compare the results of queries run against each. The queries should be run roughly simultaneously. Based on the result of the comparison, I should be able to send an email through a custom email sink app which I have created.
I envision the stream diagram to look something like this (sorry for the paint):
The problem is that SDF does not, to my knowledge, allow a stream to be composed with 2 sources. It seems to me that something like this ought to be possible without pushing the limits of the framework too far. I'm looking for answers that provide a good approach to this scenario while working within the SDF framework.
I am using Kafka as a message broker and the data flow server is using mysql to persist stream information.
I have considered creating a custom Source app which polls two datasources and sends the messages on the output channel. This would eliminate my requirement of 2 sources, but it looks like it would require a significant amount of customization of the jdbc source application.
Thanks in advance.
I have not really tried this, but you should be able to use named destinations to achieve that. Take a look here: http://docs.spring.io/spring-cloud-dataflow/docs/current-SNAPSHOT/reference/htmlsingle/#spring-cloud-dataflow-stream-advanced
stream create --name jdbc1 --definition "jdbc > :dbSource"
stream create --name jdbc2 --definition "jdbc > :dbSource"
stream create --name processor --definition ":dbSource > aggregator | sink"
I am new in Learning of Spring cloud Task and SCDF so asking this.
I wand to execute my SCT based on an event (say a message is posted into Rabbit MQ), so I am think it can be done in two ways:
Create a source which polls message from RabbitMQ and sends the data to stream, now create a sink which reads data from stream and as soon as data comes to sink (from source stream) Task will be launched.
create steam producer --definition "rabbitproducer | streamconsumer (This is #TaskEnabled)"
Not sure if this is possible?
Other way could be to use task launcher. Here task launcher will be configured with a stream and a listener will be polling message from rabbitMQ. so when a message is received then trigger will initiate the process and tasklauncher will launch the task. But here not sure how will i get the message data into my task? Do I have to add the data into TaskLaunchRequest?
create stream mystream --definition "rabbitmsgtrigger --uri:my task | joblauncher"
Launching a Task by an upstream event is already supported and there are few approaches to it - please review the reference guide (and the sample) for more details.
Here is the complete explanation about my question's answer. Here Sabby has helped me a lot for resolving my issue.
Problem: I was not able to trigger my task using tasklauncher/task-sink. In the log also I was not getting correct details and I even did not know how to set to set log level correctly.
Solution: With the help of Sabby and documentations provided on SCT site I could resolve this and have moved ahead in my POC work. Below are the detailed steps I did.
Started my SCDF with postgresql database by referring to the property file and setting log level change as
--logging.level.org.springframework.cloud=DEBUG
--spring.config.location=file://scdf.properties
Imported apps from bitly.
app import --uri [stream applications link][1]
Registered task sink app
app register --name task-sink --type sink --uri file://tasksink-1.1.0.BUILD-SNAPSHOT.jar
Created stream as:
stream create mytasklaunchertest --definition "triggertask --triggertask.uri=https://my-archiva/myproject-scdf-task1/0.0.1-SNAPSHOT/myproject-scdf-task1-0.0.1-20160916.143611-1.jar --trigger.fixed-delay=5 | task-sink"
Deployed Stream:
stream deploy foo --properties "app.triggertask.spring.rabbitmq.host=host,app.triggertask.spring.rabbitmq.username=user,app.triggertask.spring.rabbitmq.password=pass,app.triggertask.spring.rabbitmq.port=5672,app.triggertask.spring.rabbitmq.virtual-host=xxx"
I have a stream as follows,
source(jms-ibmmq) -> Process -> Process -> sink(jdbc-oracle)
Data ingestion works fine. But as part of my stream there is a possibility that my sink(jdbc-oracle) will be down (or) that there is some problem in the network which prevents persistence to the oracle db.
What I am asking is how to handle this failure and what option spring xd can provide out of the box? Is there a pattern thats commonly used to handle these failures in the streams which has caused processing / sink modules?
Please see the comments on this JIRA issue they explain documentation changes we are adding to explain how to configurre dead-lettering in the message bus.
In addition, we have provided mechanisms such that, if all four modules are deployed to the same container (and all containers that match the deployment criteria), we will directly connect the modules such that an error in the sink will be thrown back to the source (causing the JMS message to be rolled back in your case).
This is achieved by setting the module count property to 0 (meaning deploy on all containers that match the criteria - if any - or all containers, if no criteria).
This feature is available on master (it was added after M7).