Is it possible to create a Kafka Connector without REST request?
I have started my worker (distributed) through a java code and want my connector also gets started along with it. I dont want to use REST call (not from browser and not from code) to create my connector.
I just want a simple kafka api to invoke which creates my connector.
Any help will be appreciated.
Related
i have to leverage kafka connector as source but creating connector by REST api in production is something I want to avoid, is it possible to create connector during startup without using REST api?
I understand REST api provides the flexibility to dynamically create/configure connectors and size the tasks, but really want to if the same can be done during startup either by providing any configuration problem.
Currently I start connector in distributed mode by supplying properties file and I want to mention the database, filters, transformers and other details there itself.
Let me know if there is a way to achieve.
I'm trying to find examples of kafka connect with springboot. It looks like there is no spring boot integration for kafka connect. Can some one point me in the right direction to be able to listen to changes on mysql db?
Kafka Connect doesn't really need Spring Boot because there is nothing for you to code for it, and it really works best when ran in distributed mode, as a cluster, not embedded within other (single-instance) applications. I suppose if you did want to do it, then you could copy relevent portions of the source code, but that of course isn't using Spring Boot, and you'd have to wire it all yourself
The framework itself consists of a few core Java dependencies that have already been written (Debezium or Confluent JDBC Connector, for your mysql example), and two config files. One for Kafka Connect to know the bootstrap servers, serializers, etc. and another for the actual MySQL connector. So, if you want to use Kafka Connect, run it by itself, then just write the consumer in the Spring app.
The alternatives to Kafka Connect itself would be to use Apache Camel within a Spring application (Spring Integration) or Spring Cloud Dataflow and interfacing with those Kafka "components" (which aren't using the Connect API, AFAIK)
Another option, specific for listening to MySQL, is to use Debezium Engine within your code.
I'm trying to run kafka connect in distributed mode. Is REST the only way to start the connector? Otherwise, what commands should i use ?
The Kafka Connect API is part of Apache Kafka. You can run it standalone, or as part of Confluent Platform.
Information on how to configure it is here. If you are using distributed mode, you have to use the REST API to configure it.
I am trying to use Spring Boot 1.5.2.RELEASE + Camel (Spring Boot starter) 2.19.2 to listen to ActiveMQ queue and then post the message to a rest endpoint URL (POST method) as its body. What would be the best possible way to achieve this?
I have gathered pieces of information and am trying to tie it all together but getting a bit confused.
Here is what I have gathered for Camel Rest DSL, I am not too sure if camel below is creating these rest services via this or is it just an already exposed endpoint, in my case it is an already exposed endpoint
rest("/basePath")
post("/someEndpoint").to("direct:restEndpoint")
Using the above is what I have gathered for ActiveMQ which I am not too sure is correct
from("activemq:queue:<queue_name>").to("direct:restEndpoint")
But again, I am not too sure how to listen to the ActiveMQ queue for new messages or is it something that Camel would do by default always? Additionally, I need to pass the message as a post body to my rest endpoint. I also saw some references to camel-http4 and camel-http as well and I am completely confused.
Any assistance would be greatly appreciated.
Some confusion is common when starting to use Camel, but your final solution will look something like:
from("activemq:queue:my-route")
.process(/* change the in/out messages if you need to */)
.to("http4://your-endpoint.com");
Don't try to simply copy/paste this code until it works. My Camel rule of thumb is: always read the component documentation and try playing with it using it in your software. In your case I suggest:
Read ActiveMQ component docs and try reading from ActiveMQ / writing to a Log;
Generate some input from a Timer and send to your Rest endpoint using HTTP4 Component;
Your first routes will take some time for simple things but you will get on flow quickly.
I need to know which framework or API to use for my following requirement. I am currently using native java code for all this.
Requirement
I have an application where there could be multiple JMS/Rest/TCP connections. These connections can grow at runtime. User wil have a screen to define new incoming or outgoing connections. I am using Native and works fine but I want to make use of an efficient framework or API like Spring, Camel etc ?
Need Guidance.
I have been able to get this all working. There are multiple solutions to get do dynamic JMS
1. I used Spring JMS API and created the dynamic JMS connections by loading the dynamic child context into application. For this I followed the spring's dynamic FTP sample and inserted JMS beans in the example instead of FTP ones.
Spring Dynamic FTP Sample