How to consume messages from a Debezium topic from Quarkus? - quarkus
I'm trying to set up an application which produces change events with MySQL+Debezium+Kafka. I'd like to consume messages from the Debezium topic with a Quarkus Microprofile application.
I'm using the following configuration on the Quarkus side to capture incoming messages:
mp.messaging.incoming.customers.connector=smallrye-kafka
mp.messaging.incoming.customers.topic=dbserver1.inventory.customers
mp.messaging.incoming.customers.value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
That works, however the change event, when captured with a StringDeserializer, does not just contain the changed record:
{"schema":{"type":"struct","fields":[{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"first_name"},{"type":"string","optional":false,"field":"last_name"},{"type":"string","optional":false,"field":"email"}],"optional":true,"name":"dbserver1.inventory.customers.Value","field":"before"},{"type":"struct","fields":[{"type":"int32","optional":false,"field":"id"},{"type":"string","optional":false,"field":"first_name"},{"type":"string","optional":false,"field":"last_name"},{"type":"string","optional":false,"field":"email"}],"optional":true,"name":"dbserver1.inventory.customers.Value","field":"after"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"version"},{"type":"string","optional":false,"field":"connector"},{"type":"string","optional":false,"field":"name"},{"type":"int64","optional":false,"field":"ts_ms"},{"type":"string","optional":true,"name":"io.debezium.data.Enum","version":1,"parameters":{"allowed":"true,last,false"},"default":"false","field":"snapshot"},{"type":"string","optional":false,"field":"db"},{"type":"string","optional":true,"field":"table"},{"type":"int64","optional":false,"field":"server_id"},{"type":"string","optional":true,"field":"gtid"},{"type":"string","optional":false,"field":"file"},{"type":"int64","optional":false,"field":"pos"},{"type":"int32","optional":false,"field":"row"},{"type":"int64","optional":true,"field":"thread"},{"type":"string","optional":true,"field":"query"}],"optional":false,"name":"io.debezium.connector.mysql.Source","field":"source"},{"type":"string","optional":false,"field":"op"},{"type":"int64","optional":true,"field":"ts_ms"},{"type":"struct","fields":[{"type":"string","optional":false,"field":"id"},{"type":"int64","optional":false,"field":"total_order"},{"type":"int64","optional":false,"field":"data_collection_order"}],"optional":true,"field":"transaction"}],"optional":false,"name":"dbserver1.inventory.customers.Envelope"},"payload":{"before":null,"after":{"id":1005,"first_name":"myname","last_name":"myusername","email":"amail#mail.com"},"source":{"version":"1.3.0.Final","connector":"mysql","name":"dbserver1","ts_ms":1603634203000,"snapshot":"false","db":"inventory","table":"customers","server_id":223344,"gtid":null,"file":"mysql-bin.000003","pos":364,"row":0,"thread":6,"query":null},"op":"c","ts_ms":1603634203419,"transaction":null}}
How can I extract the changed data from this huge JSON?
which in my case is:
{"id":1005,"first_name":"myname","last_name":"myusername","email":"amail#mail.com"}
Should I keep using a StringDeserializer and use JSONB and iterate through the JSON Payload? or is there a better solution?
I don't think there's a better approach for that, however, as the text is a JSON, using a custom Deserializer that extends would JsonbDeserializer work:
#RegisterForReflection
public class CustomerDeserializer extends JsonbDeserializer<Customer> {
#Override
public Customer deserialize(String topic, byte[] data) {
JsonReader reader = Json.createReader(new StringReader(new String(data)));
JsonObject jsonObject = reader.readObject();
JsonObject payload = jsonObject.getJsonObject("payload");
String firstName = payload.getJsonObject("after").getString("first_name");
String lastName = payload.getJsonObject("after").getString("last_name");
String email = payload.getJsonObject("after").getString("email");
return new Customer(firstName,lastName,email);
}
}
Edit: You can find the full Debezium example here.
Related
Publish multiple events shares some attributes in one kafka topic
I need to publish multiple messages from the same project which represents employee journey events, and i need to use one topic only to publish these messages as they are representing the same project, but in some cases the message may contain extra fields for example: All messages share (id, name, type, date) and may some events have more fields like (course id, course name), so I am intending to use one parent object called "Journey", contains "Event" object, and I will create multiple children objects like 'LMSEvent' that extends this Event, etc if needed. Also using the Jackson + spring boot over rest APIs to do the needed cast based on type attribute. Finally, then this message to Kafka directly, so, each object contains its own properties. For the consumer, I will do some strategy patterns and do the required logic per each type if needed. The message size will not be very big and i don't expect to have more different attributes per each event. I am looking to know if this approach is good or not and in case is not, what is the alternative.
I think that in general it is good approach. Having single message schema on topic or multiple schemas is always good question and both has some bright sights and drawbacks, you can read more about it in Martin Kleppmann article.
When you decided to have multiple events on single topic, starting from rest api and next by Kafka producer and consumer you can use the same approach of serializing and deserializing events, #JsonTypeInfo and #JsonSubTypes does the job:
#JsonTypeInfo(
use = JsonTypeInfo.Id.NAME,
include = JsonTypeInfo.As.EXISTING_PROPERTY,
property = "type")
#JsonSubTypes({
#JsonSubTypes.Type(value = LMSEvent.class, name = "LMSEvent"),
#JsonSubTypes.Type(value = YetAnotherEvent.class, name = "YetAnotherEvent")
})
public interface Event {
String getType();
default boolean hasType(String type) {
return getType().equalsIgnoreCase(type);
}
default <T> T getConcreteEvent(Class<T> clazz) {
return clazz.cast(this);
}
}
When you consume that type of messages using spring-kafka you can define some very neat code, where each method is consuming concrete event type, so you don't need to write some dirty casting by your own:
#KafkaListener(topics = "someEvents", containerFactory = "myKafkaContainerFactory")
public class MyKafkaHandler {
#KafkaHandler
void handleLMSEvent(LMSEvent event) {
....
}
#KafkaHandler
void handleYetAnotherEvent(YetAnotherEvent yetAnotherEvent) {
...
}
#KafkaHandler(isDefault = true)
void handleDefault(#Payload Object unknown,
#Header(KafkaHeaders.OFFSET) long offset,
#Header(KafkaHeaders.RECEIVED_PARTITION) int partitionId,
#Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
logger.info("Server received unknown message {},{},{}", offset, partitionId, topic);
}
}
Full code
#InboundChannelAdapter in Spring-integration is not running continously?
i am working in spring cloud data flow,there i am having a scenario like reading from the database and send the data to the kafka topic using the #InboundChannelAdapter
Below is the strategy i followed.
->Created common list to store the objects if the list was empty
->if the list have the data i won't poll
->i am sending the values to kafka one by one by using index and after that i will remove the index
if i keep the #Bean it is inserting only the first object in the list to kafka topic.
{"id":101443442,"name":"Mobile1","price":8000}
if i remove the #Bean then it will insert all empty data into kafka.
{}
public static List<Product> products;
#Bean
public void initList() {
products = new ArrayList<>();
}
#Bean
#InboundChannelAdapter(channel = TbeSource.PR1)
public MessageSource<Product> addProducts() {
if (products.size() == 0) {
products.add(new Product(101443442, "Mobile1", 8000));
products.add(new Product(102235434, "book111", 6000));
}
MessageBuilder<Product> message = MessageBuilder.withPayload(products.get(0));
products.remove(0);
return message::build;
}
what am i doing wrong?
i need to send the data frequently by reading from db ?
Really not clear what you are asking. If you talk about JDBC then you may consider to use a JDBC Source from tout-of-the-box applications for Data Flow. If you are doing logic yourself to take data from data base, you may consider to use a JdbcPollingChannelAdapter from Spring Integration for the same #InboundChannelAdapter reason. The rest of your logic with that list is not clear. It is strange to see a #Bean on a void method. If you need to initialize that products and get access from the MessageSource implementation, you just need to do private List<Product> products = new ArrayList<>();. Having property as public is really a bad practice.
Spring-AMQP - routing based on message headers
As per the documentation: https://docs.spring.io/spring-amqp/docs/2.2.5.RELEASE/reference/html/#async-annotation-driven
We can have different handlers for messages based on it's converted class type like:
#RabbitListener(id="multi", queues = "someQueue")
#SendTo("my.reply.queue")
public class MultiListenerBean {
#RabbitHandler
public String thing2(Thing2 thing2) {
...
}
#RabbitHandler
public String cat(Cat cat) {
...
}
#RabbitHandler
public String hat(#Header("amqp_receivedRoutingKey") String rk, #Payload Hat hat) {
...
}
#RabbitHandler(isDefault = true)
public String defaultMethod(Object object) {
...
}
}
This I believe won't be great in performance since it has to do a trial and error to cast the incoming payload.
Instead, how to filter based on a condition say a header value? If header['operation']="order" then cast the message payload to Order class.
This I believe won't be great in performance since it has to do a trial and error to cast the incoming payload. Usually, type information is conveyed in headers and the MessageConverter uses that information to create the payload - there is no "trial and error". If you don't use one of the supplied converters, you can create your own, based on header['operation'].
recieve data as json and process at server side (java spring)
Send the parameters to server(spring framework) via get request, i am thinking of making a json object of all those parameters and send in get request so that in java spring i can recieve at as a map at the controller class in spring , how to achieve this
I am new to spring please help me out
I so far tried to send those parameters singly like(pram1,param2,param3,param4)
and recieve at the server side as string by setting param to string in type script before making get request to the server->i recieved parameters as map in controller
but i dont think it is a best way
{
param1: "param1"
param2: "param2
paramn: "paramn"
}
Send the above to server in the controller class ↓
#RequestParam MultiValueMap<String, String> requestMap
I want to recieve parameters as
String param1= requestMap.get("param1");
String param2=requestMap.get("param2");
If map type was an object it would be great so that i can recive any kind of object
example
at client side i am sending {param1: "myName", id: 0001}
at server side requestMap.get("param1"); requestMap.get("id");
As suggested by chrylis there's no need to manually extract parameters you can define a DTO/Request/POJO class, and Spring will map it automatically.
public class SampleDTO{
private String param1;
private String param2;
.
.
//getters and setters
}
if you specify RequestParam as hashmap, it gets automatically converted from json by jackson. Alternatively, if you are using String as the param, you can use ObjectMapper to convert it to a Map and get values from there.
You can map your incoming json to a Hashmap like so
#RequestMapping(method = RequestMethod.GET)
public String yourMethod(#RequestParam Map<String, String> parameters) {
String name = parameters.get("A"); //If your URL is http://test.com?A=ABC
...
}
Get request body as string/json to validate with a json schema- Spring boot REST API
I'm trying to validate JSON (passed by a client as a request body) before it is converted into a model in Controller method. If validation passes then return nothing, let the process continue as it was (spring boot to convert JSON into a model marked as #RequestBody). Throw error in case validation fails (everit-org/json-schema). I tried to two way: Implement HandlerMethodArgumentResolver, but resolveArgument() doesn't give request body details as it is already read and stored in ContentCachingRequestWrapper. NOTE: inputStream in ContentCachingRequestWrapper doesn't have any request body details. Using spring Interceptor. But this doesn't help me to find request body type passed in the request. As JSON schema is different for each request. Any other approaches I can try with?
I cannot add a comment ... so ...
What kind of validation do you need? If you only want to validate the fields like length of a string or range of a number and so on. I recommend you use #Validated on controller mehtod parameter, and model:
#NotNull
#Size(min = 32, max = 32)
private String id;
controller:
#PatchMapping
public Object update(#RequestBody #Validated User user, Errors errors) {
...
}
If there is something wrong, errors.hasErrors() will return true.
edit:
OK, I did some tests, in a filter :
HttpServletRequest httpServletRequest = (HttpServletRequest)request;
ServletInputStream inputStream = httpServletRequest.getInputStream();
byte[] a = new byte[1024];
inputStream.read(a);
System.out.println(IOUtils.toString(a));
I got a json string (a piece of request body) :
{"template":"5AF78355A4F0D58E03CE9F55AFA850F8","bd":"" ...