Currently I have 2 #KafkaListener methods that consume events from two different topics. The problem is that I need to "make" one of them execute first, always, and then the second one. I tried #DependsOn. Thanks
Update: Description of my system is following
I have kafka topic Account and TransactionRequest. I have microservice called datagenerator which on start generates 10.000 accounts, and 1 transaction every 300ms that includes accountid, amount etc. The problem is that microservice validationService listens for Accounts and TransactionRequests, and in my #KafkaListener method called validate, i have method that validates incoming transactions. Another #KafkaListener listens for accounts and as soon as they start generating, this method insertAccounts inserts them into cassandra DB. The problem is, first kafkaListener validate tries to check amount on account in DB, but they havent been inserted because #KafkaListener responsible for that didnt fire yet.
Related
I wonder, how do you avoid notifying all services that consume the same event? For example, service A and service B both of them consume event X. Based on some rules you want to send the event X only for service A. I am not talking about consumer groups (kafka) or even correlation-Id. As I am using Event-Driven microservices with the approach of command & event.
I think it's quite easy by using Kafka partitions/partition-key.
For example: In your topic X, you could just create it with many partitions. For each invocation, every service must specify its key, so based on the key, Kafka will do the rest of the job. So every time Service A sends a command, the consumer Service (the one who handles the command) will send the event with the same key. So in the end, Service A (the producer of the command) will receive the event on its own partition and will be the only service receiving it. So based on the Command/Event approach it may work.
On the other hand, by doing so, you are limiting one of the main benefits of partition which is allowing the scalability.
In Kafka for Spring, I see by default the max-poll-records value is 500.
So my question is suppose if 500 messages are not present in the Topic will the consumer will wait to get 500 records then the poll method will run and fetch the batch of records.
I am a bit confused here like what are all the checks before pulling the message from Topic.
Kafka operates with hybrid strategies of polling. Usually, it is a combination of the number of records (or bytes) and time interval.
All the properties can be overridden to fit your expectations for consumption.
I already have micro-services running and would like to add event(Kafka).
For example, I have a customer service with 10000 customers in the db. I will be adding an event to the customer service so that whenever a new user is created, it publishes an event in which will be consumed by consumers (like recommendation-service, statistics-service, etc.)
I think the above is clear to me. However, I am not sure how to handle the already-registered customers (10000 customers) as the event will only be triggered when 'NEW' customer registers.
I can 'hack' the service to sync the data manually but what does most people do in this case?
Thank you
I tried to search the topic but couldn't find the ones that I am looking for.
There are basically two strategies that you can follow here. The first is a bulk load of fake "new customer" events into the Kafka topic, as you also suggested. The second approach would be to use the change data capture (CDC) pattern where there is an initial snapshot of all the observed data and then a constant streaming of new data change events, direclty from the database internal log (WAL).
To handle your entire use case, you could use a tool like Debezium Source Connector for Kafka Connect platform, but note that you will also need to map its change event into your message format. There are plugins to do that with a configuration-driven approach, but you can also create your custom logic using single message transformations (SMT).
Problem
When my web application updates an item in the database, it sends a message containing the item ID via Camel onto an ActiveMQ queue, the consumer of which will get an external service (Solr) updated. The external service reads from the database independently.
What I want is that if the web application sends another message with the same item ID while the old one is still on queue, that the new message be dropped to avoid running the Solr update twice.
After the update request has been processed and the message with that item ID is off the queue, new request with the same ID should again be accepted.
Is there a way to make this work out of the box? I'm really tempted to drop ActiveMQ and simply implement the update request queue as a database table with a unique constraint, ordered by timestamp or a running insert id.
What I tried so far
I've read this and this page on Stackoverflow. These are the solutions mentioned there:
Idempotent consumers in Camel: Here I can specify an expression that defines what constitutes a duplicate, but that would also prevent all future attempts to send the same message, i.e. update the same item. I only want new update requests to be dropped while they are still on queue.
"ActiveMQ already does duplicate checks, look at auditDepth!": Well, this looks like a good start and definitely closest to what I want, but this determines equality based on the Message ID which I cannot set. So either I find a way to make ActiveMQ generate the Message ID for this queue in a certain way or I find a way to make the audit stuff look at my item ID field instead of the Message ID. (One comment in my second link even suggests using "a well defined property you set on the header", but fails to explain how.)
Write a custom plugin that redirects incoming messages to the deadletter queue if they match one that's already on the queue. This seems to be the most complete solution offered so far, but it feels so overkill for what I perceive as a fairly mundane and every-day task.
PS: I found another SO page that asks the same thing without an answer.
What you want is not message broker functionality, repeat after me, "A message broker is not a database, A message broker is not a database", repeat as necessary.
The broker's job is get messages reliably from point A to point B. The client offers some filtering capabilities via message selectors but this is minimal and mainly useful in keeping only specific messages that a single client is interested in from flowing there and not others which some other client might be in charge of processing.
Your use case calls for a more stateful database centric solution as you've described. Creating a broker plugin to walk the Queue to check for a message is reinventing the wheel and prone to error if the Queue depth is large as ActiveMQ might not even page in all the messages for you based on memory constraints.
Application server creates a new transaction before calling MDB's onMessage method. Also I am processing database update in onMessage method. Transactions create additional overhead and processing several message in one transaction could increase performance.
Is it possible to make App server to use one transaction for several messages. Or maybe there are other approaches to this problem?
And, by the way, I can't use multiple instances, cause I need to preserve the sequence order.
I guess you can store the messages in a list and depending upon how many messages you want to process in one transaction you can check the size of the list and process the messages.