Use case: I have a use case where the client send a request to an microservice endpoint (the producer) which does some operations and produce a message to the kafka to be consumed by a consumer which stores some data into his own database. Immediately after this, the client send another request to the consumer microservice to get latest updates (which should include data stored previously).
But the problem is that the client is sending the second request without waiting for consumer to finish storing data (for the first request).
Question: how should I handle it? Can somehow wait for consumer to finish storing data?
What I tried: I tried to add thread.sleep into producer endpoint, but I don't like that solution.
Thanks.
Related
I'm using Spring Boot 2.0.7 and Spring-Kafka to create a request/reply pattern. Basically the frontend UI makes a request to an API which puts a message on to a request Kafka queue, the message is processed by a backend process and when complete a message is put onto a reply queue.
I want to provide the frontend UI an api which waits until the response is ready. The UI in this time will just show a processing message. If the response is not available (e.g. after 2 minutes), the API should just return a message not available error where we can instruct the user to come back later.
I'm a bit new to Spring-Kafka. Does it allow me to create a polling API? If so, any example code would be very much appreciated.
It's not as simple as polling a topic for a reply because you have to correlate requests/replies.
You can use ReplyingKafkaTemplate.sendAndReceive() and keep checking the isDone() method on the Future<?>.
If you want to poll yourself, you would have to create a consumer object from the consumer factory.
I am looking for a way for each consumer instance to receive a message that is published to RabbitMQ via MassTransit. The scenario would be, we have multiple microservices that need to invalidate a cache on notification. Pub-Sub won't work in this instance as there will be 5 consumers of the same type as its the same code per service instance, so only one would receive the message in a traditional PubSub.
Message observation could be an option but this means the messages would never be consumed and hang around forever on the bus.
Can anyone suggest a pattern to use in the context of MassTransit?
Thanks in advance.
You should create a management endpoint in each service, which could even be a temporary queue (just request a receive endpoint without a queue name and one will be dynamically generated). Then, put your queue invalidation consumers on that endpoint. Each service instance will receive a unique instance of the message (when Publish is called), and those queues and bindings will automatically be removed once the service exits.
This is exactly how the bus endpoint works, but in your case, you're creating a receive endpoint which can have consumer message type bindings, so that published messages are received, one copy per service.
cfg.ReceiveEndpoint(cfg => { ... });
Note that the queue name is not specified, and will be automatically generated uniquely.
Hi I am new to Camel and have a design question related to JMS queues.
I am receiving set of data. These data have a reference date. These data are sent every 15 minutes by a batch process.
I have to process the received data and forward them to another route.
If a given data cannot be processed, I need to reprocess it. And I have to ensure it is processed before the next data set is processed.
So I was thinking of creating a JMS route to receive these data before processing. Then process it. Then send it to another queue.
FTP --> Process data rows (A) --> JMS Queue --> Processor (B) --> direct:call
If processor B fails I want the data to be processed before the next data set is sent by FTP. (because second data set may contain an update of the data of the first dataset)
So I was thinking using a queue, to make sure they are always processed in the order they are being received.
But my experience with JMS, without Camel, is that once the object is consumed from the queue it is not in the queue anymore.
Is it also the case with Camel?
In this case to I have to retry to process the data, or put them back in the queue?
This "recovery" part is not clear to me and I'd like to understand the patterns that do support this.
Many thanks for your help
Gilles
This part "once the object is consumed from the queue it is not in the queue anymore." is not fully correct. Actually, when you are subscribing to the queue and getting a message you need to process it and send acknowledge back to the JMS broker. If acknowledge is successful then the message will be removed from the queue. But if acknowledge will be not successful or if your process will die and connection to the broker will break then the message will not be removed from the queue and will be passed to another consumer.
Often most of the JMS libraries are using mode when acknowledgement is sent right when message was received by consumer but you always have possibility to change this mode and send acknowledgement manually when your processing part will be finished successfully.
What about camel jms (http://camel.apache.org/jms.html) you can use endpoint option "acknowledgementModeName" which has some different possible values like:
AUTO_ACKNOWLEDGE (default) - acknowledgement will be sent right after corresponded "from" in your route
CLIENT_ACKNOWLEDGE - allows the application to control when the acknowledgment is sent and if there are no exceptions will be thrown during exchange processing then message will be acknowledged and removed from queue.
I want to make a req/res request from IConsumer.Consume() method, but I don't see any method on ConsumeContext<> that returns a reference to IRequestClient<,>. Do I need to hold a reference to IBusControl somewhere and use it or I can use context somehow?
In this case, it is best to create the request client outside of the consumer and pass it as a dependency to the consumer as an IRequestClient<,> interface. The request client is created with IBus, which is outside of the consumer context.
It also ensures that the request will be less likely to deadlock with the broker because responses are received on the bus endpoint, and not the receive endpoint of the consumer (which, if you had a concurrency limit of 1, would never finish).
It's also not possible to connect consumers to a started receive endpoint, which is a requirement of the request/response handling (it happens under the covers). The bus, however, can connect consumers for messages which are sent (responses are sent to the bus address, versus being published) which is why it is used for responses to the requests.
To keep message tracking continuous, it may be nice to set the InitiatorId on the outbound request to the CorrelationId of the consumer message, as well as copying the ConversationId. This helps with tracing and keeping track of the overall message command/event flow.
I'm designing a quite complicated system and was wondering what the best way is to put a jms consumer (activemq, vm protocol, non persitent) inside a netty handler.
Let me explain, i have several clients connecting to my netty server using websockets. For every client connection i create a jms consumer that listens for interesting messages on one or more topics. If a interesting message arrives i need to do a extra step (additional filtering) before sending the message to the client using the websocket.
Is the following a good way to do this:
inside a SimpleChannelInboundHandler i declare a private non static consumer
the consumer is initialized in channelActive
the consumer is destroyed in channelInactive
when a message is received by consumer i do the extra filter a send it using ctx.channel().write()
In this setup i'm a bit worried that the consumer might turn into slow consumer and slow everything down, cause the websocket goes over the internet.
I came up with a more complex one to decouple the "receiving of message by consumer" and "sending of message through a websocket".
inside a SimpleChannelInboundHandler i declare a private non static consumer
the consumer is initialized in channelActive
the consumer is destroyed in channelInactive
when a message is received by consumer i put it in a blockedqueue
every minute i let a thread (created for every client) look in the queue and send the found messages to the client using ctx.channel().write().
At this point i'm a bit worried about the extra thread per client.
Or is there maybe a better way to accomplish this task?
This is a classic slow consumer problem and the first step to resolving it is to determine what the appropriate action is when a slow consumer is detected. If it is acceptable that the slow consumer misses messages then the solution is some variation on dropping messages or unsubscribing them from the feed. For example, if it's acceptable that the client misses messages then, when one is received from JMS, check if the channel is writable. If it isn't, drop the message. If you want to give yourself a bit more of a buffer (although OS buffers are quite large) you can track the number of write completion future's that haven't completed (ie the messages haven't been written to the OS send buffer) and drop messages if there are too many outstanding write requests.
If the client may not miss messages, and is consistently slow, then the problem is more difficult. One option might be to divert messages to a JMS queue with a specific header value, then open a new consumer that reads messages from that queue using a JMS selector. This will put more load on the JMS server but might be appropriate for temporary slowness and hopefully it won't interfere with you main topic feeds. Alternatively you might want to stash the messages in a different store, such as a database, so you can poll for messages when they can be sent. If you do this right a single polling thread can cope with many clients (query for clients which have outstanding messages, then for each client, load a bunch of messages). However this isn't as convenient as using JMS.
I wouldn't go with option 2 because the blocking queue is only going to solve the problem temporarily, and you can achieve the same thing by tracking how many write operations are waiting to complete.