I'm trying to find a solution to a problem to guarantee that a message is only over processed completely by a single consumer.
Lots of messages on a queue and a number of consumers read messages and process them writing out to a database. My messages are transacted so that if a consumer dies then the message goes back onto the queue for another consumer to process.
We have to have an active/passive configuration for activemq and this is causing the issue. If I stop the active activemq then the consumer reconnects to the other activemq as I am using the failover transport. This is fine but during the reconnect, the message is put back on the queueand the consumer is not made aware of this reconnection and continues to process. This leads to the situation where 2 consumers process the same message.
I would have liked to use a distributed transaction manager and this may happen in the future but for now I need a different solution.
If I don't use failover transport then I can hook into a JMSException listener and abort the consumer. Unfortunately this does not work when using failover transport.
I would like either to use failover transport for the initial connect (discover which of the activemqs are running) and then force failover not to reconnect... or use a different transport that allows a list of server to try but doesn't reconnect... or find away to listen to the reconnect.
Note that this happens sometimes with just one server using failover (reconnect).
I could do my over initial connect logic (hunting for the active server) but was going to check if there is another option
You can listen to transport events on the ActiveMQConnection by using a listener:
connection = (ActiveMQConnection)factory.createConnection();
connection.addTransportListener(new TransportListener() {
public void onCommand(Object command) {
// Do something
}
public void onException(IOException error) {
// Do something
}
public void transportInterupted() {
// Do something
}
public void transportResumed() {
// Do something
}
});
connection.start();
Note that in this example the listener is set on the Connection directly; however, you can set an instance on the ActiveMQConnectionFactory which will be assigned to each Connection instance that it creates.
Related
I am looking for detailed description on how IBM MQ Client or listener get the messages from MQ Server when new messages are placed on MQ Queue or Topic.
How the connection between MQ Client and MQ Server are created?
Does MQ Client initiates the connection with Server or does server initiates the connection to its consumer?
In case we have connection pool defined on MQ Client, how client knows that it has to create more connections with Server as the messages are increasing on Server? How Client know about the messages on Server?
Is there a communication from Server to Client which tells client that new messages have arrived?
I am looking for these details not the details on how this is setup or how to setup MQ Channels or Listeners. I am looking for how it works behind the scene.
If someone can point me to the right direction or documentation, it would be great.
It's hard to speak definitively about how the IBM WebSphereMQ client & server work since they're closed source, but based on my experience with other messaging implementations I can provide a general explanation.
A JMS connection is initiated by a client to a server. A JMS client uses a javax.jms.ConnectionFactory to create a javax.jms.Connection which is the connection between the client and server.
Typically when a client uses a pool the pool is either filled "eagerly" (which means a certain number of connections are created when the pool is initialized to fill it to a certain level) or "lazily" (which means the pool is filled with connections one-by-one as clients request them from the pool). If a client requests a connection from the pool and if all the connections in the pool are being used and the maximum size of connections allowed by the pool hasn't been reached then another connection will be created. If the pool has reached its maximum allowed size (i.e no more connections can be created) then the client requesting a connection will have to wait for another client to return their connection to the pool at which point the pool will then give it to the waiting client.
A JMS client can find out about messages on the server in a few different ways.
If the JMS client wants to ask the server occasionally about the messages it has on a particular queue it can create a javax.jms.Consumer and use the receive() method. This method can wait forever for a message to arrive on the queue or it can take a timeout parameter so that if a message doesn't arrive in the specified timeout the call to receive() will return.
If the JMS client wants to receive a message from a particular queue as soon as the message arrives on the queue then it can create a javax.jms.MessageListener implementation and register it on the queue. When such a listener is registered on a queue then when a message arrives on the queue the server will send the message to the listener. This is sometimes referred to as a "callback" since the server is "calling back" to the client.
The first thing you should do is take a course on JMS/IBM MQ or go to the new IBM conference called: Integration Technical Conference
Ok, now to your questions:
How the connection between MQ Client and MQ Server are created?
You simply issue the createQueueConnection method of QueueConnectionFactory class and specify the credentials.
QueueConnection conn = cf.createQueueConnection("myUserId", "myPwd");
Does MQ Client initiates the connection with Server or does server initiates the connection to its consumer?
MQ client application starts the connection - always.
In case we have connection pool defined on MQ Client, how client knows that it has to create more connections with Server as the messages are increasing on Server? How Client know about the messages on Server?
It is up to the team's architect or lead developer to understand message flow and message patterns. Hence, they will know what to set the pool count at. Also, lots and lots of testing too. Some client applications will only require a pool count of 10 whereas other applications may need a pool count of 50 because it is a heavy flow.
Is there a communication from Server to Client which tells client that new messages have arrived?
You use the createReceiver method of the QueueSession class to retrieve a message. Set a timeout value for the createReceiver method rather than continuously polling the queue manager.
Again, some training on the use of JMS/IBM MQ is strongly recommended.
I have a producer send msg to a ActiveMQ broker, and a consumer subscribe from the same topic. I know that the broker can persist the msg, so as soon as the msg reaches the broker, it can be guaranteed available for the consumer. (right?)
But what happen if the network between producer and ActiveMQ broker is broken, in this case is there a way to guarantee the at-least-once sent? like, the producer itself can persist msg before sending to the ActiveMQ broker?
In order for the guarantee to be established the message must reach the broker, if the network is down then the burden is on you to capture the send error and retry the send at a later time. The ActiveMQ client offers no persistent storage for what it sends, you can use the failover transport and the client will retry the send on reconnect but it the application is shutdown and you haven't provided some means of recovery on restart then there isn't anything more the client can do.
I am using Spring to setup Stomp server endpoints (extending AbstractWebSocketMessageBrokerConfigurer)
#Override
public void configureMessageBroker(MessageBrokerRegistry config) {
config.enableStompBrokerRelay("/topic","/queue")
.setRelayHost(<rmqhost>);
}
#Override
public void registerStompEndpoints(StompEndpointRegistry registry) {
registry.addEndpoint("/myapp/websockets").setAllowedOrigins("*");
}
The objective is that I can have multiple servers, and a client will connect to any one of them for a specific topic: /topic/topic-id-1
Any of the server (at a time) can send a message for this topic using Spring's SimpMessagingTemplate
messagingTemplate.convertAndSend(destination, message);
where destination = "/topic/topic-id-1".
For ex: I have 2 server nodes and a client connecting to each one of them, subscribing to the same topic (/topic/topic-id-1). The objective is that if server 1 sends a message for topic-id-1, it should relay via rabbitmq to both clients subscribing to the same topic. I see a queue being created with routing key as "topic-id-1", but only the client connecting to the server sending out the message explicitly receives it. Am I missing something here? Isn't RMQ stomp broker supposed to relay the message send by one server for a subscription, across all the subscriptions for the same topic? Does the server need to do something else to get messages sent by other node?
I met the same problem. After a whole day explored, I found the solution finally!! It's easy to configure though.
registry.enableStompBrokerRelay("/topic/", "/queue/", "/exchange/")
.setUserDestinationBroadcast("/topic/log-unresolved-user")
.setUserRegistryBroadcast("/topic/log-user-registry")
The only thing you need to do is configure setUserDestinationBroadcast and setUserRegistryBroadcast when you enable the StompBrokerRelay. And it works!
I found the solution from here. Thinks that guy!
I'm not sure if this is exactly the same thing but I just solved a very similar problem. I posted my answer here: Sending STOMP messages from other layers of an application
I decided to split the implementation of the relay server into it's own setup and then manually forward messages between the rabbitmq server and the websocket subscribers on each of the servers.
Hopefully this can be of some use for you.
So this is a strange one. I have a basic Spring 4 websockets application running on Glassfish 4 using RabbitMQ for the message broker, nothing fancy. I was testing the durability of the websocket clients (one in java and one in javascript using stomp.js and socks.js) and noticed that when I undeployed the application from glassfish both clients would think the websocket was still up. For fun I added a recurring ping request from each client to the server to mimic a heartbeat. When the application is up the ping request works great and I get pong responses from the server, but when I undeploy the app from glassfish (to simulate a disconnect) I still get successful ping and pong messages from the server. It seems to me that when the application is undeployed it should send out disconnect messages to all connected clients which would invoke their reconnect logic to hit another server in the cluster. Has anyone seen similar behavior??? Thanks for any help!
I think I have this one figured out. I failed to set the heartbeat configuration on the STOMP connection. Once I set these values I began seeing heartbeats sent to the client by the server, and when I pulled the plug on the web socket application the heartbeats stopped, as they should. After that point is was very easy to implement some reconnect logic based on the last time I received a heartbeat and if it was too old or not. Here is some sample code for configuring the STOMP client, most of this I pulled from the spring stock-portfolio stomp client example.
Inside the StompWebSocketHandler class you simply add this block of code. You would obviously set the heartbeatInterval variable to whatever value you desire.
public void afterConnectionEstablished(WebSocketSession session) throws IOException {
StompHeaderAccessor headers = StompHeaderAccessor.create(StompCommand.CONNECT);
headers.setAcceptVersion("1.1,1.2");
headers.setHeartbeat(heartbeatInterval, heartbeatInterval);
Message<byte[]> message = MessageBuilder.withPayload(new byte[0]).setHeaders(headers).build();
TextMessage textMessage = new TextMessage(new String(encoder.encode(message), DEFAULT_CHARSET));
session.sendMessage(textMessage);
}
I'm designing a distributed network of sensing device. These devices will generate logs and send them to a central database. I'm using JMS for transporting log messages. The main database server will be running MDB(Message Driven Bean) to process incoming messages. The clients are sending data with GPRS. The problem is I don't want my clients to process network problems. I need some relay service that runs locally on client machine and gets the message from client immediately without blocking it and try on behalf of it.(if network is down, try sending again after some time).
message is a simple java object:
public class Message {
public int x;
public int y;
public int z;
}
client:
Message msg = new Message();
while (True) {
/* sense data */
msg = get_data_from_environment();
/* send data to local relay service
* This is non blocking call */
relay_service.send(msg);
}
local relay service:
while (True) {
/* get message from local client */
msg = get_message_from_local_client();
result = send_msg_to_JMS_server(msg);
/* if not successful, persist it on a local queue and try some other time */
if (result.is_sent() != True)
store_msg_on_disk(msg);
}
Is there a message service like this or I should write relay service myself?
is this good to use JMS in this case? Should I design my own socket level protocol to send messages?
EDIT
Is there a message service like this or I should write relay service myself?
Typically these type of relay services you have to code your self, unless you are able to find a software that does exactly what you want it to do. This is not unusual to be done in these cases.
Is this good to use JMS in this case?
Yes, JMS is a very good solution to use as a middleware. You can have many clients connect to JMS and send messages to it. While you have a server program running reading the messages off the JMS and processing it and handling network problems if there are any. Also as a bonus the server program and send back messages to the client in case of complete failure.
Should I design my own socket level protocol to send messages?
I still do not know what kind of messages you want to send. If you are using a standard transport like SMTP or SMS or HTTP or something like that, there are libraries to help you send and verify delivery. If you have to send using a custom protocol then you would have to write you own socket level code.
Seeing your code examples shows me that you want to know if your client was successful in sending his message to the JMS. If it was not sent then save to disk and try again later.
JMS server will auto-acknowledge if the message received. You can check this from the JMS message or if it fails you will get a JMSException. If you save messages on disk you will need to know when to re-send them. You would need a timer or re-send on next message to send.