I'm designing a distributed network of sensing device. These devices will generate logs and send them to a central database. I'm using JMS for transporting log messages. The main database server will be running MDB(Message Driven Bean) to process incoming messages. The clients are sending data with GPRS. The problem is I don't want my clients to process network problems. I need some relay service that runs locally on client machine and gets the message from client immediately without blocking it and try on behalf of it.(if network is down, try sending again after some time).
message is a simple java object:
public class Message {
public int x;
public int y;
public int z;
}
client:
Message msg = new Message();
while (True) {
/* sense data */
msg = get_data_from_environment();
/* send data to local relay service
* This is non blocking call */
relay_service.send(msg);
}
local relay service:
while (True) {
/* get message from local client */
msg = get_message_from_local_client();
result = send_msg_to_JMS_server(msg);
/* if not successful, persist it on a local queue and try some other time */
if (result.is_sent() != True)
store_msg_on_disk(msg);
}
Is there a message service like this or I should write relay service myself?
is this good to use JMS in this case? Should I design my own socket level protocol to send messages?
EDIT
Is there a message service like this or I should write relay service myself?
Typically these type of relay services you have to code your self, unless you are able to find a software that does exactly what you want it to do. This is not unusual to be done in these cases.
Is this good to use JMS in this case?
Yes, JMS is a very good solution to use as a middleware. You can have many clients connect to JMS and send messages to it. While you have a server program running reading the messages off the JMS and processing it and handling network problems if there are any. Also as a bonus the server program and send back messages to the client in case of complete failure.
Should I design my own socket level protocol to send messages?
I still do not know what kind of messages you want to send. If you are using a standard transport like SMTP or SMS or HTTP or something like that, there are libraries to help you send and verify delivery. If you have to send using a custom protocol then you would have to write you own socket level code.
Seeing your code examples shows me that you want to know if your client was successful in sending his message to the JMS. If it was not sent then save to disk and try again later.
JMS server will auto-acknowledge if the message received. You can check this from the JMS message or if it fails you will get a JMSException. If you save messages on disk you will need to know when to re-send them. You would need a timer or re-send on next message to send.
Related
We are building a websocket server via golang+gin+json+gorilla websocket to push messages from server side to browser.
We plan to provide frontend with some subscription command, which means messages from server side will be sent to those users who subscribed target topic.
My confusion is whether we need add Ack mechanism here? For example, when client subscribe one topic, the server saved this mapping: user --> topic.
Is it necessary for the server to send a response for each subscription request to clients (Like that we do for an RPC request)? And how to do that? Below is my consumption
type MsgHeader struct {
ReqId string `json: reqId`
Cmd string `json: cmd`
// either of "req" or "rsp"
// is it necessary to have this field???
Type string `json: type`
}
I mean the application level acknowledgement, like what we do for RPC requests. For RPC request, we send responses even when the response itself is empty, something like:
type SubscriptionRsp struct {
Code int
Msg string
Data interface{}
}
No, it's not necessary.
The Websocket specs (RFC 6455) does not mandate this.
A data frame MAY be transmitted by either the client or the server at
any time after opening handshake completion and before that endpoint
has sent a Close frame
Nothing else about acknowledging messages is said in the Sending and Receiving Data section.
Therefore any ACK is entirely an implementation detail of your application. It may be useful if you develop a resilient client that retries failed messages, where "failed" could a message that is successfully sent to the server but not processed as expected.
I have an UI application (displays streaming) which makes a WebSocket connection to the Spring Boot microservice (multiple JVM'S) and this service forwards the request to one of the upstream servers and listens to the responses on a JMS queue coming from upstream server, which then response messages had to be returned to the socket.
Issue we are facing is since the socket is point to point, and the Spring Boot application is running on multiple instances which all are listening to the same JMS queue we are unable to serve the data back to the WebSocket when a message is received on a instance which the request to upstream wasn't made.
Here's the basic flow:
WebSocket -> instance1, instance2, instance3 -> Data provider
Instance1 made the request to data provider.
Data provider sends the data back to the queue
Instance 3 receives the message, but it doesn't have the socket connection to send the data back.
We had an interim solution using correlation id in JMS headers and selectors on the queue however now the data publisher is not able to provide the correlation id to depend on.
Does anybody have a better suggestion to address this?
Since you're using a request/reply pattern with JMS you must either use a correlation ID or a unique temporary queue for the response.
You indicated that, "the data publisher is not able to provide the correlation id to depend on." However, your application actually provides the correlation ID. The "data provider" in this case just needs to take it from the message it receives and put it into the response message. The process only requires 2 method calls by the "data provider" - javax.jms.Message.getJMSCorrelationID and javax.jms.Message.setJMSCorrelationID.
If the "data provider" can't do this then it's doubtful they will be able to accomplish the other option of using a unique temporary queue for the response. However, it's worth explaining in any case. When one of your "instance" servers sends the request message it first needs to use javax.jms.Session.createTemporaryQueue to create a temporary queue and then take the return parameter of that method and set it on the request message using javax.jms.Message.setJMSReplyTo. When the "data provider" receives the message they will get this value using javax.jms.Message.getJMSReplyTo and then send the response to this queue where the "instance" will then retrieve it.
These are the two generally accepted ways to implement a request/response pattern with JMS. I don't know of any other ways to implement such a pattern.
I'm new to ZeroMQ and trying to figure out a design issue. My scenario is that I have one or more clients sending requests to a single server. The server will process the requests, do some stuff, and send a reply to the client. There are two conditions:
The replies must go to the clients that sent the request.
If the client disconnects, the server should queue messages for a period of time so that if the client reconnects, it can receive the messages it missed.
I am having a difficult time figuring out the simplest way to implement this.
Things I've tried:
PUB/SUB - I could tag replies with topics to ensure only the subscribers that sent their request (with their topic as their identifier) would receive the correct reply. This takes care of the routing issue, but since the publisher is unaware of the subscribers, it knows nothing about clients that disconnect.
PUSH/PULL - Seems to be able to handle the message queuing issue, but looks like it won't support my plan of having messages sent to specific clients (based on their ID, for example).
ROUTER/DEALER - Design seemed like the solution to both, but all of the examples seem pretty complex.
My thinking right now is continuing with PUB/SUB, try to implement some sort of heartbeat on the client end (allowing the server to detect the client's presence), and when the client no longer sends a heartbeat, it will stop sending messages tagged with its topic. But that seems sub-optimal and would also involve another socket.
Are there any ideas or suggestions on any other ways I might go about implementing this? Any info would be greatly appreciated. I'm working in Python but any language is fine.
To prepare the best proposition for your solution, more data about your application requirements. I have made a little research about your conditions and connnect it with my experience about ZMQ, here I present two possibilities:
1) PUSH/PULL pattern in two direction, bigger impact on scalability, but messages from server will be cached.
Server has one PULL socket to register each client and get all messages from clients. Each message should have client ID to for server knowledge where send response.
For each client - server create PUSH socket to send responses. Socket configuration was sent in register message. You can use also REQ/REP pattern for register clients (assign socket number).
Each client has own PULL socket, which configuration was sent to server in register message.
It means that server with three clients required to (example port numbers in []):
server: 1 x PULL[5555] socket, 3 x PUSH[5560,5561,5562] sockets (+ optional 1 X REQ[5556] socket for registrations, but I think it depends how you prepare client identity)
client: 1 x PUSH[5555] socket, 1 x PULL[5560|5561|5562] (one per client) (+ optional 1 X REP[5556])
You have to connect server to multiple client sockets to send responses but if client disconnects, messages will not lost. Client will get their own messages when it reconnect to their PULL socket. The disadvantage is requirements of creating few PUSH sockets on server side (number of clients).
2) PUB/SUB + PUSH/PULL or REQ/REP, static cocket configuration on server side (only 2), but server has to prepare some mechanism for retransmit or cache messages.
Server create PUB socket and PULL or REQ. Client register it identity by PULL or REQ socket. server will publish all messages to client with this identity as filter. Server use monitor() function on PUB socket to count number of connected and disconnected clients (actions: 'accept' and 'disconnect'). After 'disconnect' action server publish message to all clients to register again. For clients which not re-register, server stop publish messages.
Client create SUB socket and PUSH or REQ to register and send requests.
This solution requires maybe some cache on server side. Client could confirm each message after get it from SUB socket. It is more complicated and have to be connected with your requirement. If you just would like to know that client lost message. Client could send timestamps of last message received from server during registration. If you need guarantee that clients get all messages, you need some cache implementation. Maybe other process which subscribe all messages and delete each confirmed by client.
In this solution server with three clients required to (example port numbers in []):
server: 1 x PUB[5555] socket, 1 x REP or PULL[5560] socket + monitoring PUB socket
client: 1 x SUB[5555] socket and own identity for filter, 1 x REQ or PUSH[5560] socket
About monitoring you could read here: https://github.com/JustinTulloss/zeromq.node#monitoring (NodeJS implementation, but Python will be similar)
I think about other patterns, but I am not sure that ROUTER/DEALER or REQ/REP will cover your requirements. You should read more about patterns, because each of it is better for some solutions. Look here:
official ZMQ guide (a lot of examples and pictures)
easy ROUTER/DEALER example: http://blog.scottlogic.com/2015/03/20/ZeroMQ-Quick-Intro.html
I am using spring/stomp/websocket framework to notify users of messages asynchronously. I have done this successfully. However, I would be get ACK from the client so that some server side action can take place when this is done.
The flow is roughly as flows:
Service notifies a specific user about a decision and updates a record in the DB with status = "notified"
Client receives the message (using stompClient.subscribe(...))
Client acknowledges that the message was received.
The service "knows" that this message was acknowledged and updates the status to "ACK" in the DB.
stompClient.connect({login:'guest', passcode:'guest'},
function(frame) {
setConnected(true);
**var headers = {ack: 'client'};**
...
stompClient.subscribe('/user/guest/response',function(notification) {
//doSomething
}), **headers**);
}
In the service, the message is sent:
this.messagingTemplate.convertAndSendToUser(user, "/response",msg, map);
Is there a way to handle the client ACK on the server side?
Alternatively, I tried to do a
stompClient.send("/app/response/ack/"+messageId);
on the client, in the method that handles the subscription, but in vain.
Can someone please tell me what is standard way to handle acknowledgments? I have been struggling with this for a a couple of days and any thoughts would be very helpful.
Thanks!
Use the ACK frame as per spec. The server sends an ack:some_id header, the client uses that some_id in the ACK frame.
The answer is no for simple broker.
https://docs.spring.io/spring/docs/current/spring-framework-reference/html/websocket.html
The simple broker is great for getting started but supports only a
subset of STOMP commands (e.g. no acks, receipts, etc.), relies on a
simple message sending loop, and is not suitable for clustering. As an
alternative, applications can upgrade to using a full-featured message
broker.
I need suggestion how to implement, if it is possible, with the Spring integration the following TCP flow:
Only the server side is need.
The TCP server waits for the incoming connection
On connection of the client, server sends data to the client
Client replies with response
Server may reply immediately with the new data or wait for external application events to send new packages to the client.
In groovy the code could be demonstrated as follow:
def serverSocket = new ServerSocket(...)
def connSocket = serverSocket.accept()
connSocket.outputStream.write(...)
while(true) {
def readBuffer = new byte[256]
connSocket.inputStream.read(readBuffer)
if(needToSendBack(readBuffer)) {
connSocket.outputStream.write(...)
}
}
def sendByDemand(def data) {
connSocket.outputStream.write(data)
}
The method sendByDemand could be invoked from the separate thread.
Here is a list of problems which I marked for myself, which prevents me to implement it with the Spring Integration (2.x version):
As far as I understand, the standard "Service Activator" approach cannot work in this scenario, since it is "connection events" driven. So when the application decides to send the new data to the client it cannot use the Service Activator
I have no "On TCP connection" event. I found that version 3.0 comes with the events support in this area, but since I cannot upgrade to 3.0, I implemented the connection check with the interceptors on the connection factory. However, when I know that client is connected, trying using the Direct Channels to send message fails with "no subscribers" error.
If someone could post possible Spring configuration for this scenario or point to the similar flow example it may be very helpful.
Your use case is possible, but it would make your life easier if you could upgrade to 3.0.
'Dispatcher has no subscribers' means there is no consumer subscribed to that channel.
You need to show your configuration; you must use collaborating channel adapters for this (not a gateway).
You need to capture the connectionId of the connection when it is established, and use it to populate the ip_connectionId header so the outbound channel adapter knows which socket to which to write the message.