protocol buffer as message over active Mq - jms

I am designing an app that has multiple components, mainly written in java and python.
I am thinking of using "JMS-Active MQ" as Message Oriented Middleware for components and "protocol buffers".
1) Is it good way to go ahead? In our case "message size" can go well over 10MB, will protocol buffer still have advantages for cross component communication? Are there any better communication "protocols" for cross platform applications that can handle "huge amounts of data"?
2) I created a Proof of concept my sending a "protocol buff" as message over the "ActiveMQ", I am using the sample proto file, in google's java tutorial.
AddressBook.Builder book = AddressBook.newBuilder();
Person.Builder person = Person.newBuilder();
person.setName("mayank");
person.setId(2);
book.addPerson(person);
TextMessage message = session.createTextMessage();
message.setText(book.build().toString());
In another java app, I listen to this message and try to deserialize it back into AddressBook object:
public void onMessage(Message message) {
TextMessage msg = (TextMessage) message;
try {
System.out.println(msg.getText());
CodedInputStream stream =CodedInputStream.newInstance(msg.getText().getBytes());
AddressBook book = AddressBook.parseFrom(stream);
}
catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
This causes an exception:
com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol
message, the input ended unexpectedly in the middle of a field. This could
mean either than the input has been truncated or that an embedded message
misreported its own length.
at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:49)
I do not know what's wrong ..?

Regarding 1), protocol buffers' documentation talks about transferring large messages here.
About 2), the problem seems to be the way you are transferring book.
Take a look at how the tutorial you mention writes the message to an OutputStream. Instead of a TextMessage, you should use a binary one, for example by writing the bytes first to a ByteArrayOutputStream and then to the message.

Main issue is that for some reason you are using TextMessage instead of correct choice of BytesMessage: protobuf is binary encoding and not textual one.
And if you absolutely want to misuse TextMessage for non-textual message, you MUST specify encoding to use for getBytes(); and encoding to use MUST match whatever was used to convert binary payload to bytes. And if you use UTF-8, you have probably already corrupted message... (ISO-8859-1, aka Latin-1 actually would work because it is single-byte encoding).
Beyond this, you can use protobuf as well as any number of other formats over JMS. I prefer JSON for readability and extensibility, but if you are already using protobuf, and this is internal system (not something exposed to external parties), protobuf works too.

Related

IBM MQ remove unwanted data

Could anyone explain how can we reduce total message data send to a topic, like unnecessary message headers. I am trying to send a string of message to a topic and retrieve from that topic using an MQGET call. https://www.ibm.com/docs/en/ibm-mq/9.2?topic=calls-mqget-get-message as defined in this link, datalength is the total size of message received from the topic. The difference of datalength and actual string size is very different. datalength is much more higher than the actual string I am sending.
So it must be that IBM MQ is padding the message with headers and properties which are not required for just sending a string to a topic.
Can we disable the unused headers and properties so that the datalength can come down?
EDIT:
Publisher's side code. here protomsg is the google's protobuf.
string buffer; // message buffer
protomsg.SerializeToString(&buffer);
long n =buffer.length();
char *char_array;
char_array = &buffer[0];
MQPUT(Hcon,
Hobj,
&md,
&pmo,
n,
char_array,
&CompCode,
&Reason);
You can use MQGMO properties options to control which properties are included and whether the properties are returned as message headers or in a separate message handle.
You might want to set the MQGMO_NO_PROPERTIES option, documented here: https://www.ibm.com/docs/en/ibm-mq/9.2?topic=mqgmo-options-mqlong

Spring boot and CloudEvent AMQP binding

I'm trying to implement some CloudEvent demo.
I have a hew spring boot services with RabbitMQ as a message bus they all send messages to a queue and one listens to the queue messages.
I try to wrap my messages as CloudEvent to make them more standard.
I use the following code to wrap the message (data) as a CloudEvent.
try {
inputEvent = CloudEventBuilder.v1()
.withSource(new URI("app://" + messageData.getChangeRequestId().toString()))
.withDataContentType("application/json")
.withId(messageData.myId().toString())
.withType("com.data.BaseMessageData")
.withData(objMapper.writeValueAsBytes(eventData))
.build();
} catch (Exception e) {
throw new MyMessagingException("Failed to convert the message into json. (See inner exception for further details)", e);
}
The data is converted to bytes since the message CloudEventData is based on bytes.
Of course, that on my listener method I get exception since SimpleMessageConverter can't handle bytes array.
Now, I can try and implement some custom message handler or try to check out CloudEvent AMQP suggested binding solution but I'm not keen with the amount of code it involves and I don't want to involve more technologies if not absolutely necessary.
Should I go down this path and implement a custom message conveter?
Is there any other standard solution for standardizing services messaging over qeueus?
You will need a custom message converter; but see this blog post:
https://spring.io/blog/2020/12/10/cloud-events-and-spring-part-1

How to filter in PUB/SUB with protobuf binaries?

Suppose I want to serialize and transmit protobuf binaries with ZMQ using a protocol defined in cake.proto:
syntax = "proto3";
message Cake {
int32 radius = 1;
}
I can find plenty of examples for the PUB/SUB pattern where a subscriber filters a topic with a string:
socket.setsockopt_string(zmq.SUBSCRIBE, "abc")
But how does subscribing to topics work when it comes to protobuf binaries? Do I use the bytes themselves or does ZMQ provide an wrapper for a message with a header I can use for cases like this?
There is no wrapper for this, the subject is just the first frame of the zeromq message.
If you are confident your protobuf messages will always start with the specific sequence of bytes (that make your subject) then yes you can just subscribe to that byte prefix pattern.
The other option is to copy the subject pattern into an initial frame then add the protobuf frame(s) via ZMQ_SNDMORE. If you can pack many protobuf frames into that same zmq message then the efficiency is good. If each protobuf message has its own "subject" then you will have the overhead of an extra subject frame per protobuf.

Sending JMS message over corrupted network

I am performing some simple tests with ActiveMQ to see how it performs on a non stable network. The first test consists in a producer that sends messages to a remote queue. The message is of type ObjectMessage with serializable content inside (a list of Objects).
With a good network everything works correctly, but when I launch the same tests using netem to simulate packages losses, delays and corruptions I get the following error when consuming the messages when trying to extract the content of the Message:
2011-03-16 11:59:21,791 ERROR [com.my.MessageConsumer] Failed to build body from bytes. Reason: java.io.StreamCorruptedException: invalid handle value: 017E0007
javax.jms.JMSException: Failed to build body from bytes. Reason: java.io.StreamCorruptedException: invalid handle value: 017E0007
So it seems like the message was corrupted while sending to the remote Queue but anyway stored, and only when is consumed the consumer see that the message is corrupted.
After this I will use a local Queue and a Network Connector to forward the messages to the remote Queue, and that I hope it solve the problem, but I was surprised that there was not any kind of validation between the producer and the destination (at least a checksum or something like that) that guarantees a correct delivery, am I doing something wrong or is the normal behaviour?
I don't have the code here right now, but it was super simple, just a MessageListener:
public class myMessageConsumer implements MessageListener{
public void onMessage(Message message){
try
{
if (message instanceof ObjectMessage){
ObjectMessage myMessage = (ObjectMessage) message;
List dtoList = (List) myMessage.getObject();
}
} catch(Exception ex){
ex.printStackTrace();
}
}
}
If the exact code is needed I'll put it when I go back from holidays, but it was exactly like that.
The broker isn't going to validate the contents of each and every message that it processes, that would be a tremendous waste of time and slow down message dispatch significantly. The client received a bad message and threw a JMSException to indicate that the message contents were corrupted which should be sufficient for your app to respond correctly.
Where's your code?
If that exception comes from your code, seems like it's possible that you've got a bug. For example, getting some JMS error receiving the message but messing up error handling and trying to process the results anyway. For a test like you describe, you'd need a good focus on error handling in your clients.
I don't have experience w/ ActiveMQ, but it seems very surprising that it'd allow corrupt message delivery. Not that I'm wanting the JMS implementation to unpack the ObjectMessage to check. Just that it should deliver a byte-for-byte uncorrupted copy of what was sent. Or error out if it can't.

How can I know what message I've received while using Protocol Buffers library?

It seems I don't understand something simple about Protocol Buffers, but this is very important question for me and for my real use-case.
While reading documentation about Protocol Buffers I don't understand how one know which message you should decode from the stream? All examples about some defined Message, but if you have defined several completelly different messages and you want to send those messages between 2 processes -- how do you know which message you have just received?
Or maybe Protocol Buffers do not try to address this problem and leave this question for another abstraction level?
Or maybe I should pack the message into structure like that:
message wrapper {
required string message_name = 1;
string packed_message = 2;
}
And then I should decode message in 2 stages: get the message_name at first, and then decode real packed message at second stage, should not I?
Look at self describing messages section

Resources