Could anyone explain how can we reduce total message data send to a topic, like unnecessary message headers. I am trying to send a string of message to a topic and retrieve from that topic using an MQGET call. https://www.ibm.com/docs/en/ibm-mq/9.2?topic=calls-mqget-get-message as defined in this link, datalength is the total size of message received from the topic. The difference of datalength and actual string size is very different. datalength is much more higher than the actual string I am sending.
So it must be that IBM MQ is padding the message with headers and properties which are not required for just sending a string to a topic.
Can we disable the unused headers and properties so that the datalength can come down?
EDIT:
Publisher's side code. here protomsg is the google's protobuf.
string buffer; // message buffer
protomsg.SerializeToString(&buffer);
long n =buffer.length();
char *char_array;
char_array = &buffer[0];
MQPUT(Hcon,
Hobj,
&md,
&pmo,
n,
char_array,
&CompCode,
&Reason);
You can use MQGMO properties options to control which properties are included and whether the properties are returned as message headers or in a separate message handle.
You might want to set the MQGMO_NO_PROPERTIES option, documented here: https://www.ibm.com/docs/en/ibm-mq/9.2?topic=mqgmo-options-mqlong
Related
We have a compute node that copies OutputRoot from InputRoot having MQRFH2 header and then sets MQMD format as 'MQSTR ' before sending it to an MQOutput node.
On my local system(MQ and WMB 7.0) when I try to test the flow and browse output queue from RFHUTILC the message still have MQMD format as MQHRF2 with proper MQRFH2 header details.
While this same flow when tested in other test regions is giving an output message with blank MQMD format and RFH2 details comes in message payload.
What can be the reason of this difference?
The input message will have the RFH2 header at InputRoot.MQRFH2, or at InputRoot.MQRFH2C if you use the compact parser for RFH2, which will be copied to the OutputRoot, and Broker sets the MQMD Format to MQHRF2 when it sees that header in the OutputRoot.
To delete the RFH2 header you should do something like this:
SET OutputRoot.MQMD.Format = OutputRoot.MQRFH2.Format;
SET OutputRoot.MQRFH2 = NULL;
I can't think of an explanation for the different behavior you see, and I don't have a V7 Broker to test. By the way you shouldn't have one either as it is out of support.
I am Passing data as JSONObject as String in Header and in next module/processor using below code but data on next module i received as "java.io.DataInputStream#7c6a857" and not able to convert it back to JSONObject as String.
messageOut = MessageBuilder.withPayload(message-payload).copyHeaders(payload.getHeaders()).setHeader("ActualRecord", JSONObject).build();
I assume you are using RabbitMQ as the transport.
The DefaultMessagePropertiesConverter used within the bus limits headers to 1024 bytes by default; anything larger is left as a DataInputStream in the headers. The MessagePropertiesConverter is configurable for this size, but XD does not currently expose a setting to allow this to be increased.
You need to read from the DataInputStream into a byte[] to restore the JSON.
I have opened a JIRA Issue to support configuration of the LongString limit.
I have a WTX map which puts a message on WMQ "Q1". There is some other application which reads the message from "Q1" and then processes the message and places the response on the queue specified in "ReplyToQ" available on MQ Header information.
I am not able to find a command parameter to add the ReplyToQ in the message WTX map is placing on "Q1".
Any thoughts?
Thanks for taking the time to look at this question and helping out!
The ReplyToQueue is sent on the message header, so you must get the message with the header (-HDR) and parse it from there on your input card.
Here's the doc about the adapter's command:
http://pic.dhe.ibm.com/infocenter/wtxdoc/v8r4m1/index.jsp
You should have a type tree on the Examples to read the message with a header.
Regards,
Bruno.
I am designing an app that has multiple components, mainly written in java and python.
I am thinking of using "JMS-Active MQ" as Message Oriented Middleware for components and "protocol buffers".
1) Is it good way to go ahead? In our case "message size" can go well over 10MB, will protocol buffer still have advantages for cross component communication? Are there any better communication "protocols" for cross platform applications that can handle "huge amounts of data"?
2) I created a Proof of concept my sending a "protocol buff" as message over the "ActiveMQ", I am using the sample proto file, in google's java tutorial.
AddressBook.Builder book = AddressBook.newBuilder();
Person.Builder person = Person.newBuilder();
person.setName("mayank");
person.setId(2);
book.addPerson(person);
TextMessage message = session.createTextMessage();
message.setText(book.build().toString());
In another java app, I listen to this message and try to deserialize it back into AddressBook object:
public void onMessage(Message message) {
TextMessage msg = (TextMessage) message;
try {
System.out.println(msg.getText());
CodedInputStream stream =CodedInputStream.newInstance(msg.getText().getBytes());
AddressBook book = AddressBook.parseFrom(stream);
}
catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
This causes an exception:
com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol
message, the input ended unexpectedly in the middle of a field. This could
mean either than the input has been truncated or that an embedded message
misreported its own length.
at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:49)
I do not know what's wrong ..?
Regarding 1), protocol buffers' documentation talks about transferring large messages here.
About 2), the problem seems to be the way you are transferring book.
Take a look at how the tutorial you mention writes the message to an OutputStream. Instead of a TextMessage, you should use a binary one, for example by writing the bytes first to a ByteArrayOutputStream and then to the message.
Main issue is that for some reason you are using TextMessage instead of correct choice of BytesMessage: protobuf is binary encoding and not textual one.
And if you absolutely want to misuse TextMessage for non-textual message, you MUST specify encoding to use for getBytes(); and encoding to use MUST match whatever was used to convert binary payload to bytes. And if you use UTF-8, you have probably already corrupted message... (ISO-8859-1, aka Latin-1 actually would work because it is single-byte encoding).
Beyond this, you can use protobuf as well as any number of other formats over JMS. I prefer JSON for readability and extensibility, but if you are already using protobuf, and this is internal system (not something exposed to external parties), protobuf works too.
It seems I don't understand something simple about Protocol Buffers, but this is very important question for me and for my real use-case.
While reading documentation about Protocol Buffers I don't understand how one know which message you should decode from the stream? All examples about some defined Message, but if you have defined several completelly different messages and you want to send those messages between 2 processes -- how do you know which message you have just received?
Or maybe Protocol Buffers do not try to address this problem and leave this question for another abstraction level?
Or maybe I should pack the message into structure like that:
message wrapper {
required string message_name = 1;
string packed_message = 2;
}
And then I should decode message in 2 stages: get the message_name at first, and then decode real packed message at second stage, should not I?
Look at self describing messages section