The transaction start monitoring event of an IIB flow with a MQ Input node produces a base64 encoded byte array of the MQ message. Now I would like to write a Java program that reconstructs this byte array so I can read the headers and body.
The base64 MQ messages looks like this:
TUQgIAIAAAAAAAAACAAAAP////8AAAAAEQEAALgEAABNUUhSRjIgIAQAAAABAAAAQU1RIENFTUJSQSAgICAgIKVV+Fslx7YCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIENFTUJSQSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhbmllbCAgICAgIBYBBRUAAADiboF1+wHSKOpNUf3pAwAAAAAAAAAAAAALICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICALAAAAMC45XGNvbW1vblxqZGtcYmluXGphdmF3LmV4ZTIwMTgxMTI1MTQzNjEyNDcgICAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAA/////1JGSCAAAAACAAAAvAAAAREAAAS4TVFTVFIgICAAAAAAAAAEuAAAACA8bWNkPjxNc2Q+am1zX3RleHQ8L01zZD48L21jZD4gIAAAAEg8am1zPjxEc3Q+cXVldWU6Ly8vTU9OSTwvRHN0PjxUbXM+MTU0MzE1NjU3MjQ1NjwvVG1zPjxEbHY+MjwvRGx2Pjwvam1zPiAAAAAkPHVzcj48VGhlS2V5PlRoZVZhbHVlPC9UaGVLZXk+PC91c3I+PGZvbz5iYXI8L2Zvbz4=
I made following tests to parse this in Java:
import java.io.ByteArrayInputStream;
import java.io.DataInputStream;
import javax.xml.bind.DatatypeConverter;
import org.apache.commons.io.IOUtils;
import org.junit.Assert;
import org.junit.Test;
import com.ibm.mq.headers.CCSID;
import com.ibm.mq.headers.MQHeaderList;
import com.ibm.mq.headers.MQMD;
import com.ibm.mq.headers.MQRFH2;
public class MqMsgTest {
#Test
public void allGood() throws Exception {
String msgBase64 = IOUtils.toString(getClass().getResourceAsStream("/mq-msg.base64"), "UTF-8");
byte[] msgBytes = DatatypeConverter.parseBase64Binary(msgBase64);
DataInputStream msgStream = new DataInputStream(new ByteArrayInputStream(msgBytes));
MQMD mqmd = new MQMD(msgStream);
Assert.assertEquals("MQHRF2 ", mqmd.getFormat());
Assert.assertEquals("daniel ", mqmd.getUserIdentifier());
MQRFH2 mqrfh2 = new MQRFH2(msgStream);
Assert.assertEquals("TheValue", mqrfh2.getStringFieldValue("usr", "TheKey"));
String body = IOUtils.toString(msgStream, CCSID.getCodepage(mqrfh2.nextCharacterSet()));
Assert.assertEquals("<foo>bar</foo>", body);
}
#Test
public void doesNotWork() throws Exception {
String msgBase64 = IOUtils.toString(getClass().getResourceAsStream("/mq-msg.base64"), "UTF-8");
byte[] msgBytes = DatatypeConverter.parseBase64Binary(msgBase64);
DataInputStream msgStream = new DataInputStream(new ByteArrayInputStream(msgBytes));
MQHeaderList headers = new MQHeaderList(msgStream, true);
Assert.assertEquals(2, headers.size());
}
}
The allGood() test parses the headers and body nicely. But it would fail if the message would not contain a RFH2 header. The doesNotWork() test should parse the headers in a generic way, but it does not work.
How can I parse the base64 encoded MQ message in a flexible way so I have access to the headers and the body?
The transaction start monitoring event of an IIB flow with a MQ Input
node produces a base64 encoded byte array of the MQ message. Now I
would like to write a Java program that reconstructs this byte array
so I can read the headers and body.
Why? Why do you combine the MQMD and the MQRFH2 structures then encode it with Base64 only to want to extract it with a Java program?
It sounds like a very poor design.
I dropped your Base64 message into a program that outputs a hex dump of it:
000000: 4D442020 02000000 00000000 08000000 MD ............
000010: FFFFFFFF 00000000 11010000 B8040000 ????........?...
000020: 4D514852 46322020 04000000 01000000 MQHRF2 ........
000030: 414D5120 43454D42 52412020 20202020 AMQ CEMBRA
000040: A555F85B 25C7B602 00000000 00000000 ?U?[%??.........
000050: 00000000 00000000 00000000 00000000 ................
000060: 00000000 20202020 20202020 20202020 ....
000070: 20202020 20202020 20202020 20202020
000080: 20202020 20202020 20202020 20202020
000090: 20202020 43454D42 52412020 20202020 CEMBRA
0000a0: 20202020 20202020 20202020 20202020
0000b0: 20202020 20202020 20202020 20202020
0000c0: 20202020 64616E69 656C2020 20202020 daniel
0000d0: 16010515 000000E2 6E8175FB 01D228EA .......?n?u?.?(?
0000e0: 4D51FDE9 03000000 00000000 0000000B MQ??............
0000f0: 20202020 20202020 20202020 20202020
000100: 20202020 20202020 20202020 20202020
000110: 0B000000 302E395C 636F6D6D 6F6E5C6A ....0.9\common\j
000120: 646B5C62 696E5C6A 61766177 2E657865 dk\bin\javaw.exe
000130: 32303138 31313235 31343336 31323437 2018112514361247
000140: 20202020 00000000 00000000 00000000 ............
000150: 00000000 00000000 00000000 01000000 ................
000160: 00000000 00000000 FFFFFFFF 52464820 ........????RFH
000170: 00000002 000000BC 00000111 000004B8 .......?.......?
000180: 4D515354 52202020 00000000 000004B8 MQSTR .......?
000190: 00000020 3C6D6364 3E3C4D73 643E6A6D ... <mcd><Msd>jm
0001a0: 735F7465 78743C2F 4D73643E 3C2F6D63 s_text</Msd></mc
0001b0: 643E2020 00000048 3C6A6D73 3E3C4473 d> ...H<jms><Ds
0001c0: 743E7175 6575653A 2F2F2F4D 4F4E493C t>queue:///MONI<
0001d0: 2F447374 3E3C546D 733E3135 34333135 /Dst><Tms>154315
0001e0: 36353732 3435363C 2F546D73 3E3C446C 6572456</Tms><Dl
0001f0: 763E323C 2F446C76 3E3C2F6A 6D733E20 v>2</Dlv></jms>
000200: 00000024 3C757372 3E3C5468 654B6579 ...$<usr><TheKey
000210: 3E546865 56616C75 653C2F54 68654B65 >TheValue</TheKe
000220: 793E3C2F 7573723E 3C666F6F 3E626172 y></usr><foo>bar
000230: 3C2F666F 6F3E </foo>
The decoded message starts off with the MQMD which is 364 bytes then the MQRFH2 structures which is 202 bytes.
The MQHeaderList class is extremely picky about parsing an MQMessage. The message MUST begin with one of the known MQ classes or it will throw an exception. Using MQHeaderList class for problem messages will probably just lead to more problems.
It would be better to simply dump the message in hex like I did then let the support person or developer figure out what the issue is.
MQHeaderList class is not a magic wand, although, I wish IBM would actually expand its use to include EVERY MQ header. Your problem is that you are assuming MQHeaderList class can do more than what it can. MQHeaderList class can only handle 11 internal MQ headers/structures (aka classes). See here: https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.dev.doc/q030880_.htm
i.e. MQRFH, MQRFH2, MQCIH, MQDLH, MQIIH, MQRMH, MQSAPH, MQWIH, MQXQH, MQDH & MQEPH
As you can see, MQMD is NOT on the list. So, your decoded Base64 message will NOT work with MQHeaderList class.
Also, you should read my comments about MQRFH2 being an embedded message here: Issue While Setting MQRFH2 header in IBM MQ
You have to plug and chug through the data stream that IIB created. Here's some basic code to do the job:
import java.io.ByteArrayInputStream;
import java.io.DataInputStream;
import javax.xml.bind.DatatypeConverter;
import com.ibm.mq.headers.MQMD;
import com.ibm.mq.headers.MQRFH2;
public class Test_IIB_Data
{
public static void main(String[] args)
{
String msgBase64 = "TUQgIAIAAAAAAAAACAAAAP////8AAAAAEQEAALgEAABNUUhSRjIgIAQAAAABAAAAQU1RIENFTUJSQSAgICAgIKVV+Fslx7YCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIENFTUJSQSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhbmllbCAgICAgIBYBBRUAAADiboF1+wHSKOpNUf3pAwAAAAAAAAAAAAALICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICALAAAAMC45XGNvbW1vblxqZGtcYmluXGphdmF3LmV4ZTIwMTgxMTI1MTQzNjEyNDcgICAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAA/////1JGSCAAAAACAAAAvAAAAREAAAS4TVFTVFIgICAAAAAAAAAEuAAAACA8bWNkPjxNc2Q+am1zX3RleHQ8L01zZD48L21jZD4gIAAAAEg8am1zPjxEc3Q+cXVldWU6Ly8vTU9OSTwvRHN0PjxUbXM+MTU0MzE1NjU3MjQ1NjwvVG1zPjxEbHY+MjwvRGx2Pjwvam1zPiAAAAAkPHVzcj48VGhlS2V5PlRoZVZhbHVlPC9UaGVLZXk+PC91c3I+PGZvbz5iYXI8L2Zvbz4=";
try
{
byte[] msgBytes = DatatypeConverter.parseBase64Binary(msgBase64);
DataInputStream msgStream = new DataInputStream(new ByteArrayInputStream(msgBytes));
MQMD md = new MQMD(msgStream);
System.out.println("md.getFormat="+md.getFormat());
System.out.println("md.getPutApplName="+md.getPutApplName());
System.out.println("md.getUserIdentifier="+md.getUserIdentifier());
MQRFH2 rfh2 = new MQRFH2(msgStream);
System.out.println("rfh2.getFormat="+rfh2.getFormat());
System.out.println("rfh2.usr.TheKey="+rfh2.getStringFieldValue("usr", "TheKey"));
int bodyLen = msgBytes.length - rfh2.getStrucLength() - md.size();
byte[] body = new byte[bodyLen];
msgStream.read(body);
System.out.println("body="+new String(body));
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
And when you run it, the output should be:
md.getFormat=MQHRF2
md.getPutApplName=0.9\common\jdk\bin\javaw.exe
md.getUserIdentifier=daniel
rfh2.getFormat=MQSTR
rfh2.usr.TheKey=TheValue
body=<foo>bar</foo>
Related
I send a simple text message to an MQ Queue (MQ 7.0.1):
"abc"
Using spring JMS the total length of the message is: 291
But putting the same message in the queue using IBM MQ libraries the total length of the message is: 3
How can I get total data length 3 with JMS?
Spring JMS code:
#EnableJms
public class JMSTestController {
...
#Autowired
private JmsTemplate jmsTemplate;
#Autowired
JmsMessagingTemplate jmsMessagingTemplate;
...
public String send() throws JMSException{
jmsTemplate.setReceiveTimeout(10000);
jmsMessagingTemplate.setJmsTemplate(jmsTemplate);
Session session = jmsMessagingTemplate.getConnectionFactory().createConnection()
.createSession(false, Session.AUTO_ACKNOWLEDGE);
Queue entryQueue = session.createQueue("hereQueueName");
Queue replyQueue = session.createQueue("hereReplyQueueName");
TextMessage message = session.createTextMessage("abc");
message.setJMSDeliveryMode(DeliveryMode.NON_PERSISTENT);
message.setJMSDestination(entryQueue);
message.setIntProperty(WMQConstants.JMS_IBM_CHARACTER_SET, 819);
message.setIntProperty(WMQConstants.JMS_IBM_ENCODING, 273);
jmsMessagingTemplate.convertAndSend(entryQueue, message);
String messageId = message.getJMSMessageID();
...
}
Native code:
MQQueueManager qm = createQueueManager(queueManager, host, port,
channel, username, password, connectionType);
MQQueue m_receiver = null;
MQMessage msg = new MQMessage();
msg.format = MQC.MQFMT_STRING;
msg.expiry = timeout / 1000;
msg.replyToQueueName = qReceiver;
msg.replyToQueueManagerName = queueManager;
msg.write("abc".getBytes());
MQPutMessageOptions pmo = new MQPutMessageOptions();
try {
qm.put(qSender, msg, pmo);
} catch (MQException e) {
MQTalkerException ex = new MQTalkerException(
"An error happened sending a message", e);
logger.error(ex);
throw ex;
}
Solution
Following JoshMc's comment I made the following modification and reached the expected result:
Check out these answers, you want to set targetClient to MQ to remove
those properties. There are many ways to accomplish this, changing
your CreateQueue to use a URI is probably the easiest.
JMS transport v/s MQ transport
That is, modify the creation of the queue using the URI instead of just its name.
Queue entryQueue = session.createQueue("queue:///QUEUE_NAME?targetClient=1");
I reached the solution by following JoshMc's comment. That is, modify the creation of the queue using the URI instead of just its name.
Queue entryQueue = session.createQueue("queue:///QUEUE_NAME?targetClient=1");
This removes the MQRFH2 header (the extra bytes I didn't know where they came from)
and with that the message has a total length of 3 bytes.
Spring is counting the bytes of the message body (aka data)
IBM MQ native is counting the bytes of the message headers plus body
In your screenshot, the field directly above shows '3' bytes.
Longitud dataos = length of body = 3
Longitud total = length of headers + body = 291
I am having issue with push utf8 encode message using Parse Rest, here is my body
{"where":{"$and":[{"email":{"$in":["phaxxx#gmail.com","nhungxxx#gmail.com"]}},{"deviceType":{"$in":["ios"]}}]},"data":{"alert":"TEST: Giảm 40% Khi Mua Sách Harry Potter","sound":"default","page_type":"cms_key","page_value":"harry-potter"}}
Does anyone know how to encode utf8 message?
Javascript code:
public bool SendPushNotification(string jsonContent)
{
...
request.Headers.Add("X-Parse-Application-Id", appId);
request.Headers.Add("X-Parse-REST-API-KEY", restApiKey);
System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
Byte[] byteArray = encoding.GetBytes(jsonContent);
...
}
I added a SpringContextProcessor to my NiFi flow, it executes as expected and updates the FlowFile content and attributes. But in the data provenance section of NiFi instead of seeing SEND/RECEIVE, I am seeing
03/27/2017 11:47:57.164 MDT RECEIVE 42fa1c3f-edde-4cb7-8e73-ce752f7e3d66
03/27/2017 11:47:57.163 MDT DROP 667094a7-8eef-4657-981a-dc9fdc6c4056
03/27/2017 11:47:57.163 MDT SEND 667094a7-8eef-4657-981a-dc9fdc6c4056
Looks like the original message is being dropped and replaced by a new message. I haven't seen this behavior in other components, i.e. they all seem to preserve the original Flow File UUID. Simplified version of the Spring processor code:
#ServiceActivator(inputChannel = "fromNiFi", outputChannel = "toNiFi")
public Message<byte[]> process1(Message<byte[]> inMessage) {
String inMessagePayload = new String(inMessage.getPayload());
String userId = getUserIdFromDb(inMessagePayload);
String outMessagePayload = inMessagePayload + userId;
return MessageBuilder.withPayload(outMessagePayload.getBytes())
.copyHeaders(inMessage.getHeaders())
.setHeader("userId", userId)
.build();
}
Is there a way to preserve the original Flow File UUID in the outgoing message?
This is probably an oversight on our end, so yes please do raise a JIRA.
However as a workaround you can try to extract FlowFile attributes from the incoming Message headers and then propagate them back to the outgoing message.
public Message<byte[]> process1(Message<byte[]> inMessage) {
String myHeader = inMessage.getHeader("someHeader");
. . .
return MessageBuilder.withPayload(outMessagePayload.getBytes())
.copyHeaders(inMessage.getHeaders())
.setHeader("userId", userId)
.setHeader("someHeader", myHeader)
.build();
}
I have a C++ component which passes message to JMS broker via tcp using ActiveMQ. My broker is written in JAVA. I want these two to communicate; to C++ component send messages to the JAVA broker.
On C++ side what I am doing is creating a message (using protocol buffer), converting that to vector of bytes and passing it to the broker. On JAVA side (broker), I am constantly listening and acting upon a received message.
Now, I can tell that the system somehow works, since when I execute my C++ component (and when it passes the messages), I see my JAVA broker printing an error message: unexpected error:null per each message that I am sending from my C++ component. This means that at least my messages do reach to the broker, but somehow they cannot be decrypted, hence the null issue.
I am using the following for composing the message from C++ side:
// convert pointmsg to byte
int size = pointmsg.ByteSize();
char* byteArray = new char[size];
pointmsg.SerializeToArray(byteArray, size);
// convert bytearray to vector
vector<unsigned char> v(byteArray, byteArray + sizeof byteArray / sizeof byteArray[0]);
// pass as a vector
BytesMessage *message = session->createBytesMessage();
message->writeBytes(v);
producer->send(message);
printf("Sent message #%d from thread %s\n", ix + 1, threadIdStr.c_str());
pointmsg is just an object that I create and fill in, and it is not null, I tested it already and it has data in it. I am converting pointmsg to byte array since this is the way to pass an object as far as I read in the documantation. And since the writeBytes() function expects a vector, I am converting the byte array into a vector. I suspect there might be some problem in this part.
On JMS side I am simply listening the upcoming messages with:
public void onMessage(final javax.jms.Message message) {
final Timer traceTimer = new Timer();
final long messageReceived = System.currentTimeMillis();
try {
if (message instanceof ActiveMQBytesMessage) {
final ActiveMQBytesMessage amqBytesMsg = (ActiveMQBytesMessage) message;
final byte[] buffer = amqBytesMsg.getContent().data;
final String msgType = amqBytesMsg.getStringProperty(LLCom.MSG_PROP_CLASS_NAME);
final String topic = amqBytesMsg.getStringProperty(LLCom.MSG_PROP_TOPIC_NAME);
String msgLookUpType;
if (topic == null || topic.isEmpty()) {
// get message class name: foo.bar$MessageMsg
msgLookUpType = msgType.split("\\$")[1];
} else {
// it's a topic we assume, that all subscribers have the
// correct type
msgLookUpType = topic;
}
if (logger.isDebugEnabled())
logger.debug("Router(" + name + ") received message ( " + buffer.length + "bytes)of type or topic " + msgLookUpType);
final Message req = parsers.createMessage(buffer, msgType);
// process explicit topic/queue subscriber
processServiceMessage(msgLookUpType, messageReceived, amqBytesMsg, req, traceTimer);
} else {
logger.error("Not supported JMS MessageType: " + message);
}
} catch (final Exception e) {
logger.error("Unexpected error: " + e.getMessage());
// e.printStackTrace();
}
}
When I debug it I can see that msgType and topic variables (on JMS side) are coming as null, which means that activemq message is somehow not decrypted. What could be the reason for that? I can see the message is being sent, received but not understood.
Any thoughts?
Update: I noticed that I am expecting to get stringProperties on JMS side, which I am not setting on C++ side, but I am not sure whether it causes the problem or not.
Ok, It seems the error was related to set properties, msgType and topic, I gave them the expected strings with using setStringProperty() on C++ side with the required methods, and now that initial error is gone.
message->setStringProperty();
How can I set start attribute in the content-type for mime multipart/related message?
To be more precise, I want to know how I can set the start attribute for the root attachment in the below example taken from https://www.rfc-editor.org/rfc/rfc6362:
--OUTER-BOUNDARY
Content-type: multipart/related; boundary="INNER-BOUNDARY";
start="<root.attachment>"; type="application/xml"
--INNER-BOUNDARY
Content-type: application/xml
Content-ID: <root.attachment>
[XML DOCUMENT]
--INNER-BOUNDARY
Content-type: application/pdf
Content-ID: <2nd.attachment>
[PDF DOCUMENT]
--INNER-BOUNDARY--
--OUTER-BOUNDARY
I am not able to find it in the javax.mail api. Please help.
I've been struggling with that recently, the code below is the best I could come up with (actually nothing else worked for me):
public class MultipartGenerator {
//Let's assume the static members below
//hold our message parts content
//an the instances of arrays of byte
private static final byte [] ROOT_BYTES = new byte[]{/* ... bytes ... */};
private static final byte [] ATTCH_1_BYTES = new byte[]{/* ... bytes ... */};
private static final byte [] ATTCH_2_BYTES = new byte[]{/* ... bytes ... */};
/**
* Generate multipart with headers
*
* #return javax.mail.Multipart instance
*/
public static Multipart generateMultipart() {
//This is our root MimeBodyPart,
//content-id equals 'rootcid'
//content-type equals 'roottype/rootsubtype'
InternetHeaders ih0 = new InternetHeaders();
ih0.addHeader("Content-Type", "roottype/rootsubtype");
ih0.addHeader("Content-Transfer-Encoding", "binary");
ih0.addHeader("Content-ID", "rootcid");
MimeBodyPart rootBodyPart = new MimeBodyPart(ih0, ROOT_BYTES);
//This is a body part wrapping first message attachment
InternetHeaders ih1 = new InternetHeaders();
ih1.addHeader("Content-Type", "text/plain; name=attachment1.txt");
ih1.addHeader("Content-Transfer-Encoding", "binary");
ih1.addHeader("Content-Location", "attachment1.txt");
ih1.addHeader("Content-ID", "a00");
MimeBodyPart attch1BodyPart = new MimeBodyPart(ih1, ATTCH_1_BYTES);
//This is a body part wrapping second message attachment
InternetHeaders ih2 = new InternetHeaders();
ih2.addHeader("Content-Type", "text/plain; name=attachment2.txt");
ih2.addHeader("Content-Transfer-Encoding", "binary");
ih2.addHeader("Content-Location", "attachment2.txt");
ih2.addHeader("Content-ID", "a01");
MimeBodyPart attch2BodyPart = new MimeBodyPart(ih2, ATTCH_2_BYTES);
//This is our desired multipart, this is where things turn a bit dirty
//No success with setting the parameters in a different way
Multipart multipart = new MimeMultipart("related;start=\"<rootcid>\";type=\"roottype/rootsubtype\"");
multipart.addBodyPart(rootBodyPart,0);
multipart.addBodyPart(attch1BodyPart);
multipart.addBodyPart(attch2BodyPart);
return multipart;
}
}
There's probably a better way to handle this task, however I am not able to find one.