I have a use case where I need to convert a message from one type to another (i.e. TextMessage -> ObjectMessage).
I found that when diverting between queues there is an option to transform the message. I have implemented the Transformer interface as instructed in the documentation.
import org.apache.activemq.artemis.api.core.Message;
import org.apache.activemq.artemis.core.server.transformer.Transformer;
import javax.jms.ObjectMessage;
public class TypeTransformer implements Transformer {
#Override
public Message transform(Message message) {
return message;
}
}
But I am now beginning to realize that it might be impossible to convert from a org.apache.activemq.artemis.api.core.Message to an javax.jms.ObjectMessage?
Is this right? That it cannot be done or is there some other way?
It should technically be possible to convert a javax.jms.TextMessage to a javax.jms.ObjectMessage, but it may be cumbersome. Here are some important things to note:
javax.jms.TextMessage, javax.jms.ObjectMessage, and org.apache.activemq.artemis.api.core.Message are all just interfaces. The javax version is what you use on the client and Message is what is used on the broker. The data for each type of message is stored differently in the underlying message implementation.
The class for the Java object that you wish to put into the ObjectMessage will need to be on the broker's classpath. This isn't required under normal circumstances as the broker itself will never serialize or deserialize the object.
You should really try to avoid ObjectMessage whenever possible. ObjectMessage objects depend on Java serialization to marshal and unmarshal their object payload. This process is generally considered unsafe (and slow!), because a malicious payload can exploit the host system. Lots of CVEs have been created for this. For this reason, most JMS providers force users to explicitly whitelist packages that can be exchanged using ObjectMessage messages. For example, here's the related documentation for ActiveMQ Artemis. There are a number of other issues with using JMS ObjectMessage not related to security that you should read about.
Granted you understand all that you should be able to convert the message using code something like this:
import java.io.ByteArrayOutputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
import org.apache.activemq.artemis.api.core.ICoreMessage;
import org.apache.activemq.artemis.api.core.Message;
import org.apache.activemq.artemis.api.core.SimpleString;
import org.apache.activemq.artemis.core.server.transformer.Transformer;
public class TypeTransformer implements Transformer {
#Override
public Message transform(Message message) {
ICoreMessage coreMessage = message.toCore();
try {
// get the data from the TextMessage
SimpleString mySimpleString = coreMessage.getBodyBuffer().readNullableSimpleString();
if (mySimpleString == null) {
// no text in the message so no transformation can be done
return message;
}
String myString = mySimpleString.toString();
// parse the data from the TextMessage and set it on the serializable object
Serializable object = new MySerializable();
// turn serializable object into byte array and write it to the message
ByteArrayOutputStream baos = new ByteArrayOutputStream(1024);
ObjectOutputStream oos = new ObjectOutputStream(baos);
oos.writeObject(object);
oos.flush();
byte[] data = baos.toByteArray();
coreMessage.getBodyBuffer().clear();
coreMessage.getBodyBuffer().writeInt(data.length);
coreMessage.getBodyBuffer().writeBytes(data);
coreMessage.setType(Message.OBJECT_TYPE);
return coreMessage;
} catch (Exception e) {
e.printStackTrace();
return message;
}
}
}
Related
I am writing a small spring boot application that is supposed to monitor queues on an external IBM Queue installation.
I am able to connect via MQXAQueueConnectionFactory, but I have not found a way to discover all remote queues/destinations on that Host programmatically. I don't want to add them fix in my code.
How can I get a list of all existing queues in order to add listeners? I have to mention that an access via REST-API is not possible because this feature has been disabled by the administration.
You can use the IBM MQ Programmable Command Formats. If you installed the IBM MQ samples, the tools/pcf/samples/PCF_DisplayActiveLocalQueues.java gives you an idea for your use case.
Here is how I use it in my unit tests to find all the queues with messages:
import java.io.IOException;
import com.ibm.mq.MQException;
import com.ibm.mq.MQGetMessageOptions;
import com.ibm.mq.MQMessage;
import com.ibm.mq.MQQueue;
import com.ibm.mq.MQQueueManager;
import com.ibm.mq.constants.CMQC;
import com.ibm.mq.constants.CMQCFC;
import com.ibm.mq.constants.MQConstants;
import com.ibm.mq.headers.MQDataException;
import com.ibm.mq.headers.pcf.PCFMessage;
import com.ibm.mq.headers.pcf.PCFMessageAgent;
public class MqUtils {
public static void queuesWithMessages(MQQueueManager qmgr) {
try {
PCFMessageAgent agent = new PCFMessageAgent(qmgr);
try {
PCFMessage request = new PCFMessage(CMQCFC.MQCMD_INQUIRE_Q);
// NOTE: You can not use a queue name pattern like "FOO.*" together with
// the "addFilterParameter" method. This is a limitation of PCF messages.
// If you want to filter on queue names, you would have to do it in the
// for loop after sending the PCF message.
request.addParameter(CMQC.MQCA_Q_NAME, "*");
request.addParameter(CMQC.MQIA_Q_TYPE, MQConstants.MQQT_LOCAL);
request.addFilterParameter(CMQC.MQIA_CURRENT_Q_DEPTH, CMQCFC.MQCFOP_GREATER, 0);
for (PCFMessage response : agent.send(request)) {
String queueName = (String) response.getParameterValue(CMQC.MQCA_Q_NAME);
if (queueName == null
|| queueName.startsWith("SYSTEM")
|| queueName.startsWith("AMQ")) {
continue;
}
Integer queueDepth = (Integer) response.getParameterValue(CMQC.MQIA_CURRENT_Q_DEPTH);
// Do something with this queue that has messages
}
} catch (MQException | IOException e) {
throw new RuntimeException(e);
} finally {
agent.disconnect();
}
} catch (MQDataException e) {
throw new RuntimeException(e);
}
}
}
And this should give you ideas how to configure the MQQueueManager (see also IBM docs):
import com.ibm.mq.MQEnvironment;
import com.ibm.mq.MQException;
import com.ibm.mq.MQQueueManager;
#Configuration
static class MQConfig {
#Bean(destroyMethod = "disconnect")
public MQQueueManager mqQueueManager() throws MQException {
MQEnvironment.hostname = "the.host.com";
MQEnvironment.port = 1415;
MQEnvironment.channel = "xxx.CL.FIX";
return new MQQueueManager("xxx");
}
}
The chapter Using with IBM MQ classes for JMS explains how you can use PCF messages in pure JMS.
I am developing a springboot application that reads messages from a topic. Messages are managed in transaction and read as string in batch mode and then deserialized to an object. This operation may fail but I don't want to discard all the batch but rather I would move failed messages to DLQ.
As I am using spring-kafka 2.6.5 I found out that I can use BatchToRecordAdapter in order to achieve this purpose. However I did not find out how to know when I am reading the last message of any batch.
I would like to read one message at a time, serialize it and then store in an ArrayList; when listener reads the last message I want to make some processing and finally commit the transaction.
Thanks,
Giuseppe.
UPDATE
In order to achieve this purpose I override BatchToRecordAdapter and added headers that allow me to know the position in a batch of every element.
package com.doxee.commons.lifecycle.kafka;
import java.util.List;
import lombok.extern.slf4j.Slf4j;
import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.listener.ConsumerRecordRecoverer;
import org.springframework.kafka.listener.adapter.BatchToRecordAdapter;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.messaging.Message;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.util.Assert;
/*
* Insert a description here.
*
* Bugs: none known
*
* #author gmiano gmiano#doxee.com
* #createDate 25/01/21
*
* Copyright (C) 2021 Doxee S.p.A. C.F. - P.IVA: IT02714390362. All Rights Reserved
*/
#Slf4j
public class BatchToEnrichedRecordAdapter<K, V> implements BatchToRecordAdapter<K, V> {
private final ConsumerRecordRecoverer recoverer;
public BatchToEnrichedRecordAdapter(ConsumerRecordRecoverer recoverer) {
Assert.notNull(recoverer, "'recoverer' cannot be null");
this.recoverer = recoverer;
}
#Override
public void adapt(List<Message<?>> messages, List<ConsumerRecord<K, V>> records,
Acknowledgment ack, Consumer<?, ?> consumer, Callback<K, V> callback) {
for (int i = 0; i < messages.size(); ++i) {
Message enrichedMessage = MessageBuilder.fromMessage(messages.get(i))
.setHeader(MyHeaders.BATCH_SIZE, messages.size())
.setHeader(MyHeaders.MESSAGE_BATCH_POSITION, i + 1)
.build();
try {
callback.invoke(records.get(i), ack, consumer, enrichedMessage);
} catch (Exception var9) {
this.recoverer.accept(records.get(i), var9);
}
}
}
}
with this bean as recoverer
#Bean
ConsumerRecordRecoverer recoverer(KafkaOperations<?, ?> template) {
return new DeadLetterPublishingRecoverer(template, (record, ex) -> {
String srcTopic = record.topic();
String srcKey = record.key().toString();
log.error("Failed consume of message {} from topic {}", srcKey, srcTopic, ex);
String dstTopic;
if (ex.getCause() instanceof ClientResumableException) {
dstTopic = srcTopic.concat(".RECOVERABLE");
} else {
dstTopic = srcTopic.concat(".DLT");
}
log.error("Cannot retry. Try to write message to topic: {}", dstTopic);
return new TopicPartition(dstTopic, 0);
});
}
Is this the proper solution?
I'm trying to send messages with different priority with the JmsTemplate using Spring Boot and ActiveMQ and it's not working.
I tried this :
MessageCreator mc = session -> {
TextMessage tm = session.createTextMessage("hello");
tm.setJMSPriority(6);
return tm;
};
jmsTemplate.send((Queue) () -> "box", mc);
The priority inside the ActiveMQ broker is still 4 (default value).
The only way I found to actually change the priority for a message is by change the priority at the JmsTemplate level.
jmsTemplate.setPriority(3);
The problem here is, now, all messages sent after that will have the priority 3.
I know I can reset the JmsTemplate priority after each send, but it's not "clean" and what about concurrency?
How can I set the priority for each message and get the message with the highest priority using #JmsListener?
I just ran into the same issue.
I tested your point about setting the priority of the jmsTemplate and your assumption is correct. It is not handled properly with concurrency.
The solution I found that works (albeit not ideal) is to extend JmsTemplate and override the doSend method to copy the JmsPriority from the message to the producer. This isnt ideal, extending the class make break over spring boot releases (I've tested this on 2.1.7) and there are some additional steps to register the new JmsTemplate. But it does work and I've tested it under load.
Steps.....
Create a new class that extends JmsTemplate overriding the doSend method to copy the priority from the message
import java.io.Serializable;
import javax.jms.ConnectionFactory;
import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageProducer;
import org.springframework.jms.core.JmsTemplate;
public class RcsJmsTemplate extends JmsTemplate implements Serializable {
public RcsJmsTemplate() {
}
public RcsJmsTemplate(ConnectionFactory connectionFactory) {
super(connectionFactory);
}
/**
* Actually send the given JMS message.
*
* AF: EXTENDED TO COPY THE PRIORITY FROM THE MESSAGE TO THE PRODUCER
*
* #param producer the JMS MessageProducer to send with
* #param message the JMS Message to send
* #throws JMSException if thrown by JMS API methods
*/
#Override
protected void doSend(MessageProducer producer, Message message) throws JMSException {
if (getDeliveryDelay() >= 0) {
producer.setDeliveryDelay(getDeliveryDelay());
}
producer.send(message, getDeliveryMode(), message.getJMSPriority(), getTimeToLive());
}
}
Add a bean (to your App.java, or appropriate config class) You may not need to pass through a message converter (Im using Jackson in my project) There may also be other configurations you need to apply to the new JmsTemplate.
#Bean
public JmsTemplate jmsTemplate(ConnectionFactory connectionFactory, MessageConverter messageConverter) {
RcsJmsTemplate rcsJmsTemplate = new RcsJmsTemplate(connectionFactory);
rcsJmsTemplate.setMessageConverter(messageConverter);
return rcsJmsTemplate;
}
Then as in your question set the JmsPriority attribute of the message. Your using a MessageCreator but in my project Im using a message post processoe
public void convertAndSendWithPriority(JmsTemplate jmsTemplate, String destination, Object message, int priority) {
jmsTemplate.convertAndSend(destination, message, (Message jmsMessage) -> {
jmsMessage.setJMSPriority(priority);
return jmsMessage;
});
}
For completeness you should add the property:
spring.jms.template.qos-enabled=true
Thats it. Hope it helps
(Actually I hope someone comes up with a better answer)
Thanks
I need some help with performance tuning of a use case. In this use case the Camel route is tailing status lines in a log file and sends each line as a message to a JMS queue. I have implemented the use case like this:
package tests;
import java.io.File;
import java.net.URI;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.broker.BrokerFactory;
import org.apache.activemq.broker.BrokerService;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.component.sjms.SjmsComponent;
import org.apache.camel.main.Main;
public class LinesToQueue {
public static void main() throws Exception {
final File file = new File("data/log.txt");
final String uri = "tcp://127.0.0.1:61616";
final BrokerService jmsService = BrokerFactory.createBroker(new URI("broker:" + uri));
jmsService.start();
final SjmsComponent jmsComponent = new SjmsComponent();
jmsComponent.setConnectionFactory(new ActiveMQConnectionFactory(uri));
final Main main = new Main();
main.bind("jms", jmsComponent);
main.addRouteBuilder(new RouteBuilder() {
#Override
public void configure() throws Exception {
fromF("stream:file?fileName=%s&scanStream=true&scanStreamDelay=0", file.getAbsolutePath())
.routeId("LinesToQueue")
.to("jms:LogLines?synchronous=false");
}
});
main.enableHangupSupport();
main.run();
}
}
When I run this use case with a file already filled with 1.000.000 lines the overall performance I get in the route is about 313 lines/second. This means that it takes about 55 minutes to process the file.
As some sort of reference I also have created another use case. In this use case the Camel route is tailing status lines in a log file and sends each line as a document to an Elasticsearch index. I have implemented the use case like this:
package tests;
import java.io.File;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.main.Main;
public class LinesToIndex {
public static void main() throws Exception {
final File file = new File("data/log.txt");
final String uri = "local";
final Main main = new Main();
main.addRouteBuilder(new RouteBuilder() {
#Override
public void configure() throws Exception {
fromF("stream:file?fileName=%s&scanStream=true&scanStreamDelay=0", file.getAbsolutePath())
.routeId("LinesToIndex")
.bean(new LineConverter())
.toF("elasticsearch://%s?operation=INDEX&indexName=log&indexType=line", uri);
}
});
main.enableHangupSupport();
main.run();
}
}
When I run this use case with a file already filled with 1.000.000 lines the overall performance I get in the route is about 8333 lines/second. This means that it takes about 2 minutes to process the file.
I understand that there is a huge difference between a JMS queue and an Elasticsearch index but how can have the JMS use case above to perform better?
Update #1:
It seems to be the persistence in the JMS service that is the bottleneck in my first use case above. If I disable the persistence in the JMS service then the performance in the route is about 11111 lines/second. Which persistence storage for the JMS service will give me a better performance?
a couple of things to consider...
ActiveMQ producer connections are expensive, make sure you use a pooled connection factory...
consider using the VM transport for an in process ActiveMQ instance
consider using an external ActiveMQ broker over TCP (so it doesn't compete for resources with your test)
setup/tune KahaDB or LevelDB to optimize persistent storage for your use case
Maybe somebody can help me find out how to solve this.
I am using jersey-apache-client 1.17
I tried to use Jersey client to build a standalone application (no Servlet container or whatever, just the Java classes) which communicates with a RESTFUL API, and everything worked fine until I tried to handle the mediatype "text/csv; charset=utf-8" which is a CSV stream sent by the server.
The thing is that I can read this stream with the following code:
InputStreamReader reader = new InputStreamReader(itemExportBuilder
.get(ClientResponse.class).getEntityInputStream());
Csv csv = new Csv();
Input input = csv.createInput(reader);
try {
String[] readLine;
while ((readLine = input.readLine()) != null) {
LOG.debug("Reading CSV: {}", readLine);
}
} catch (IOException e) {
e.printStackTrace();
}
try {
input.close();
} catch (IOException e) {
e.printStackTrace();
}
But I'd like to encapsulate it and put it into a MessageBodyReader. But after writing this code, I just can't make the client use the following class:
package client.response;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.lang.annotation.Annotation;
import java.lang.reflect.Type;
import java.util.ArrayList;
import java.util.List;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.ext.MessageBodyReader;
import javax.ws.rs.ext.Provider;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
#Provider
public class ItemExportMessageBodyReader implements MessageBodyReader<ItemExportResponse> {
private static final Logger LOG = LoggerFactory.getLogger(ItemExportMessageBodyReader.class);
private static final Integer SKU = 0;
private static final Integer BASE_SKU = 1;
public boolean isReadable(Class<?> paramClass, Type type, Annotation[] annotations,
MediaType mediaType) {
LOG.info("Cheking if content is readable or not");
return paramClass == ItemExportResponse.class && !mediaType.isWildcardType()
&& !mediaType.isWildcardSubtype()
&& mediaType.isCompatible(MediaType.valueOf("text/csv; charset=utf-8"));
}
public ItemExportResponse readFrom(Class<ItemExportResponse> paramClass, Type paramType,
Annotation[] paramArrayOfAnnotation, MediaType paramMediaType,
MultivaluedMap<String, String> paramMultivaluedMap, InputStream entityStream)
throws IOException, WebApplicationException {
InputStreamReader reader = new InputStreamReader(entityStream);
Csv csv = new Csv();
Input input = csv.createInput(reader);
List<Item> items = new ArrayList<Item>();
try {
String[] readLine;
while ((readLine = input.readLine()) != null) {
LOG.trace("Reading CSV: {}", readLine);
Item item = new Item();
item.setBaseSku(readLine[BASE_SKU]);
items.add(item);
}
} catch (IOException e) {
LOG.warn("Item export HTTP response handling failed", e);
} finally {
try {
input.close();
} catch (IOException e) {
LOG.warn("Could not close the HTTP response stream", e);
}
}
ItemExportResponse response = new ItemExportResponse();
response.setItems(items);
return response;
}
}
The following documentation says that the preferred way of making this work in a JAX-RS client to register the message body reader with the code below:
Using Entity Providers with JAX-RS Client API
Client client = ClientBuilder.newBuilder().register(MyBeanMessageBodyReader.class).build();
Response response = client.target("http://example/comm/resource").request(MediaType.APPLICATION_XML).get();
System.out.println(response.getStatus());
MyBean myBean = response.readEntity(MyBean.class);
System.out.println(myBean);
Now the thing is that I can't use the ClientBuilder. I have to extend from a specific class which constructs the client another way, and I have no access to change the construction.
So when I receive the response from the server, the client fails with the following Exception:
com.sun.jersey.api.client.ClientHandlerException: A message body reader for Java class client.response.ItemExportResponse, and Java type class client.response.ItemExportResponse, and MIME media type text/csv; charset=utf-8 was not found
Any other way to register my MessageBodyReader?
OK. If anybody would bump into my question I solved this mystery by upgrading from Jersey 1.17 to version 2.9. The documentation I linked above also covers this version not the old one, this is where the confusion stems from.
Jersey introduced backward INCOMPATIBLE changes starting from version 2, so I have no clue how to configure it in version 1.17.
In version 2 the proposed solution worked fine.