I'd like to process messages in a queue in parallel using a simple Processor/AsyncProcessor as a destination. The processor takes a little time per message, but each message can be handled seperately, and thus at the same time (within healthy boundaries).
I'm having a hard time finding examples, especially about the xml configuration of camel routes.
So far, I've defined a threadpool, route and processor:
<threadPool id="smallPool" threadName="MyProcessorThread" poolSize="5" maxPoolSize="50" maxQueueSize="100"/>
<route>
<from uri="broker:queue:inbox" />
<threads executorServiceRef="smallPool">
<to uri="MyProcessor" />
</threads>
</route>
<bean id="MyProcessor" class="com.example.java.MyProcessor" />
and my processor looks like:
public class MyProcessor implements Processor {
#Override
public void process(Exchange exchange) throws Exception {
Message in = exchange.getIn();
String msg = in.getBody(String.class);
System.out.println(msg);
try {
Thread.sleep(10 * 1000); // Do something in the background
} catch (InterruptedException e) {}
System.out.println("Done!");
}
}
Unfortunatly, when I post messages to the queue, they are still processed one by one, each delayed by 10 seconds (my "background task").
Can anyone point me to the right direction to have the messages processed using the defined threadpool or explain what I am doing wrong?
You should use the concurrentConsumers options as said in the comments,
<route>
<from uri="broker:queue:inbox?concurrentConsumers=5" />
<to uri="MyProcessor" />
</route>
Notice there is also maxConcurrentConsumers you can set to use a min/max range of concurrent consumers, so Camel will automatic grow/shrink depending on load.
See more details in the JMS docs at
http://camel.apache.org/jms
Related
This is my client side code :
public class ABCServlet extends HttpServlet {
protected void doGet(HttpServletRequest request,
HttpServletResponse response){
//do blah blah
String msg = null;
java.io.OutputStream os = response.getOutputStream();
java.io.ObjectOutputStream oos = new java.io.ObjectOutputStream(os);
oos.writeObject(msg);
msg = null;
oos.flush();
oos.close();
}
I don't know how using the above code my listener gets kicked off -
public class ABCListener implements MessageListener {
#Override
public void onMessage(Message arg0) {
AbstractJDBCFacade façade = null;
try{
façade = something;
throw new UserException();
}catch(UserException ex){
log.error("ABC Exception " + ex);
}
Configuration :
<bean id="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">....
<bean id="jmsQueue" class="org.springframework.jndi.JndiObjectFactoryBean">
<bean id="listenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer102">
I have 3 questions :
1. without putting it on the queue explicitly , how a listener gets invoked?
2. When onMessage method throws UserException, instead of logging I want to pass the message to the client. How can I do that ?
3. Why would someone use JndiObjectFactoryBean instead of ActiveMQ...
JMS by design was supposed to be asynchronous and one-way. Even "synchronous" jms with using receive method of consumer will internally turn into creating a new temporary queue. And here we come to the second point about it's one-way nature. JMS queue was supposed to be one-way and that's why it is called point-to-point (http://www.enterpriseintegrationpatterns.com/patterns/messaging/PointToPointChannel.html). Of course technically with some dancing you will manage to achieve what you want but it is bad practice which will also lead to performance degradation due to the fact that you will need filtering.
To get this thing work fast the best way will be to have exactly one logical receiver (of course you can use concurrent cosumers for one receiver but that should be one logical consumer without any need of filtering the message).
without putting it on the queue explicitly , how a listener gets invoked?
Listener get invoked only when a message comes to a queue. Thats the only way to get it work as it was supposed to work.
In general there are two types of message consuming models: push (also known as event-driven consuming) and poll. In case of using push model all listeners (according to canonical observer pattern) got registered somewhere in the broker and then, when broker receive new message in some queue, it executes listener's method. On the others side in polling model consumer take care itself about receiving messages. So with some interval it comes to a broker and checks the queue for new messages.
Push model: http://www.enterpriseintegrationpatterns.com/patterns/messaging/EventDrivenConsumer.html
Poll model: http://www.enterpriseintegrationpatterns.com/patterns/messaging/PollingConsumer.html
When onMessage method throws UserException, instead of logging I want to pass the message to the client. How can I do that ?
Thats a very bad practice. Of course technically you can achieve it with dirty tricks but thats not the right way of using jms. When onMessage throws the exception then message wont be taken from the queue (of course if u did not reconfigured acknowledge mods or used another tricks). So the best way of solving your probem fmpv is to use redelivery limit on message and a dead letter queue(http://www.enterpriseintegrationpatterns.com/patterns/messaging/DeadLetterChannel.html). If system was not able to process the message after some attempts (redelivery limit shows exactly this) then broker remove message from the queue and send it to a so-called dead letter queue where all failed (from the point of broker) messages are stored. And then client can read that queue and decide what to do with message.
In amq: http://activemq.apache.org/message-redelivery-and-dlq-handling.html
If you want to use so-called "synchronous" features in JMS and really there is no way of using dead letter queue or smth like that then actually you can use consumer.recieve method on the client. But in this case you should send response on every message. In case of success you can send one message and in case of failure error messages. And so a client will be able to understand what is going on. But i dont think that you need such a huge overhead cause actually you need only failure messages. Also in this case you will have to take care about appropriate receive timeouts.
Why would someone use JndiObjectFactoryBean instead of ActiveMQ...
That's cause you are using Spring and there are additional features especially for spring.
PS:
1. For consuming:
How can I send a message using just this piece of code? Don't I need
to put this on a queue? java.io.OutputStream os =
response.getOutputStream(); java.io.ObjectOutputStream oos = new
java.io.ObjectOutputStream(os); oos.writeObject(msg);
For receiving smth like this:
`
<bean id="connectionFactory" class="org.springframework.
jndi.JndiObjectFactoryBean">
<property name="jndiTemplate" ref="baseJNDITemplate"/>
<property name="jndiName"
value="weblogic.jms.ConnectionFactory"/>
</bean>
<bean id="queue" class="org.springframework.
jndi.JndiObjectFactoryBean">
<property name="jndiTemplate" ref="baseJNDITemplate"/>
<property name="jndiName" value="#{properties.queueName}"/>
</bean>
<bean id="messageListenerContainer"
class="org.springframework.jms.listener.
DefaultMessageListenerContainer">
<property name="connectionFactory" ref="connectionFactory"/>
<property name="destination" ref="queue"/>
<property name="messageListener" ref="messageListener"/>
<property name="sessionTransacted" value="true"/>
</bean>
<bean id="messageListener" class="com.example.ABCListener"/>
And then simply all logic for message processing will be in the listener.
For sending smth like this in config:
<bean id="jmsQueueTemplate"
class="org.springframework.
jms.core.JmsTemplate">
<property name="connectionFactory">
<ref bean="jmsConnectionFactory"/>
</property>
<property name="destinationResolver">
<ref bean="jmsDestResolver"/>
</property>
...
</bean>
<bean id="jmsDestResolver"
class=" org.springframework.jms.support.destination.
JndiDestinationResolver"/>
<bean id="jmsConnectionFactory"
class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:comp/env/jms/myCF"/>
<property name="lookupOnStartup" value="false"/>
<property name="cache" value="true"/>
<property name="proxyInterface" value="amq con fact here"/>
</bean>
and in code simply use jmsTemplate.send(queue, messageCreator) method:
#Autowired
ConnectionFactory connectionFactory;
#Test(enabled = false)
public void testJmsSend(final String msg) throws Exception {
JmsTemplate template = new JmsTemplate(connectionFactory);
template.send("test_queue", new MessageCreator() {
#Override
public Message createMessage(Session session)
throws JMSException {
return session.createTextMessage(msg);
}
});
}
https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/cspr_data_access_jms.html
I believe Dead channel comes in picture only when the message is not properly received by the receiver. In my case, the receiver
received it and processed it, however while processing it failed with
some exception. I want to let the sender know that there was a
exception and the message did not process successfully. I can do this
using a response queue but I don't want to do that, can the receiver
receive a message from the sender on the same queue ? How?
Dead letter channel is a kind of error handling for message processing also. If message processing had failed then after the limit end it got transferred there. It is not actually only for transport issues but also for processing issues. If the message processing got failed with exception then message will stay in the queue and wont be acked by default. So what we should do with this message? For example if it failed due to our database error or smth like this? We should initiate error-handling process, notify assurance systems and stakeholders, collect all necessary info and preserve the message. Due to this kind of queues, which was create d exactly for that, it is much easier. And then customer support team will investigate error queue for further analysis of what has happened. Also we have monitoring tools for notifications and statistics collection on such errors. After understanding what has happened message got removed from the queue and archived.
After processing a message, the consumer is responsible for deleting
the message. If the consumer doesn't delete the message, for example
because because it crashed while processing the message, the message
becomes visible again after the message's Visibility Timeout expires.
Each time this happens, the message's receive count is increased.
When this count reaches a configured limit, the message is placed in a
designated Dead Letter Queue.
http://www.enterpriseintegrationpatterns.com/patterns/messaging/DeadLetterChannel.html
I can do this using a response queue but I don't want to do that, can
the receiver receive a message from the sender on the same queue ?
How?
For you it will look like it's the same queue but internally new temporary queue will be created. To achieve that you should use jms request\reply message pattern. More here: http://activemq.apache.org/how-should-i-implement-request-response-with-jms.html
The only part still confuses me is : If I expect my JMS listener
(receiver) to listen to the queue, then my sender should also
implement JMS and connect to the same queue and send a message. But in
the ABCListener application that I am supporting does not have any
configuration where the sender is configured to the queue. All the
sender does is 3 lines of code : java.io.OutputStream os =
response.getOutputStream(); java.io.ObjectOutputStream oos = new
java.io.ObjectOutputStream(os); oos.writeObject(msg); Literally, that
is it. I don't know how it still works!
Of course 3 lines of code with outputstream do nothing except populating msg string. To send any jms message to the queue you anyway will have to use JMS Api or some library like Spring which wrap it by adding additional features.
I've wrote simple samples to get it more clear.
Modified servlet for asynchronous processing with dead letter queue (for dlq you should create also another listener ofc)
public class AsynchronousJmsSenderServlet extends HttpServlet {
#Override
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
String msg = null;
try (java.io.OutputStream os = response.getOutputStream()) {
try(java.io.ObjectOutputStream oos = new java.io.ObjectOutputStream(os)) {
oos.writeObject(msg);
}
}
sendJmsMessage(msg);
}
private void sendJmsMessage(final String msg) {
ConnectionFactory connectionFactory = null; //here get it in some way from spring
JmsTemplate template = new JmsTemplate(connectionFactory);
template.send("your_queue_name", new MessageCreator() {
#Override
public Message createMessage(Session session)
throws JMSException {
return session.createTextMessage(msg);
}
});
}
}
And here is the code for "synchronous" processing and status reply messages
public class SynchronousJmsSenderServlet extends HttpServlet {
#Override
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
String msg = null;
try (java.io.OutputStream os = response.getOutputStream()) {
try(java.io.ObjectOutputStream oos = new java.io.ObjectOutputStream(os)) {
oos.writeObject(msg);
}
}
sendJmsMessage(msg);
}
private void sendJmsMessage(final String msg) {
ConnectionFactory connectionFactory = null; //here get it in some way from spring
JmsTemplate template = new JmsTemplate(connectionFactory);
Message reply = template.sendAndReceive("your_queue_name", new MessageCreator() {
#Override
public Message createMessage(Session session)
throws JMSException {
return session.createTextMessage(msg);
}
});
if(reply instanceof TextMessage) {
try {
String status = ((TextMessage) reply).getText();
//do error handling if status is error
} catch (JMSException ex) {
throw new RuntimeException("Unable to get status message", ex);
}
} else {
throw new RuntimeException("Only text messages are supported");
}
}
}
public class SynchronousJmsMessageListener implements SessionAwareMessageListener {
#Override
public void onMessage(Message request, Session session) throws JMSException {
try {
//do some processing
sendReply(request, session, "OK");
} catch (Exception ex) {
sendReply(request, session, "Error: " + ex.toString());
}
}
private void sendReply(Message request, Session session, String status) {
try {
TextMessage reply = null; //for example you can use ActiveMQTextMessage here
reply.setJMSCorrelationID(request.getJMSCorrelationID());
reply.setText(status);
MessageProducer producer = session.createProducer(reply.getJMSReplyTo());
producer.send(reply);
} catch (JMSException exception) {
throw new RuntimeException("Unable to send reply", exception);
}
}
}
You will need Spring 5 to have sendAndReceive method on jmsTemplate. Or you will have to do all that manually.
PS1: Please let me know if that will work
I am new to rabbitmq and am trying the following scenario
--> producer sends message
--> consumer receives the message
-- Execute my own logic
if the logic fails - requeue
--> requeue the message if the consumer fails(machine goes down)
I have implemented the basic sender using Spring rabbitTemplate
rabbitTemplate.convertAndSend(.....);
and for consumer i implemented a message listener
public class CustomMessageListener implements MessageListener {
#Override
public void onMessage(Message message) {
//** my own logic**
}
}
and added it to the container through spring
<bean id="aListener" class="com.sample.CustomMessageListener" autowire="byName"/>
<rabbit:listener-container id="myListenerContainer" connection-factory="connectionFactory" acknowledge="auto" prefetch="750" concurrency="5" >
<rabbit:listener ref="aListener" queues="reportQueue"/>
</rabbit:listener-container>
Its working fine till this part.
now if ** my own logic** mentioned in the listener fails. i want to requeue the message. how can i implement this. From the blogs that i have gone through it looks like returnedMessage needs to overridden. But am not sure how it can be done through listener.
With acknowledge="auto", the message won't be ack'd until the listener exits normally, so there's nothing extra that you need to do; if your listener throws an exception or the server crashes, the message will remain in the queue.
i am asked to create a 2 projects, project A and B, that makes use of JMS. B is a WEB project with a simple page containing a form where i input two dates(start date- end date). After submitting the form, B will ask A to process a task (Query a files) based on the two dates. B will then display the result in a web page. the caveat is that i need to do this in jms.
in my B controller im thinking about this (simplified. i'm using spring )
#controller
Mycontroller{
MyMessageProducer mp;
#RequestMapping(....)
public String(...){
mp.sendMessage(...);
//wait for the response here and render?
}
}
now i'm stuck with how to implement project A. if a use a point-to-point messaging (using queue), then that means that A will have to explicitly get the message from the queue(im using activemq). Thats bad because A should be automatically listening for request, shouldn't it? However, if i use publisher-subscriber, in this case the publisher would be B, the client (because B sends message to A), which i think is a bad solution. which strategy should i use?
Now suppose that A successfully receives the message and query the file, how will i send the result back to B such the B will be able to display the result in a web page? is there a way to do this?
(PS i'm new to JMS though i've already implement a simple producers and receivers based on tutorials)
The easiest way, IMO, is to use Spring Integration which has the concept of Gateways (see also Enterprise Integration Patterns). You can just specify a service-interface that has a method with a return value. Something like
public interface MessageProducer {
#Gateway
public String sendMessageAndGetReply(String name);
}
A proxy of the interface will be created. when you reference it from the <int:gateway> element. Something like
<int:channel id="requestChannel"/>
<int:channel id="replyChannel"/>
<int:gateway id="messageProducerGateway" default-request-channel="requestChannel"
default-reply-channel="replyChannel"
service-interface="demo.MessageProducer">
</int:gateway>
<int-jms:outbound-gateway id="outboundJmsGateway"
connection-factory="connectionFactory" reply-channel="replyChannel"
request-channel="requestChannel" reply-destination-name="reply.queue"
request-destination-name="request.queue">
</int-jms:outbound-gateway>
The <int-jms:outbound-gateway> will send the message out the jms queue and receive a reply. connectionFactory is just your usual ConnectionFactory instances (i.e. ActiveMQConnectionFactory, CachingConnectionFactory)
On the "server" side of the jms interaction, you'll use a <int-jms:inbound-gateway>, Something like
<int-jms:inbound-gateway id="inboundJmsGateway"
request-channel="requestChannel" acknowledge="client"
connection-factory="connectionFactory" request-destination-name="request.queue" />
<int:service-activator id="messageHandler" ref="serverMessageHandler"
input-channel="requestChannel">
</int:service-activator>
The serverMessageHandler is just a simple component with a method to handle and return a reply back to the gateway.
#Component
public class ServerMessageHandler {
#ServiceActivator
public String handleMessage(String message) {
return "Hello, " + message;
}
}
Running a simple demo
AbstractApplicationContext client = new ClassPathXmlApplicationContext("demo-gateway.xml");
AbstractApplicationContext server = new ClassPathXmlApplicationContext("demo-gateway-server.xml");
MessageProducer producer = client.getBean(MessageProducer.class);
String returnedMessage = producer.sendMessageAndGetReply("StackOverflow");
System.out.println(returnedMessage);
you get "Hello, StackOverflow". There's not really much to it, once you get a basic understanding of the framework. The example I gave a synchronous example. You can see a full example (along with other examples) at spring-integration-samples at github. I'd take some time to go over the reference guide to get familiar with the basics.
I need to be able to poll a directory for a specific file using SCP, and once the file has been processed, it needs to keep polling.
Is this possible with Spring Batch?
The normal way to handle this is using Spring Integration. The way I'd address it is with a Spring Integration flow that uses a SFTP Inbound Channel Adapter to retrieve the files, then passes the transferred name to Spring Batch to launch. The flow would actually be similar to the sample in the SpringBatchIntegration in my Spring Batch Webinar here: https://github.com/mminella/SpringBatchWebinar
In that example, I use Twitter to launch the job. The only thing you'd need to change is the twitter piece for the SFTP.
I had to solve the same question (but just accessing to the local filesystem) and I did not find any solution in the framework, so I ended up creating my own class which polls for the file and creates a resource.I know this is just a workaround, but I haven't found a better way to do that so far.
I can't remember where (maybe in the "retry handling" part) but I read in the documentation something like "batch jobs should not try to solve issues like files not found, connections down and so, these kind of errors should make the job raise an error to be handled by operators" so I gave up...
On the other hand Spring Retry was part of Spring batch and now is a new separate library, maybe you just can assume the file is there and if the reader does not find it, let the step fail and establish a "retry policy" for that step, but for me that's overkill.
This is what I did:
<bean id="resourceFactory"
class="com.mycompany.batch.zip.ResourceFactory">
<property name="retryAttemps" value="${attemps}" />
<property name="timeBetweenAttemps" value="${timeBetweenAttemps}"/>
</bean>
<bean id="myResource"
factory-bean="resourceFactory" factory-method="create" scope="step">
<constructor-arg value="${absolutepath}" type="java.lang.String" />
</bean>
<!--step scope to avoid looking for the file when deployment-->
<bean id="myReader"
class="org.springframework.batch.item.xml.StaxEventItemReader" scope="step">
<property name="fragmentRootElementName" value="retailer" />
<property name="unmarshaller" ref="reportUnmarshaller" />
<property name="resource" ref="myResource" />
</bean>
And this is my class:
public class ResourceFactory {
public static final Logger LOG= LoggerFactory.getLogger(ResourceFactory.class);
private int retryAttemps;
private long timeBetweenAttemps;
public Resource create(String resource) throws IOException, InterruptedException {
Resource r;
File f=new File(resource);
int attemps=1;
while (!f.exists()) {
if (attemps<this.retryAttemps) {
attemps++;
LOG.warn("File "+resource+" not found, waiting "+timeBetweenAttemps+
" before retrying. Attemp: "+attemps+" of "+this.retryAttemps);
Thread.sleep(this.timeBetweenAttemps);
} else {
throw new FileNotFoundException(resource);
}
if (resource!=null && resource.endsWith(".zip")) {
ZipFile zipFile = new ZipFile(resource);
ZipEntry entry=zipFile.entries().nextElement();
if (entry==null) {
throw new FileNotFoundException("The zip file has no entries inside");
}
//TODO Test if a buffered Stream is faster than the raw InputStream
InputStream is=new BufferedInputStream(zipFile.getInputStream(entry));
r= new InputStreamResource(is);
if (LOG.isInfoEnabled()) {
int size=(int)entry.getSize();
LOG.info("Opening a compressed file of "+size+" bytes");
}
} else {
LOG.info("Opening a regular file");
r= new FileSystemResource(f);
}
}
return r;
}
}
If anyone knows a better way to do that, I'll gladly remove this answer (and implement the new solution)
PS: BTW, I've found some faults in my code when reviewing this post, so for me this is being helpful even with no other answers :)
I have this use case.
First chain:
<int:chain input-channel="inserimentoCanaleActivate" output-channel="inserimentoCanalePreRouting">
<int:service-activator ref="inserimentoCanaleActivator" method="activate" />
</int:chain>
This is the relative code:
#Override
#Transactional(propagation = Propagation.REQUIRES_NEW)
public EventMessage<ModificaOperativitaRapporto> activate(EventMessage<InserimentoCanale> eventMessage) {
...
// some Database changes
dao.save(myObject);
}
All is working great.
Then I have another chain:
<int:chain id="onlineCensimentoClienteChain" input-channel="ONLINE_CENSIMENTO_CLIENTE" output-channel="inserimentoCanaleActivate">
<int:service-activator ref="onlineCensimentoClienteActivator" method="activate" />
<int:splitter expression="payload.getPayload().getCanali()" />
</int:chain>
And the relative activator:
#Override
public EventMessage<CensimentoCliente> activate(EventMessage<CensimentoCliente> eventMessage) {
...
// some Database changes
dao.save(myObject);
}
The CensimentoCliente payload as described below has a List of payload of the first chain, so with a splitter I split on the list and reuse the code of the first chain.
public interface CensimentoCliente extends Serializable {
Collection<? extends InserimentoCanale> getCanali();
void setCanali(Collection<? extends InserimentoCanale> canali);
...
}
But since every activator gets his transaction definition (since the first one can live without the second one) I have a use case where the transactions are separated.
The goal is to have the db modifies of the two chains been part of the same transaction.
Any help?
Kind regards
Massimo
You can accomplish this by creating a custom channel (or other custom component, but this is the simplest approach) that wraps the message dispatch in a TransactionTemplate callback execution:
public class TransactionalChannel extends AbstractSubscribableChannel {
private final MessageDispatcher dispatcher = new UnicastingDispatcher();
private final TransactionTemplate transactionTemplate;
TransactionalChannel(TransactionTemplate transactionTemplate) {
this.transactionTemplate = transactionTemplate;
}
#Override
protected boolean doSend(final Message<?> message, long timeout) {
return transactionTemplate.execute(new TransactionCallback<Boolean>() {
#Override
public Boolean doInTransaction(TransactionStatus status) {
return getDispatcher().dispatch(message);
}
});
}
#Override
protected MessageDispatcher getDispatcher() {
return dispatcher;
}
}
In your XML, you can define your channel and transaction template and reference your custom channel just as you would any other channel:
<bean id="transactionalChannel" class="com.stackoverflow.TransactionalChannel">
<constructor-arg>
<bean class="org.springframework.transaction.support.TransactionTemplate">
<property name="transactionManager" ref="transactionManager"/>
<property name="propagationBehavior" value="#{T(org.springframework.transaction.TransactionDefinition).PROPAGATION_REQUIRES_NEW}"/>
</bean>
</constructor-arg>
</bean>
For your example, you could perhaps use a bridge to pass the message through the new channel:
<int:bridge input-channel="inserimentoCanaleActivate" output-channel="transactionalChannel" />
<int:chain input-channel="transactionalChannel" output-channel="inserimentoCanalePreRouting">
<int:service-activator ref="inserimentoCanaleActivator" method="activate" />
</int:chain>
You you have <service-activator> and #Transactional on service method, the transaction will be bounded only to that method invocation.
If you want to have a transction for entire message flow (or its part) you should declare TX advice somewhere before.
If your channels are direct all service invocations will be wrapped with the same transaction.
The most simple way to accomplish your wishes, write simple #Gateway interface with #Transactional and call it from the start of your message flow.
To clarify a bit regarding transactions
Understanding Transactions in Message flows
Are these modifying 2 separate relational databases ? If so you are looking at an XA transaction. Now if you are running this on a non XA container like tomcat, all of this must be done in a single thread that is watched by a transaction manager - (you will have to piggy back on the transaction manager that actually triggers these events). The transaction manager can be a JMS message or a poller against some data source. Also this processing must be done in a single thread so that spring can help you run the entire process in a single transaction.
As a final note , do not introduce threadpools / queues between service activators. This can cause the activators to run in separate threads