i am asked to create a 2 projects, project A and B, that makes use of JMS. B is a WEB project with a simple page containing a form where i input two dates(start date- end date). After submitting the form, B will ask A to process a task (Query a files) based on the two dates. B will then display the result in a web page. the caveat is that i need to do this in jms.
in my B controller im thinking about this (simplified. i'm using spring )
#controller
Mycontroller{
MyMessageProducer mp;
#RequestMapping(....)
public String(...){
mp.sendMessage(...);
//wait for the response here and render?
}
}
now i'm stuck with how to implement project A. if a use a point-to-point messaging (using queue), then that means that A will have to explicitly get the message from the queue(im using activemq). Thats bad because A should be automatically listening for request, shouldn't it? However, if i use publisher-subscriber, in this case the publisher would be B, the client (because B sends message to A), which i think is a bad solution. which strategy should i use?
Now suppose that A successfully receives the message and query the file, how will i send the result back to B such the B will be able to display the result in a web page? is there a way to do this?
(PS i'm new to JMS though i've already implement a simple producers and receivers based on tutorials)
The easiest way, IMO, is to use Spring Integration which has the concept of Gateways (see also Enterprise Integration Patterns). You can just specify a service-interface that has a method with a return value. Something like
public interface MessageProducer {
#Gateway
public String sendMessageAndGetReply(String name);
}
A proxy of the interface will be created. when you reference it from the <int:gateway> element. Something like
<int:channel id="requestChannel"/>
<int:channel id="replyChannel"/>
<int:gateway id="messageProducerGateway" default-request-channel="requestChannel"
default-reply-channel="replyChannel"
service-interface="demo.MessageProducer">
</int:gateway>
<int-jms:outbound-gateway id="outboundJmsGateway"
connection-factory="connectionFactory" reply-channel="replyChannel"
request-channel="requestChannel" reply-destination-name="reply.queue"
request-destination-name="request.queue">
</int-jms:outbound-gateway>
The <int-jms:outbound-gateway> will send the message out the jms queue and receive a reply. connectionFactory is just your usual ConnectionFactory instances (i.e. ActiveMQConnectionFactory, CachingConnectionFactory)
On the "server" side of the jms interaction, you'll use a <int-jms:inbound-gateway>, Something like
<int-jms:inbound-gateway id="inboundJmsGateway"
request-channel="requestChannel" acknowledge="client"
connection-factory="connectionFactory" request-destination-name="request.queue" />
<int:service-activator id="messageHandler" ref="serverMessageHandler"
input-channel="requestChannel">
</int:service-activator>
The serverMessageHandler is just a simple component with a method to handle and return a reply back to the gateway.
#Component
public class ServerMessageHandler {
#ServiceActivator
public String handleMessage(String message) {
return "Hello, " + message;
}
}
Running a simple demo
AbstractApplicationContext client = new ClassPathXmlApplicationContext("demo-gateway.xml");
AbstractApplicationContext server = new ClassPathXmlApplicationContext("demo-gateway-server.xml");
MessageProducer producer = client.getBean(MessageProducer.class);
String returnedMessage = producer.sendMessageAndGetReply("StackOverflow");
System.out.println(returnedMessage);
you get "Hello, StackOverflow". There's not really much to it, once you get a basic understanding of the framework. The example I gave a synchronous example. You can see a full example (along with other examples) at spring-integration-samples at github. I'd take some time to go over the reference guide to get familiar with the basics.
Related
I'm trying to setup a project with Springboot cloud Stream with Kafka. I managed to build a simple example, where a listener gets messages from a topic and after processed it, it sends the output to another topic.
My listener and channels are configured like this:
#Component
public class FileEventListener {
private FileEventProcessorService fileEventProcessorService;
#Autowired
public FileEventListener(FileEventProcessorService fileEventProcessorService) {
this.fileEventProcessorService = fileEventProcessorService;
}
#StreamListener(target = FileEventStreams.INPUT)
public void handleLine(#Payload(required = false) String jsonData) {
this.fileEventProcessorService.process(jsonData);
}
}
public interface FileEventStreams {
String INPUT = "file_events";
String OUTPUT = "raw_lines";
#Input(INPUT)
SubscribableChannel inboundFileEventChannel();
#Output(OUTPUT)
MessageChannel outboundRawLinesChannel();
}
The problem with this example is that when the service starts, it doesn't check for messages that already exist in the topic, it only process those messages that are sent after it started. I'm very new to Springboot stream and kafka, but for what I've read, this behavior may correspond to the fact that I'm using a SubscribableChannel. I tried to use a QueueChannel for example, to see how it works but I found the following exception:
Error creating bean with name ... nested exception is java.lang.IllegalStateException: No factory found for binding target type: org.springframework.integration.channel.QueueChannel among registered factories: channelFactory,messageSourceFactory
So, my questions are:
If I want to process all messages that exists in the topic once the application starts (and also messages are processed by only one consumer), I'm on the right path?
Even if QueueChannel is not the right choice for achieve the behavior explained in 1.) What do I have to add to my project to be able to use this type of channel?
Thanks!
Add spring.cloud.stream.bindings.file_events.group=foo
anonymous groups consume from the end of the topic only, bindings with a group consume from the beginning, by default.
You cannot use a PollableChannel for a binding, it must be a SubscribableChannel.
I have a scenario where I need to invoke A and B system's REST call in parallel and aggregate the responses and transform into single FinalResponse.
To achieve this, I am using Spring Integration splitter and agreggator and the configuration is as given below.
I have exposed a REST endpoint, when the request(request has co-relationId in the header) comes to the controller, we invoke gateway and the splitter sends requests to A and B channels .
Service activator A listens to channel A and invokes A system's REST call and service Activator B listens to B channel and invokes B system's REST call.
Then I need to aggregate the responses from A and B system and then transform it into FinalResponse. Currently the aggregation and transformation is working fine.
Sometimes when multiple requests come to controller, the FinalResponse takes more time when compared to single request to controller. All the responses to the requests come almost at the same time not sure why (even though the last request to controller was sent 6-7 secs after the 1st request). Is there something wrong in my configuration related to threads? Not sure why it takes more time to respond when multiple requests comes to the controller.
Also, I am not using any CorrelationStrategy, do we need to use it? Will I face any issues in multi threading environment with the below configuration? Any feedback on the configuration would be helpful
// Controller
{
FinalResponse aggregatedResponse = gateway.collateServiceInformation(inputData);
}
//Configuration
#Autowired
Transformer transformer;
//Gateway
#Bean
public IntegrationFlow gatewayServiceFlow() {
return IntegrationFlows.from("input_channel")
.channel("split_channel").get();
}
//splitter
#Bean
public IntegrationFlow splitAggregatorFlow() {
return IntegrationFlows.from("split_channel").
.split(SomeClass.class, SomeClass::getResources)
.channel(c -> c.executor(Executors.newCachedThreadPool()))
.<Resource, String>route(Resource::getName,
mapping -> mapping.channelMapping("A", "A")
.channelMapping("B", "B"))
.get();
}
//aggregator
#Bean
public IntegrationFlow aggregateFlow() {
return IntegrationFlows.from("aggregate_channel").aggregate()
.channel("transform_channel").transform(transformer).get();
}
.
.
.
//Transformer
#Component
#Scope("prototype")
public class Transformer {
#Transformer
public FinalResponse transform(final List<Result> responsesFromAAndB) {
//transformation logic and then return final response
}
}
The splitter provides a default strategy for correlation details in headers. The Aggregator will use them afterwards. What you talk about is called scatter-gather: https://docs.spring.io/spring-integration/docs/5.0.8.RELEASE/reference/html/messaging-routing-chapter.html#scatter-gather. There is a Java DSL equivalent.
I think your problem that some request in the splitted set fails, so am Aggregator can’t finish a group for that request. Nothing obvious so far in your config...
maybe someone has an idea to my following problem:
I am currently on a project, where i want to use the AWS SQS with Spring Cloud integration. For the receiver part i want to provide a API, where a user can register a "message handler" on a queue, which is an interface and will contain the user's business logic, e.g.
MyAwsSqsReceiver receiver = new MyAwsSqsReceiver();
receiver.register("a-queue-name", new MessageHandler(){
#Override
public void handle(String message){
//... business logic for the received message
}
});
I found examples, e.g.
https://codemason.me/2016/03/12/amazon-aws-sqs-with-spring-cloud/
and read the docu
http://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_sqs_support
But the only thing i found there to "connect" a functionality for processing a incoming message is a annotation on a method, e.g. #SqsListener or #MessageMapping.
These annotations are fixed to a certain queue-name, though. So now i am at a loss, how to dynamically "connect" my provided "MessageHandler" (from my API) to the incoming message for the specified queuename.
In the Config the example there is a SimpleMessageListenerContainer, which gets a QueueMessageHandler set, but this QueueMessageHandler does not seem
to be the right place to set my handler or to override its methods and provide my own subclass of QueueMessageHandler.
I already did something like this with the Spring Amqp integration and RabbitMq and thought, that it would be also similar here with AWS SQS.
Does anyone have an idea, how to accomplish this?
thx + bye,
Ximon
EDIT:
I found, that Spring JMS could actually do that, e.g. www.javacodegeeks.com/2016/02/aws-sqs-spring-jms-integration.html. Does anybody know, what consequences using JMS protocol has here, good or bad?
I am facing the same issue.
I am trying to go in an unusual way where I set up an Aws client bean at build time and then instead of using sqslistener annotation to consume from the specific queue I use the scheduled annotation which I can programmatically pool (each 10 secs in my case) from which queue I want to consume.
I did the example that iterates over queues defined in properties and then consumes from each one.
Client Bean:
#Bean
#Primary
public AmazonSQSAsync awsSqsClient() {
return AmazonSQSAsyncClientBuilder
.standard()
.withRegion(Regions.EU_WEST_1.getName())
.build();
}
Consumer:
// injected in the constructor
private final AmazonSQSAsync awsSqsClient;
#Scheduled(fixedDelay = 10000)
public void pool() {
properties.getSqsQueues()
.forEach(queue -> {
val receiveMessageRequest = new ReceiveMessageRequest(queue)
.withWaitTimeSeconds(10)
.withMaxNumberOfMessages(10);
// reading the messages
val result = awsSqsClient.receiveMessage(receiveMessageRequest);
val sqsMessages = result.getMessages();
log.info("Received Message on queue {}: message = {}", queue, sqsMessages.toString());
// deleting the messages
sqsMessages.forEach(message -> {
val deleteMessageRequest = new DeleteMessageRequest(queue, message.getReceiptHandle());
awsSqsClient.deleteMessage(deleteMessageRequest);
});
});
}
Just to clarify, in my case, I need multiple queues, one for each tenant, with the queue URL for each one passed in a property file. Of course, in your case, you could get the queue names from another source, maybe a ThreadLocal which has the queues you have created in runtime.
If you wish, you can also try the JMS approach where you create message consumers and add a listener to each one you wish (See the doc Aws Jms documentation).
When we do Spring and SQS we use the spring-cloud-starter-aws-messaging.
Then just create a Listener class
#Component
public class MyListener {
#SQSListener(value="myqueue")
public void listen(MyMessageType message) {
//process the message
}
}
We have a consumer on some RabbitMQ queues which reads messages and then indexes this data in Elasticsearch. The implementation is done using spring-amqp. In order to increase our performance, we plan to aggregate the messages at the consumer level and do a bulk insert in Elasticsearch (this would really increase performance).
Do you have any sugestions on how to implement this? Also, another sensitive problem is how to handle responses. Each message has a "reply_to" header and we use an inbound gateway with a reply-channel, so for each message a response is supposed to be delivered.
I am thinking of using the aggregator from spring integration, with a release strategy based on a batch size and a period of time when the MessageGroupStore will expire (and of course the reaper). The inbound gateway has a task executor of 20 let's say and a prefetch count of 20 also. Whenever a request comes, the message will be added into the group store and when the canRelease() condition is ok, the reaper ot one of the threads that came with the request will do a bulk operation. But what I do with the other threads, which will have to wait for a response which will never come. Also, I don't know how to break the response for the big, aggregated message so each small request will have a response.
Another problem, how do I ack the messages? From what I read transactions will decrease performance on the RabbitMQ side, so I not that pleased of using the "tx-size" attribute. Also this attribute might do a wrong count if the timeout is too small.
The answer to the question regarding consumer and aggregator:
The config to consume messages from AMQP and aggregate.
The aggregate strategy is based on the Transction commit:
<amqp:inbound-channel-adapter queue-names="myQueue"
transaction-manager="transactionManager"
channel-transacted="true"
channel="aggregateChannel"
advice-chain="aggregatorReaperAdvice"
concurrent-consumers="4"
tx-size="100"/>
<aggregator input-channel="aggregateChannel" output-channel="storeChannel"
expire-groups-upon-completion="true"
correlation-strategy-expression="T(Thread).currentThread().id"
release-strategy-expression="^[payload.equals(#AGGREGATOR_RELEASE_MARK)] != null"
expression="?[!payload.equals(#AGGREGATOR_RELEASE_MARK)].![payload]"/>
The ReaperAdvice (Groovy code):
#Service
class AggregatorReaperAdvice implements MethodBeforeAdvice, InitializingBean {
private static final TRANSACTION_RESOURCE_MARK = 'TRANSACTION_RESOURCE_MARK'
public static final AGGREGATOR_RELEASE_MARK = 'AGGREGATOR_RELEASE_MARK'
MessagingTemplate messagingTemplate
#Autowired
MessageChannel aggregateChannel
#Override
void afterPropertiesSet() throws Exception {
Assert.notNull aggregateChannel, "aggregateChannel must not be null"
messagingTemplate = new MessagingTemplate(aggregateChannel)
}
#Override
void before(Method method, Object[] args, Object target) {
if (!TransactionSynchronizationManager.hasResource(AggregatorReaperAdvice)) {
TransactionSynchronizationManager.bindResource(AggregatorReaperAdvice, TRANSACTION_RESOURCE_MARK)
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronizationAdapter() {
#Override
void beforeCommit(boolean readOnly) {
messagingTemplate.send(MessageBuilder.withPayload(AGGREGATOR_RELEASE_MARK).build())
}
#Override
void afterCompletion(int status) {
TransactionSynchronizationManager.unbindResource(AggregatorReaperAdvice)
}
})
}
}
}
Let me know if it isn't clear.
All other question, will be addressed soon.
For manual ack you can use channel.basicAck(deliveryTag, true); - to ack on the last deliveryTag for all previous messages.
For the headers["reply_to"] case... I think you should provide custom AbstractAggregatingMessageGroupProcessor for the aggregator and kill two birds: the cumulative result of aggregator and iteration over MessageGroup.getMessages() to send each of them for the reply process to the provided MessageChannel. It is a quick solution for your case.
Something similar but more loosely-coupled solution maybe based on the result from aggregator and its MessageGroupStore, where you extract correlationKey to retrieve group and its messages to do the desired reply logic. In this case you shouldn't remove group from store with aggregator, but manually after that group retrieval.
I am in the process of learning Spring Integration and using it to implement a basic email service in Grails. What I want to be able to do is call my email service but only have the email be sent if the transaction trying to send the email is successful. Although this is being done in Grails, it really shouldn't be different from a regular Spring app except for using the BeanBuilder DSL instead of the XML configuration.
Anyway, here is my configuration for the channel:
beans = {
xmlns integration:'http://www.springframework.org/schema/integration'
integration.channel(id: 'email')
}
Here is my service:
class MailService {
#ServiceActivator(inputChannel = "email")
MailMessage sendMail(Closure callable) {
//sending mail code
}
}
Now what I expect to happen is that when I inject this MailService into another service and call send mail, that will place a message on the email channel, which will only get published if my transaction completes. What leads me to believe this is the section on UserProcess here: http://docs.spring.io/spring-integration/reference/html/transactions.html, which states that a user started process will have all the transactional properties that Spring provides.
I am attempting to test this with an integration test:
void "test transactionality"() {
when:
assert DomainObject.all.size() == 0
DomainObject.withNewTransaction { status ->
DomainObject object = buildAndSaveNewObject()
objectNotificationService.sendEmails(object) //This service injects emailService and calls sendMail
throw new Exception()
}
then:
thrown(Exception) // This is true
DomainObject.all.size() == 0 // This is true
greenMail.receivedMessages.length == 0 // This fails
}
What this does is create and save an object and send emails all within the same transaction. I then throw an exception to cause that transaction to fail. As expected, none of my domain objects are persisted. However, I still receive emails.
I am quite new to Spring Integration and Spring in general, so it's possible I'm misunderstanding how this is all supposed to work, but I would expect the sendMail message to never be placed on the email channel.
It turns out that I don't think Spring Integration is the best way to achieve "only perform on commit" functionality (but if you do, Gary Russell's answer is the way to go.) You should instead use the TransactionSynchronizationManager provided as part of the Spring transaction management framework.
As an example, I created a TransactionService in grails:
class TransactionService {
def executeAfterCommit(Closure c) {
TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronizationAdapter() {
#Override
void afterCommit() {
c.call()
}
})
}
}
You can then inject this anywhere you need it and use it like so:
def transactionService
transactionService.executeAfterCommit {
sendConfirmationEmail()
}
I don't know how this would be done in Grails, but in Java, you could use a transaction synchronization factory whereby you can take different actions depending on success/failure...
<int:transaction-synchronization-factory id="syncFactory">
<int:after-commit expression="payload.renameTo('/success/' + payload.name)" channel="committedChannel" />
<int:after-rollback expression="payload.renameTo('/failed/' + payload.name)" channel="rolledBackChannel" />
</int:transaction-synchronization-factory>
The result of the expression evaluation is sent to the channel, where you can have your outbound mail adapter.