We are using spring-integration with S3. We have s3-inbound-streaming-channel-adapter to read from S3. What is happening is that if the "get" fails the s3-inbound-streaming-channel-adapter put the filename in "acceptOnceFilter" and doesn't retry on failure.
Q1. What we want is when the s3-inbound-streaming-channel-adapter "gets" a file from S3 and say for some reason this "get" fails... how do we get the s3-inbound-streaming-channel-adapter to retry this "get" request again for the same file?
Q2. On failure, an exception is sent to default "errorChannel" from s3-inbound-streaming-channel-adapter. Would the Message in the exception contain "filename" that failed?
<int:channel id="s3FileProcessingChannel">
<int:queue capacity="15"/>
</int:channel>
<bean id="metadataStore" class="org.springframework.integration.metadata.SimpleMetadataStore"/>
<bean id="acceptOnceFilter"
class="org.springframework.integration.aws.support.filters.S3PersistentAcceptOnceFileListFilter">
<constructor-arg index="0" ref="metadataStore"/>
<constructor-arg index="1" value="streaming"/>
</bean>
<int-aws:s3-inbound-streaming-channel-adapter id="s3Region1"
channel="s3FileProcessingChannel"
session-factory="s3SessionFactory"
filter="acceptOnceFilter"
remotedirectoryexpression="'${s3.sourceBucket}/emm'">
<int:poller fixed-delay="1000" max-messages-per-poll="15"/>
</int-aws:s3-inbound-streaming-channel-adapter>
Thanks
GM
The S3PersistentAcceptOnceFileListFilter implements:
/**
* A {#link FileListFilter} that can be reset by removing a specific file from its
* state.
* #author Gary Russell
* #since 4.1.7
*
*/
public interface ResettableFileListFilter<F> extends FileListFilter<F> {
/**
* Remove the specified file from the filter so it will pass on the next attempt.
* #param f the element to remove.
* #return true if the file was removed as a result of this call.
*/
boolean remove(F f);
}
And S3StreamingMessageSource populates headers like these:
return getMessageBuilderFactory()
.withPayload(session.readRaw(remotePath))
.setHeader(IntegrationMessageHeaderAccessor.CLOSEABLE_RESOURCE, session)
.setHeader(FileHeaders.REMOTE_DIRECTORY, file.getRemoteDirectory())
.setHeader(FileHeaders.REMOTE_FILE, file.getFilename())
.setHeader(FileHeaders.REMOTE_FILE_INFO,
this.fileInfoJson ? file.toJson() : file);
When error happened you just need to use that FileHeaders.REMOTE_FILE to call the mention above remove() and your failed file is going to be picked up from the S3 on the next poll cycle.
Related
I created an application that calls a gateway asynchronously using Splitter / Aggregator. In my concfiguration file, I invoke the process via InvestmentMessagingGateway that proceeds on calling the splitter. Every splitted message calls a service activator in parallel and pass it inn aggregator.
I placed an error channel in the InvestmentMessagingGateway and transform every failed message to pass to the aggregator as well.
I collect every successful and failed message in aggregator as a compilation for the response.
But when I tried to place an exception in one or more of the messages, I get an error in my aggregator,
Reply message received but the receiving thread has already received a reply.
<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="......."">
<context:component-scan base-package="com.api.investments"/>
<!--The gateway to be called in parallel-->
<gateway id="InvestmentGateway" service-interface="com.api.investments.gateways.InvestmentGateway"/>
<channel id="investmentDetailChannel"/>
<service-activator input-channel="investmentDetailChannel" ref="investmentService" method="getAccountPortfolio"/>
<!--Inbound gateway to invoke Splitter / Aggregator-->
<gateway id="InvestmentMessageGateway" service-interface="com.api.investments.gateways.InvestmentMessageGateway"
default-reply-channel="investmentAsyncReceiver" error-channel="investmentAsyncException"/>
<channel id="investmentAsyncSender"/>
<channel id="investmentAsyncReceiver"/>
<!-- Splitter for Invesment Details-->
<splitter input-channel="investmentAsyncSender" output-channel="investmentSplitChannel" id="investmentDetailsSplitter" ref="investmentComponentsSplitter" />
<channel id="investmentSplitChannel">
<queue />
</channel>
<!--Calls the Investment Gateway asynchronously using split messages ad send the response in aggregator-->
<service-activator input-channel="investmentSplitChannel" output-channel="investmentAggregateChannel" ref="investmentAsyncActivator" method="retrieveInvestmentDetailsAsync" requires-reply="true">
<poller receive-timeout="5000" task-executor="investmentExecutor" fixed-rate="50"/>
</service-activator>
<channel id="investmentAsyncException"/>
<!--Handles failed messages and pass it in aggregator-->
<transformer input-channel="investmentAsyncException" output-channel="investmentAggregateChannel" ref="invesmentErrorLogger" method="logError"/>
<!--Aggreggates successfull and failed messaged-->
<publish-subscribe-channel id="investmentAggregateChannel"/>
<aggregator input-channel="investmentAggregateChannel" output-channel="investmentAsyncReceiver" id="investmentAggregator"
ref="investmentComponentsAggregator" correlation-strategy="investmentComponentsCorrelationStrategy"
expire-groups-upon-completion="true"
send-partial-result-on-expiry="true" />
<task:executor id="investmentExecutor" pool-size="10-1000"
queue-capacity="5000"/>
</beans:beans>
I tried putting my error channel in the poller of the service activator and but the error is still the same but this time it didn't went to the aggregator.
I also tried putting a mid-gateway for the service activator like this
but the error became null.
<gateway id="InvestmentAsyncActivatorGateway" service-interface="com.api.investments.gateways.InvestmentAsyncActivatorGateway"
default-reply-channel="investmentAggregateChannel" error-channel="investmentAsyncException"/>
----UPDATE------
This is the transformer that handles every error message
#Component("invesmentErrorLogger")
public class InvesmentErrorLoggerImpl implements InvestmentErrorLogger {
private final Logger logger = LoggerFactory.getLogger(Application.class.getName());
/**
* handles all error messages in InvestmentMessageGateway
* Creates an error message and pass it in the aggregator channel
* #param invesmentMessageError
* #return errorMessage
*/
#Override
public Message<ErrorDetails> logError(Message<?> invesmentMessageError) {
if(invesmentMessageError.getPayload().getClass().equals(MessagingException.class)) {
MessagingException messageException = (MessagingException) invesmentMessageError.getPayload();
AccountPortfolioRequest failedMsgPayload = (AccountPortfolioRequest) messageException.getFailedMessage().getPayload();
String logError = "Exception occured in Account Number: " + failedMsgPayload.getiAccNo();
logger.error(logError);
ErrorDetails productErrorDetail = new ErrorDetails();
productErrorDetail.setCode(InvestmentAPIErrorMessages.SVC_ERR_INQACCNTPORTFOLIO);
productErrorDetail.setMessage(InvestmentAPIErrorMessages.SVC_ERR_INQACCNTPORTFOLIO_DESC + ". Problem occured in Account Number: " + failedMsgPayload.getiAccNo());
Message<ErrorDetails> errorMessage = MessageBuilder.withPayload(productErrorDetail)
.setHeaderIfAbsent(InvestmentInquiryConstants.INV_CORRELATION_STRATEGY, InvestmentInquiryConstants.INV_CORRELATION_STRATEGY_VALUE)
.build();
return errorMessage;
}
else if(invesmentMessageError.getPayload().getClass().equals(MessageDeliveryException.class)) {
MessageDeliveryException messageException = (MessageDeliveryException) invesmentMessageError.getPayload();
AccountPortfolioRequest failedMsgPayload = (AccountPortfolioRequest) messageException.getFailedMessage().getPayload();
String logError = "Exception occured in Account Number: " + failedMsgPayload.getiAccNo();
logger.error(logError);
ErrorDetails productErrorDetail = new ErrorDetails();
productErrorDetail.setCode(InvestmentAPIErrorMessages.SVC_ERR_INQACCNTPORTFOLIO);
productErrorDetail.setMessage(InvestmentAPIErrorMessages.SVC_ERR_INQACCNTPORTFOLIO_DESC + ". Problem occured in Account Number: " + failedMsgPayload.getiAccNo());
Message<ErrorDetails> errorMessage = MessageBuilder.withPayload(productErrorDetail)
.setHeaderIfAbsent(InvestmentInquiryConstants.INV_CORRELATION_STRATEGY, InvestmentInquiryConstants.INV_CORRELATION_STRATEGY_VALUE)
.build();
return errorMessage;
}
else {
Exception messageException = (Exception) invesmentMessageError.getPayload();
String logError = "Exception occured in Investment Gateway ";
logger.error(logError);
logger.equals(messageException.getMessage());
ErrorDetails productErrorDetail = new ErrorDetails();
productErrorDetail.setCode(InvestmentAPIErrorMessages.SVC_ERR_INQACCNTPORTFOLIO);
productErrorDetail.setMessage(InvestmentAPIErrorMessages.SVC_ERR_INQACCNTPORTFOLIO_DESC + " " + messageException.getMessage());
Message<ErrorDetails> errorMessage = MessageBuilder.withPayload(productErrorDetail)
.setHeaderIfAbsent(InvestmentInquiryConstants.INV_CORRELATION_STRATEGY, InvestmentInquiryConstants.INV_CORRELATION_STRATEGY_VALUE)
.build();
return errorMessage;
}
}
}
Reply message received but the receiving thread has already received a reply.
As the error suggests, you can't send multiple replies (or errors) for a single request; it is strictly one reply per request.
You need another gateway between the splitter and the service.
The mid-flow gateway should have no service-interface so it uses the RequestReplyExchanger.
In the signup process of Spring Lemon, I'm trying to send the verification email.
I've put messages_en.properties and messages_fr.properties in my resourcesfolder.
Here is the relevant content of messages_en.properties
com.naturalprogrammer.spring.verifyEmail: Hi,<br/><br/>Your email id at XYZ is unverified. Please click the link below to get verified:<br/><br/>{0}<br/><br/>
But when I look at the logs, it sends the mail without replacing the {0} by the verifyLink.
I looked at the code and figured out that this portion of LemonService is the isse :
// send the mail
mailSender.send(user.getEmail(),
LemonUtil.getMessage("com.naturalprogrammer.spring.verifySubject"),
LemonUtil.getMessage(
"com.naturalprogrammer.spring.verifyEmail", verifyLink));
But the actual work is being done by this code in LemonUtil.java :
/**
* Gets a message from messages.properties
*
* #param messageKey the key of the message
* #param args any arguments
*/
public static String getMessage(String messageKey, Object... args) {
// http://stackoverflow.com/questions/10792551/how-to-obtain-a-current-user-locale-from-spring-without-passing-it-as-a-paramete
return messageSource.getMessage(messageKey, args,
LocaleContextHolder.getLocale());
}
I managed to solve it somehow by deleting the {0} in the .properties, and by adding the link myself like this :
// send the mail
mailSender.send(user.getEmail(),
LemonUtil.getMessage("com.naturalprogrammer.spring.verifySubject"),
LemonUtil.getMessage(
"com.naturalprogrammer.spring.verifyEmail", verifyLink) + verifyLink);
I think the getMessage method of org.springframework.context.MessageSource is not working properly.
My question is : what could prevent messageSource from working ?
am new to Spring Integration
my objective is passing message to one channel to another (chain process)
Channel1 ---> chennal2 --> chennal3 ---> chennal4.
(Msg1) (Msg2) (Msg3) (Msg4)
\ \ / /
\ \ / /
errorChennal (Msg5)
1. Msg1(EmployeeObject), Msg2 (DetailsObjet), Msg3(VerificationObject), Msg4(FinalObject), Msg5(ErrorObject)
Each channel payloads will have different Class Objects.
All the channel need to be communicated to "errorChennal" in case of Exception, validation error, etc.
Tried:
1. when I tried with #transformer am not able to communicate to "erroeChannel".
when I tried with #Router(header-value-router) Message transform not happing. Msg1 object is to all object
Question:
How do i route one channel to specific channel with message transformer?
FOUND ANSWER
Configuration:
<int:channel id="channel1"/>
<int:channel id="channel2"/>
<int:channel id="channel3"/>
<int:channel id="channel4"/>
<int:service-activator input-channel="channel1" ref="firstChannel" method="doProcess1" />
<int:service-activator input-channel="channel2" ref="secondChannel" method="doProcess2" />
<int:service-activator input-channel="channel3" ref="thirdChannel" method="doProcess3" />
<int:service-activator input-channel="channel4" ref="forthChannel" method="doProcess4" />
<int:service-activator input-channel="errorChannel" ref="errorHandlerChannel" method="doErrorProcess" />
Java Code:
public FirstChannel {
private Map<String, MessageChannel> msgChannels;
boolean isError = false;
#Autowired
public ScheduleParser(Map<String, MessageChannel> msgChannels) {
super();
this.msgChannels = msgChannels;
}
public void doprocess1(Message<?> message){
File file = (File) message.getPayload();
//business code
if(!isError)
//transforming the messae
msgChannels.get("channel2").send(new GenericMessage(EmployeeVO , headersMap));
else
msgChannels.get("errorChannel").send(new GenericMessage(ObjectVO , headersMap));
}
}
Same way Other channels Code will be
Channels aren't connected to each other they are connected to endpoints.
Transformers don't route, routers route.
You need something like
channel1->payload-type-router
type1Channel->...
type2Channel->...
...
possibly with a different transformer for each payload type downstream.
It's generally better to show the configuration you have tried rather than some abstraction like you have shown.
The error flow configuration depends on what starts your flow (gateway, poller, etc) - i.e. what is upstream of channel1.
EDIT
Your configuration is completely wrong. For example, you currently have two subscribers on channel1 - a header value router and service activator.
Messages arriving on channel1 will alternately go to one or the other - messages going to the service activator will never go to the router.
Without knowing the complete picture, I am guessing you need something like...
channel1->serviceActivator1->channel1toRouter->router(channel2 or failed)
channel2->serviceActivator2->channel2toRouter->router(channel2 or failed)
...
I am working on camel’s dynamic router to derive and dispatch to the relevant queue by referring http://camel.apache.org/dynamic-router.html
Following is the camel config xml:
<route id="dynamicRouter" autoStartup="true">
<from uri="vm://service1?concurrentConsumers=10&timeout=0" />
<choice>
<when>
<simple>${body.documentStatus} == 'SUCCESS' </simple>
<log message="routing to dynamic router" />
<dynamicRouter>
<!-- use a method call on a bean as dynamic router -->
<method ref="messageRouter" method="route" />
</dynamicRouter>
</when>
</choice>
</route>
Following is my dynamic router bean:
public class MessageRouter {
private static final String QUEUE_BROKER = "activemq";
private static final String DEFAULT_QUEUE = "queue";
/**
*
* #param obj
* message
* #return the dynamically generated queue name
*/
public String route(ServiceObject obj) {
RecordObject record = obj.getRecordObject();
if(record != null){
return QUEUE_BROKER + ":"
+ record.getId() + "." + DEFAULT_QUEUE;
}
return null;
}
}
I am able to en-queue messages in dynamically created queue but messages are getting enqueued infinitely. Any help in identifying what could be the reason would be of great help.
Thanks in advance!!!
Read the documentation about dynamic router at
http://camel.apache.org/dynamic-router.html
And see the beware box, it says the method must return null to signal to the dynamic router to break out.
So in your code above, then this code line
RecordObject record = obj.getRecordObject();
... the record object is never null and therefore the dynamic router keeps going forever.
If you only want this dynamic routing one time then use the dynamic receipient list eip instead, see
http://camel.apache.org/how-to-use-a-dynamic-uri-in-to.html
http://camel.apache.org/recipient-list.html
And your xml route is
<recipientList>
<!-- use a method call on a bean as dynamic router -->
<method ref="messageRouter" method="route" />
</recipientList>
I am using inbound-channel-adapter of spring integration. I want to poll under two different directories -one per file category- and parse the files that are located there. The code that i use is:
<int:channel id="inputChannel"/>
<file:inbound-channel-adapter id="fileInOne"
directory="myDirOne"
auto-create-directory="true"
channel = "inputChannel">
<int:poller id="one" cron="1/10 * * * * *"/>
</file:inbound-channel-adapter>
<file:inbound-channel-adapter id="fileInTwo"
directory="myDirTwo"
auto-create-directory="true"
channel = "inputChannel">
<int:poller id="two" cron="1/10 * * * * *"/>
</file:inbound-channel-adapter>
Both inbound-channel-adapters use the same channel. So I want to know from which inbound-channel-adapter the file was loaded.
These are two ways that I can think of:
a. Pass each of the flows through a header enricher, add a custom header which tells you which directory you started from, and then to the inputChannel.
<file:inbound-channel-adapter id="fileInOne"
directory="myDirOne"
auto-create-directory="true"
channel = "dirOneEnricher">
<int:poller id="one" cron="1/10 * * * * *"/>
</file:inbound-channel-adapter>
<int:header-enricher input-channel="dirOneEnricher" output-channel="inputChannel">
<int:header name="fileCategory" value="dirOneTypeCategory"/>
</int:header-enricher>
..
b. Since the payload is a java.io.File, you can use the API's to find out which directory this file belongs to and take some action.