I'd like to utilize Spring Integration to initiate messages about files that appear in a remote location, without actually transferring them. All I require is the generation of a Message with, say, header values indicating the path to the file and filename.
What's the best way to accomplish this? I've tried stringing together an FTP inbound channel adapter with a service activator to write the header values I need, but this causes the file to be transferred to a local temp directory, and by the time the service activator sees it, the message consists of a java.io.File that refers to the local file and the remote path info is gone. It is possible to transform the message prior to this local transfer occurring?
We have similar problem and we solved it with filters. On inbound-channel-adapter you can set custom filter implementation. So before polling your filter will be called and you will have all informations about files, from which you can decide will that file be downloaded or not, for example;
<int-sftp:inbound-channel-adapter id="test"
session-factory="sftpSessionFactory"
channel="testChannel"
remote-directory="${sftp.remote.dir}"
local-directory="${sftp.local.dir}"
filter="customFilter"
delete-remote-files="false">
<int:poller trigger="pollingTrigger" max-messages-per-poll="${sftp.max.msg}"/>
</int-sftp:inbound-channel-adapter>
<beans:bean id="customFilter" class="your.class.location.SftpRemoteFilter"/>
Filter class is just implementation of the FileListFilter interface. Here it is dummy filter implementation.
public class SftpRemoteFilter implements FileListFilter<LsEntry> {
private static final Logger log = LoggerFactory.getLogger(SftpRemoteFilter.class);
#Override
public final List<LsEntry> filterFiles(LsEntry[] files) {
log.info("Here is files.");
//Do something smart
return Collections.emptyList();
}
}
But if you want to do that as you described, I think it is possible to do it by setting headers on payloads and then using same headers when you are using that payload, but in that case you should use Message<File> instead File in your service activator method.
Related
I started using spring integration SFTP and I have some questions.
Filters not working. I have example configuration:
Sftp.inboundAdapter(ftpFileSessionFactory())
.preserveTimestamp(true)
.deleteRemoteFiles(false)
.remoteDirectory(integrationProperties.getRemoteDirectory())
.filter(sftpFileListFilter()) // doesn't work
.patternFilter("*.xlsx") // doesn't work
And my ChainFileListFilter:
private ChainFileListFilter<ChannelSftp.LsEntry> sftpFileListFilter() {
ChainFileListFilter<ChannelSftp.LsEntry> chainFileListFilter = new ChainFileListFilter<>();
chainFileListFilter.addFilter(new SftpPersistentAcceptOnceFileListFilter(metadataStore(), "INT"));
chainFileListFilter.addFilter(new SftpSimplePatternFileListFilter("*.xlsx"));
return chainFileListFilter;
}
If I understand correctly, only the XLSX file should be saved in the local directory. If yes it doesn't work with this configuration. Am I doing something wrong or misunderstood this?
How I can configure SFTP that each downloaded file emit message? I see in the doc two params max-messages-per-poll and max-fetch-size, but I don't know how to set it up so that every file emits a message. I would like to sync files once every 24 hours and produce batch job queue. Maybe there is a workaround?
Is there built-in filter which allow me fetch only files with changed content? The best solution would be to check the checksums of the files.
I will be grateful for your help and explanations.
You cannot combine filter() and patternFilter(). Only one of them can be used: the last one overrides whatever you used before. In other words: or filter() or patternFilter() - not both. By default the logic is like this:
public SftpInboundChannelAdapterSpec patternFilter(String pattern) {
return filter(composeFilters(new SftpSimplePatternFileListFilter(pattern)));
}
private CompositeFileListFilter<ChannelSftp.LsEntry> composeFilters(FileListFilter<ChannelSftp.LsEntry>
fileListFilter) {
CompositeFileListFilter<ChannelSftp.LsEntry> compositeFileListFilter = new CompositeFileListFilter<>();
compositeFileListFilter.addFilters(fileListFilter,
new SftpPersistentAcceptOnceFileListFilter(new SimpleMetadataStore(), "sftpMessageSource"));
return compositeFileListFilter;
}
So, technically you don't need your custom one, if you don't use external persistent MetadataStore. But if you do, think about flipping SftpSimplePatternFileListFilter with SftpPersistentAcceptOnceFileListFilter. Since it is better to check for the pattern before storing the file into MetadataStore.
It is the fact that every synched remote file, passed those filters, is stored into local dir and the message for that local file is emitted immediately when the poller does a request.
The maxFetchSize plays the role when we load remote files into a local dir. The maxMessagesPerPoll is used from the poller, but those are already built from the local files. The message is emitted per local file, not as a batch for all of them. That's not what messaging is designed for.
Please, share more info what does not work with files. The SftpPersistentAcceptOnceFileListFilter checks not only file name, but also mtime of the file. So, that it not about any checksum, but more last modified timestamp of the file.
I am using int-sftp:outbound-gateway to download remote files. File download is working. I need to call another method after file is downloaded for both success as well as failure. In that method I need status (success or failure) and name of the file that was requested to be downloaded. Then from that method I will initiate a post download flow depending on the status like - moving file to different location, notifying the user, sending email, etc.
I have used AfterReturningAdviceInterceptor to call my own method defined in MyAfterReturningAdvice which implements AfterReturningAdvice interface. With this my method to initiate the post download flow. It does execute and I do get filename in GenericMessage's payload. My question is, do we have a better way to implement this flow.
I tried using ExpressionEvaluatingRequestHandlerAdvice's onSuccessExpression but from that I cannot call another method. All I can do is manipulate the inputMessage(GenericMessage instance).
In future sprints I will have compare checksum of downloaded file with expected checksum and re-download file for a fixed number of times if there is checksum mismatch. As soon as checksum matches I again need to call post download flow. If the download fails even at last retry, then I need to call another flow (send email, update db, notify user of failure,etc.)
I am asking this question just to make sure that my current implementation fits overall requirements.
<int:gateway id="downloadGateway" service-interface="com.rizwan.test.sftp_outbound_gateway.DownloadRemoteFileGateway"
default-request-channel="toGet"/>
<bean id="myAfterAdvice" class="org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor">
<constructor-arg>
<bean class="com.rizwan.test.sftp_outbound_gateway.MyAfterReturningAdvice">
</bean>
</constructor-arg>
</bean>
<int-sftp:outbound-gateway id="gatewayGet"
local-directory="C:\sftp-outbound-gateway"
session-factory="sftpSessionFactory"
request-channel="toGet"
remote-directory="/si.sftp.sample"
command="get"
command-options="-P"
expression="payload"
auto-create-local-directory="true">
<int-sftp:request-handler-advice-chain>
<ref bean="myAfterAdvice" />
</int-sftp:request-handler-advice-chain>
</int-sftp:outbound-gateway>
public class MyAfterReturningAdvice implements AfterReturningAdvice {
#Override
public void afterReturning(Object returnValue, Method method, Object[] args, Object target) throws Throwable {
//update db, send email, notify user.
}
}
The ExpressionEvaluatingRequestHandlerAdvice.onSuccessExpression() is the best choice for you. Its EvaluationContext is BeanFactory-aware, therefore you definitely can call any bean from that expression. The Message provided there as a root object is a good candidate to get an information about a downloaded file.
So, this is what you can do there:
<bean class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpressionString" value="#myBean.myMethod(#root)"/>
</bean>
The same you can do with the onFailureExpression.
On the other hand you may even don't need to worry about the bean access from the expression. The ExpressionEvaluatingRequestHandlerAdvice has successChannel and failureChannel options. So, the message with the result can be send there and some <service-activator> with your bean can handle a message on that channel.
Is it possible to extend the visibility time out of a message that is in flight.
See:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html.
Section: Changing a Message's Visibility Timeout.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQSClient.html#changeMessageVisibility-com.amazonaws.services.sqs.model.ChangeMessageVisibilityRequest-
In summary I want to be able to extend the first set visibility timeout for a given message that is in flight.
Example if 15secs have passed I then want to extend the timeout by another 20secs. Better example in java docs above.
From my understanding in the links above you can do this on the amazon side.
Below are my current settings;
SqsMessageDrivenChannelAdapter adapter =
new SqsMessageDrivenChannelAdapter(queue);
adapter.setMessageDeletionPolicy(SqsMessageDeletionPolicy.ON_SUCCESS);
adapter.setMaxNumberOfMessages(1);
adapter.setSendTimeout(2000);
adapter.setVisibilityTimeout(200);
adapter.setWaitTimeOut(20);
Is it possible to extend this timeout?
Spring Cloud AWS supports this starting with Version 2.0. Injecting a Visiblity parameter in your SQS listener method does the trick:
#SqsListener(value = "my-sqs-queue")
void onMessageReceived(#Payload String payload, Visibility visibility) {
...
var extension = visibility.extend(20);
...
}
Note, that extend will work asynchronously and will return a Future. So if you want to be sure further down the processing, that the visibility of the message is really extended at the AWS side of things, either block on the Future using extension.get() or query the Future with extension.isDone()
OK. Looks like I see your point.
We can change visibility for particular message using API:
AmazonSQS.changeMessageVisibility(String queueUrl, String receiptHandle, Integer visibilityTimeout)
For this purpose in downstream flow you have to get access to (inject) AmazonSQS bean and extract special headers from the Message:
#Autowired
AmazonSQS amazonSqs;
#Autowired
ResourceIdResolver resourceIdResolver;
...
MessageHeaders headers = message.getHeaders();
DestinationResolver destinationResolver = new DynamicQueueUrlDestinationResolver(this.amazonSqs, this.resourceIdResolver);
String queueUrl = destinationResolver.resolveDestination(headers.get(AwsHeaders.QUEUE));
String receiptHandle = headers.get(AwsHeaders.RECEIPT_HANDLE);
amazonSqs.changeMessageVisibility(queueUrl, receiptHandle, YOUR_DESIRED_VISIBILITY_TIMEOUT);
But eh, I agree that we should provide something on the matter as out-of-the-box feature. That may be even something similar to QueueMessageAcknowledgment as a new header. Or even just one more changeMessageVisibility method to this one.
Please, raise a GH issue for Spring Cloud AWS project on the matter with link to this SO topic.
You can specify whether JavaMail emits a protocol-level trace either when setting the Properties for your Session (by setting the "mail.debug" property to "true") or by calling Session.setDebug before you do the store connect.
However, when the Protocol object gets instantiated, it creates a "protocol" TraceLogger that persists for the lifetime of the protocol object. Which appears to mean that you can't temporarily disable protocol-level debug logging on a connection once you start using it.
There is a Protocol.suspendTracing method. And it does allow you to temporarily turn off protocol trace output. A bunch of the IMAP auth methods use it to keep your credentials out of the logfile. But suspendTracing is protected, so it's not callable from regular user code.
Is there another way to temporarily turn off IMAP protocol tracing? (I'd prefer temporarily turning off just the traceInput logging, but I'm fine with disabling all logging.) Do I need to write and register a whole Protocol subclass so I can get access to Protocol.suspendTracing?
You could use Session.setDebugOut to set your own stream and control it from there.
If you're using java.util.logging, you can change the logging level at any time.
Not the best solution but, you can install a custom log filter on the com.sun.mail.imap.protocol logger to check that some specific thread is allowed to produce output. Assuming your connection is local to one thread
public class ServiceFilter implements Filter {
private static final ThreadLocal<javax.mail.Service> id = new ThreadLocal<>();
public static void suspendTracing(javax.mail.Service s) {
id.set(Objects.requireNonNull(s));
}
public static void enableTracing(javax.mail.Service s) {
if (s.equals(id.get())) {
id.remove();
}
}
#Override
public boolean isLoggable(LogRecord record) {
return id.get() == null;
}
}
The downside is that this code becomes part of your project and it is another resource that you have to manage.
How can this be done? It works fine with one int-file:outbound-channel-adapter, but I could not make it work when I add another one. I actually added another, separate set of channel/adapter but it still did not work.
In int-file:outbound-channel-adapter tag, there is actually a "directory" attribute, but it only accepts a single directory path.
Here is the code I have tried:
<int-file:outbound-channel-adapter id="outputDirectory1"
directory="${output.directory1}"
channel="fileWriterChannel1"
filename-generator- expression="headers.get('filename')"
delete-source-files="true"/>
<int-file:outbound-channel-adapter id="outputDirectory2"
directory="${output.directory2}"
channel="fileWriterChannel2"
filename-generator-expression="headers.get('filename')"
delete-source-files="true"/>
Below are the channels, while the bean is the actual writer. Note that the two channels both refer to the bean (ref="messageTransformer"):
<int:transformer id="messageToStringTransformer1"
input-channel="messageTypeChannel"
output-channel="fileWriterChannel1"
ref="messageTransformer"
method="write"/>
<int:transformer id="messageToStringTransformer2"
input-channel="messageTypeChannel"
output-channel="fileWriterChannel2"
ref="messageTransformer"
method="write"/>
<bean id="messageTransformer" class="com.message.writer.DefaultMessageWriter"/>
If I do understand you correctly, do you want to write a Message payload to a collection of directories simultaneously? In order to have multiple file adapters listen to the same channel, you have to use a Publish Subscribe Channel using the element. For more information, please see: http://static.springsource.org/spring-integration/reference/html/messaging-channels-section.html#channel-configuration-pubsubchannel
When using a File Outbound Channel Adapter, you can also use the directory-expression attribute which is available since Spring Integration 2.2. It gives you full SpEL expression support. Thus, the directory you want to write to, can be for example a provided message header. For more information, please see:
http://static.springsource.org/spring-integration/reference/html/files.html#file-writing-output-directory