Spring Integration with SFTP - spring

I'm building a little microservice to access files from a SFTP file server. I decided to use Spring Integration SFTP to get the job done. I'm new to Spring Integration and confused with how it all works.
My goal is to get a list of files a directory on a SFTP server and present them to the user interface. From there a user will select a file for download and I'll use the filename to stream the file from the SFTP server to the user interface.
I'm using the following code which does work.
Entire class to handle SFTP with SSH
#Slf4j
#Configuration
public class SftpConfig {
#Value("${sftp.host}")
private String sftpHost;
#Value("${sftp.port:22}")
private int sftpPort;
#Value("${sftp.user}")
private String sftpUser;
#Value("${sftp.privateKey:#{null}}")
private Resource sftpPrivateKey;
#Value("${sftp.privateKeyPassphrase:}")
private String sftpPrivateKeyPassphrase;
#Value("${sftp.password:#{null}}")
private String sftpPasword;
#Value("${sftp.remote.directory:/}")
private String sftpRemoteDirectory;
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(sftpHost);
factory.setPort(sftpPort);
factory.setUser(sftpUser);
if (sftpPrivateKey != null) {
factory.setPrivateKey(sftpPrivateKey);
factory.setPrivateKeyPassphrase(sftpPrivateKeyPassphrase);
} else {
factory.setPassword(sftpPasword);
}
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
#ServiceActivator(inputChannel = "ftpLS")
#Bean
public SftpOutboundGateway getFtpLS() {
SftpOutboundGateway gateway = new SftpOutboundGateway(sftpSessionFactory(), "ls", "'" + sftpRemoteDirectory + "' + payload");
gateway.setOption(AbstractRemoteFileOutboundGateway.Option.NAME_ONLY);
return gateway;
}
#ServiceActivator(inputChannel = "ftpGet")
#Bean
public SftpOutboundGateway getFtpGet() {
SftpOutboundGateway gateway = new SftpOutboundGateway(sftpSessionFactory(), "get", "'" + sftpRemoteDirectory + "' + payload");
gateway.setOption(AbstractRemoteFileOutboundGateway.Option.STREAM);
return gateway;
}
#MessagingGateway(defaultRequestChannel = "ftpLS")
public interface FtpLS {
List list(String directory);
}
#MessagingGateway(defaultRequestChannel = "ftpGet")
public interface FtpGet {
InputStream get(String fileName);
}
}
Run
#Bean
public ApplicationRunner runner(SftpConfig.FtpLS ftpLS, SftpConfig.FtpGet ftpGet) {
return args -> {
List<String> list = ftpLS.list("139");
System.out.println("Result:" + list);
InputStream is = ftpGet.get("139/" + list.get(0));
String theString = IOUtils.toString(is,"UTF-8");
System.out.println("Result:" + theString);
};
}
My number one question is this the correct approach?
Secondly, do I need two interfaces in order to use the two different SftpOutboundGateway's?
Lastly, is there a better way to pass in a dynamic directory name when doing a FtsGet? Right now I'm passing I'm concatenating 139 with the base directory in a string and passing it in through the interface.

is this the correct approach?
Yes, the approach is correct. Although I would suggest do not use isSharedSession for the gateway since it might be used from different threads by different users.
do I need two interfaces?
No, you really can have one #MessagingGateway, but with several methods marked with their own #Gateway annotations.
is there a better way to pass in a dynamic directory?
No, your approach is correct. There is no something like working directory to switch automatically, like we can do in FTP.

Related

how to Spring integration SftpOutboundGateway setFilter Custom Argument?

How can I pass a parameter as a filter condition when getting the file list of the SFTP server from MessagingGateway?
My SftpMessageGateway code
#MessagingGateway
public interface SftpMessageGateway {
#Gateway(requestChannel = "getSftpChannel")
List<SftpFileInfo> getIconListByProductUiId(#Payloads("productUiId") String productUiId);
Integration Config
#Bean
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(host);
factory.setPort(port);
factory.setUser(id);
factory.setPassword(password);
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
#Bean
#ServiceActivator(inputChannel = "getSftpChannel")
public MessageHandler getMessageHandler() {
SftpOutboundGateway outboundGateway = new SftpOutboundGateway(sftpSessionFactory(), "ls", "'" + uploadPath + "'");
outboundGateway.setOption(AbstractRemoteFileOutboundGateway.Option.NAME_ONLY);
outboundGateway.setFilter(new SftpSimplePatternFileListFilter("*alpha*"));
outboundGateway.setFilter(new SftpSimplePatternFileListFilter("I want get custom argument)); <----
return outboundGateway;
}
You can set only one filter into a gateway, however there is a CompositeFileListFilter where you can combine a set of filters, include any custom impl of the FileListFilter.
See more info in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/file.html#remote-persistent-flf
You can refer following code snippet for implementing FileListFilter. My use case was to fetch most latest file uploaded in SFTP directory.
#Component
public class LastModifiedFileFilter implements FileListFilter<LsEntry> {
#Override
public List<LsEntry> filterFiles(LsEntry[] files) {
List<LsEntry> result = new ArrayList<LsEntry>();
Vector<LsEntry> list = new Vector<LsEntry>();
Collections.addAll(list, files);
ChannelSftp.LsEntry lastModifiedEntry = Collections.max(list,
(Comparator.comparingInt(entry -> entry.getAttrs().getMTime())));
result.add(lastModifiedEntry);
return result;
}
}
Once you have your own custom filter in place then you need to 'Chain' it with your other filters in SftpOutboundGateway object. For your reference, I did it this way
ChainFileListFilter<LsEntry> filterList = new ChainFileListFilter<LsEntry>();
filterList.addFilter(new SftpSimplePatternFileListFilter("*alpha*"));
filterList.addFilter(new LastModifiedFileFilter());
setFilter(filterList);
For me, it will now fetch latest file having "alpha" string present in its name. Hope this helps.

Spring Integration - Dynamic MailReceiver configuration

I'm pretty new to spring-integration anyway I'm using it in order to receive mails and elaborate them.
I used this spring configuration class:
#Configuration
#EnableIntegration
#PropertySource(value = { "classpath:configuration.properties" }, encoding = "UTF-8", ignoreResourceNotFound = false)
public class MailReceiverConfiguration {
private static final Log logger = LogFactory.getLog(MailReceiverConfiguration.class);
#Autowired
private EmailTransformerService emailTransformerService;
// Configurazione AE
#Bean
public MessageChannel inboundChannelAE() {
return new DirectChannel();
}
#Bean(name= {"aeProps"})
public Properties aeProps() {
Properties javaMailPropertiesAE = new Properties();
javaMailPropertiesAE.put("mail.store.protocol", "imap");
javaMailPropertiesAE.put("mail.debug", Boolean.TRUE);
javaMailPropertiesAE.put("mail.auth.debug", Boolean.TRUE);
javaMailPropertiesAE.put("mail.smtp.socketFactory.fallback", "false");
javaMailPropertiesAE.put("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory");
return javaMailPropertiesAE;
}
#Bean(name="mailReceiverAE")
public MailReceiver mailReceiverAE(#Autowired MailConfigurationBean mcb, #Autowired #Qualifier("aeProps") Properties javaMailPropertiesAE) throws Exception {
return ConfigurationUtil.getMailReceiver("imap://USERNAME:PASSWORD#MAILSERVER:PORT/INBOX", new BigDecimal(2), javaMailPropertiesAE);
}
#Bean
#InboundChannelAdapter( autoStartup = "true",
channel = "inboundChannelAE",
poller = {#Poller(fixedRate = "${fixed.rate.ae}",
maxMessagesPerPoll = "${max.messages.per.poll.ae}") })
public MailReceivingMessageSource pollForEmailAE(#Autowired MailReceiver mailReceiverAE) {
MailReceivingMessageSource mrms = new MailReceivingMessageSource(mailReceiverAE);
return mrms;
}
#Transformer(inputChannel = "inboundChannelAE", outputChannel = "transformerChannelAE")
public MessageBean transformitAE( MimeMessage mailMessage ) throws Exception {
// amministratore email inbox
MessageBean messageBean = emailTransformerService.transformit(mailMessage);
return messageBean;
}
#Splitter(inputChannel = "transformerChannelAE", outputChannel = "nullChannel")
public List<Message<?>> splitIntoMessagesAE(final MessageBean mb) {
final List<Message<?>> messages = new ArrayList<Message<?>>();
for (EmailFragment emailFragment : mb.getEmailFragments()) {
Message<?> message = MessageBuilder.withPayload(emailFragment.getData())
.setHeader(FileHeaders.FILENAME, emailFragment.getFilename())
.setHeader("directory", emailFragment.getDirectory()).build();
messages.add(message);
}
return messages;
}
}
So far so good.... I start my micro-service and there is this component listening on the specified mail server and mails are downloaded.
Now I have this requirement: mail server configuration (I mean the string "imap://USERNAME:PASSWORD#MAILSERVER:PORT/INBOX") must be taken from a database and it can be configurable. In any time a system administrator can change it and the mail receiver must use the new configuration.
As far as I understood I should create a new instance of MailReceiver when a new configuration is present and use it in the InboundChannelAdapter
Is there any best practice in order to do it? I found this solution: ImapMailReceiver NO STORE attempt on READ-ONLY folder (Failure) [THROTTLED];
In this solution I can inject the ThreadPoolTaskScheduler if I define it in my Configuration class; I can also inject the DirectChannel but every-time I should create a new MailReceiver and a new ImapIdleChannelAdapter without considering this WARN message I get when the
ImapIdleChannelAdapter starts:
java.lang.RuntimeException: No beanfactory at org.springframework.integration.expression.ExpressionUtils.createStandardEvaluationContext(ExpressionUtils.java:79) at org.springframework.integration.mail.AbstractMailReceiver.onInit(AbstractMailReceiver.java:403)
Is there a better way to satisfy my scenario?
Thank you
Angelo
The best way to do this is to use the Java DSL and dynamic flow registration.
Documentation here.
That way, you can unregister the old flow and register a new one, each time the configuration changes.
It will automatically handle injecting dependencies such as the bean factory.

Spring Integration: how to access the returned values from last Subscriber

I'm trying to implement a SFTP File Upload of 2 Files which has to happen in a certain order - first a pdf file and after successfull upload of that an text file with meta information about the pdf.
I followed the advice in this thread, but can't get it to work properly.
My Spring Boot Configuration:
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
final DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
final Properties jschProps = new Properties();
jschProps.put("StrictHostKeyChecking", "no");
jschProps.put("PreferredAuthentications", "publickey,password");
factory.setSessionConfig(jschProps);
factory.setHost(sftpHost);
factory.setPort(sftpPort);
factory.setUser(sftpUser);
if (sftpPrivateKey != null) {
factory.setPrivateKey(sftpPrivateKey);
factory.setPrivateKeyPassphrase(sftpPrivateKeyPassphrase);
} else {
factory.setPassword(sftpPasword);
}
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
#Bean
#BridgeTo
public MessageChannel toSftpChannel() {
return new PublishSubscribeChannel();
}
#Bean
#ServiceActivator(inputChannel = "toSftpChannel")
#Order(0)
public MessageHandler handler() {
final SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
handler.setRemoteDirectoryExpression(new LiteralExpression(sftpRemoteDirectory));
handler.setFileNameGenerator(message -> {
if (message.getPayload() instanceof byte[]) {
return (String) message.getHeaders().get("filename");
} else {
throw new IllegalArgumentException("File expected as payload.");
}
});
return handler;
}
#ServiceActivator(inputChannel = "toSftpChannel")
#Order(1)
public String transferComplete(#Payload byte[] file, #Header("filename") String filename) {
return "The SFTP transfer complete for file: " + filename;
}
#MessagingGateway
public interface UploadGateway {
#Gateway(requestChannel = "toSftpChannel")
String upload(#Payload byte[] file, #Header("filename") String filename);
}
My Test Case:
final String pdfStatus = uploadGateway.upload(content, documentName);
log.info("Upload of {} completed, {}.", documentName, pdfStatus);
From the return of the Gateway upload call i expect to get the String confirming the upload e.g. "The SFTP transfer complete for file:..." but I get the the returned content of the uploaded File in byte[]:
Upload of 123456789.1.pdf completed, 37,80,68,70,45,49,46,54,13,37,-30,-29,-49,-45,13,10,50,55,53,32,48,32,111,98,106,13,60,60,47,76,105,110,101,97,114,105,122,101,100,32,49,47,76,32,50,53,52,55,49,48,47,79,32,50,55,55,47,69,32,49,49,49,55,55,55,47,78,32,49,47,84,32,50,53,52,51,53,57,47,72,32,91,32,49,49,57,55,32,53,51,55,93,62,62,13,101,110,100,111,98,106,13,32,32,32,32,32,32,32,32,32,32,32,32,13,10,52,55,49,32,48,32,111,98,106,13,60,60,47,68,101,99,111,100,101,80,97,114,109,115,60,60,47,67,111,108,117,109,110,115,32,53,47,80,114,101,100,105,99,116,111,114,32,49,50,62,62,47,70,105,108,116,101,114,47,70,108,97,116,101,68,101,99,111,100,101,47,73,68,91,60,57,66,53,49,56,54,69,70,53,66,56,66,49,50,52,49,65,56,50,49,55,50,54,56,65,65,54,52,65,57,70,54,62,60,68,52,50,68,51,55,54,53,54,65,67,48,55,54,52,65,65,53,52,66,52,57,51,50,56,52,56,68,66 etc.
What am I missing?
I think #Order(0) doesn't work together with the #Bean.
To fix it you should do this in that bean definition istead:
final SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory());
handler.setOrder(0);
See Reference Manual for more info:
When using these annotations on consumer #Bean definitions, if the bean definition returns an appropriate MessageHandler (depending on the annotation type), attributes such as outputChannel, requiresReply etc, must be set on the MessageHandler #Bean definition itself.
In other words: if you can use setter, you have to. We don't process annotations for this case because there is no guarantee what should get a precedence. So, to avoid such a confuse we have left for you only setters choice.
UPDATE
I see your problem and it is here:
#Bean
#BridgeTo
public MessageChannel toSftpChannel() {
return new PublishSubscribeChannel();
}
That is confirmed by the logs:
Adding {bridge:dmsSftpConfig.toSftpChannel.bridgeTo} as a subscriber to the 'toSftpChannel' channel
Channel 'org.springframework.context.support.GenericApplicationContext#b3d0f7.toSftpChannel' has 3 subscriber(s).
started dmsSftpConfig.toSftpChannel.bridgeTo
So, you really have one more subscriber to that toSftpChannel and it is a BridgeHandler with an output to the replyChannel header. And a default order is like private volatile int order = Ordered.LOWEST_PRECEDENCE; this one becomes as a first subscriber and exactly this one returns you that byte[] just because it is a payload of request.
You need to decide if you really need such a bridge. There is no workaround for the #Order though...

How to use encrypted store-uri in Spring ImapIdleChannelAdapter

Sample spring configuration is as below.
<int-mail:imap-idle-channel-adapter id="mailAdapter"
store-uri="imaps://${"username"}:${"password"}#imap-server:993/INBOX"
java-mail-properties="javaMailProperties"
channel="emails"
should-delete-messages="false"
should-mark-messages-as-read="true">
</int-mail:imap-idle-channel-adapter>
I wish to keep the password field encrypted in properties file and decrypt it in the code. I am not sure on how to set mailReceiver property of ImapIdleChannelAdapter to my custom version of ImapMailReceiver.
Please let me know if there is any way to do this.
All of my configurations are in XML as described above.
Above solution of adding the defifnation did not work may be I am doing something wrong. Then I tried using XML + Java configuration, as below.
#Configuration
public class EmailConfiguration {
#Bean
public ImapIdleChannelAdapter customAdapter() {
ImapIdleChannelAdapter adapter = new ImapIdleChannelAdapter(mailReceiver());
adapter.setOutputChannel(outputChannel());
adapter.setErrorChannel(errorChannel());
adapter.setAutoStartup(true);
adapter.setShouldReconnectAutomatically(true);
adapter.setTaskScheduler(taskScheduler());
return adapter;
}
#Bean
public TaskImapMailReceiver mailReceiver() {
TaskImapMailReceiver mailReceiver = new TaskImapMailReceiver("imaps://[username]:[password]#imap.googlemail.com:993/inbox");
mailReceiver.setShouldDeleteMessages(false);
mailReceiver.setShouldMarkMessagesAsRead(true);
//mailReceiver.setJavaMailProperties(javaMailProperties());
mailReceiver.setMaxFetchSize(Integer.MAX_VALUE);
return mailReceiver;
}
}
Also created empty errorChannel,outputChannel etc. I observed that Spring creates two instances one with xml config and other with java #Configuration. Where it was expected to use only java configuration. If I remove the xml config tag
then it provides sigle imap instance with my mailReceiver but runs only once does not go periodic. also does not show IMAPS logs.
Just wondering if I need to do so much to encrypt the password. Is somthing wrong with my approach.
Use Java configuration instead of XML...
#Configuration
public class MyConfigClass {
#Bean
public MyMailReceiver receiver() {
...
}
#Bean
public ImapIdleChannelAdapter adapter() {
ImapIdleChannelAdapter adapter = new ImapIdleChannelAdapter(receiver());
...
return adapter;
}
}
If you are using XML for everything else, simply add this class as a <bean/> to your XML.
EDIT
Here's an example that works fine for me...
#SpringBootApplication
public class So42298254Application {
public static void main(String[] args) {
SpringApplication.run(So42298254Application.class, args);
}
#Bean
public TestMailServer.ImapServer imapServer() {
return TestMailServer.imap(0);
}
#Bean
public ImapMailReceiver receiver() {
ImapMailReceiver imapMailReceiver = new ImapMailReceiver(imapUrl("user", "pw"));
imapMailReceiver.setHeaderMapper(new DefaultMailHeaderMapper()); // converts the MimeMessage to a String
imapMailReceiver.setUserFlag("testSIUserFlag"); // needed by the SI test server
Properties javaMailProperties = new Properties();
javaMailProperties.put("mail.debug", "true");
imapMailReceiver.setJavaMailProperties(javaMailProperties);
return imapMailReceiver;
}
private String imapUrl(String user, String pw) {
return "imap://"
+ user + ":" + pw
+ "#localhost:" + imapServer().getPort() + "/INBOX";
}
#Bean
public ImapIdleChannelAdapter adapter() {
ImapIdleChannelAdapter adapter = new ImapIdleChannelAdapter(receiver());
adapter.setOutputChannelName("handleMail");
return adapter;
}
#ServiceActivator(inputChannel = "handleMail")
public void handle(String mail, #Header(MailHeaders.FROM) Object from) {
System.out.println(mail + " from:" + from);
imapServer().resetServer(); // so we'll get the email again
}
}
My intention was to use encrypted passwords in properties files.
So I changed my approach of getting into email receiving classes. I added inherited PropertyPlaceholderConfigurer and implemented method convertPropertyValue() as below.
public class EncryptationAwarePropertyPlaceholderConfigurer extends PropertyPlaceholderConfigurer {
private static final Logger logger = LoggerFactory.getLogger(EncryptationAwarePropertyPlaceholderConfigurer.class);
#Override
protected String convertPropertyValue(String originalValue) {
if (originalValue.contains("{<ENC>}") && originalValue.contains("{</ENC>}")) {
String encryptedTaggedValue = originalValue.substring(originalValue.indexOf("{<ENC>}"), originalValue.indexOf("{</ENC>}") + 8);
String encryptedValue = originalValue.substring(originalValue.indexOf("{<ENC>}") + 7, originalValue.indexOf("{</ENC>}"));
try {
String decryptedValue = EncrypDecriptUtil.decrypt(encryptedValue);//EncrypDecriptUtil is my class for encription and decryption
originalValue = originalValue.replace(encryptedTaggedValue, decryptedValue);
} catch (GeneralSecurityException e) {
logger.error("failed to decrypt property returning original value as in properties file.", e);
}
}
return originalValue;
}
}
And changed properties file to enclose encrypted value in custuom ENC tag
as
mail.imap.task.url=imap://username:{<ENC>}encryptedPassword{</ENC>}#imap.googlemail.com:993/inbox

How to consume a spring data rest service with java?

I have the following spring boot + data Rest repository:
#RepositoryRestResource(collectionResourceRel = "dto", path = "produtos")
public interface ProdutoRepository extends CrudRepository<Produto, Integer> {
#Query("SELECT p FROM Produto p where descricao LIKE CONCAT(UPPER(:like),'%')")
List<Produto> findByLike(#Param("like") String like);
}
I also have a java client that access this method (this is my example of doing it):
String url = "http://localhost:8080/produtos/search/findByLike?like={like}";
RestTemplate t = new RestTemplate();
ProdutoDto resp = t.getForObject(url, ProdutoDto.class, txtLoc.getText());
ProdutoDto (this one is not totally necessary):
public class ProdutoDto extends HalDto<Produto> {}
HalDto:
public class HalDto<T extends ResourceSupport> extends ResourceSupport {
#JsonProperty("_embedded")
private EmbeddedDto<T> embedded;
public EmbeddedDto<T> getEmbedded() {
return embedded;
}
public void setEmbedded(EmbeddedDto<T> embedded) {
this.embedded = embedded;
}
}
EmbeddedDto:
public class EmbeddedDto<T> {
#JsonProperty("dto")
private List<T> dtoList;
public List<T> getDtoList()
{
return dtoList;
}
public void setDto(List<T> dtoList) {
this.dtoList = dtoList;
}
}
Those classes are necessary (i think) because Spring Data returns data in the HAL (https://en.wikipedia.org/wiki/Hypertext_Application_Language) format.
Note: Produto must extend ResourceSupport.
Caveats: All collectionResourceRel must be named "dto" and it only works for collections (may be adjusted).
Is this the proper way to do this?
I have googled around and found plenty of examples of doing the server side, but almost nothing on building clients.
Thanks.
This is a solution that I have found which seems to work well.
First, setup your RestTemplate so that it expects JSON/HAL and knows what to do with it:
#Bean
public RestTemplate restTemplate() {
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.registerModule(new Jackson2HalModule());
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
MappingJackson2HttpMessageConverter messageConverter =
new MappingJackson2HttpMessageConverter();
messageConverter.setObjectMapper(objectMapper);
messageConverter.setSupportedMediaTypes(Arrays.asList(MediaTypes.HAL_JSON, MediaType.APPLICATION_JSON_UTF8));
return new RestTemplate(Arrays.asList(messageConverter));
}
Then you can use the exchange method of the RestTemplate to specify that you want your result to be ResponseEntity<PagedResources<Producto>>
ResponseEntity<PagedResources<Producto>> resultResponse = restTemplate.exchange(uri, HttpMethod.GET, HttpEntity.EMPTY, new ParameterizedTypeReference<PagedResources<Producto>>(){});
if(resultResponse.getStatusCode() == HttpStatus.OK){
Collection<Producto> results = resultResponse.getBody().getContent();
log.info("{} results obtained", results.size());
}
You can instantiate restTemplate by either calling the restTemplate() method defined above or you can inject (autowire) it.

Resources