Create multiple beans of SftpInboundFileSynchronizingMessageSource dynamically with InboundChannelAdapter - spring
I am using spring inbound channel adapter to poll files from sftp server. Application needs to poll from multiple directories from single sftp server. Since Inbound channel adapter does not allow to poll multiple directories I tried creating multiple beans of same type with different values. Since number of directories can increase in future, I want to control it from application properties and want to register beans dynamically.
My code -
#Override
public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
beanFactory.registerSingleton("sftpSessionFactory", sftpSessionFactory(host, port, user, password));
beanFactory.registerSingleton("sftpInboundFileSynchronizer",
sftpInboundFileSynchronizer((SessionFactory) beanFactory.getBean("sftpSessionFactory")));
}
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory(String host, String port, String user, String password) {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(host);
factory.setPort(Integer.parseInt(port));
factory.setUser(user);
factory.setPassword(password);
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
private SftpInboundFileSynchronizer sftpInboundFileSynchronizer(SessionFactory sessionFactory) {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sessionFactory);
fileSynchronizer.setDeleteRemoteFiles(true);
fileSynchronizer.setPreserveTimestamp(true);
fileSynchronizer.setRemoteDirectory("/mydir/subdir);
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter("*.pdf"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "sftpChannel", poller = #Poller(fixedDelay = "2000"))
public MessageSource<File> sftpMessageSource(String s) {
SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource(
(AbstractInboundFileSynchronizer<ChannelSftp.LsEntry>) applicationContext.getBean("sftpInboundFileSynchronizer"));
source.setLocalDirectory(new File("/dir/subdir"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<>());
source.setMaxFetchSize(Integer.parseInt(maxFetchSize));
source.setAutoCreateLocalDirectory(true);
return source;
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler handler() {
return message -> {
LOGGER.info("Payload - {}", message.getPayload());
};
}
This code works fine. But If I create sftpMessageSource dynamically, then #InboundChannelAdapter annotation won't work. Please suggest a way to dynamically create sftpMessageSource and handler beans also and add respective annotations.
Update:
Following Code Worked :
#PostConstruct
void init() {
int index = 0;
for (String directory : directories) {
index++;
int finalI = index;
IntegrationFlow flow = IntegrationFlows
.from(Sftp.inboundAdapter(sftpSessionFactory())
.preserveTimestamp(true)
.remoteDirectory(directory)
.autoCreateLocalDirectory(true)
.localDirectory(new File("/" + directory))
.localFilter(new AcceptOnceFileListFilter<>())
.maxFetchSize(10)
.filter(new SftpSimplePatternFileListFilter("*.pdf"))
.deleteRemoteFiles(true),
e -> e.id("sftpInboundAdapter" + finalI)
.autoStartup(true)
.poller(Pollers.fixedDelay(2000)))
.handle(handler())
.get();
this.flowContext.registration(flow).register();
}
}
#Bean
public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(host);
factory.setPort(Integer.parseInt(port));
factory.setUser(user);
factory.setPassword(password);
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
Annotations in Java are static. You can't add them at runtime for created objects. Plus the framework reads those annotation on application context startup. So, what you are looking for is just not possible with Java as language per se.
You need consider to switch to Java DSL in Spring Integration to be able to use its "dynamic flows": https://docs.spring.io/spring-integration/docs/5.3.1.RELEASE/reference/html/dsl.html#java-dsl-runtime-flows.
But, please, first of all study more what Java can do and what cannot.
Related
how to Spring integration SftpOutboundGateway setFilter Custom Argument?
How can I pass a parameter as a filter condition when getting the file list of the SFTP server from MessagingGateway? My SftpMessageGateway code #MessagingGateway public interface SftpMessageGateway { #Gateway(requestChannel = "getSftpChannel") List<SftpFileInfo> getIconListByProductUiId(#Payloads("productUiId") String productUiId); Integration Config #Bean public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() { DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true); factory.setHost(host); factory.setPort(port); factory.setUser(id); factory.setPassword(password); factory.setAllowUnknownKeys(true); return new CachingSessionFactory<>(factory); } #Bean #ServiceActivator(inputChannel = "getSftpChannel") public MessageHandler getMessageHandler() { SftpOutboundGateway outboundGateway = new SftpOutboundGateway(sftpSessionFactory(), "ls", "'" + uploadPath + "'"); outboundGateway.setOption(AbstractRemoteFileOutboundGateway.Option.NAME_ONLY); outboundGateway.setFilter(new SftpSimplePatternFileListFilter("*alpha*")); outboundGateway.setFilter(new SftpSimplePatternFileListFilter("I want get custom argument)); <---- return outboundGateway; }
You can set only one filter into a gateway, however there is a CompositeFileListFilter where you can combine a set of filters, include any custom impl of the FileListFilter. See more info in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/file.html#remote-persistent-flf
You can refer following code snippet for implementing FileListFilter. My use case was to fetch most latest file uploaded in SFTP directory. #Component public class LastModifiedFileFilter implements FileListFilter<LsEntry> { #Override public List<LsEntry> filterFiles(LsEntry[] files) { List<LsEntry> result = new ArrayList<LsEntry>(); Vector<LsEntry> list = new Vector<LsEntry>(); Collections.addAll(list, files); ChannelSftp.LsEntry lastModifiedEntry = Collections.max(list, (Comparator.comparingInt(entry -> entry.getAttrs().getMTime()))); result.add(lastModifiedEntry); return result; } } Once you have your own custom filter in place then you need to 'Chain' it with your other filters in SftpOutboundGateway object. For your reference, I did it this way ChainFileListFilter<LsEntry> filterList = new ChainFileListFilter<LsEntry>(); filterList.addFilter(new SftpSimplePatternFileListFilter("*alpha*")); filterList.addFilter(new LastModifiedFileFilter()); setFilter(filterList); For me, it will now fetch latest file having "alpha" string present in its name. Hope this helps.
Accessing file from SFTP without downloading it to local using Spring Integration
I currently have the following configuration: #Bean public SessionFactory<ChannelSftp.LsEntry> sftpSessionFactory() { DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true); factory.setHost(sftpHost); factory.setPort(sftpPort); factory.setUser(sftpUser); if (null != sftpPrivateKey) { factory.setPrivateKey(sftpPrivateKey); factory.setPrivateKeyPassphrase(sftpPrivateKeyPassphrase); } else { factory.setPassword(sftpPassword); } factory.setAllowUnknownKeys(true); return new CachingSessionFactory<>(factory); } #Bean public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() { SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory()); // fileSynchronizer.setDeleteRemoteFiles(true); fileSynchronizer.setRemoteDirectory(sftpRemoteDirectory); fileSynchronizer .setFilter(new SftpSimplePatternFileListFilter(sftpRemoteDirectoryFilter)); return fileSynchronizer; } #Bean #InboundChannelAdapter(channel = "fromSftpChannel", poller = #Poller(cron = "0/5 * * * * *")) public MessageSource<File> sftpMessageSource() { SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource( sftpInboundFileSynchronizer()); source.setLocalDirectory(new File(sftpLocalDirectory)); source.setAutoCreateLocalDirectory(true); source.setLocalFilter(new AcceptOnceFileListFilter<>()); return source; } #Bean #ServiceActivator(inputChannel = "fromSftpChannel") public MessageHandler resultFileHandler() { return message -> System.err.println(message.getPayload()); } This one downloads anything from the remote directory to a local directory. But I have a rest controller and I would like to stream back a byte array of the file from the SFTP server instead of downloading it to a local machine. Is it possible in Spring Integration/Boot? Do you have some code examples, please?
Since you say that you have a REST controller to make requests for SFTP files, then I would recommend to look into an SftpOutboundGateway, which indeed designed for requests and replies. See its Command.GET and Option.STREAM: /** * (-stream) Streaming 'get' (returns InputStream); user must call {#link Session#close()}. */ STREAM("-stream"), See more in docs: https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#using-the-get-command Not sure what led you to SftpInboundFileSynchronizingMessageSource for your request-reply task...
You can use the FtpClient and call retrieveFileStream to read a file from the remote sftp server. See https://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/ftp/FTPClient.html#retrieveFileStream-java.lang.String-
I think you can achieve this by using RemoteFileTemplate
Spring Integration - Dynamic MailReceiver configuration
I'm pretty new to spring-integration anyway I'm using it in order to receive mails and elaborate them. I used this spring configuration class: #Configuration #EnableIntegration #PropertySource(value = { "classpath:configuration.properties" }, encoding = "UTF-8", ignoreResourceNotFound = false) public class MailReceiverConfiguration { private static final Log logger = LogFactory.getLog(MailReceiverConfiguration.class); #Autowired private EmailTransformerService emailTransformerService; // Configurazione AE #Bean public MessageChannel inboundChannelAE() { return new DirectChannel(); } #Bean(name= {"aeProps"}) public Properties aeProps() { Properties javaMailPropertiesAE = new Properties(); javaMailPropertiesAE.put("mail.store.protocol", "imap"); javaMailPropertiesAE.put("mail.debug", Boolean.TRUE); javaMailPropertiesAE.put("mail.auth.debug", Boolean.TRUE); javaMailPropertiesAE.put("mail.smtp.socketFactory.fallback", "false"); javaMailPropertiesAE.put("mail.imap.socketFactory.class", "javax.net.ssl.SSLSocketFactory"); return javaMailPropertiesAE; } #Bean(name="mailReceiverAE") public MailReceiver mailReceiverAE(#Autowired MailConfigurationBean mcb, #Autowired #Qualifier("aeProps") Properties javaMailPropertiesAE) throws Exception { return ConfigurationUtil.getMailReceiver("imap://USERNAME:PASSWORD#MAILSERVER:PORT/INBOX", new BigDecimal(2), javaMailPropertiesAE); } #Bean #InboundChannelAdapter( autoStartup = "true", channel = "inboundChannelAE", poller = {#Poller(fixedRate = "${fixed.rate.ae}", maxMessagesPerPoll = "${max.messages.per.poll.ae}") }) public MailReceivingMessageSource pollForEmailAE(#Autowired MailReceiver mailReceiverAE) { MailReceivingMessageSource mrms = new MailReceivingMessageSource(mailReceiverAE); return mrms; } #Transformer(inputChannel = "inboundChannelAE", outputChannel = "transformerChannelAE") public MessageBean transformitAE( MimeMessage mailMessage ) throws Exception { // amministratore email inbox MessageBean messageBean = emailTransformerService.transformit(mailMessage); return messageBean; } #Splitter(inputChannel = "transformerChannelAE", outputChannel = "nullChannel") public List<Message<?>> splitIntoMessagesAE(final MessageBean mb) { final List<Message<?>> messages = new ArrayList<Message<?>>(); for (EmailFragment emailFragment : mb.getEmailFragments()) { Message<?> message = MessageBuilder.withPayload(emailFragment.getData()) .setHeader(FileHeaders.FILENAME, emailFragment.getFilename()) .setHeader("directory", emailFragment.getDirectory()).build(); messages.add(message); } return messages; } } So far so good.... I start my micro-service and there is this component listening on the specified mail server and mails are downloaded. Now I have this requirement: mail server configuration (I mean the string "imap://USERNAME:PASSWORD#MAILSERVER:PORT/INBOX") must be taken from a database and it can be configurable. In any time a system administrator can change it and the mail receiver must use the new configuration. As far as I understood I should create a new instance of MailReceiver when a new configuration is present and use it in the InboundChannelAdapter Is there any best practice in order to do it? I found this solution: ImapMailReceiver NO STORE attempt on READ-ONLY folder (Failure) [THROTTLED]; In this solution I can inject the ThreadPoolTaskScheduler if I define it in my Configuration class; I can also inject the DirectChannel but every-time I should create a new MailReceiver and a new ImapIdleChannelAdapter without considering this WARN message I get when the ImapIdleChannelAdapter starts: java.lang.RuntimeException: No beanfactory at org.springframework.integration.expression.ExpressionUtils.createStandardEvaluationContext(ExpressionUtils.java:79) at org.springframework.integration.mail.AbstractMailReceiver.onInit(AbstractMailReceiver.java:403) Is there a better way to satisfy my scenario? Thank you Angelo
The best way to do this is to use the Java DSL and dynamic flow registration. Documentation here. That way, you can unregister the old flow and register a new one, each time the configuration changes. It will automatically handle injecting dependencies such as the bean factory.
Spring Integration: how to access the returned values from last Subscriber
I'm trying to implement a SFTP File Upload of 2 Files which has to happen in a certain order - first a pdf file and after successfull upload of that an text file with meta information about the pdf. I followed the advice in this thread, but can't get it to work properly. My Spring Boot Configuration: #Bean public SessionFactory<LsEntry> sftpSessionFactory() { final DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true); final Properties jschProps = new Properties(); jschProps.put("StrictHostKeyChecking", "no"); jschProps.put("PreferredAuthentications", "publickey,password"); factory.setSessionConfig(jschProps); factory.setHost(sftpHost); factory.setPort(sftpPort); factory.setUser(sftpUser); if (sftpPrivateKey != null) { factory.setPrivateKey(sftpPrivateKey); factory.setPrivateKeyPassphrase(sftpPrivateKeyPassphrase); } else { factory.setPassword(sftpPasword); } factory.setAllowUnknownKeys(true); return new CachingSessionFactory<>(factory); } #Bean #BridgeTo public MessageChannel toSftpChannel() { return new PublishSubscribeChannel(); } #Bean #ServiceActivator(inputChannel = "toSftpChannel") #Order(0) public MessageHandler handler() { final SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory()); handler.setRemoteDirectoryExpression(new LiteralExpression(sftpRemoteDirectory)); handler.setFileNameGenerator(message -> { if (message.getPayload() instanceof byte[]) { return (String) message.getHeaders().get("filename"); } else { throw new IllegalArgumentException("File expected as payload."); } }); return handler; } #ServiceActivator(inputChannel = "toSftpChannel") #Order(1) public String transferComplete(#Payload byte[] file, #Header("filename") String filename) { return "The SFTP transfer complete for file: " + filename; } #MessagingGateway public interface UploadGateway { #Gateway(requestChannel = "toSftpChannel") String upload(#Payload byte[] file, #Header("filename") String filename); } My Test Case: final String pdfStatus = uploadGateway.upload(content, documentName); log.info("Upload of {} completed, {}.", documentName, pdfStatus); From the return of the Gateway upload call i expect to get the String confirming the upload e.g. "The SFTP transfer complete for file:..." but I get the the returned content of the uploaded File in byte[]: Upload of 123456789.1.pdf completed, 37,80,68,70,45,49,46,54,13,37,-30,-29,-49,-45,13,10,50,55,53,32,48,32,111,98,106,13,60,60,47,76,105,110,101,97,114,105,122,101,100,32,49,47,76,32,50,53,52,55,49,48,47,79,32,50,55,55,47,69,32,49,49,49,55,55,55,47,78,32,49,47,84,32,50,53,52,51,53,57,47,72,32,91,32,49,49,57,55,32,53,51,55,93,62,62,13,101,110,100,111,98,106,13,32,32,32,32,32,32,32,32,32,32,32,32,13,10,52,55,49,32,48,32,111,98,106,13,60,60,47,68,101,99,111,100,101,80,97,114,109,115,60,60,47,67,111,108,117,109,110,115,32,53,47,80,114,101,100,105,99,116,111,114,32,49,50,62,62,47,70,105,108,116,101,114,47,70,108,97,116,101,68,101,99,111,100,101,47,73,68,91,60,57,66,53,49,56,54,69,70,53,66,56,66,49,50,52,49,65,56,50,49,55,50,54,56,65,65,54,52,65,57,70,54,62,60,68,52,50,68,51,55,54,53,54,65,67,48,55,54,52,65,65,53,52,66,52,57,51,50,56,52,56,68,66 etc. What am I missing?
I think #Order(0) doesn't work together with the #Bean. To fix it you should do this in that bean definition istead: final SftpMessageHandler handler = new SftpMessageHandler(sftpSessionFactory()); handler.setOrder(0); See Reference Manual for more info: When using these annotations on consumer #Bean definitions, if the bean definition returns an appropriate MessageHandler (depending on the annotation type), attributes such as outputChannel, requiresReply etc, must be set on the MessageHandler #Bean definition itself. In other words: if you can use setter, you have to. We don't process annotations for this case because there is no guarantee what should get a precedence. So, to avoid such a confuse we have left for you only setters choice. UPDATE I see your problem and it is here: #Bean #BridgeTo public MessageChannel toSftpChannel() { return new PublishSubscribeChannel(); } That is confirmed by the logs: Adding {bridge:dmsSftpConfig.toSftpChannel.bridgeTo} as a subscriber to the 'toSftpChannel' channel Channel 'org.springframework.context.support.GenericApplicationContext#b3d0f7.toSftpChannel' has 3 subscriber(s). started dmsSftpConfig.toSftpChannel.bridgeTo So, you really have one more subscriber to that toSftpChannel and it is a BridgeHandler with an output to the replyChannel header. And a default order is like private volatile int order = Ordered.LOWEST_PRECEDENCE; this one becomes as a first subscriber and exactly this one returns you that byte[] just because it is a payload of request. You need to decide if you really need such a bridge. There is no workaround for the #Order though...
Spring-Boot MQTT Configuration
I have a requirement to send payload to a lot of devices whose names are picked from Database. Then, i have to send to different topics, which will be like settings/{put devicename here}. Below is the configuration i was using which i got from spring-boot reference documents. MQTTConfiguration.java #Configuration #IntegrationComponentScan public class MQTTConfiguration { #Autowired private Settings settings; #Autowired private DevMqttMessageListener messageListener; #Bean MqttPahoClientFactory mqttClientFactory() { DefaultMqttPahoClientFactory clientFactory = new DefaultMqttPahoClientFactory(); clientFactory.setServerURIs(settings.getMqttBrokerUrl()); clientFactory.setUserName(settings.getMqttBrokerUser()); clientFactory.setPassword(settings.getMqttBrokerPassword()); return clientFactory; } #Bean MessageChannel mqttOutboundChannel() { return new DirectChannel(); } #Bean #ServiceActivator(inputChannel = "mqttOutboundChannel") public MessageHandler mqttOutbound() { MqttPahoMessageHandler messageHandler = new MqttPahoMessageHandler("dev-client-outbound", mqttClientFactory()); messageHandler.setAsync(true); messageHandler.setDefaultTopic(settings.getMqttPublishTopic()); return messageHandler; } #MessagingGateway(defaultRequestChannel = "mqttOutboundChannel") public interface DeviceGateway { void sendToMqtt(String payload); } } Here, i am sending to only 1 topic. So i added the bean like below to send to multiple number of topics; #Bean public MqttClient mqttClient() throws MqttException { MqttClient mqttClient = new MqttClient(settings.getMqttBrokerUrl(), "dev-client-outbound"); MqttConnectOptions connOptions = new MqttConnectOptions(); connOptions.setUserName(settings.getMqttBrokerUser()); connOptions.setPassword(settings.getMqttBrokerPassword().toCharArray()); mqttClient.connect(connOptions); return mqttClient; } and i send using, try { mqttClient.publish(settings.getMqttPublishTopic()+device.getName(), mqttMessage); } catch (MqttException e) { LOGGER.error("Error While Sending Mqtt Messages", e); } Which works. But my question is, Can i achieve the same, using output channel for better performance? If yes, any help is greatly appreciated. Thank You.
MqttClient is synchronous. The MqttPahoMessageHandler uses an MqttAsyncClient and can be configured (set async to true) to not wait for the confirmation, but publish the confirmation later as an application event. If you are using your own code and sending multiple messages in a loop, it will probably be faster to use an async client, and wait for the IMqttDeliveryToken completions later.