I want to create a flow using DSL. The flow is from the adapter, message will flow to channel.
#Bean
public IntegrationFlow sftpInboundFlow() {
prepareSftpServer();
return IntegrationFlows
.from(Sftp.inboundAdapter(this.sftpSessionFactory).getId("SftpInboundAdapter")
.preserveTimestamp(true)
.remoteDirectory("sftpSource")
.regexFilter(".*\\.txt$")
.localFilenameExpression("#this.toUpperCase() + '.a'").localDirectory(file).channel(MessageChannels.queue("sftpInboundResultChannel"))
.get());
}
Not sure of compilation error at getId() method . tried to convert from Java 8 lambda to Java 7
I think you want to add an id attribute for your component to register it with that bean name in the application context. You config must look like:
return IntegrationFlows
.from(Sftp.inboundAdapter(this.sftpSessionFactory)
.preserveTimestamp(true)
.remoteDirectory("sftpSource")
.regexFilter(".*\\.txt$")
.localFilenameExpression("#this.toUpperCase() + '.a'")
.localDirectory(file),
new Consumer<SourcePollingChannelAdapterSpec>() {
#Override
public void accept(SourcePollingChannelAdapterSpec e) {
e.id("SftpInboundAdapter");
}
})
.channel(MessageChannels.queue("sftpInboundResultChannel"))
.get();
There is no such a getId(String) method.
Yes I'll fix its JavaDocs eventuelly, but you are facing really compilation error, hence wrong language usage.
Related
I have a Spring SFTP output adapter that I start via "adapter.start()" in my main program. Once started, the adapter transfers and uploads all the files in the specified directory as expected. But I want to stop the adapter after all the files have been transferred. How do I detect if all the files have been transferred so I can issue an adapter.stop()?
#Bean
public IntegrationFlow sftpOutboundFlow() {
return IntegrationFlows.from(Files.inboundAdapter(new File(sftpOutboundDirectory))
.filterExpression("name.endsWith('.pdf') OR name.endsWith('.PDF')")
.preventDuplicates(true),
e -> e.id("sftpOutboundAdapter")
.autoStartup(false)
.poller(Pollers.trigger(new FireOnceTrigger())
.maxMessagesPerPoll(-1)))
.log(LoggingHandler.Level.INFO, "sftp.outbound", m -> m.getPayload())
.log(LoggingHandler.Level.INFO, "sftp.outbound", m -> m.getHeaders())
.handle(Sftp.outboundAdapter(outboundSftpSessionFactory())
.useTemporaryFileName(false)
.remoteDirectory(sftpRemoteDirectory))
.get();
}
#Artem Bilan has already given the answer. But here's kind of a concrete implementation of what he said - for those who are a Spring Integration noob like me:
Define a service to get the PDF files on demand:
#Service
public class MyFileService {
public List<File> getPdfFiles(final String srcDir) {
File[] files = new File(srcDir).listFiles((dir, name) -> name.toLowerCase().endsWith(".pdf"));
return Arrays.asList(files == null ? new File[]{} : files);
}
}
Define a Gateway to start the SFTP upload flow on demand:
#MessagingGateway
public interface SFtpOutboundGateway {
#Gateway(requestChannel = "sftpOutboundFlow.input")
void uploadFiles(List<File> files);
}
Define the Integration Flow to upload the files to the SFTP server via Sftp.outboundGateway:
#Configuration
#EnableIntegration
public class FtpFlowIntegrationConfig {
// could be also bound via #Value
private String sftpRemoteDirectory = "/path/to/remote/dir";
#Bean
public SessionFactory<ChannelSftp.LsEntry> outboundSftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost("localhost");
factory.setPort(22222);
factory.setUser("client1");
factory.setPassword("password123");
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(factory);
}
#Bean
public IntegrationFlow sftpOutboundFlow(RemoteFileTemplate<ChannelSftp.LsEntry> remoteFileTemplate) {
return e -> e
.log(LoggingHandler.Level.INFO, "sftp.outbound", Message::getPayload)
.log(LoggingHandler.Level.INFO, "sftp.outbound", Message::getHeaders)
.handle(
Sftp.outboundGateway(remoteFileTemplate, AbstractRemoteFileOutboundGateway.Command.MPUT, "payload")
);
}
#Bean
public RemoteFileTemplate<ChannelSftp.LsEntry> remoteFileTemplate(SessionFactory<ChannelSftp.LsEntry> outboundSftpSessionFactory) {
RemoteFileTemplate<ChannelSftp.LsEntry> template = new SftpRemoteFileTemplate(outboundSftpSessionFactory);
template.setRemoteDirectoryExpression(new LiteralExpression(sftpRemoteDirectory));
template.setAutoCreateDirectory(true);
template.afterPropertiesSet();
template.setUseTemporaryFileName(false);
return template;
}
}
Wiring up:
public class SpringApp {
public static void main(String[] args) {
final MyFileService fileService = ctx.getBean(MyFileService.class);
final SFtpOutboundGateway sFtpOutboundGateway = ctx.getBean(SFtpOutboundGateway.class);
// trigger the sftp upload flow manually - only once
sFtpOutboundGateway.uploadFiles(fileService.getPdfFiles());
}
}
Import notes:
1.
#Gateway(requestChannel = "sftpOutboundFlow.input")
void uploadFiles(List files);
Here the DirectChannel channel sftpOutboundFlow.input will be used to pass message with the payload (= List<File> files) to the receiver. If this channel is not created yet, the Gateway is going to create it implicitly.
2.
#Bean
public IntegrationFlow sftpOutboundFlow(RemoteFileTemplate<ChannelSftp.LsEntry> remoteFileTemplate) { ... }
Since IntegrationFlow is a Consumer functional interface, we can simplify the flow a little using the IntegrationFlowDefinition. During the bean registration phase, the IntegrationFlowBeanPostProcessor converts this inline (Lambda) IntegrationFlow to a StandardIntegrationFlow and processes its components. An IntegrationFlow definition using a Lambda populates DirectChannel as an inputChannel of the flow and it is registered in the application context as a bean with the name sftpOutboundFlow.input in the sample above (flow bean name + ".input"). That's why we use that name for the SFtpOutboundGateway gateway.
Ref: https://spring.io/blog/2014/11/25/spring-integration-java-dsl-line-by-line-tutorial
3.
#Bean
public RemoteFileTemplate<ChannelSftp.LsEntry> remoteFileTemplate(SessionFactory<ChannelSftp.LsEntry> outboundSftpSessionFactory) {}
see: Remote directory for sftp outbound gateway with DSL
Flowchart:
But I want to stop the adapter after all the files have been transferred.
Logically this is not for what this kind of component has been designed. Since you are not going to have some constantly changing local directory, probably it is better to think about an even driver solution to list files in the directory via some action. Yes, it can be a call from the main, but only once for all the content of the dir and that's all.
And for this reason the Sftp.outboundGateway() with a Command.MPUT is there for you:
https://docs.spring.io/spring-integration/reference/html/sftp.html#using-the-mput-command.
You still can trigger an IntegrationFlow, but it could start from a #MessagingGateway interface to be called from a main with a local directory to list files for uploading:
https://docs.spring.io/spring-integration/reference/html/dsl.html#java-dsl-gateway
I am writing a spring integration DSL flow.
Which will look like below diagram.
As you can see in flow I need to read 1 mil entities from database. I want o avoid reading those in single go.
I want to implement polling which will read N entities in fixed interval and send it for processing.
In the examples I read for polling, The polling is used as the first step of the Flow. in my case I want to implement in in middle of the flow.
Please let me know how do I implement this.
Any help is appreciated.
Thanks in Advance.
If you want to trigger the start of some polled flow using some external stimulus, use a control bus:
#SpringBootApplication
public class So63337649Application {
public static void main(String[] args) {
SpringApplication.run(So63337649Application.class, args);
}
#Bean
IntegrationFlow trigger(ConnectionFactory connectionFactory) {
return IntegrationFlows.from(Amqp.inboundAdapter(connectionFactory, "foo"))
.transform(msg -> "#poller.start()")
.channel("control.input")
.get();
}
#Bean
IntegrationFlow control() {
return f -> f.controlBus();
}
#Bean
IntegrationFlow mainFlow() {
return IntegrationFlows.from(() -> "foo", e -> e
.id("poller")
.autoStartup(false)
.poller(Pollers.fixedDelay(5000)))
.handle(System.out::println)
.get();
}
}
I'm using Spring Cloud Stream with Spring Boot. My application is very simple:
ExampleService.class:
#EnableBinding(Processor1.class)
#Service
public class ExampleService {
#StreamListener(Processor1.INPUT)
#SendTo(Processor1.OUTPUT)
public String dequeue(String message){
System.out.println("New message: " + message);
return message;
}
#SendTo(Processor1.OUTPUT)
public String queue(String message){
return message;
}
}
Procesor1.class:
public interface Processor1 {
String INPUT = "input1";
String OUTPUT = "output1";
#Input(Processor1.INPUT)
SubscribableChannel input1();
#Output(Processor1.OUTPUT)
MessageChannel output1();
}
application.properties:
spring.cloud.stream.bindings.input1.destination=test_input
spring.cloud.stream.bindings.input1.group=test_group
spring.cloud.stream.bindings.input1.binder=binder1
spring.cloud.stream.bindings.output1.destination=test_output
spring.cloud.stream.bindings.output1.binder=binder1
spring.cloud.stream.binders.binder1.type=rabbit
spring.cloud.stream.binders.binder1.environment.spring.rabbitmq.host=localhost
Scenarios:
1) When I push a message in 'test_input.test_group' queue, message is correctly printed and correctly sent to 'test_output' exchange. So ExampleService::dequeue works well.
2) When I invoke ExampleService::queue method (from outside the class, in a test), message is never sent to 'test_output' exchange.
I'm working with Spring Boot 2.0.6.RELEASE and Spring Cloud Stream 2.0.2.RELEASE.
Anybody knows why scenario 2) is not working? Thanks in advance.
What leads you to believe that #SendTo on its own is supported? #SendTo is a secondary annotation used by many projects, not just Spring Cloud Stream; as far as I know, there is nothing that will look for it on its own.
Try Spring Integration's #Publisher annotation instead (with #EnablePublisher).
EDIT
To force proxying with CGLIB instead of a JDK proxy, you can do this...
#Bean
public static BeanFactoryPostProcessor bfpp() {
return bf -> {
bf.getBean(IntegrationContextUtils.PUBLISHER_ANNOTATION_POSTPROCESSOR_NAME,
PublisherAnnotationBeanPostProcessor.class).setProxyTargetClass(true);
};
}
I am currently new to Spring Integration.
Basically trying to poll onto multiple file locations asynchronously with Java Spring integration DSL. I am required to get the file name and perform some operations with filename and push the file to S3 finally, my question is can these tasks of performing operations with file be performed in the task executor or the service activator handler . I am not sure which is the right place.
#Autowired
private AWSFileManager awsFileManager;
#Bean
public IntegrationFlow inboundChannelFlow(#Value("${file.poller.delay}") long delay,
#Value("${file.poller.messages}") int maxMsgsPerPoll,
TaskExecutor taskExecutor, MessageSource<File> fileSource)
{
return IntegrationFlows.from(fileSource,
c -> c.poller(Pollers.fixedDelay(delay)
.taskExecutor(taskExecutor)
.maxMessagesPerPoll(maxMsgsPerPoll)))
.handle("AWSFileManager", "fileUpload")
.channel(ApplicationConfiguration.inboundChannel)
.get();
}
#Bean
TaskExecutor taskExecutor(#Value("${file.poller.thread.pool.size}") int poolSize) {
ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
//Runnable task1 = () -> {this.methodsamp();};
taskExecutor.setCorePoolSize(poolSize);
//taskExecutor.execute(task1);
return taskExecutor;
}
#Async
public void methodsamp()
{
try
{
awsFileManager.fileUpload();
System.out.println("test");
}
catch(Exception ex)
{
}
I have attached the sample code here.
Also is there a way I could retrieve the filename of the files in the channel as I need to pass this as parameter to the fileUpload method.
Please advise.
Your question isn't clear. The TaskExecutor is for the thread context in the flow. The Service Activator (.handle()) is exactly for your business logic method. This one can be performed on a thread from the executor. And you really use them in your IntegrationFlow correctly.
The FileReadingMessageSource produces message with the java.io.File as a payload. So, that is the way to get a file name - just from File.getName()!
I'm new to Spring Framework and, indeed, I'm learning and using Spring Boot. Recently, in the app I'm developing, I made Quartz Scheduler work, and now I want to make Spring Integration work there: FTP connection to a server to write and read files from.
What I want is really simple (as I've been able to do so in a previous java application). I've got two Quartz Jobs scheduled to fired in different times daily: one of them reads a file from a FTP server and another one writes a file to a FTP server.
I'll detail what I've developed so far.
#SpringBootApplication
#ImportResource("classpath:ws-config.xml")
#EnableIntegration
#EnableScheduling
public class MyApp extends SpringBootServletInitializer {
#Autowired
private Configuration configuration;
//...
#Bean
public DefaultFtpsSessionFactory myFtpsSessionFactory(){
DefaultFtpsSessionFactory sess = new DefaultFtpsSessionFactory();
Ftp ftp = configuration.getFtp();
sess.setHost(ftp.getServer());
sess.setPort(ftp.getPort());
sess.setUsername(ftp.getUsername());
sess.setPassword(ftp.getPassword());
return sess;
}
}
The following class I've named it as a FtpGateway, as follows:
#Component
public class FtpGateway {
#Autowired
private DefaultFtpsSessionFactory sess;
public void sendFile(){
// todo
}
public void readFile(){
// todo
}
}
I'm reading this documentation to learn to do so. Spring Integration's FTP seems to be event driven, so I don't know how can I execute either of the sendFile() and readFile() from by Jobs when the trigger is fired at an exact time.
The documentation tells me something about using Inbound Channel Adapter (to read files from a FTP?), Outbound Channel Adapter (to write files to a FTP?) and Outbound Gateway (to do what?):
Spring Integration supports sending and receiving files over FTP/FTPS by providing three client side endpoints: Inbound Channel Adapter, Outbound Channel Adapter, and Outbound Gateway. It also provides convenient namespace-based configuration options for defining these client components.
So, I haven't got it clear as how to follow.
Please, could anybody give me a hint?
Thank you!
EDIT:
Thank you #M. Deinum. First, I'll try a simple task: read a file from the FTP, the poller will run every 5 seconds. This is what I've added:
#Bean
public FtpInboundFileSynchronizer ftpInboundFileSynchronizer() {
FtpInboundFileSynchronizer fileSynchronizer = new FtpInboundFileSynchronizer(myFtpsSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(false);
fileSynchronizer.setPreserveTimestamp(true);
fileSynchronizer.setRemoteDirectory("/Entrada");
fileSynchronizer.setFilter(new FtpSimplePatternFileListFilter("*.csv"));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "ftpChannel", poller = #Poller(fixedDelay = "5000"))
public MessageSource<File> ftpMessageSource() {
FtpInboundFileSynchronizingMessageSource source = new FtpInboundFileSynchronizingMessageSource(inbound);
source.setLocalDirectory(new File(configuracion.getDirFicherosDescargados()));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
return source;
}
#Bean
#ServiceActivator(inputChannel = "ftpChannel")
public MessageHandler handler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
Object payload = message.getPayload();
if(payload instanceof File){
File f = (File) payload;
System.out.println(f.getName());
}else{
System.out.println(message.getPayload());
}
}
};
}
Then, when the app is running, I put a new csv file intro "Entrada" remote folder, but the handler() method isn't run after 5 seconds... I'm doing something wrong?
Please add #Scheduled(fixedDelay = 5000) over your poller method.
You should use SPRING BATCH with tasklet. It is far easier to configure bean, crone time, input source with existing interfaces provided by Spring.
https://www.baeldung.com/introduction-to-spring-batch
Above example is annotation and xml based both, you can use either.
Other benefit Take use of listeners and parallel steps. This framework can be used in Reader - Processor - Writer manner as well.