I have 8 files that I want to upload to an FTP server using sftp in the spring batch. I am not able to configure the Tasklet for that can anyone tell me how to do that. Also, the file name should remain the same as it was in local. I am new to Spring so please help.
#Configuration
public class FTPSonfigurations {
#Bean
public DefaultSftpSessionFactory gimmeFactory(){
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory();
factory.setHost("");
factory.setUser("");
factory.setPassword("");
return factory;
}
#Bean
#ServiceActivator(inputChannel = "uploadfile")
SftpMessageHandler uploadHandler(DefaultSftpSessionFactory factory){
SftpMessageHandler messageHandler = new SftpMessageHandler(factory);
messageHandler.setRemoteDirectoryExpression(new LiteralExpression("/upload/ "));
return messageHandler;
}
}
#MessagingGateway
public interface UploadMessagingGateway {
#Gateway(requestChannel = "uploadfile")
public void uploadFile(File file);
}
public class MyTasklet implements Tasklet {
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
//What to do here???
return null;
}
}
Just auto wire the gateway into the tasklet and call it.
#Autowired
UploadMessagingGateway gw;
...
gw.uploadFile(file);
Related
I have codes for Inbound and Outbound channel adapter over SFTP. I want to call those method via spring boot scheduler not using polling. Looking for example how to call resultFileHandler() method
public class SftpConfig {
#Value("${nodephone.directory.sftp.host}")
private String sftpHost;
#Value("${nodephone.directory.sftp.port}")
private int sftpPort;
#Value("${nodephone.directory.sftp.user}")
private String sftpUser;
#Value("${nodephone.directory.sftp.password}")
private String sftpPasword;
#Value("${nodephone.directory.sftp.remote.directory.download}")
private String sftpRemoteDirectoryDownload;
#Value("${nodephone.directory.sftp.remote.directory.upload}")
private String sftpRemoteDirectoryUpload;
#Value("${nodephone.directory.sftp.remote.directory.filter}")
private String sftpRemoteDirectoryFilter;
#Value("${nodephone.directory.sftp.remote.directory.localDirectory}")
private String sftpLocalDirectory;
// private FtpOrderRequestHandler handler;
#Bean
public SessionFactory<LsEntry> sftpSessionFactory() {
DefaultSftpSessionFactory factory = new DefaultSftpSessionFactory(true);
factory.setHost(sftpHost);
factory.setPort(sftpPort);
factory.setUser(sftpUser);
factory.setPassword(sftpPasword);
factory.setAllowUnknownKeys(true);
return new CachingSessionFactory<LsEntry>(factory);
}
#Bean
public SftpInboundFileSynchronizer sftpInboundFileSynchronizer() {
SftpInboundFileSynchronizer fileSynchronizer = new SftpInboundFileSynchronizer(sftpSessionFactory());
fileSynchronizer.setDeleteRemoteFiles(true);
fileSynchronizer.setRemoteDirectory(sftpRemoteDirectoryDownload);
fileSynchronizer.setFilter(new SftpSimplePatternFileListFilter(sftpRemoteDirectoryFilter));
return fileSynchronizer;
}
#Bean
#InboundChannelAdapter(channel = "fromSftpChannel", poller = #Poller(cron = "0/5 * * * * *"))
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source = new SftpInboundFileSynchronizingMessageSource(
sftpInboundFileSynchronizer());
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new AcceptOnceFileListFilter<File>());
source.setLocalDirectory(new File("/local"));
return source;
}
#Bean
#ServiceActivator(inputChannel = "fromSftpChannel")
public MessageHandler resultFileHandler() {
return new MessageHandler() {
#Override
public void handleMessage(Message<?> message) throws MessagingException {
System.out.println("********************** " + message.getPayload());
}
};
}
I have tested with Configuration annotation and it reads the file from the server, but I want to run this from Cron instead of polling, how do I call the method resultFileHandler()
I've never done this using Spring Integration in any production code although I did something like below, to download files from remote servers using sftp/ftp.
I'm only using the SftpOutboundGateway (there could be better ways), to call the "mget" method and fetch the payload (file).
#Configuration
#ConfigurationProperties(prefix = "sftp")
#Setter
#Getter
#EnableIntegration
public class RemoteFileConfiguration {
private String clients;
private String hosts;
private int ports;
private String users;
private String passwords;
#Bean(name = "clientSessionFactory")
public SessionFactory<LsEntry> clientSessionFactory() {
DefaultSftpSessionFactory sf = new DefaultSftpSessionFactory();
sf.setHost(hosts);
sf.setPort(ports);
sf.setUser(users);
sf.setPassword(passwords);
sf.setAllowUnknownKeys(true);
return new CachingSessionFactory<>(sf);
}
#Bean
#ServiceActivator(inputChannel = "sftpChannel")
public MessageHandler clientMessageHandler() {
SftpOutboundGateway sftpOutboundGateway = new SftpOutboundGateway(
clientSessionFactory(), "mget", "payload");
sftpOutboundGateway.setAutoCreateLocalDirectory(true);
sftpOutboundGateway.setLocalDirectory(new File("/users/localPath/client/INPUT/"));
sftpOutboundGateway.setFileExistsMode(FileExistsMode.REPLACE_IF_MODIFIED);
sftpOutboundGateway.setFilter(new AcceptOnceFileListFilter<>());
return sftpOutboundGateway;
}
}
#MessagingGateway
public interface SFTPGateway {
#Gateway(requestChannel = "sftpChannel")
List<File> get(String dir);
}
To ensure we use cron to execute this, I have used a Tasklet which is executed by Spring Batch, when I need it to be using a cron expression.
#Slf4j
#Getter
#Setter
public class RemoteFileInboundTasklet implements Tasklet {
private RemoteFileTemplate remoteFileTemplate;
private String remoteClientDir;
private String clientName;
private SFTPGateway sftpGateway;
#Override
public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext)
throws Exception {
List<File> files = sftpGateway.get(remoteClientDir);
if (CollectionUtils.isEmpty(files)) {
log.warn("No file was downloaded for client {}.", clientName);
return RepeatStatus.FINISHED;
}
log.info("Total file: {}", files.size());
return RepeatStatus.FINISHED;
}
}
NOTE: If you don't want to use Batch's Tasklet, you can use your #Component class and inject the Gateway to call "get" method.
#Autowired
private SFTPGateway sftpGateway;
I try to test a simple MQTT listener created with Spring Boot and Spring Integration, but don't get it working. I've tried many approaches. The most promising was:
#RunWith(SpringRunner.class)
#SpringBootTest
public class BasicMqttTest {
#Value("${mqtt.client.id}")
private String mqttClientId;
#Value("${mqtt.state.topic}")
private String mqttTopic;
#Autowired
MqttPahoClientFactory mqttPahoClientFactory;
protected IMqttClient client;
#Before
public void setUp() throws Exception {
client = mqttPahoClientFactory.getClientInstance(mqttPahoClientFactory.getConnectionOptions().getServerURIs()[0], mqttClientId);
client.connect();
}
#After
public void tearDown() throws Exception {
client.disconnect();
client.close();
}
#Test
public void contextLoads() throws Exception {
MqttMessage mqttMessage = new MqttMessage();
mqttMessage.setPayload("MQTT!".getBytes());
client.publish(mqttTopic, mqttMessage);
}
}
However, the test runs with
2018-07-12 16:53:50.937 ERROR 21160 --- [T Rec: consumer] .m.i.MqttPahoMessageDrivenChannelAdapter : Lost connection: Verbindung wurde getrennt; retrying...
, and I don't see anything printed out.
The code is mostly from: https://github.com/spring-projects/spring-integration-samples/tree/master/basic/mqtt
. The example works nicely, but I need to be able to write proper integration tests.
The configuration is:
#Value("${mqtt.server.uri}")
private String mqttServerUri;
#Value("${mqtt.username}")
private String mqttUsername;
#Value("${mqtt.password}")
private String mqttPassword;
#Value("${mqtt.client.id}")
private String mqttClientId;
#Value("${mqtt.state.topic}")
private String mqttTopic;
#Value("${mqtt.completion.timeout}")
private Integer mqttCompletionTimeout;
#Value("${mqtt.quality.of.service}")
private Integer mqttQualityOfService;
#Bean
public MqttPahoClientFactory mqttClientFactory() {
DefaultMqttPahoClientFactory factory = new DefaultMqttPahoClientFactory();
MqttConnectOptions options = new MqttConnectOptions();
options.setServerURIs(new String[]{mqttServerUri});
options.setUserName(mqttUsername);
options.setPassword(mqttPassword.toCharArray());
factory.setConnectionOptions(options);
return factory;
}
#Bean
#Qualifier("MqttInboundChannel")
public MessageProducerSupport mqttInbound() {
MqttPahoMessageDrivenChannelAdapter adapter = new MqttPahoMessageDrivenChannelAdapter(
mqttClientId,
mqttClientFactory(),
mqttTopic
);
adapter.setCompletionTimeout(mqttCompletionTimeout);
adapter.setConverter(new DefaultPahoMessageConverter());
adapter.setQos(mqttQualityOfService);
return adapter;
}
#Bean
public IntegrationFlow myMqttInFlow() {
return IntegrationFlows.from(mqttInbound)
.handle(message -> {
System.out.println(message);
}).get();
}
UPDATE:
Didn't work either:
#Autowired
protected MessageHandler mqttOutbound;
#Test
public void test0() throws Exception {
mqttOutbound.handleMessage(MessageBuilder.withPayload("test").build());
}
#SpringBootApplication
static class MqttSourceApplication {
#Autowired
private MqttPahoClientFactory mqttClientFactory;
#Bean
public MessageHandler mqttOutbound() {
MqttPahoMessageHandler messageHandler = new MqttPahoMessageHandler("test", mqttClientFactory);
messageHandler.setAsync(true);
messageHandler.setDefaultTopic("test");
messageHandler.setConverter(pahoMessageConverter());
return messageHandler;
}
#Bean
public DefaultPahoMessageConverter pahoMessageConverter() {
DefaultPahoMessageConverter converter = new DefaultPahoMessageConverter(1, false, "UTF-8");
return converter;
}
}
UPDATE
Even simpler ... same error:
#Autowired
MqttPahoClientFactory mqttPahoClientFactory;
private MessageHandler mqttOutbound;
#Before
public void setUp() throws Exception {
MqttPahoMessageHandler messageHandler = new MqttPahoMessageHandler(mqttClientId, mqttPahoClientFactory);
messageHandler.setAsync(false);
messageHandler.setDefaultTopic(mqttTopic);
messageHandler.setConverter(new DefaultPahoMessageConverter());
mqttOutbound = messageHandler;
}
#Test
public void test0() throws Exception {
mqttOutbound.handleMessage(MessageBuilder.withPayload("test").build());
Thread.sleep(10000L);
}
Ok, resolved: Paho apparently closed my connection since both the test and main used the same client id. The solution was to replace the clientId with MqttAsyncClient.generateClientId() as suggested here: https://stackoverflow.com/a/48232793/2739681
I have a Spring Boot application which listens to messages on a Kafka queue. To filter those messages, have the following two classs
#Component
public class Listener implements MessageListener {
private final CountDownLatch latch1 = new CountDownLatch(1);
#Override
#KafkaListener(topics = "${spring.kafka.topic.boot}")
public void onMessage(Object o) {
System.out.println("LISTENER received payload *****");
this.latch1.countDown();
}
}
#Configuration
#EnableKafka
public class KafkaConfig {
#Autowired
private Listener listener;
#Bean
public FilteringMessageListenerAdapter filteringReceiver() {
return new FilteringMessageListenerAdapter(listener, recordFilterStrategy() );
}
public RecordFilterStrategy recordFilterStrategy() {
return new RecordFilterStrategy() {
#Override
public boolean filter(ConsumerRecord consumerRecord) {
System.out.println("IN FILTER");
return false;
}
};
}
}
While messages are being processed by the Listener class, the RecordFilterStrategy implementation is not being invoked. What is the correct way to use FilteringMessageListenerAdapter?
Thanks
The solution was as follows:
No need for the FilteringMessageListenerAdapter class.
Rather, create a ConcurrentKafkaListenerContainerFactory, rather than relying on what Spring Boot provides out of the box. Then, set the RecordFilterStrategy implementation on this class.
#Bean
ConcurrentKafkaListenerContainerFactory<Integer, String>
kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setRecordFilterStrategy(recordFilterStrategy());
return factory;
}
I'm using Spring Batch with a AsyncItemProcessor and things are behaving unexpectedly. Let me show first the code:
Followed a simple example as shown on the Spring Batch project:
#EnableBatchProcessing
#SpringBootApplication
#Import({HttpClientConfigurer.class, BatchJobConfigurer.class})
public class PerfilEletricoApp {
public static void main(String[] args) throws Exception {// NOSONAR
System.exit(SpringApplication.exit(SpringApplication.run(PerfilEletricoApp.class, args)));
//SpringApplication.run(PerfilEletricoApp.class, args);
}
}
-- EDIT
If I just sleep the main process go give a few seconds to slf4j to write the flush the logs, everything works as expected.
#EnableBatchProcessing
#SpringBootApplication
#Import({HttpClientConfigurer.class, BatchJobConfigurer.class})
public class PerfilEletricoApp {
public static void main(String[] args) throws Exception {// NOSONAR
//System.exit(SpringApplication.exit(SpringApplication.run(PerfilEletricoApp.class, args)));
ConfigurableApplicationContext context = SpringApplication.run(PerfilEletricoApp.class, args);
Thread.sleep(1000 * 5);
System.exit(SpringApplication.exit(context));
}
}
-- ENDOF EDIT
I'm reading a text file with a field and then using a AsyncItemProcessor to get a multithreaded processing, which consists of a Http GET on a URL to fetch some data, I'm also using a NoOpWriter to do nothing on the write part. I'm saving the results of the GET on the Processor part of the job (using log.trace / log.warn).
#Configuration
public class HttpClientConfigurer {
// [... property and configs omitted]
#Bean
public CloseableHttpClient createHttpClient() {
// ... creates and returns a poolable http client etc
}
}
As for the Job:
#Configuration
public class BatchJobConfigurer {
#Autowired
private JobBuilderFactory jobs;
#Autowired
private StepBuilderFactory steps;
#Value("${async.tps:10}")
private Integer tps;
#Value("${com.bemobi.perfilelerico.sourcedir:/AppServer/perfil-eletrico/source-dir/}")
private String sourceDir;
#Bean
public ItemReader<String> reader() {
MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
reader.setResources( new Resource[] { new FileSystemResource(sourceDir)});
reader.setDelegate((ResourceAwareItemReaderItemStream<? extends String>) flatItemReader());
return reader;
}
#Bean
public ItemReader<String> flatItemReader() {
FlatFileItemReader<String> itemReader = new FlatFileItemReader<>();
itemReader.setLineMapper(new DefaultLineMapper<String>() {{
setLineTokenizer(new DelimitedLineTokenizer() {{
setNames(new String[] { "sample-field-001"});
}});
setFieldSetMapper(new SimpleStringFieldSetMapper<>());
}});
return itemReader;
}
#Bean
public ItemProcessor asyncItemProcessor(){
AsyncItemProcessor<String, OiPaggoResponse> asyncItemProcessor = new AsyncItemProcessor<>();
asyncItemProcessor.setDelegate(processor());
asyncItemProcessor.setTaskExecutor(getAsyncExecutor());
return asyncItemProcessor;
}
#Bean
public ItemProcessor<String,OiPaggoResponse> processor(){
return new PerfilEletricoItemProcessor();
}
/**
* Using a NoOpItemWriter<T> so we satisfy spring batch flow but don't use writer for anything else.
* #return a NoOpItemWriter<OiPaggoResponse>
*/
#Bean
public ItemWriter<OiPaggoResponse> writer() {
return new NoOpItemWriter<>();
}
#Bean
protected Step step1() throws Exception {
/*
Problem starts here, If Use the processor() everything ends nicely, but if I insist on the asyncItemProcessor(), the job ends and the logs from processor are not stored on the disk.
*/
return this.steps.get("step1").<String, OiPaggoResponse> chunk(10)
.reader(reader())
.processor(asyncItemProcessor())
.build();
}
#Bean
public Job job() throws Exception {
return this.jobs.get("consulta-perfil-eletrico").start(step1()).build();
}
#Bean(name = "asyncExecutor")
public TaskExecutor getAsyncExecutor()
{
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(tps);
executor.setMaxPoolSize(tps);
executor.setQueueCapacity(tps * 1000);
executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
executor.setThreadNamePrefix("AsyncExecutor-");
return executor;
}
}
-- UPDATED WITH AsyncItemWriter (Working version)
/*Wrapped Writer*/
#Bean
public ItemWriter asyncItemWriter(){
AsyncItemWriter<OiPaggoResponse> asyncItemWriter = new AsyncItemWriter<>();
asyncItemWriter.setDelegate(writer());
return asyncItemWriter;
}
/*AsyncItemWriter defined on the steps*/
#Bean
protected Step step1() throws Exception {
return this.steps.get("step1").<String, OiPaggoResponse> chunk(10)
.reader(reader())
.processor(asyncItemProcessor())
.writer(asyncItemWriter())
.build();
}
--
Any thoughts on why the AsyncItemProcessor don't wait for all the children to to complete before send a OK-Completed signal to the context?
The issue is that the AsyncItemProcessor is creating Futures that no one is waiting for. Wrap your NoOpItemWriter in the AsyncItemWriter so that someone is waiting for the Futures. That will cause the job to complete as expected.
I'm currently trying to create a batch with the spring annotations but the batch is never called. No error occurs, my batch isn't called. Its a simple batch that retrieves values from the database and add messages in a queue (rabbitmq).
The main configuration class:
#Configuration
#EnableBatchProcessing
public class BatchInfrastructureConfiguration {
#Bean
public JobLauncher getJobLauncher() throws Exception {
SimpleJobLauncher jobLauncher = new SimpleJobLauncher();
jobLauncher.setJobRepository(getJobRepository());
jobLauncher.afterPropertiesSet();
return jobLauncher;
}
public JobRepository getJobRepository() throws Exception {
MapJobRepositoryFactoryBean factory = new MapJobRepositoryFactoryBean();
factory.setTransactionManager(new ResourcelessTransactionManager());
factory.afterPropertiesSet();
return (JobRepository) factory.getObject();
}
}
The configuration class specific to my batch
#Configuration
#Import(BatchInfrastructureConfiguration.class)
public class PurchaseStatusBatchConfiguration {
#Inject
private JobBuilderFactory jobBuilders;
#Inject
private StepBuilderFactory stepBuilders;
#Bean
public Job purchaseStatusJob(){
return jobBuilders.get("purchaseStatusJob")
.start(step())
.build();
}
#Bean
public Step step(){
return stepBuilders.get("purchaseStatusStep")
.tasklet(new PurchaseStatusBatch())
.build();
}
}
The batch class:
public class PurchaseStatusBatch implements Tasklet {
#Inject
private PurchaseRepository purchaseRepository;
#Inject
#Qualifier(ApplicationConst.BEAN_QUALIFIER_PURCHASE_QUEUE)
private RabbitTemplate rabbitTemplate;
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
PurchaseDto purchaseDto;
PurchaseMessage purchaseMessage;
List<Purchase> notVerifiedPurchase = purchaseRepository.findByVerified(false);
for (Purchase purchase : notVerifiedPurchase) {
purchaseDto = new PurchaseDto();
purchaseDto.setOrderId(purchase.getOrderId());
purchaseDto.setProductId(purchase.getProductId());
purchaseDto.setPurchaseToken(purchase.getPurchaseToken());
purchaseDto.setUserScrapbookKey(purchase.getUserScrapbookKey());
purchaseMessage = new PurchaseMessage();
purchaseMessage.setPurchaseDto(purchaseDto);
rabbitTemplate.convertAndSend(purchaseMessage);
}
return null;
}
}
The job runner (class calling the batch):
#Service
public class PurchaseStatusJobRunner {
#Inject
private JobLocator jobLocator;
#Inject
private JobLauncher jobLauncher;
//#Scheduled(fixedDelay = 3000L)
//#Scheduled(cron="* * * * *") // every 1 minute
#Scheduled(fixedDelay = 3000L)
public void runJob() throws JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException, JobParametersInvalidException, NoSuchJobException {
jobLauncher.run(jobLocator.getJob("purchaseStatusJob"), new JobParameters());
}
}
Short answer: The problem is that your PurchaseStatusBatch instance is not a spring bean... then all its attributes are null (they have not been injected)
Try this :
#Bean
public Step step(){
return stepBuilders.get("purchaseStatusStep")
.tasklet(purchaseStatusBatch())
.build();
}
#Bean
public PurchaseStatusBatch purchaseStatusBatch() {
return new PurchaseStatusBatch()
}
When you want to have feedback on your job execution, just use the JobExecution instance returned by the JobLauncher. You can get the ExitStatus, you can get all exceptions catched by the JobLauncher, and more information.
Another solution to have feedback would be providing a real database to your JobRepository, and then check execution statuses in it.