I have a Spring Batch service containg ItemWriter to write the data to the CSV.
I have used the example give by Spring Batch guide. https://spring.io/guides/gs/batch-processing/
I tried to modify the ItemWriter to create the CSV again.
The Problems which I am facing are -
It is not creating the CSV file if it is not present.
If I made it available before hand it is not writing data to it.
#Bean
public ItemWriter<Person> writer(DataSource dataSource) {
FlatFileitemWriter<Person> csvWriter = new FlatFileItemWriter<Person>();
csvWriter.setResource(new ClassPathResource("csv/new-data.csv"));
csvWriter.setShouldDeleteIfExists(true);
DelimitedLineAggregator<Person> lineAggregator = new DelimitedLineAggregator<Person>();
lineAggregator.setDelimiter(",");
BeanWrapperFieldExtractor<Person> fieldExtractor = new BeanWrapperFieldExtractor<Person>();
String[] names = {"firstName", "lastName"};
fieldExtractor.setNames(names);
lineAggregator.setFieldExtractor(fieldExtractor);
csvWriter.setLineAggregator(lineAggregator);
return csvWriter;
}
I have gone through various links but they show the example with XML based configuration. How to do it completely in JAVA ?
You are using a ClassPathResource to write. I'm not sure, but I don't think you can write to a ClassPathResource. Try using a normal FileSystemResource and try again.
Moreover, how do you inject the writer? are you sure that it really is instantiated as spring bean?
Why do you have DataSource as a parameter since you don't need a datasource to instantiate a FlatFileItemWriter.
Related
I have came across a situation where I need to fetch cron expression from database and then schedule it in Spring boot. I am fetching the data using JPA. Now the problem is in spring boot when I use #Scheduled annotation it does not allow me to use the db value directly as it is taken only constant value. So, what I am planning to do is to dynamically generate properties file and read cron expression from properties file. But here also I am facing one problem.The dynamically generated properties file created in target directory.
So I cant use it the time of program loading.
So can anyone assist me to read the dynamically generated file from the resource folder or how to schedule cron expression fetching from DB in spring boot?
If I placed all the details of corn expression in properties file I can schedule the job.
Latest try with dynamically generate properties file.
#Configuration
public class CronConfiguration {
#Autowired
private JobRepository jobRepository;
#Autowired
private ResourceLoader resourceLoader;
#PostConstruct
protected void initialize() {
updateConfiguration();
}
private void updateConfiguration() {
Properties properties = new Properties();
List<Job> morningJobList=new ArrayList<Job>();
List<String> morningJobCornExp=new ArrayList<String>();
// Map<String,String> map=new HashMap<>();
int num=1;
System.out.println("started");
morningJobList= jobRepository.findByDescriptionContaining("Morning Job");
for(Job job:morningJobList) {
//morningJobURL.add(job.getJobUrl());
morningJobCornExp.add(job.getCronExp());
}
for(String cron:morningJobCornExp ) {
properties.setProperty("cron.expression"+num+"=", cron);
num++;
}
Resource propertiesResource = resourceLoader.getResource("classpath:application1.properties");
try (OutputStream out = new BufferedOutputStream(new FileOutputStream(propertiesResource.getFile()))) {
properties.store(out, null);
} catch (Exception ex) {
// Handle error
ex.printStackTrace();
}
}
}
Still it is not able to write in properties file under resource folder.
Consider using Quartz Scheduler framework. It stores scheduler info in DB. No need to implement own DB communication, it is already provided.
Found this example: https://www.callicoder.com/spring-boot-quartz-scheduler-email-scheduling-example/
In Spring batch I configure a file write as such:
#Bean
public FlatFileItemWriter<MyObject> flatFileItemWriter() throws Exception{
FlatFileItemWriter<MyObject> itemWriter = new FlatFileItemWriter();
// pass through aggregator just calls toString on any item pass in.
itemWriter.setLineAggregator(new PassThroughLineAggregator<>());
String outputPath = File.createTempFile("output", ".out").getAbsolutePath();
System.out.println(">>output path=" + outputPath);
itemWriter.setResource(new FileSystemResource(outputPath));
itemWriter.afterPropertiesSet();
return itemWriter;
}
What happens if MyObject is a complex structure that can vary depending on configuration settings etc and I want to generate different parts of that structure to different files.
How do I do this?
Have you looked at CompositeItemWriter? You may need to have CompositeLineMapper in your reader as well as ClassifierCompositeItemProcessor depending on your needs.
Below is example of a CompositeItemWriter
#Bean
public ItemWriter fileWriter() {
CompositeItemWriter compWriter = new CompositeItemWriter();
FlatFileItemWriter<MyObject_data> dataWriter = new FlatFileItemWriter<MyObject_data>();
FlatFileItemWriter<MyObject_otherdata> otherWriter = new FlatFileItemWriter<MyObject_otherdata>();
List<ItemWriter> iList = new ArrayList<ItemWriter>();
iList.add(dataWriter);
iList.add(otherWriter);
compWriter.setDelegates(iList);
return compWriter;
}
I am new to Spring. I have a use case where I need to execute same multiple sql queries and return same POJO for every query. I would like to write one Item reader and change the query in each step. Is there a way to do this?
You can use spring batch late binding by adding #StepScope in your reader
Sample code
#StepScope
#Bean
public ItemReader<Pojo> myReader() {
JdbcCursorItemReader<Pojo> reader = new JdbcCursorItemReader<>();
reader.setDataSource(basicDataSource);
//You can inject sql as per you need
//Some expamles
//using #{jobParameters['']}
//using {jobExecutionContext['input.file.name']}"
//using #{stepExecutionContext['input.file.name']}"
reader.setSql("Your-SQL");
reader.setRowMapper(new MyMapper());
return reader;
}
check section 5.4
https://docs.spring.io/spring-batch/reference/html/configureStep.html
How to poll multiple files from a particular file directory using Spring Integration Api without xml configuration preferably using Java annotations based approach?
I want to get the polled files list and iterate through them and process further. This is the requirement. Any Sample
Code available to meet this requirement. Thanks in Advance. Below is the code fragment I used.
#Bean
#InboundChannelAdapter(value = "fileInputChannel", poller = #Poller(fixedDelay = "60000",maxMessagesPerPoll="5"))
public MessageSource<File> fileReadingMessageSource() {
txtSource = new FileReadingMessageSource();
txtSource.setDirectory(inputDir);
txtSource.setFilter(new SimplePatternFileListFilter("*.txt"));
txtSource.setScanEachPoll(true);
return txtSource;
}
#Bean
#Transformer(inputChannel = "fileInputChannel", outputChannel = "processFileChannel")
public FileToStringTransformer fileToStringTransformer() {
Message<File> message1 = txtSource.receive();
File file1 = message1.getPayload();
return new FileToStringTransformer();
}
But irrespective of no of files in the input dir the message source instance always fetch only one file. Not sure how to make it work for multiple files to be fetched.
You can try adding a task-executor to your implementation
Follow this post for polling multiple files concurrently, but it doesn't follow the annotation based approach. And this post for sequential polling of files. And this also involves the use of annotations.
I'm writing a jar intended to be used with Spring and Ehcache. Spring requires that there be a cache defined for each element, so I was planning to have an Ehcache defined for the jar, preferably as a resource in the jar that could be imported into the primary Ehcache configuration for the app. However, my reading of the example Ehcache config file and my Google searches have not turned up any way to import a sub Ehcache config file.
Is there a way to import a sub Ehcache config file, or is there some other way to solve this problem?
What I did to do something similar (replace some placeholders in my Ehcache xml file - a import statement is more or less a placeholder if you will) is to extend (more or less copy to be honest) from Springs EhCacheManagerFactoryBean and create the final Ehcache xml config file on the fly.
For creating the CacheManager instance in afterPropertiesSet() you just hand over a InputStream which points to your config.
#Override
public void afterPropertiesSet() throws IOException, CacheException {
if (this.configLocation != null) {
InputStreamSource finalConfig = new YourResourceWrapper(this.configLocation); // put your custom logic here
InputStream is = finalConfig.getInputStream();
try {
this.cacheManager = (this.shared ? CacheManager.create(is) : new CacheManager(is));
} finally {
IOUtils.closeQuietly(is);
}
} else {
// ...
}
// ...
}
For my filtering stuff I internally used a ByteArrayResource to keep the final config.
data = IOUtils.toString(source.getInputStream()); // get the original config (configLocation) as string
// do your string manipulation here
Resource finalConfigResource = new ByteArrayResource(data.getBytes());
For "real" templating one could also think of using a real template engine like FreeMarker (which Spring has support for) to do more fancy stuff.