How to Write into multiple files with Spring Batch using writer, Multiple files depends on data return from Database. Any better solution to implement in Spring Batch
Use a ClassifierCompositeItemWriter
Calls one of a collection of ItemWriters for each item, based on a router pattern implemented through the provided Classifier.
Related
As the title says, I am trying to read a cvs file that contains thousands of ip addresses to their respective country. I want to import the cvs file into a MongoDB using WebFlux. I haven't been able to find any resources on how to do this. I have come across Spring Batch but I don't believe it supports WebFlux.
One way I thought of achieving this is just read the CVS file, parse the file, create DTO with values then save it into the database, however, I worry about performance.
Spring WebFlux is the alternative to Spring MVC module. It is not suitable for data processing. So if you want to solve your problem, use this way:
One way I thought of achieving this is just read the CVS file, parse the file, create DTO with values then save it into the database, however, I worry about performance.
And "reactive way" won't be faster than batch processing just because it's "reactive".
I'm dipping my toes into the microservices, is spring boot batch applicable to the following requirements?
Files of one or multiple are read from a specific directory in Linux.
Several operations like regex, build new files, write the file and ftp to a location
Send email during a process fail
Using spring boot is confirmed, now the question is
Should I use spring batch or just core spring framework?
I need to integrate with Control-M to trigger the job. Can the Control-M be completely removed by using Spring batch library? As we don't know when to expect the files in the directory.
I've not seen a POC with these requirements. Would someone provide an example POC or an affirmation this could be achieved with Spring batch?
I would use Spring Batch for that use case. Not only does it provide out of the box components for reading, processing, and writing files, it adds a lot more for error handling, scalability, etc. All of those things you'd probably end up wiring up by yourself if you go without Spring Batch.
As for being launched via Control-M, yes MANY large customers use Control-M to launch their jobs. Unfortunately, I've never done it myself so I cannot provide any details on the mechanics, but if Control-M can either launch a script or call a REST API, you can launch a job with it.
I would suggest you, go for spring batch as it has much-inbuilt functionality which will be provided to you for file reading and writing to your required location. Even you will be able to handle record skipping requirement. Your mail triggering requirement will be handled by Control M. You just need to decide one exit code for your handled exception and on the basis of that exit code you can trigger the mail to respective members. And there are many other features which will be helpful if you go for spring batch.
I am using spring batch partitioning and i would like to send a file list to each partition for processing.
I need to know how to invoke SftpOutboundGateway and get the list of remote files names from the spring batch Partitioner.
If that's all you need to do, consider just using FtpRemoteFileTemplate.list() or even Session.listNames().
I have been trying to fetch details from multiple tables in a database and provide the output in the form of XML file using Spring batch. Is there any working code to do the above task or any idea to do so?
Following links should guide you.
If simple then xml then:
http://www.mkyong.com/spring-batch/spring-batch-example-mysql-database-to-xml/
For complex xml using spring, refer to these threads;
Complex XML using Spring Batch; StaxEventItemWriter ; Jaxb2Marshaller
Build non trivial XML file with StaxEventItemWriter
We are trying to develop a framework on top of spring batch, basically it has to read the data from database like fields, fields order, file location..etc..
Is there any existing frameworks to achieve this, otherwise please shed some light on this...
Thanks,
MK
I don't think there is any such extension available over the framework. You might have to write your own customizations to achieve a database driven configuration for Spring Batch.
What do you want Spring Batch to read in? Have you looked at ItemReaders? http://docs.spring.io/spring-batch/trunk/reference/html/readersAndWriters.html
In your ItemReader constructor you can read in whatever configuration you require.
And see:
Reading Records From a Database in Spring Batch