Here is my scenario..
I need to read csv file and store output to ElasticSearch. I am using Spring Batch to read csv file. can anyone give me example how to save in elasticsearch using Spring Batch or Spring Batch Extension?
Its an old question and probably you might have found answer by now but here it goes...
To work with ElasticSearch, you need Spring Data and you simply write items from your writer as you normally do but with a repository instance like - repository.save(list) where list is a List of items passed from Spring Batch processor to writer.
Where repository is basically a ElasticsearchRepository from Spring Data. You need to define repositories for your items.
You need to provide your ElasticsearchRepository definitions to ElasticSearch instance definitions by editing - #EnableElasticsearchRepositories in and you define persistent layer as done here. Edit #EnableElasticsearchRepositories to actual repository package location of your project.
Hope it helps !!
Actually, I worked with a similar project but instead of importing data from a CSV file, I imported it from a relational database MySQL, reading and filtering data with the spring batch and write it into elasticsearch , this is the link of the project in the GitHub read carefully the readme.md file you will find all the required configuration :
the github project link
Related
I found a solution to do it via Java code here:
https://docs.spring.io/spring-batch/docs/4.2.x/reference/html/job.html#configuringJobRepository
But, I want to do it if possible in a simple way via configuration in yaml format in the batch configuration file.
Thank you.
As far as I know, there is currently no such property in yaml available.
There is an open feature request in Spring Boot (https://github.com/spring-projects/spring-boot/issues/28802) that may result in a property like spring.batch.jdbc.isolation-level-for-create in the future. Until then, you'll need to use Java (or XML) configuration.
I'm fairly new to Spring Boot and MongoDB. Currently I have a project that can send data back and forth to a server that is running locally on my computer, but I want to change this and make it edit and retrieve data from an externally running database. In other tutorials I have followed I have had an application.properties file that I can edit details in to get it to connect, but I can't find this in any of the sub folders (I pulled the code from a tutorial) and I can't find anything to say that it is specifically connecting to the local instance.
Would it be okay to just create the application.properties file in the right sub folder and enter the external database's details there? Or am I going to have to try a separate method and tutorial to try and connect to the external database in another way?
I have a feeling that to answer it you will need to see/understand more of the code, but I'm not sure how to summarise anything else or what would actually be relevant. Thank you.
Spring Boot has several default folders, where it searches for properties.
One of those places is for example src/main/resources/application.properties, there you can just create this file.
An overview of other possible places for Spring Boot properties can be found here:
https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html
In my German blog I wrote an article about how to use Spring-data to access MongoDB - there I used also application.properties file:
https://agile-coding.blogspot.com/2020/10/keine-ahnung-von-mongodb-dann-nimm.html
As the title says, I am trying to read a cvs file that contains thousands of ip addresses to their respective country. I want to import the cvs file into a MongoDB using WebFlux. I haven't been able to find any resources on how to do this. I have come across Spring Batch but I don't believe it supports WebFlux.
One way I thought of achieving this is just read the CVS file, parse the file, create DTO with values then save it into the database, however, I worry about performance.
Spring WebFlux is the alternative to Spring MVC module. It is not suitable for data processing. So if you want to solve your problem, use this way:
One way I thought of achieving this is just read the CVS file, parse the file, create DTO with values then save it into the database, however, I worry about performance.
And "reactive way" won't be faster than batch processing just because it's "reactive".
I am aware of spring batch metadata tables are written to Relational database tables say MySQL,H2 etc. But my question is whether spring batch metadata tables can be written to elasticsearch . If so how to proceed further?
have you checked the Spring Batch Extensions module? This module provides an ItemReader and an ItemWriter for interacting with Elasticsearch. Hope it helps.
We are trying to develop a framework on top of spring batch, basically it has to read the data from database like fields, fields order, file location..etc..
Is there any existing frameworks to achieve this, otherwise please shed some light on this...
Thanks,
MK
I don't think there is any such extension available over the framework. You might have to write your own customizations to achieve a database driven configuration for Spring Batch.
What do you want Spring Batch to read in? Have you looked at ItemReaders? http://docs.spring.io/spring-batch/trunk/reference/html/readersAndWriters.html
In your ItemReader constructor you can read in whatever configuration you require.
And see:
Reading Records From a Database in Spring Batch