This is regarding Spring Batch - https://github.com/spring-projects/spring-batch/issues/1844, extending this question further here. I am looking to delete all history from Spring Batch metadata tables older than 30 days since its no longer needed to hold that info and this will save some space at db side.
How can we do this programmatically through Spring Batch code?
Related
I'm trying to convert an older application that so far only uses Spring to Spring Boot. I managed to get the application to run again. But I'm having problems with Spring Batch.
The application uses Spring + Spring Batch + Spring Security + Hibernate + Wicket + ojdbc and is running on Tomcat with a Oracle 19c database.
I'm fighting these Spring Batch problems:
Massive performance drop: A Batch Step that took 2 minutes before is now running for an hour. This seems to me like it's caused by a slow database connection. Spring Batch spends a lot of time creating the objects with the ItemReader.
The Spring Job fails at the end with a "ObjectOptimisticLockingFailureException: Batch update returned unexpected row count from update" There is a versioned table that contains the status of the calculations. This status is updated multiple times during the Batch Job. The status update works in the beginning, but near the end the EntityManager returns an result with an outdated version. Updating causes the exception.
Do you have any ideas or hints what could cause these problems? The functionality was not changed, this should only be a migration to Spring Boot.
I have a working Spring Boot application which embeds a Spring Batch Job. The job is not run on a schedule, instead we kick it with an endpoint. It is working as it should. The basics of the batch are
Kick the endpoint to start the job
Reader reads from input file
Processor reads from oracle database using jpa repository and simple spring datasource config
Writer writes to output file
However there are new requirements:
The schema of the repository database is from here on unknown on application startup. The tables are the same, it is just an unknown schema. This fact is out of our control and you might think it is stupid but there are reasons for it and this cant be changed. This means that with current functionality we need to reconfigure the datasource when we know the new schema name, and restart the application. This is a job that we will run for a number of times when migrating from one system to another, so it has a limited lifecycle and we just need a "quick fix" to be able to use it without rewriting the whole app. So what I would like to do is:
Send the schema name as a query param to the application, put it in job parameters and then - get a new datasource when the processor reads from the repository. Would this be doable at all using Spring Batch? Any help appreciated!
My application is using spring batch , in the first run the tables of the batch were created in the database , but in the other attempts to run it is trying to recreate the existing batch tables which causes me errors .
Is there any way to prevent spring batch from recreating these tables ?
Spring Batch does not create any table, you should be talking about Spring Boot. There is a way to prevent boot from creating the tables using spring.batch.initialize-schema=never, but there is no way to prevent re-creating them. Ideally, the scripts should include "if does not exist" clause (there is an open issue for that here: https://github.com/spring-projects/spring-batch/issues/1855), but this is not the case.
I am working on spring batchupdate ,I search on google I didn't find any solution for my problem.
I have two databases(MySQL,ORALCE) I want to read data from mysql and write into oracle by using batch update .
Your problem is unclear.
You can first read the data from MySQL with one Spring JdbcTemplate object initialized with MySql data source, and then use another JdbcTemplate object, initialized with Oracle data source, to write the data.
If you want to do it in one transaction, you will have to use distributed transactions/XA libraries, such as Atomicos, and Spring distributed transaction manager. See here https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-jta.html for details on Spring integration with distributed transactions libraries.
I am aware of spring batch metadata tables are written to Relational database tables say MySQL,H2 etc. But my question is whether spring batch metadata tables can be written to elasticsearch . If so how to proceed further?
have you checked the Spring Batch Extensions module? This module provides an ItemReader and an ItemWriter for interacting with Elasticsearch. Hope it helps.