Is there any configuration for spring-batch, or in the datasources configurations, where I can disable/ignore all commits?
Basically, I want to execute my batch without updating the database just for testing. This would be perfect if I could disable and enable commits with a property
Edit: I don't want to use an in memory database because I want to process millions of lines that I have in the database. The batch reads lines from database and changes the state those lines after processing and I don't want to persist this changes.
You can use the ResourcelessTransactionManager for that. It is a No-Op implementation of PlatformTransactionManager which means there will be no real transaction ongoing against a transactional resource (hence the term Resourceless). You can use it when you don't really need (or care about) transactional behaviour, for example in tests/prototypes or when using a non-transactional job repository.
Related
I'm developing a migration app to migrate data from one database to another database, with different schema's - so we have some processors in between. Currently, we're using the JdbcCursorItemReader in our steps. I'm trying to avoid a temporary network issue resulting in a failed job, hours into a migration job.
I tried extending the JdbcCursorItemReader and overriding the 'open(ExecutionContext ec)' and also the 'read()' methods and annotating them with #Retryable. However, when an exception is thrown in either the open or read methods, the job fails - the exception is not caught and retry is not triggered.
I'm now starting to wonder if the JdbcCursorItemReader can encounter transient errors, which would need to be retried. As I understand it, a long running connection is opened and the results are streamed. Am I wasting my time trying to make the JdbcCursorItemReader retryable? Is it even possible?
If I used the JdbcPagingItemReader, could I make its read() method retryable?
I'm not too experienced with Spring Batch, any guidance on making the reader more resilient would be greatly appreciated!
Regards,
David
The retry policy of a fault-tolerant chunk-oriented step is not applied to the item reader, see Retry not working with Spring Batch with Java Config and Spring Batch: Using SkipPolicy and RetryPolicy with SpringBoot. The reason for that is because the contract of an item reader is "forward only", there is no way (in the current interface) to go back to a previous position in case of a retry.
So the path you are exploring is the way to go. You can add retry capabilities to your custom reader either declaratively with an annotation or programmatically with a RetryTemplate.
I've been researching various batch frameworks and I appreciate the abstraction and out-of-the-box features such as skip, retry, listeners etc. that Spring Batch brings with it and I'm familiar with Spring framework so this is a natural choice.
However, the batch flows I intent to create do not have transactional databases on either end of the reader-process-write flow. I desire to use Spring Batch to connect two systems through API's and still leverage the batch framework for tracking job executions and back the batch app with the Spring Batch database.
The remote API's support their own batching concepts, so, we can be sure to process 100 records and attempt to batch insert where the entire batch fails when one record is invalid. In that way, I would still like Spring Batch to "rollback" and retry each record individually.
Is it possible to leverage Spring Batch with its backing batch metadata database, to use skip and retry, without holding database connections or using transactions during chunk based processing?
Edit:
Based on Mahmoud's comment, I can use a DataSourceTransactionManager with the JobRepository and a ResourcelessTransactionManager with the chunk steps. So I will define a custom StepBuilderFactory:
#Component
public class MyStepBuilderFactory extends StepBuilderFactory {
public MyStepBuilderFactory(JobRepository jobRepository) {
super(jobRepository, new ResourcelessTransactionManager());
}
}
A chunk-oriented tasklet requires a transaction manager to handle the "rollback" semantics you are looking for. If your step does not interact with a transactional resource, you can configure it to use a ResourcelessTransactionManager. This transaction manager is a NoOp implementation that can be used in such cases.
You can always use a DataSourceTransactionManager with the job repository to track job/step execution meta-data.
Is there any way to prevent spring data from closing the underlying hibernate session after an #Transactional method in a #Service component? I am writing a simple command line app that reads data from a web service, does some processing and writes to the database. Some of the operations span multiple entities and must be transactional.
However, I am running into too many LazyLoadingExceptions. Apparently the hibernate session is automatically closed by spring after the transaction commits.
In other words, what I am trying to achieve is something like OpenSessionInView, but for a command line app. I could achieve this by using pure JPA and manually handling the entity manager and transactions. However, I would like to know if is there a solution which enables to leverage the spring-data and still achieve the desired behavior.
Two spring boot applications are connected to the common database.
I just wanted to know, how to handle the transaction if both the application try to update the record at the same time?
Since you seem to use JPA (via Spring Data JPA) there isn't much to handle.
The database itself will prevent two transactions to update the record at the same time. So one will always be first.
If you use optimistic locking (which is the default with JPA) the second transaction will notice the modified row and rollback.
Without that the second transaction will simply overwrite the changes with it's own changes.
I have a spring application which is based on Spring Batch. By default spring batch introduces transaction for its steps (i.e. at reader,writer and processor) . There are certain stages where I don't really need transaction to be enabled. Because transaction is enabled unnecessary for these methods its giving me some runtime errors as I am making call to two different databases in one method.
Is there any annotation which spring provides to DISABLE transaction for a specific set of methods ?
OR is there anything available in spring batch which can allow me to get rid of transaction either completely or declarative
I am even open to the solution which can disable transaction globally.
Any link , paper will greatly be appreciated.
Thanks in advance
Samir
Spring Batch is inherently transactional. Even if your datasources are not transactional, the semantics of the JobRepository require it. The closest you can get with Spring Batch and not being transactional is using the ResourcelessTransactionManager. This transaction manager is essentially a no-op transaction manager that just keeps track of if an operation is within the scope of a transaction or not.