Spring JPA transaction partially retry - spring

I am trying to use Spring retry for my Spring web application (with JPA, hibernate).
Currently, I have a long transaction chain (marked functions as #Transactional) in my application, intuitively:
start transction -> A() -> B() -> C() -> D() - E() -> commit/rollback
now I want to have D() to be retried if any concurrency failure occurred (marked D() with #retryable), but remain A(), B(), C() in current states
I failed, the function D() is not retried at all and just throwed a concurrency failure error (eg. ObjectOptimisticLockingFailureException)
For me, if I want to do such things in database, I have to make a new transaction try catch block with a cursor while loop to handle retries. I wonder is there a simple way I can handle this "partial" transaction retry in Spring?
An example code would be:
#RestController
public DimensionController()
{
...
#Autowired
private TblDimensionService dimensionService;
...
#PutMapping(...)
public ResponseEntity<TblDimensionDTO> update(#Valid #RequestBody TblDimensionDTO dimensionDTO)
{
...
dimensionService.update(dimensionDTO);
...
}
}
#Transactional //transaction in service level
#Service
public TblDimensionService()
{
...
#Autowired
private RetryService retryService;
...
public TblDimensionDTO update(TblDimensionDTO dimensionDTO) throws InterruptedException
{
if (dimensionDTO.getId() == null)
{
throw new ResponseStatusException(HttpStatus.NOT_FOUND, "not found");
}
TblDimension dimension = findByIdOrElseThrow(dimensionDTO.getId()); //call another function to retrieve entity by id with JPA repository
dimension = retryService.updateEntity(dimension, dimensionDTO);
return tblDimensionMapper.toDto(dimension);
}
...
}
#Transactional //transaction in service level
#Service
public RetryService()
{
...
#Autowired
private TblDimensionRepository dimensionRepository;
...
//I want to only retry this part
//but this retry does not work
#Retryable(value = {ConcurrencyFailureException.class})
public TblDimension updateEntity(TblDimension dimension, TblDimensionDTO dimensionDTO) throws InterruptedException
{
Thread.sleep(3000);
dimension.setHeight(dimension.getHeight() + 1);
Thread.sleep(3000);
return dimensionRepository.save(dimension);
}
...
}

Related

Failure in inner #Transactional(readOnly = false, rollbackFor = Exception.class) rolls back entire outside persistence

I have a job method in a class-annotated #Transactional. This job method calls inner methods for persistence of individual records. If I simulate an error in the following inner update() method somewhere in the middle of my result set processing, I see that all successful records before/after this exception do not get saved after job completion. Why is that? All outside persistence should remain, with the exception of the individual record that failed. The inner update alone has rollbackFor.
#Service("mailService")
#Transactional
#EnableScheduling
public class MailServiceImpl implements MailService {
#Override
#Scheduled(cron = "${mail.cron.pubmed.autosynch.job}")
public void autoSynchPubMedJob() {
//... Fetch result set
for (Result r: resultset) {
try {
pubService.updatePublication(r);
} catch (Exception e) {
// Silently log and continue
log.error("Error on record: ", e);
}
}
}
The updatePublication method, this is the one with rollbackFor:
#Override
#Transactional(readOnly = false, rollbackFor = Exception.class)
public void updatePublication(Publication publication) throws Exception {
dao.update1(..);
dao.update2(..);
// Simulate exception for a specific record for testing
if (publication.getId() == 123) {
throw new Exception("Test Exception");
}
}
Result: no successful data persisted at all at the end of job completion. There should be partial persistence (for other successful records).
When I remove this Exception simulation, all data is successfully persisted at the end. Also, all data is persisted if I remove the inner call's rollbackFor.
Probaby because it uses existing transaction. Try opening a new one with propagation = REQUIRES_NEW.
Note: New transaction won't be opened if you call the method from the same service. You should use either self-reference call or extract logic to another #Service.

How to rollback child transaction if any exception in parent transaction?

I have two transaction manager for two database. I need to persist same data into both databases. If one transaction failed, other one need rollback. I have done like below
public interface DataService {
void saveData();
}
#Service
public class DataServiceImpl implements DataService {
#Autowired
private DataRepository dataRepository;
#Autowired
private OrDataRepository orDataRepository;
#Autowired
#Qualifier("orService")
private OrService orDataServiceImpl;
#Override
#Transactional(transactionManager = "transactionManager", rollbackFor = {RuntimeException.class})
public void saveData() {
Data data = new Data();
data.setCompKey(UUID.randomUUID().toString().substring(1,5));
data.setName("data");
dataRepository.save(data);
orDataServiceImpl.save();
//throw new RuntimeException("");
}
}
public interface OrService {
void save();
}
#Service("orService")
public class OrDataServiceImpl implements OrService {
#Autowired
private OrDataRepository orDataRepository;
#Override
#Transactional(rollbackFor = {RuntimeException.class})
public void save() {
OrData data = new OrData();
data.setCompKey(UUID.randomUUID().toString().substring(1,5));
data.setName("ordata");
orDataRepository.save(data);
}
}
I have two transaction manager (entityManager & orEntityManager) for two different DB.
If any exception in OrDataServiceImpl save method, data is not getting persisted in both DB. But if any exception in DataServiceImpl saveData method, data is getting persisted into OrData table.
I want to rollback the data from both DB if any exception.
chainedTransactionManager is deprecated. So can't use. atomikos and bitronix also can't use due to some restrictions. Kindly suggest better way to achieve distributed transation
The code need to be refactored, edit the DataServiceImpl.save() method.
Comment the orDataServiceImpl.save() line
public void saveData() {
Data data = new Data();
data.setCompKey(UUID.randomUUID().toString().substring(1,5));
data.setName("data");
dataRepository.save(data);
//orDataServiceImpl.save();
//throw new RuntimeException("");
}
Refactor/Edit the OrDataService Interface
public interface OrDataService {
void save(String uuid);
void delete(String uuid);
//will be use for compensating transaction
}
Update the OrDataServiceImpl class to implement above interface
Write new orchestration Method and use compensating transaction to rollback
pseudo code
call OrDataServiceImpl.save()
if step#1 was success
-> DataServiceImpl.saveData()
if Exception at step#3,
->OrDataServiceImpl.delete() [//to rollback]
else if, Exception at step#1
//do nothing

Why is exception in Spring Batch AsycItemProcessor caught by SkipListener's onSkipInWrite method?

I'm writing a Spring Boot application that starts up, gathers and converts millions of database entries into a new streamlined JSON format, and then sends them all to a GCP PubSub topic. I'm attempting to use Spring Batch for this, but I'm running into trouble implementing fault tolerance for my process. The database is rife with data quality issues, and sometimes my conversions to JSON will fail. When failures occur, I don't want the job to immediately quit, I want it to continue processing as many records as it can and, before completion, to report which exact records failed so that I, and or my team, can examine these problematic database entries.
To achieve this, I've attempted to use Spring Batch's SkipListener interface. But I'm also using an AsyncItemProcessor and an AsyncItemWriter in my process, and even though the exceptions are occurring during the processing, the SkipListener's onSkipInWrite() method is catching them - rather than the onSkipInProcess() method. And unfortunately, the onSkipInWrite() method doesn't have access to the original database entity, so I can't store its ID in my list of problematic DB entries.
Have I misconfigured something? Is there any other way to gain access to the objects from the reader that failed the processing step of an AsynItemProcessor?
Here's what I've tried...
I have a singleton Spring Component where I store how many DB entries I've successfully processed along with up to 20 problematic database entries.
#Component
#Getter //lombok
public class ProcessStatus {
private int processed;
private int failureCount;
private final List<UnexpectedFailure> unexpectedFailures = new ArrayList<>();
public void incrementProgress { processed++; }
public void logUnexpectedFailure(UnexpectedFailure failure) {
failureCount++;
unexpectedFailure.add(failure);
}
#Getter
#AllArgsConstructor
public static class UnexpectedFailure {
private Throwable error;
private DBProjection dbData;
}
}
I have a Spring batch Skip Listener that's supposed to catch failures and update my status component accordingly:
#AllArgsConstructor
public class ConversionSkipListener implements SkipListener<DBProjection, Future<JsonMessage>> {
private ProcessStatus processStatus;
#Override
public void onSkipInRead(Throwable error) {}
#Override
public void onSkipInProcess(DBProjection dbData, Throwable error) {
processStatus.logUnexpectedFailure(new ProcessStatus.UnexpectedFailure(error, dbData));
}
#Override
public void onSkipInWrite(Future<JsonMessage> messageFuture, Throwable error) {
//This is getting called instead!! Even though the exception happened during processing :(
//But I have no access to the original DBProjection data here, and messageFuture.get() gives me null.
}
}
And then I've configured my job like this:
#Configuration
public class ConversionBatchJobConfig {
#Autowired
private JobBuilderFactory jobBuilderFactory;
#Autowired
private StepBuilderFactory stepBuilderFactory;
#Autowired
private TaskExecutor processThreadPool;
#Bean
public SimpleCompletionPolicy processChunkSize(#Value("${commit.chunk.size:100}") Integer chunkSize) {
return new SimpleCompletionPolicy(chunkSize);
}
#Bean
#StepScope
public ItemStreamReader<DbProjection> dbReader(
MyDomainRepository myDomainRepository,
#Value("#{jobParameters[pageSize]}") Integer pageSize,
#Value("#{jobParameters[limit]}") Integer limit) {
RepositoryItemReader<DbProjection> myDomainRepositoryReader = new RepositoryItemReader<>();
myDomainRepositoryReader.setRepository(myDomainRepository);
myDomainRepositoryReader.setMethodName("findActiveDbDomains"); //A native query
myDomainRepositoryReader.setArguments(new ArrayList<Object>() {{
add("ACTIVE");
}});
myDomainRepositoryReader.setSort(new HashMap<String, Sort.Direction>() {{
put("update_date", Sort.Direction.ASC);
}});
myDomainRepositoryReader.setPageSize(pageSize);
myDomainRepositoryReader.setMaxItemCount(limit);
// myDomainRepositoryReader.setSaveState(false); <== haven't figured out what this does yet
return myDomainRepositoryReader;
}
#Bean
#StepScope
public ItemProcessor<DbProjection, JsonMessage> dataConverter(DataRetrievalSerivice dataRetrievalService) {
//Sometimes throws exceptions when DB data is exceptionally weird, bad, or missing
return new DbProjectionToJsonMessageConverter(dataRetrievalService);
}
#Bean
#StepScope
public AsyncItemProcessor<DbProjection, JsonMessage> asyncDataConverter(
ItemProcessor<DbProjection, JsonMessage> dataConverter) throws Exception {
AsyncItemProcessor<DbProjection, JsonMessage> asyncDataConverter = new AsyncItemProcessor<>();
asyncDataConverter.setDelegate(dataConverter);
asyncDataConverter.setTaskExecutor(processThreadPool);
asyncDataConverter.afterPropertiesSet();
return asyncDataConverter;
}
#Bean
#StepScope
public ItemWriter<JsonMessage> jsonPublisher(GcpPubsubPublisherService publisherService) {
return new JsonMessageWriter(publisherService);
}
#Bean
#StepScope
public AsyncItemWriter<JsonMessage> asyncJsonPublisher(ItemWriter<JsonMessage> jsonPublisher) throws Exception {
AsyncItemWriter<JsonMessage> asyncJsonPublisher = new AsyncItemWriter<>();
asyncJsonPublisher.setDelegate(jsonPublisher);
asyncJsonPublisher.afterPropertiesSet();
return asyncJsonPublisher;
}
#Bean
public Step conversionProcess(SimpleCompletionPolicy processChunkSize,
ItemStreamReader<DbProjection> dbReader,
AsyncItemProcessor<DbProjection, JsonMessage> asyncDataConverter,
AsyncItemWriter<JsonMessage> asyncJsonPublisher,
ProcessStatus processStatus,
#Value("${conversion.failure.limit:20}") int maximumFailures) {
return stepBuilderFactory.get("conversionProcess")
.<DbProjection, Future<JsonMessage>>chunk(processChunkSize)
.reader(dbReader)
.processor(asyncDataConverter)
.writer(asyncJsonPublisher)
.faultTolerant()
.skipPolicy(new MyCustomConversionSkipPolicy(maximumFailures))
// ^ for now this returns true for everything until 20 failures
.listener(new ConversionSkipListener(processStatus))
.build();
}
#Bean
public Job conversionJob(Step conversionProcess) {
return jobBuilderFactory.get("conversionJob")
.start(conversionProcess)
.build();
}
}
This is because the future wrapped by the AsyncItemProcessor is only unwrapped in the AsyncItemWriter, so any exception that might occur at that time is seen as a write exception instead of a processing exception. That's why onSkipInWrite is called instead of onSkipInProcess.
This is actually a known limitation of this pattern which is documented in the Javadoc of the AsyncItemProcessor, here is an excerpt:
Because the Future is typically unwrapped in the ItemWriter,
there are lifecycle and stats limitations (since the framework doesn't know
what the result of the processor is).
While not an exhaustive list, things like StepExecution.filterCount will not
reflect the number of filtered items and
itemProcessListener.onProcessError(Object, Exception) will not be called.
The Javadoc states that the list is not exhaustive, and the side-effect regarding the SkipListener that you are experiencing is one these limitations.

Testing that delete is correctly rolled back with DataIntegrityViolationException junit, spring, #Transactional

I have a category -> subCategory -> products hierarchy in my application. If a subcategory has no products, you are allowed to delete it. If a subCategory has products, the DAO throws a DataIntegrityViolationException and the transaction should be rolled back.
In my tests, I have:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(classes = {TestTransactionManagement.class})
public class BusinessSubCategoryCRUDTest {
#Autowired
public void setCRUD(BusinessSubCategoryCRUD crud) {
this.crud = crud;
}
// #Transactional
#Test
public void testDeleteBusinessSubCategoryInUseCanNotBeDeleted() {
final long id = 1;
BusinessSubCategory subCategoryBeforeDelete =
crud.readBusinessSubCategory(id);
final int numCategoriesBeforeDelete =
subCategoryBeforeDelete.getBusinessCategories().size();
try {
crud.deleteBusinessSubCategory(
new BusinessSubCategory(id, ""));
} catch (DataIntegrityViolationException e) {
System.err.println(e);
}
BusinessSubCategory subCategoryAfterDeleteFails =
crud.readBusinessSubCategory(id);
// THIS next assertion is the source of my angst.
// At this point the the links to the categories will have been
// been deleted, an exception will have been thrown but the
// Transaction is not yet rolled back if the test case (or test
// class) is marked with #Transactional
assertEquals(
numCategoriesBeforeDelete,
subCategoryAfterDeleteFails.getBusinessCategories().size());
}
}
However, if I uncomment the #Transactional above #Test, it fails. I think the DAO is using the transaction from the #Test and so the transaction doesn't roll back until AFTER I check to be sure the transaction has been rolled back.
#Transactional(readOnly = false, propagation =
Propagation.REQUIRED)
public boolean deleteBusinessSubCategory(
BusinessSubCategory businessSubCategory) {
BeanPropertySqlParameterSource paramMap = new
BeanPropertySqlParameterSource(businessSubCategory);
namedJdbcTemplate.update(
DELETE_CATEGORY_SUB_CATEGORY_BY_ID_SQL,
paramMap);
return 0 != namedJdbcTemplate.update(
DELETE_SUB_CATEGORY_BY_ID_SQL,
paramMap);
}
So, how do I have the DAO code still inherit the transaction from the context it is running in (in production it inherits the transaction from the service it is running in) but still be able to test it. I want to put #Transactional on the entire test class, but that then leaves my test either failing or incomplete.
For completeness, here is my configuration class for the test.
#Configuration
#EnableTransactionManagement
public class TestTransactionManagement {
#Bean
public EmbeddedDatabase getDataSource() {
EmbeddedDatabaseBuilder builder = new EmbeddedDatabaseBuilder();
EmbeddedDatabase db = builder
.setType(EmbeddedDatabaseType.HSQL) //.H2 or .DERBY
.addScript("sql/create-db.sql")
.addScript("sql/create-test-data.sql")
.build();
return db;
}
#Bean
public DataSourceTransactionManager transactionManager() {
return new DataSourceTransactionManager(getDataSource());
}
#Bean
public BusinessSubCategoryCRUD getCRUD() {
return new BusinessSubCategoryCRUD(getDataSource());
}
}
The "solution" or workaround was to reset the database before each test. Then there was no need for an #Transactional on the test, the rollback could be tested, and the test suite ran slighly slower due to the additional database setup.
#Before
public void setUp() {
Connection conn = DataSourceUtils.getConnection(dataSource);
ScriptUtils.executeSqlScript(
conn, new ClassPathResource("sql/create-test-data.sql"));
DataSourceUtils.releaseConnection(conn, dataSource);
}

Spring Integration - JdbcPollingChannelAdapter commit instead of rollback on handled Exceptions

I am using Spring 4.1.x APIs, Spring Integration 4.1.x APIs and Spring Integration Java DSL 1.0.x APIs for an EIP flow where we consume messages from an Oracle database table using a JdbcPollingChannelAdpater as the entry point into the flow.
Even though we have an ErrorHandler configured on the JdbcPollingChannelAdapter's Poller, we are seeing that a session's Transaction is still rolled back and not committed when a RuntimeException is thrown and correctly handled by the ErrorHandler.
After reading through this thread: Spring Transactions - Prevent rollback after unchecked exceptions (RuntimeException), I get the feeling that it is not possible to prevent a rollback and instead force a commit. Is this correct? And, if there is a way, what is the cleanest way to force a commit instead of a rollback when an error is safely handled?
Current Configuration:
IntegrationConfig.java:
#Bean
public MessageSource<Object> jdbcMessageSource() {
JdbcPollingChannelAdapter adapter = new JdbcPollingChannelAdapter(
dataSource,
"select * from SERVICE_TABLE where rownum <= 10 for update skip locked");
adapter.setUpdateSql("delete from SERVICE_TABLE where SERVICE_MESSAGE_ID in (:id)");
adapter.setRowMapper(serviceMessageRowMapper);
adapter.setMaxRowsPerPoll(1);
adapter.setUpdatePerRow(true);
return adapter;
}
#SuppressWarnings("unchecked")
#Bean
public IntegrationFlow inFlow() {
return IntegrationFlows
.from(jdbcMessageSource(),
c -> {
c.poller(Pollers.fixedRate(100)
.maxMessagesPerPoll(10)
.transactional(transactionManager)
.errorHandler(errorHandler));
})
.channel(inProcessCh()).get();
}
ErrorHandler.java
#Component
public class ErrorHandler implements org.springframework.util.ErrorHandler {
#Autowired
private PlatformTransactionManager transactionManager;
private static final Logger logger = LogManager.getLogger();
#Override
public void handleError(Throwable t) {
logger.trace("handling error:{}", t.getMessage(), t);
// handle error code here...
// we want to force commit the transaction here?
TransactionStatus txStatus = transactionManager.getTransaction(null);
transactionManager.commit(txStatus);
}
}
--- EDITED to include ExpressionEvaluatingRequestHandlerAdvice Bean ---
#Bean
public Advice expressionEvaluatingRequestHandlerAdvice() {
ExpressionEvaluatingRequestHandlerAdvice expressionEvaluatingRequestHandlerAdvice = new ExpressionEvaluatingRequestHandlerAdvice();
expressionEvaluatingRequestHandlerAdvice.setTrapException(true);
expressionEvaluatingRequestHandlerAdvice.setOnSuccessExpression("payload");
expressionEvaluatingRequestHandlerAdvice
.setOnFailureExpression("payload");
expressionEvaluatingRequestHandlerAdvice.setFailureChannel(errorCh());
return expressionEvaluatingRequestHandlerAdvice;
}
--- EDITED to show Dummy Test Message handler ---
.handle(Message.class,
(m, h) -> {
boolean forceTestError = m.getHeaders().get("forceTestError");
if (forceTestError) {
logger.trace("simulated forced TestException");
TestException testException = new TestException(
"forced test exception");
throw testException;
}
logger.trace("simulated successful process");
return m;
}, e-> e.advice(expressionEvaluatingRequestHandlerAdvice())
--- EDITED to show ExecutorChannelInterceptor method ---
#Bean
public IntegrationFlow inFlow() {
return IntegrationFlows
.from(jdbcMessageSource(), c -> {
c.poller(Pollers.fixedRate(100).maxMessagesPerPoll(10)
.transactional(transactionManager));
})
.enrichHeaders(h -> h.header("errorChannel", errorCh(), true))
.channel(
MessageChannels.executor("testSyncTaskExecutor",
syncTaskExecutor()).interceptor(
testExecutorChannelInterceptor()))
.handle(Message.class, (m, h) -> {
boolean forceTestError = m.getHeaders().get("forceTestError");
if (forceTestError) {
logger.trace("simulated forced TestException");
TestException testException = new TestException(
"forced test exception");
throw testException;
}
logger.trace("simulated successful process");
}).channel("nullChannel").get();
}
It won't work just because your ErrorHandler works already after the finish of TX.
Here is a couple lines of source code (AbstractPollingEndpoint.Poller):
#Override
public void run() {
taskExecutor.execute(new Runnable() {
#Override
public void run() {
.............
try {
if (!pollingTask.call()) {
break;
}
count++;
}
catch (Exception e) {
....
}
}
}
});
}
Where:
The ErrorHandler is applied for the taskExecutor (SyncTaskExecutor) by default.
TransactionInterceptor being as Aspect is applied for the Proxy around that pollingTask.
Therefore TX is done around the pollingTask.call() and goes out. And only after that your ErrorHandler starts to work inside taskExecutor.execute().
To fix your issue, you need to figure out which downstream flow part isn't so critical for TX rallback and make there some try...catch or use ExpressionEvaluatingRequestHandlerAdvice to "burke" that RuntimeException.
But as you have noticed by my reasoning that must be done within TX.

Resources