I am using spring batch for reading data from csv.
The Item Class is EmployeeItem where I set the values from CSV.
During persist when any exception is thrown for example DataIntegrity Violation, the process is skipped and the skipped data is written to the error file.
onSkipInWrite where I will get only the Employee entity.
public void onSkipInWrite(Employee item, Throwable t) {
errorItemWriter.write(items);
}
If its a validation error during the process time
public void onSkipInProcess(EmployeeItem item, Throwable t) {
}
I will get an EmployeeItem object which I can directly write the skipped data to the error file, but in the case of onSkipInWrite I will get the Employee entity only.
I have to reset the values from Employee to EmployeeItem
Any possibility to get the EmployeeItem in onSkipInWrite?
Related
I have a simple listener with 3 methods. and a repository with autowired on that. While saving an object from afterWrite it works nicely. but when saving item from onError methods no exception occurs, however it is not saving any data. Thankful for suggestions.
public class WriteListener implements ItemWriteListener{
public void beforeWrite(List items) {
System.out.println("Going to write following items: "+ items.toString());
}
public void onWriteError(Exception exception, List items) {
System.out.println("Error occurred when writing items!");
testRepository.save(items.get(0)); //not working
}
public void afterWrite(List items) {
testRepository.save(items.get(0)); //not nicely and save data
Based on the limited information provided, most likely the cause is the exception itself. The exception would have marked current transaction as dirty thus spring would have rolled it back.
If you still want to store data in your listener despite existing exception, use it in a separate transaction context. Simplest way for that would be to use #Async annotation on your listener and marking it Transactional explicitly to ensure it initiate a new transaction. Check out Spring Event which covers this topic in little bit more depth.
I have to create a batch job where I need to fetch data from 1 DB and after processing dump that data to another DB where auto generated ID would be assigned to persisted data. I need to send that data along with generated ID to solace queue.
Reader(DB1) --data1--> Processor --data2--> Writer (DB2) --data3--> Writer (Solace Publisher)
I am using spring boot-2.2.5.RELEASE and spring-boot-starter-batch.
I have created a job having 1 step that read data from DB1 and write data to DB2 via RepositoryItemReader and RepositoryItemWriter respectively. This is working fine.
Now next task is to send persisted data having generated ID to solace stream (using spring-cloud-starter-stream-solace).
I have below questions. Please assist as I am totally new to spring batch
How can I get the complete record after it's saved to DB2 based on some parameter? Do I have to write my own RepositoryItemWriter having StepExecution Context or can I somehow use the existing RepositoryItemWriter.
Once I got the record I need to use solace stream and there I have publish method which expects argument(record) to be published. I think again I need to write my own Item Writer and either I could use the record passed from above repositoryItemWriter by StepExecutionContext or should I query into DB2 directly from here based on some parameter ?
Either of the above case I need to use stepexecution context but can I use available RepositoryItemWriter or do I have to write my own?
Is there any other concept which is handy in this handy instead of using above approaches?
Passing data to future steps is a common pattern in Spring Batch. According to the documentation https://docs.spring.io/spring-batch/docs/current/reference/html/common-patterns.html#passingDataToFutureSteps you can use stepExecution to store and retrieve your generated IDs. In your case the writers are also listeners which has before step methods annotated with #BeforeStep. For example:
public class DB2ItemWriter implements ItemWriter<Object> {
private StepExecution stepExecution;
public void write(List<? extends Object> items) throws Exception {
// ...
ExecutionContext stepContext = this.stepExecution.getExecutionContext();
stepContext.put("generatedIds", ids);
}
#BeforeStep
public void saveStepExecution(StepExecution stepExecution) {
this.stepExecution = stepExecution;
}
}
and then you retrieve the ids in the next writer
public class SolacePublisherItemWriter implements ItemWriter<Object> {
public void write(List<? extends Object> items) throws Exception {
// ...
}
#BeforeStep
public void retrieveGeneratedIds(StepExecution stepExecution) {
ExecutionContext stepExecutionContext = stepExecution.getExecutionContext();
this.generatedIds = stepExecutionContext.get("generatedIds");
}
}
I have created a job having 1 step that read data from DB1 and write data to DB2 via RepositoryItemReader and RepositoryItemWriter respectively. This is working fine.
I would add a second step that reads data from the table (in which records have been persisted by step 1 and have their IDs generated) and push it to solace using a custom writer.
Suppose I have a caller to distribute work to multiple async tasks:
public class Caller{
public boolean run() {
for (int i = 0: i< 100; i++) {
worker.asyncFindOrCreate(entites[i]);
}
return true;
}
public class Worker{
#Autowired
Dao dao;
#Async
public E asyncFindOrCreate(User entity) {
return dao.findByName(entity.getName).elseGet(() -> dao.save(entity));
}
}
If we have 2 same entities:
with the synchronized method, the first one will be created and then the second one will be retrieved from the existing entity;
with async, the second entities might pass the findByName and go to save because the first entity hasn't been saved yet, which cause the save of the second entity throws unique identifier error.
Is there a way to add some fault tolerance mechanic to have some features like retry and skipAfterRetry, in particular for database operations.
In this special case you should convert your array to a map. Use the name property as a key, so there will be no duplicated entries.
However, if this method also can be called by multiple threads (ie. it's in a web-server) or there are multiple instances running it's still not fail-safe.
In generic, you should let the DB to check the uniqueness. There is no safest/easiest way to do that. Put the save method inside a try-catch block and check/handle the unique identifier exception.
I am doing few hibernate save operations in spring's transactional service class.
My expectation is that by the time method execution finishes hibernate should write data to database.
But hibernate is actually executing those queries only when controller is about to return a response.
My sample code is as follows.
#Controller
public class TestController {
#Autowired
private SaveService saveService;
#RequestMapping(value = "saveData", method = RequestMethod.POST)
public String saveData(HttpServletRequest request, Model model) {
try {
saveService.saveData(object1, object2, object3); // LINE 1
sendEmail(); // LINE 2
//some code here // LINE 3
} catch(Exception e) {
//log exception
//return error message
}
}
}
#Service("saveService")
#Transactional
public class SaveServiceImpl implements SaveService {
#Autowired
private SaveDAO saveDAO;
public void saveData(Object objec1, Object objec2, Object objec3) {
saveDAO.save(object1);
saveDAO.save(object2);
saveDAO.save(object3);
}
}
In above code I am calling SaveService.saveData() method from controller. And if save is successful I want to go ahead and send an email. But if for some reason SaveService.saveData() throws an exception i don't want
to send the email.
When I performed some tests it was observed that even if SaveService.saveData() throws an exception it's not thrown
until the email is sent from controller. I want that if a call to saveService.saveData() at 'LINE 1' in controller
throws an exception the next line that sends email should not get executed.
I wanted to know if this is expected hibernate behavior and if it is what can I do to tell hibernate to execute
queries before exiting service methods.
Thank you.
This behavior is due to hibernate optimizations. Hibernate will wait until the last time possible to execute the sentences.
You can avoid this with session.flush(), Flushing the session forces Hibernate to synchronize the in-memory state of the Session with the database (i.e. to write changes to the database).
The problem here is when an exception occurs, your the variables/objects are not initialized in the catch block and you are accessing it.
As it looks like you have just added a snippet of the code in question, so I guess the variables object1, object2, object3 needs to initalized to null.
for example: object1=null
In the service layer, I have some method who have a transactional annotation.
#Transactional
public void process() throws ProcessPaymentException{
try{
.... do some operation
catch (ProcessPaymentException ppe) {
save db problem issue.
}
}
It seem like if there are a issue, there are roll back... and nothing is saved in the db...
ProcessPaymentException extend Exception
Is there a way to rollback the process in the try but do the save in the catch?
Edit
Nested transaction could be a solution if this link is ok
https://www.credera.com/blog/technology-insights/java/common-oversights-utilizing-nested-transactions-spring/
Existing answer of using ControllerAdvise should help in normal setup that incoming requests are coming through Spring MVC (i.e. through a Controller).
For cases that is not, or you do not want to tie your exception handling logic to Spring MVC, here are some alternatives I can think of
(Here I assume you want to rely on declarative transaction control instead of programmatically controlling transactions yourself)
Separate service/component to save error in different transaction.
In short, you can have a separate service, which create its own transaction by propagation REQUIRES_NEW. e.g.
#Service
public class FooService
#Inject
private ErrorAuditService errorAuditService;
#Transactional
public void process() throws ProcessPaymentException{
try{
.... do some operation
catch (ProcessPaymentException ppe) {
errorAuditService.saveErrorAudit(ppe.getErrorText());
throw ppe; // I guess you want to re-throw the exception
}
}
}
#Service
public class ErrorAuditService
#Transactional(propagation=REQUIRES_NEW)
public void saveErrorAudit() {
// save to DB
}
}
One step further, if the error handling it the same for different services, you may create an advise, which will be called when service method throws exception. In that advise, you can save the error in db (using ErrorAuditService), and rethrow the exception.
Because processes of try-catch are wrapped by the same transaction.
The transaction manager do rollback whenever an exception is thrown. So, not thing would be saved.
Is there a way to rollback the process in the try but do the save in the catch?
Yes. Create Exception Handler to save db problem issue after rollback.
this is the idea
#ControllerAdvice
public class HandlerName {
#ExceptionHandler(ProcessPaymentException.class)
public void saveDbIssue(ProcessPaymentException ex) {
// save db problem issue.
}
But it only works if u want to save static data.