I am doing some ETL that process some input CSV files and load then to Neo4j using Spring data Neo4j.
I have 2 routes one that takes the input CSV split by line and send to the second route that does the load line by line in transnational mode.
The following is the first route
#Override
void configure() throws Exception {
from(endpoint)
.id('CSV_ROUTE')
.unmarshal(buildCsvDataFormat())
.split(body())
.streaming()
.parallelProcessing()
.recipientList(header('IMPORTER_ROUTE'))
And the following is the second route
#Override
void configure() throws Exception {
from(endpoint)
.transacted()
.id(routeId)
.bean(importer)
}
How can I make transaction to commit in batches for example of 10 lines instead of every line ?
Thank you
Luis Oscar
You cannot do this. Transactions is per message in Camel.
Also mind transaction is not some magic fairy dust you can turn on and then anything you touch becomes transactional.
In Java transactions often only work with transactional resources such as JDBC and JMS.
Related
I'm using SpringBatch for my app. In one of the batch jobs, I need to process multiple data. Each data requires several database updates. And I need to make one transaction for one data. Meaning, if when processing one data an exception is thrown, database updates are rolled back for that data, then keep processing the next data.
I've put all database updates in one method in service layer. In my springbatch tasklet, I call that method for each data, like this;
for (RequestViewForBatch request : requestList) {
orderService.processEachRequest(request);
}
In the service class the method is like this;
Transactional(propagation = Propagation.NESTED, timeout = 100, rollbackFor = Exception.class)
public void processEachRequest(RequestViewForBatch request) {
//update database
}
When executing the task, it gives me this error message
org.springframework.transaction.NestedTransactionNotSupportedException: Transaction manager does not allow nested transactions by default - specify 'nestedTransactionAllowed' property with value 'true'
but i don't know how to solve this error.
Any suggestion would be appreciated. Thanks in advance.
The tasklet step will be executed in a transaction driven by Spring Batch. You need to remove the #Transactional on your processEachRequest method.
You would need a fault-tolerant chunk-oriented step configured with a skip policy. In this case, only faulty items will be skipped. Please refer to the Configuring Skip Logic section of the documentation. You can find an example here.
I am in the process of implementing a spring batch job for our file upload process. My requirement is to read a flat file, apply business logic then store it in DB then post a Kafka message.
I have a single chunk-based step that uses a custom reader, processor, writer. The process works fine but takes a lot of time to process a big file.
It takes 15 mins to process a file having 60K records. I need to reduce it to less than 5 mins, as we will be consuming much bigger files than this.
As per https://docs.spring.io/spring-batch/docs/current/reference/html/scalability.html I understand making it multithreaded would give a performance boost, at the cost of restart ability. However, I am using FlatFileItemReader, ItemProcessor, ItemWriter and none of them is thread-safe.
Any suggestions as to how to improve performance here?
Here is the writer code:-
public void write(List<? extends Message> items) {
items.forEach(this::process);
}
private void process(Message message) {
if (message == null)
return;
try {
//message is a DTO that have info about success or failure.
if (success) {
//post kafka message using spring cloud stream
//insert record in DB using spring jpaRepository
} else {
//insert record in DB using spring jpaRepository
}
} catch (Exception e) {
//throw exception
}
}
Best regards,
Preeti
Please refer to below SO thread and refer the git hub source code for parallel processing
Spring Batch multiple process for heavy load with multiple thread under every process
Spring batch to process huge data
In a spring boot application, I have a class with jms listener.
public class PaymentNotification{
#JmsListener(destination="payment")
public void receive(String payload) throws Exception{
//mapstring conversion
....
paymentEvent = billingService.insert(paymentEvent); //transactional method
//call rest...
billingService.save(paymentEvent);
//send info to jms
}
}
I saw then when a error happen, data is inserted in the database, that ok, but it's like receive method is called again and again... but queue is empty when I check on the server.
If there is an error, I don't want method is called again, Is there something for that.
The JMS Message Headers might contain additional information to help with your processing. In particular JMSRedelivered could be of some value. The Oracle doc states that "If a client receives a message with the JMSRedelivered field set, it is likely, but not guaranteed, that this message was delivered earlier but that its receipt was not acknowledged at that time."
I ran the following code to explore what was available in my configuration (Spring Boot with IBM MQ).
#JmsListener(destination="DEV.QUEUE.1")
public void receive(Message message) throws Exception{
for (Enumeration<String> e = message.getPropertyNames(); e.hasMoreElements();)
System.out.println(e.nextElement());
}
From here I could find JMSXDeliveryCount is available in JMS 2.0. If that property is not available, then you may well find something similar for your own configuration.
One strategy would be to use JMSXDeliveryCount, a vendor specific property or maybe JMSRedelivered (if suitable for your needs) as a way to check before you process the message. Typically, the message would be sent to a specific blackout queue where the redelivery count exceeds a set threshold.
Depending on the messaging provider you are using it might also be possible to configure back out queue processing as properties of the queue.
Is it possible to perform an action before a jms message is received in spring boot? I know I could put it at the very top of my #JmsListener, but I have several listeners and I'd prefer avoiding adding a call to all of them.
I'm trying to use the logging MDC (a threadlocal, if you're not familiar with the MDC,) to track various things and I'd like to set some properties before beginning to process the message. I can do this on my controllers with a Filter, but does spring jms have the same concept?
I would try to start with an Before or Around (in case there should be some logic implemented after handling message as well) aspect:
#Before("#annotation(JmsListener)")
public void handle(ProceedingJoinPoint joinPoint) { ... }
#Around("#annotation(JmsListener)")
public void handle(ProceedingJoinPoint joinPoint) { ... }
Couple of links: enabling aspectj support, before advice, around advice.
With #Transaction and with trace level logging of Spring I see that Hibernate has an exception on a db constraint but it just rolls back the transaction. I tried using #Exception without success to try to catch it.
I see a suggestion online to use Spring AOP #AfterThrowing to catch such events.
This seems like a rather complex way to handle a commonplace event. It seems so much easier with try/catch and old fashioned commits/rollbacks. Is there no better way
in Spring Hibernate?
I'm processing XML messages from a queue. Depending on the type of the exception I get I might want to just catch it and put the bad incoming queue message into an db error table and continue the transaction. (If I just blindly rollback the message will get put back on the queue which might be good in some cases but not in others).
From what I've read there are ways of being made aware of some errors (for some reason not the constraint error) and logging them. Are there ways of interrupting the transaction after an exception and deciding if one wants to commit, continue or rollback?
If not, can one use old style commits and rollback with spring hibernate?
Configure SimpleMappingException resolver and log the exception there:
public class MyExceptionResolver extends SimpleMappingExceptionResolver {
private Logger logger = LoggerFactory.getLogger(MyExceptionResolver .class);
#Override
protected void logException(Exception ex, HttpServletRequest request) {
this.logger.warn(buildLogMessage(ex, request), ex);
}
<bean class="com.mypackage.MyExceptionResolver"/>
EDIT:
You can choose to do anything. The above class has nothing to do with rolling back. It's useful to intercept exceptions and do whatever you want with them.
By default, Spring rolls back for any RuntimeException.
If you want to configure rollback policy:
#Transactional(rollbackFor = { CustomException.class})
or
#Transactional(noRollBackFor= { CustomException.class})