EJB transactions slowing down in one thread - performance

Question

I have a performance problem with my process. It's an asynchronous task launched in CMT bean (on jboss server).
1 iteration performs 1 update and 3 inserts to my db via Hibernate. The process is divided into new transactions every 100 iterations.
Flush is called on EntityManager after every update/insert.
While the starting performance of first batch is satisfying (around 5-8s) it slows down drastically with time. The 30th batch takes around 30s to finish and later grows up to over 2 minutes per batch.
I tried switching FlushModeType to COMMIT, manual clearing/closing entityManagers, clearing entityManagers cache, I looked for memory leaks and can't find the reason for this slow down.
I measured little bits of code execution time and every code involving database connection slows down with time. I understand that a transaction slows with more entities processed but why is new transaction also slower?
The latest process consists of 250 000 iterations (2500 transactions in 1 thread) and takes forever to end.
If needed I'll provide more information. Any help would be appreciated.
I tried simplifying this code just to do 1 hibernate insert and no other operations and it still slows with time. This is an abstract pseudo view of what's going on inside.
Bean1
#Asynchronous
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public void mainTask(){
while(...){
subTask();
}
}
Bean2
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public void subTask(){
100.times{
3*Insert
1*Update
}
}

Answer 1

I am sure that my suggestions might not be accurate one or you tried them already, but I'd like to give a try. As you mentioned that DB connection is bottleneck, I'll go after that.
After reading the question, I find that time taken by transaction is proportional to iteration number. So it looks that entities created in first iteration are being sent to hibernate in next iteration.
For example, in 4th iteration, entities created in 1st,2nd and 3rd iterations are also being sent for update or being sent to hibernate somehow.
That could be the reason for degradation of performance as iteration progresses. As number of records to be updated/inserted/selected increases with each iteration.
I can think of following possibilities on top of my head -
Hibernate session used in first iteration is being used till end. Due to this, entities created in first iteration are being updated in later iterations as well. I read that you tried closing entity managers etc. But still, please check the place where sesssion is ending. You can create a new session in each transaction or delete the entities created in session after each transaction.
The list which filters records for each iteration is sending already processed records. That means the list should send records from 300 to 399 in 3rd iteration, but records from 0 to 399 are sent to transaction.
If you're using HQL, try using named query. Last time when I used Hibernate (about 8 months back), I noticed that when HQL is used number of objects loaded by hibernate are much more than named query.
Hibernate provides a way to print actual SQL query/parameters sent to DB. You can check actual query sent to DB. Link for this -
How to print a query string with parameter values when using Hibernate

Answer 2

We faced similar problems on couple of occasions during executing of batch programs with JPA.
Only solution we could find is use jdbc api for batch programs which involves lot of processing.

Answer 3

Turns out it's a java problem, we tested our application on different configurations and it works fine when our jboss runs on java 1.7 (on contrary to 1.6). Every batch finishes in about 5 seconds. We now stand against a choice to upgrade our java to 1.7 or to dig deeper and find what's wrong with our setup on java 1.6.