My method annotated with Spring's #Transactional executes in 1s by doing around 5K of inserts in 1 Postgres tables.
I use tableRepo.flush() and entityManager.clear() after every 10 entity inserts.
Here is the code:
#Service
public class ImportService {
...
#Transactional
public void import() {
for (int i = 0; i < 500; i++) {
for (int j = 0; j < 10; j++) {
Entity entity = new Entity();
entity.setValue(String.format("value %d - %d", i, j));
tableRepo.save(entity);
}
log.debug("Saved i = {}", i);
tableRepo.flush();
entityManager.flush();
entityManager.clear();
log.debug("Flushed and cleared i = {}", i);
}
}
}
#Service
public class MainService {
...
public void do() {
importService.import();
log.debug("Completed");
}
}
However, after that method is executed, Spring's proxy class that called that method, takes around 82s to commit all that into database.
These are the logs with relevant execution times:
[DEBUG] 14:02:54,236 ImportService - Saved i = 498
[DEBUG] 14:02:54,239 ImportService - Flushed and cleared i = 498
[DEBUG] 14:02:54,242 ImportService - Saved i = 499
[DEBUG] 14:02:54,245 ImportService - Flushed and cleared i = 499
[DEBUG] 14:02:54,245 org.hibernate.engine.transaction.internal.TransactionImpl:98 - committing
[DEBUG] 14:04:16,868 MainService - Completed
Why does it take 82s to commit those changes to database, is there anything I can do to make it faster?
For those who enter the same situation, I had a hidden trigger that was executed on Postgres side during commiting.
Since trigger function was executed 5000 times, once for each inserted row, and used unindexed selects, it prolonged the commit for such a long time
Related
Scenario : I need to modify maxAttempts value of #Retryable at runtime, so that the number of retries can be driven from database
#Service
public class PropertyHolder {
private int retryCount= 2;
public int retryCount() { return retryCount; }
#Scheduled(fixedRate=3600000) //Query DB and update retryCount every 60mins
private void updateRetryCount(int in) {
this.retryCount = 0; //Fetch retryCount from DB and update retryCount
}
}
#Service
public class SimpleService{
#Retryable( value={ Throwable.class }, maxAttemptsExpression="#{#propertyHolder.retryCount()}")
private void performTask() {
// do some opertion that throws Exception
}
}
PropertyHolder will update retryCount once in every 60 minutes.
This PropertHolder#retryCount needs to be wired to #Retryable in SimpleService#performTask .At present, it takes only the initial value of retryCount (2).Is this a right approach or Am I making some terrible mistake?
No; currently the expression is only evaluated during context initialization; there is an open feature request to add runtime evaluation.
https://github.com/spring-projects/spring-retry/issues/184
Currently you have to wire up your own interceptor with a mutable retry policy and configure it via the interceptor property in #Retryable.
I have next implementation of RecursiveAction, single purpose of this class - is to print from 0 to 9, but from different threads, if possible:
public class MyRecursiveAction extends RecursiveAction {
private final int num;
public MyRecursiveAction(int num) {
this.num = num;
}
#Override
protected void compute() {
if (num < 10) {
System.out.println(num);
new MyRecursiveAction(num + 1).fork();
}
}
}
And I thought that invoking awaitQuiescence will make current thread to wait until all tasks (submitted and forked) will be completed:
public class Main {
public static void main(String[] args) {
ForkJoinPool forkJoinPool = new ForkJoinPool();
forkJoinPool.execute(new MyRecursiveAction(0));
System.out.println(forkJoinPool.awaitQuiescence(5, TimeUnit.SECONDS) ? "tasks" : "time");
}
}
But I don't always get correct result, instead of printing 10 times, prints from 0 to 10 times.
But if I add helpQuiesce to my implementation of RecursiveAction:
public class MyRecursiveAction extends RecursiveAction {
private final int num;
public MyRecursiveAction(int num) {
this.num = num;
}
#Override
protected void compute() {
if (num < 10) {
System.out.println(num);
new MyRecursiveAction(num + 1).fork();
}
RecursiveAction.helpQuiesce();//here
}
}
Everything works fine.
I want to know for what actually awaitQuiescence waiting?
You get an idea of what happens when you change the System.out.println(num); to System.out.println(num + " " + Thread.currentThread());
This may print something like:
0 Thread[ForkJoinPool-1-worker-3,5,main]
1 Thread[main,5,main]
tasks
2 Thread[ForkJoinPool.commonPool-worker-3,5,main]
When awaitQuiescence detects that there are pending tasks, it helps out by stealing one and executing it directly. Its documentation says:
If called by a ForkJoinTask operating in this pool, equivalent in effect to ForkJoinTask.helpQuiesce(). Otherwise, waits and/or attempts to assist performing tasks until this pool isQuiescent() or the indicated timeout elapses.
Emphasis added by me
This happens here, as we can see, a task prints “main” as its executing thread. Then, the behavior of fork() is specified as:
Arranges to asynchronously execute this task in the pool the current task is running in, if applicable, or using the ForkJoinPool.commonPool() if not inForkJoinPool().
Since the main thread is not a worker thread of a ForkJoinPool, the fork() will submit the new task to the commonPool(). From that point on, the fork() invoked from a common pool’s worker thread will submit the next task to the common pool too. But awaitQuiescence invoked on the custom pool doesn’t wait for the completion of the common pool’s tasks and the JVM terminates too early.
If you’re going to say that this is a flawed API design, I wouldn’t object.
The solution is not to use awaitQuiescence for anything but the common pool¹. Normally, a RecursiveAction that splits off sub tasks should wait for their completion. Then, you can wait for the root task’s completion to wait for the completion of all associated tasks.
The second half of this answer contains an example of such a RecursiveAction implementation.
¹ awaitQuiescence is useful when you don’t have hands on the actual futures, like with a parallel stream that submits to the common pool.
Everything works fine.
No it does not, you got lucky that it worked when you inserted:
RecursiveAction.helpQuiesce();
To explain this let's slightly change your example a bit:
static class MyRecursiveAction extends RecursiveAction {
private final int num;
public MyRecursiveAction(int num) {
this.num = num;
}
#Override
protected void compute() {
if (num < 10) {
System.out.println(num);
new MyRecursiveAction(num + 1).fork();
}
}
}
public static void main(String[] args) {
ForkJoinPool forkJoinPool = new ForkJoinPool();
forkJoinPool.execute(new MyRecursiveAction(0));
LockSupport.parkNanos(TimeUnit.SECONDS.toNanos(2));
}
If you run this, you will notice that you get the result you expect to get. And there are two main reasons for this. First, fork method will execute the task in the common pool as the other answer already explained. And second, is that threads in the common pool are daemon threads. JVM is not waiting for them to finish before exiting, it exists early. So if that is the case, you might ask why it works. It does because of this line:
LockSupport.parkNanos(TimeUnit.SECONDS.toNanos(2));
which makes the main thread (which is a non daemon thread) sleep for two seconds, giving enough time for the ForkJoinPool to execute your task.
Now let's change the code closer to your example:
public static void main(String[] args) {
ForkJoinPool forkJoinPool = new ForkJoinPool();
forkJoinPool.execute(new MyRecursiveAction(0));
System.out.println(forkJoinPool.awaitQuiescence(5, TimeUnit.SECONDS) ? "tasks" : "time");
}
specifically, you use: forkJoinPool.awaitQuiescence(...), which is documented as:
Otherwise, waits and/or attempts to assist performing tasks...
It does not say that it will necessarily wait, it says it will "wait and/or attempt ...", in this case it is more or, than and. As such, it will attempt to help, but still it will not wait for all the tasks to finish. Is this weird or even stupid?
When you insert RecursiveAction.helpQuiesce(); you are eventually calling the same awaitQuiescence (with different arguments) under the hood - so essentially nothing changes; the fundamental problem is still there:
static ForkJoinPool forkJoinPool = new ForkJoinPool();
static AtomicInteger res = new AtomicInteger(0);
public static void main(String[] args) {
forkJoinPool.execute(new MyRecursiveAction(0));
System.out.println(forkJoinPool.awaitQuiescence(5, TimeUnit.SECONDS) ? "tasks" : "time");
System.out.println(res.get());
}
static class MyRecursiveAction extends RecursiveAction {
private final int num;
public MyRecursiveAction(int num) {
this.num = num;
}
#Override
protected void compute() {
if (num < 10_000) {
res.incrementAndGet();
System.out.println(num + " thread : " + Thread.currentThread().getName());
new MyRecursiveAction(num + 1).fork();
}
RecursiveAction.helpQuiesce();
}
}
When I run this, it never printed 10000, showing that the insertions of that line changes nothing.
The usual default way to handle such things is to fork then join. And one more join in the caller, on the ForkJoinTask that you get back when calling submit. Something like:
public static void main(String[] args) {
ForkJoinPool forkJoinPool = new ForkJoinPool(2);
ForkJoinTask<Void> task = forkJoinPool.submit(new MyRecursiveAction(0));
task.join();
}
static class MyRecursiveAction extends RecursiveAction {
private final int num;
public MyRecursiveAction(int num) {
this.num = num;
}
#Override
protected void compute() {
if (num < 10) {
System.out.println(num);
MyRecursiveAction ac = new MyRecursiveAction(num + 1);
ac.fork();
ac.join();
}
}
}
I am trying to configure a multithreaded environment through spring boot.
Service class
for (int i = 1; i <= 1000; j++) {
threadServiceMethod(i);
}
ThreadService class
public void threadServiceMethod(int i) {
System.out.println(i);
}
I confirmed that multithreading works by calling threadservice from service.
ex. console = 1 2 3 8 10 5 ....1000
But I want threadservice to select from DB.
So I modified the service.
Service class
for (int i = 1; i <= 30; j++) {
threadServiceMethod(i);
}
ThreadService class
public CompletableFuture<List<DTO>> threadServiceMethod(int i) {
return CompletableFuture.completedFuture(DAO method);
}
But multithreading doesn't seem to work. Because the logs are output in order.
I don't know if multithreading is working or not. I am using mybatis to make a request to the db.
What is the difference between method with #Transactional(propagation = Propagation.SUPPORTS) vs having no #Transactional annotation at all?
#Transactional(propagation = Propagation.SUPPORTS)
public void MyMethod()
vs
public void MyMethod()
Wouldn't the one without the annotation also use a transaction if one is already open, otherwise continue without any transaction?
There is a small difference. suppose we have two methods a() and b(), and a() is going to call b(). a itself is transnational and its propagation level is Propagation.REQUIRED but b one time is annotated with #Transactional(propagation = Propagation.SUPPORTS) and one time is not with annotation.
case 1:
#Transactional
public void a() {
for (int i = 0; i < 10; i++) {
try {
productRepository.b(i, "product " + i);
} catch (RuntimeException ex){
// do nothing
}
}
}
public void b(int id, String name) {
if(id > 5)
throw new RuntimeException();
String sql = "INSERT INTO test_table VALUES(?, ?)";
template.update(sql, id, name);
}
in case 1 we have aspect() -> a() -> b() and you are able to prevent RuntimeException to reach aspect and aspect inspects the transaction to see whether it is marked to rollback, so aspect consider this transaction successful and you can see we have this result in our database
0,product 0
1,product 1
2,product 2
3,product 3
4,product 4
5,product 5
even though multiple exceptions were thrown but we were able to commit the operations have been done so far.
now consider case 2:
#Transactional
public void a() {
for (int i = 0; i < 10; i++) {
try {
productRepository.b(i, "product " + i);
} catch (RuntimeException ex){
// do nothing
}
}
}
#Transactional(propagation = Propagation.SUPPORTS)
public void b(int id, String name) {
if(id > 5)
throw new RuntimeException();
String sql = "INSERT INTO test_table VALUES(?, ?)";
template.update(sql, id, name);
}
with propagation = Propagation.SUPPORTS if a transaction already exists it's going to use it. so if you throw an exception in b it is going to mark the same transaction to rollback and even if you use the try/catch block to prevent the RuntimeException to reach aspect in a() the transaction is already marked to rollback in b and you see no result in your database. aspect() -> a() -> aspect() -> b()
credit goes to Laurentiu Spilca, see this and read the comment section
From your link, it states that Propagation.SUPPORTS might have impact on
synchronization:
SUPPORTS is slightly different from no transaction at all, as it defines a transaction scope that synchronization will apply for. As a consequence, the same resources (JDBC Connection, Hibernate Session, etc) will be shared for the entire specified scope
Yes one without any annotation will use existing transaction if one is already open but what if there is not any won't you want to revert the changes if your transaction fails.
Propagation has also other attribute has well in case you method is not dependant of parent transaction and runs in individual transaction scope, probably you can go for REQUIRES_NEW this way you can have independent transaction for your method.
Is that possible to perform commit in the method that is marked as Spring's #Transactional?
#PersistenceContext
private EntityManager em;
#Transactional(propagation = Propagation.REQUIRED)
public void saveMembersWithMultipleCommits(List<Member> members)
throws HibernateException
{
Iterator<Member> it = members.iterator();
while (it.hasNext())
{
while (it.hasNext())
{
Member wsBean = it.next();
em.persist(wsBean); // overall commit will be made after method exit
log.info("Webservices record " + wsBean + " saved. " + i++);
}
}
}
I would like to have commit to DB after say each 500 items. Is that possible with aforementioned context?
No, you need to do it programatically using, for instance, the TransactionTemplate API. Read more here.
It would look something like
while (it.hasNext())
{
transactionTemplate.execute(new TransactionCallbackWithoutResult() {
protected void doInTransactionWithoutResult(TransactionStatus status) {
int counter = 0;
while (it.hasNext() && counter++ < 500) {
Member wsBean = it.next();
em.persist(wsBean);
log.info("Webservices record " + wsBean + " saved. " + i++);
}
}
);
}
Your question suggests that you have misplaced your transaction boundary.
You can move the persist call into a private method and make that method transactional instead of the outer one. This method could accept 500 members at a time and then will commit when it exits.
If you are looking forward to committing transactionally inside your other transaction, you might need to use #Transactional (propagation = Propagation.REQUIRES_NEW)
Alternate strategy is you create a method in DAO and mark it #Transactional. This method will do bulk update(for eg 500 nos). So you can have a method with code
#Transactional
public void mybatchUpdateMethod(){
StatelessSession session = this.hibernateTemplate.getSessionFactory()
.openStatelessSession();
Transaction transaction = null;
Long entryCounter = 0L;
PreparedStatement batchUpdate = null;
try {
transaction = session.beginTransaction();
batchUpdate = session.connection().prepareStatement(insertSql);
for (BatchSnapshotEntry entry : entries) {
entry.addEntry(batchUpdate);
batchUpdate.addBatch();
if (++entryCounter == 500) {
// Reached limit for uncommitted entries, so commit
batchUpdate.executeBatch();
}
}
batchUpdate.executeBatch();
batchUpdate.close();
batchUpdate = null;
}
catch (HibernateException ex) {
transaction.rollback();
transaction = null;
}
}
Every time you call this method, it will commit after 500 inserts/updates