JPARepository deleteAllInBatch() not working as expected - spring

Question

I am trying to delete some rows from my table using Spring JPA deleteAllInBatch() but when the number of rows to be deleted exceed some threshold value JPA throws error. I'm not certain the cause of this error but found a jira ticket : https://jira.spring.io/browse/DATAJPA-137.
I don't want to use deleteAll() as it deletes the data one by one and will lead to performance issues. Is this a drawback of JPA or there's some solution to it. I tried for some workaround but didn't found anything much useful. Please help me to get an efficient solution for this operation or some useful references. Thanks in advance...
DbIssueApplication.java
#SpringBootApplication
public class DbIssueApplication
{
public static void main(String[] args)
{
ApplicationContext context = SpringApplication.run(DbIssueApplication.class, args);
TestService service = context.getBean(TestService.class);
long st = System.currentTimeMillis();
List<Test> testList = new ArrayList<>();
for(int i=0;i<5000;i++)
{
testList.add(new Test(i,(i%2==0)?"field1":"field2"));
}
service.insert(testList);
service.deleteByName("field2");
System.err.println("The processing took = "+(System.currentTimeMillis()-st)+" ms");
}
}
Test.java
#Entity
#Table(name="test")
public class Test implements Serializable
{
private static final long serialVersionUID = -9182756617906316269L;
#Id
private Integer id;
private String name;
... getter,setter and constructors
}
TestRepository.java
public interface TestRepository extends JpaRepository<Test, Integer>
{
List<Test> findByName(String name);
}
TestService.java
public interface TestService
{
public void insert(List<Test> testList);
public void deleteByName(String name);
}
TestServiceImpl.java
#Service
public class TestServiceImpl implements TestService
{
#Autowired
TestRepository testRepository;
#Override
public void insert(List<Test> testList)
{
testRepository.deleteAllInBatch();
testRepository.saveAll(testList);
}
#Override
public void deleteByName(String name)
{
System.err.println("The number of rows to be deleted = "+testRepository.findByName(name).size());
testRepository.deleteInBatch(testRepository.findByName(name));
}
}
dbSchema
create table test
(
id int,
name varchar(40)
);
ErrorLog
[ main] o.h.e.t.i.TransactionImpl : begin
[ main] o.h.h.i.a.QueryTranslatorImpl : parse() - HQL: delete from com.example.demo.entity.Test x where x = ?1 or x = ?2 or x = ?3 or x = ?4 or ... x = ?2500
[ main] o.h.h.i.a.ErrorTracker : throwQueryException() : no errors
[ main] o.h.e.t.i.TransactionImpl : rolling back
[ Thread-14] o.h.i.SessionFactoryImpl : HHH000031: Closing
[ Thread-14] o.h.t.s.TypeConfiguration$Scope : Un-scoping TypeConfiguration [org.hibernate.type.spi.TypeConfiguration$Scope#6cf001] from SessionFactory [org.hibernate.internal.SessionFactoryImpl#1ad3d8a]
[ Thread-14] o.h.s.i.AbstractServiceRegistryImpl : Implicitly destroying ServiceRegistry on de-registration of all child ServiceRegistries
[ Thread-14] o.h.b.r.i.BootstrapServiceRegistryImpl : Implicitly destroying Boot-strap registry on de-registration of all child ServiceRegistries
=======================================================================================================================================================================================================================================================================================================
[ main] o.h.h.i.QueryTranslatorFactoryInitiator : QueryTranslatorFactory : org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory#5e167a
[ main] o.h.h.i.QueryTranslatorFactoryInitiator : HHH000397: Using ASTQueryTranslatorFactory
[ main] o.h.h.i.a.QueryTranslatorImpl : parse() - HQL: select generatedAlias0 from com.example.demo.entity.Test as generatedAlias0 where generatedAlias0.name=:param0
[ main] o.h.h.i.a.ErrorTracker : throwQueryException() : no errors
[ main] o.h.h.i.a.QueryTranslatorImpl : --- HQL AST ---
\-[QUERY] Node: 'query'
+-[SELECT_FROM] Node: 'SELECT_FROM'
| +-[FROM] Node: 'from'
| | \-[RANGE] Node: 'RANGE'
| | +-[DOT] Node: '.'
| | | +-[DOT] Node: '.'
| | | | +-[DOT] Node: '.'
| | | | | +-[DOT] Node: '.'
| | | | | | +-[IDENT] Node: 'com'
| | | | | | \-[IDENT] Node: 'example'
| | | | | \-[IDENT] Node: 'demo'
| | | | \-[IDENT] Node: 'entity'
| | | \-[IDENT] Node: 'Test'
| | \-[ALIAS] Node: 'generatedAlias0'
| \-[SELECT] Node: 'select'
| \-[IDENT] Node: 'generatedAlias0'
\-[WHERE] Node: 'where'
\-[EQ] Node: '='
+-[DOT] Node: '.'
| +-[IDENT] Node: 'generatedAlias0'
| \-[IDENT] Node: 'name'
\-[COLON] Node: ':'
\-[IDENT] Node: 'param0'
[ main] o.h.h.i.a.HqlSqlBaseWalker : select << begin [level=1, statement=select]
[ main] o.h.h.i.a.t.FromElement : FromClause{level=1} : com.example.demo.entity.Test (generatedAlias0) -> test0_
[ main] o.h.h.i.a.t.FromReferenceNode : Resolved : generatedAlias0 -> test0_.id
[ main] o.h.h.i.a.t.FromReferenceNode : Resolved : generatedAlias0 -> test0_.id
[ main] o.h.h.i.a.t.DotNode : getDataType() : name -> org.hibernate.type.StringType#d003cd
[ main] o.h.h.i.a.t.FromReferenceNode : Resolved : generatedAlias0.name -> test0_.name
[ main] o.h.h.i.a.HqlSqlBaseWalker : select : finishing up [level=1, statement=select]
[ main] o.h.h.i.a.HqlSqlWalker : processQuery() : ( SELECT ( {select clause} test0_.id ) ( FromClause{level=1} test test0_ ) ( where ( = ( test0_.name test0_.id name ) ? ) ) )
[ main] o.h.h.i.a.u.JoinProcessor : Using FROM fragment [test test0_]
[ main] o.h.h.i.a.HqlSqlBaseWalker : select >> end [level=1, statement=select]
[ main] o.h.h.i.a.QueryTranslatorImpl : --- SQL AST ---
\-[SELECT] QueryNode: 'SELECT' querySpaces (test)
+-[SELECT_CLAUSE] SelectClause: '{select clause}'
| +-[ALIAS_REF] IdentNode: 'test0_.id as id1_0_' {alias=generatedAlias0, className=com.example.demo.entity.Test, tableAlias=test0_}
| \-[SQL_TOKEN] SqlFragment: 'test0_.name as name2_0_'
+-[FROM] FromClause: 'from' FromClause{level=1, fromElementCounter=1, fromElements=1, fromElementByClassAlias=[generatedAlias0], fromElementByTableAlias=[test0_], fromElementsByPath=[], collectionJoinFromElementsByPath=[], impliedElements=[]}
| \-[FROM_FRAGMENT] FromElement: 'test test0_' FromElement{explicit,not a collection join,not a fetch join,fetch non-lazy properties,classAlias=generatedAlias0,role=null,tableName=test,tableAlias=test0_,origin=null,columns={,className=com.example.demo.entity.Test}}
\-[WHERE] SqlNode: 'where'
\-[EQ] BinaryLogicOperatorNode: '='
+-[DOT] DotNode: 'test0_.name' {propertyName=name,dereferenceType=PRIMITIVE,getPropertyPath=name,path=generatedAlias0.name,tableAlias=test0_,className=com.example.demo.entity.Test,classAlias=generatedAlias0}
| +-[ALIAS_REF] IdentNode: 'test0_.id' {alias=generatedAlias0, className=com.example.demo.entity.Test, tableAlias=test0_}
| \-[IDENT] IdentNode: 'name' {originalText=name}
\-[NAMED_PARAM] ParameterNode: '?' {name=param0, expectedType=org.hibernate.type.StringType#d003cd}
[ main] o.h.h.i.a.ErrorTracker : throwQueryException() : no errors
[ main] o.h.h.i.a.QueryTranslatorImpl : HQL: select generatedAlias0 from com.example.demo.entity.Test as generatedAlias0 where generatedAlias0.name=:param0
[ main] o.h.h.i.a.QueryTranslatorImpl : SQL: select test0_.id as id1_0_, test0_.name as name2_0_ from test test0_ where test0_.name=?
[ main] o.h.h.i.a.ErrorTracker : throwQueryException() : no errors
[ main] o.h.h.i.a.QueryTranslatorImpl : parse() - HQL: delete from com.example.demo.entity.Test x where x = ?1 or x = ?2 ... or x = ?2500
[ main] o.h.h.i.a.ErrorTracker : throwQueryException() : no errors
The Sample code is uploaded in github the link to which is : https://github.com/Anand450623/Stackoverflow

Answer 1

You could try using a JPQL query to make deleteAll() delete in batch rather than one by one.
However, you might actually want to drop the orm-framework entirely. A common experience is that even though it looks like a good idea in the beginning, it almost turns up with issues like the one you gave here :/ you could read https://www.toptal.com/java/how-hibernate-ruined-my-career The gist of it is: it's hard to debug, you can't avoid writing native SQL in most cases anyway, JPQL limits your expressiveness and it's extremely invasive in how you model (e.g. you can't do immutability in a lot of casee).
Spring has an excellent JdbcTemplate, but keep in mind that also have drawbacks, mainly that you have to write the mapping yourself - that said it's not that much code. That said the benefits are huge. So if a JPQL query doesn't work, consider if using JPA (hibernate?) is the right choice to begin with

Answer 2

The way I got around this was to implement my own deleteInBatch method with native sql,
something like
#Modifying
#Query(value="delete from table_x where id in (:ids)", nativeQuery=true)
void deleteBatch(List<String> ids);
The problem is this piece of code over here https://github.com/spring-projects/spring-data-jpa/blame/a31c39db7a12113b5adcb6fbaa2a92d97f1b3a02/src/main/java/org/springframework/data/jpa/repository/query/QueryUtils.java#L409
Which generates a awfull sql, not suited for large number of elements