How to execute a JPAQuery with pagination using Spring Data and QueryDSL - spring

I have this request working good with queryDSL :
Iterable<AO> query_result = new JPAQuery(entityManager).from(ao)
.leftJoin( ao.lots , lot )
.leftJoin( ao.acs , ac )
.where(where).distinct()
.list(ao);
But what is its equivalent if we use it with spring data jpa
ao_respository.findAll(Predicate arg0, Pageable arg1);
Because i want to return a Page and just with querydsl it doesn't implement Page without spring data jpa.
I try to put my where in Predicate arg0 but i got this exception
Undeclared path 'lot '. Add this path as a source to the query to be able to reference it
where lot is declared as QLot lot = QLot.lot;

I created my own Page class and executed the query like this:
JPAQuery query = new JPAQuery(entityManager).from(ao)
.leftJoin( .. ).fetch()
.leftJoin( .. ).fetch()
...
.where(where)
MaPage<AO> page = new MaPage<AO>();
page.number = pageNumber+1;
page.content = query.offset(pageNumber*pageSize).limit(pageSize).list(ao);
page.totalResult = query.count();
My Page class:
public class MaPage<T> {
public List<T> content;
public int number;
public Long totalResult;
public Long totalPages;
...
}
It works but I got this warning
nov. 21, 2014 6:48:54 AM
org.hibernate.hql.internal.ast.QueryTranslatorImpl list WARN:
HHH000104: firstResult/maxResults specified with collection fetch;
applying in memory!

Returning a Page:
JPAQuery query =
...
.orderBy(getOrderSpecifiers(pageable, MyEntity.class))
.limit(pageable.getPageSize())
.offset(pageable.getOffset());
long total = query.fetchCount();
List<MyEntity> content = query.fetch();
return new PageImpl<>(content, pageable, total);
And I created this function to get OrderSpecifier:
private OrderSpecifier[] getOrderSpecifiers(#NotNull Pageable pageable, #NotNull Class klass) {
// orderVariable must match the variable of FROM
String className = klass.getSimpleName();
final String orderVariable = String.valueOf(Character.toLowerCase(className.charAt(0))).concat(className.substring(1));
return pageable.getSort().stream()
.map(order -> new OrderSpecifier(
Order.valueOf(order.getDirection().toString()),
new PathBuilder(klass, orderVariable).get(order.getProperty()))
)
.toArray(OrderSpecifier[]::new);
}

If you have a working, complex query in querydsl and you want to use springdata pagination, you have to:
make your querydsl/repository method return Page<T>
Page<YourEntity> yourSelect(Pageable aPageable)
use querydsl offset and limit to page your result set
List<YourEntity> theResultList = jpaQueryFactory
.select(<whatever complext jpaquery you like>)
.offset(aPageable.getOffset())
.limit(aPageable.getPageSize())
.fetch();
provide a LongSuplier counting all available results with respect to your query and use PageableExecutionUtils to return the result as Page
final long theCount = jpaQueryFactory
.selectFrom(<your select to count all results>)
.fetchCount();
return PageableExecutionUtils.getPage(theResultList, aPageable, () -> theCount);

Related

InvalidPathException while sorting with org.springframework.data.domain.Pageable

I am trying to sort my table's content on the backend side, so I am sending org.springframework.data.domain.Pageable object to controller. It arrives correctly, but at the repository I am getting org.hibernate.hql.internal.ast.InvalidPathException. Somehow the field name I would use for sorting gets an org. package name infront of the filed name.
The Pageable object logged in the controller:
Page request [number: 0, size 10, sort: referenzNumber: DESC]
Exception in repository:
Invalid path: 'org.referenzNumber'","logger_name":"org.hibernate.hql.internal.ast.ErrorTracker","thread_name":"http-nio-8080-exec-2","level":"ERROR","level_value":40000,"stack_trace":"org.hibernate.hql.internal.ast.InvalidPathException: Invalid path: 'org.referenzNumber'\n\tat org.hibernate.hql.internal.ast.util.LiteralProcessor.lookupConstant(LiteralProcessor.java:111)
My controller endpoint:
#GetMapping(value = "/get-orders", params = { "page", "size" }, produces = { MediaType.APPLICATION_JSON_VALUE })
public ResponseEntity<PagedModel<KryptoOrder>> getOrders(
#ApiParam(name = "searchrequest", required = true) #Validated final OrderSearchRequest orderSearchRequest,
#PageableDefault(size = 500) final Pageable pageable, final BindingResult bindingResult,
final PagedResourcesAssembler<OrderVo> pagedResourcesAssembler) {
if (bindingResult.hasErrors()) {
return ResponseEntity.badRequest().build();
}
PagedModel<Order> orderPage = PagedModel.empty();
try {
var orderVoPage = orderPort.processOrderSearch(resourceMapper.toOrderSearchRequestVo(orderSearchRequest), pageable);
orderPage = pagedResourcesAssembler.toModel(orderVoPage, orderAssembler);
} catch (MissingRequiredField m) {
log.warn(RESPONSE_MISSING_REQUIRED_FIELD, m);
return ResponseEntity.badRequest().build();
}
return ResponseEntity.ok(orderPage);
}
the repository:
#Repository
public interface OrderRepository extends JpaRepository<Order, UUID> {
static final String SEARCH_ORDER = "SELECT o" //
+ " FROM Order o " //
+ " WHERE (cast(:partnerernumber as org.hibernate.type.IntegerType) is null or o.tradeBasis.account.retailpartner.partnerbank.partnerernumber = :partnerernumber)"
+ " and (cast(:accountnumber as org.hibernate.type.BigDecimalType) is null or o.tradeBasis.account.accountnumber = :accountnumber)"
+ " and (cast(:orderReference as org.hibernate.type.LongType) is null or o.tradeBasis.referenceNumber = :orderReference)"
+ " and (cast(:orderReferenceExtern as org.hibernate.type.StringType) is null or o.tradeBasis.kundenreferenceExternesFrontend = :orderReferenceExtern)"
+ " and (cast(:dateFrom as org.hibernate.type.DateType) is null or o.tradeBasis.timestamp > :dateFrom) "
+ " and (cast(:dateTo as org.hibernate.type.DateType) is null or o.tradeBasis.timestamp < :dateTo) ";
#Query(SEARCH_ORDER)
Page<Order> searchOrder(#Param("partnerernumber") Integer partnerernumber,
#Param("accountnumber") BigDecimal accountnumber, #Param("orderReference") Long orderReference,
#Param("orderReferenceExtern") String orderReferenceExtern, #Param("dateFrom") LocalDateTime dateFrom,
#Param("dateTo") LocalDateTime dateTo, Pageable pageable);
}
Update:
I removed the parameters from the sql query, and put them back one by one to see where it goes sideways. It seems as soon as the dates are involved the wierd "org." appears too.
Update 2:
If I change cast(:dateTo as org.hibernate.type.DateType) to cast(:dateFrom as date) then it appends the filed name with date. instead of org..
Thanks in advance for the help
My guess is, Spring Data is confused by the query you are using and can't properly append the order by clause to it. I would recommend you to use a Specification instead for your various filters. That will not only improve the performance of your queries because the database can better optimize queries, but will also make use of the JPA Criteria API behind the scenes, which requires no work from Spring Data to apply an order by specification.
Since your entity Order is named as the order by clause of HQL/SQL, my guess is that Spring Data tries to do something stupid with the string to determine the alias of the root entity.

Memory leak with Criteria API Pageable

I implemented pageable functionality into Criteria API query and I noticed increased memory usage during query execution. I also used spring-data-jpa method query to return same result, but there memory is cleaned up after every batch is processed. I tried detaching, flushing, clearing objects from EntityManager, but memory use would keep going up, occasionally it will drop but not as much as with method queries. My question is what could cause this memory use if objects are detached and how to deal with it?
Memory usage with Criteria API pageable:
Memory usage with method query:
Code
Since I'm also updating entities retrieved from DB, I use approach where I save ID of last processed entity, so when entity gets updated query doesen't skip next selected page. Below I provide code example that is not from real app I'm working on, but it just recreation of the issue I'm having.
Repository code:
#Override
public Slice<Player> getPlayers(int lastId, Pageable pageable) {
List<Predicate> predicates = new ArrayList<>();
CriteriaBuilder criteriaBuilder = entityManager.getCriteriaBuilder();
CriteriaQuery<Player> criteriaQuery = criteriaBuilder.createQuery(Player.class);
Root<Player> root = criteriaQuery.from(Player.class);
predicates.add(criteriaBuilder.greaterThan(root.get("id"), lastId));
criteriaQuery.where(criteriaBuilder.and(predicates.toArray(Predicate[]::new)));
criteriaQuery.orderBy(criteriaBuilder.asc(root.get("id")));
var query = entityManager.createQuery(criteriaQuery);
if (pageable.isPaged()) {
int pageSize = pageable.getPageSize();
int offset = pageable.getPageNumber() > 0 ? pageable.getPageNumber() * pageSize : 0;
// Fetch additional element and skip it based on the pageSize to know hasNext value.
query.setMaxResults(pageSize + 1);
query.setFirstResult(offset);
var resultList = query.getResultList();
boolean hasNext = pageable.isPaged() && resultList.size() > pageSize;
return new SliceImpl<>(hasNext ? resultList.subList(0, pageSize) : resultList, pageable, hasNext);
} else {
return new SliceImpl<>(query.getResultList(), pageable, false);
}
}
Iterating through pageables:
#Override
public Slice<Player> getAllPlayersPageable() {
int lastId = 0;
boolean hasNext = false;
Pageable pageable = PageRequest.of(0, 200);
do {
var players = playerCriteriaRepository.getPlayers(lastId, pageable);
if(!players.isEmpty()){
lastId = players.getContent().get(players.getContent().size() - 1).getId();
for(var player : players){
System.out.println(player.getFirstName());
entityManager.detach(player);
}
}
hasNext = players.hasNext();
} while (hasNext);
return null;
}
I think you are running into a query plan cache issue here that is related to the use of the JPA Criteria API and how numeric values are handled. Hibernate will render all numeric values as literals into an intermediary HQL query string which is then compiled. As you can imagine, every "scroll" to the next page will be a new query string so you gradually fill up the query plan cache.
One possible solution is to use a library like Blaze-Persistence which has a custom JPA Criteria API implementation and a Spring Data integration that will avoid these issues and at the same time improve the performance of your queries due to a better pagination implementation.
All your code would stay the same, you just have to include the integration and configure it as documented in the setup section.

JPA Criteria api - Total records for concrete query within pagination

I am programming function for pagination in my repository layer. Function receive as parameters spring's pageable object and some value like this:
public Page<Foo> filterFoo(Pageable pageable, String value) {
CriteriaBuilder cb = entityManager.getCriteriaBuilder();
CriteriaQuery<Foo> fooQuery = cb.createQuery(Foo.class);
Root<Foo> foo = fooQuery .from(Foo.class);
fooQuery .where(adding predicate for match value);
List<Foo> result = entityManager.createQuery(fooQuery )
.setFirstResult((pageable.getPageNumber() - 1) * pageable.getPageSize())
.setMaxResults(pageable.getPageSize())
.getResultList();
return new PageImpl<>(result, pageable, xxxx);
}
Function return spring's PageImpl object filled with my result. To PageImpl I also need set total count of objects which suit predicates. This count number have to be of course without maxResult and firstResult. Is possible create another database call with my fooQuery to get total database records for that query without limit? What is the best practise to use pageable and criteria api in JPA? Thank you in advice.
Because generated SQL uses aliases - you may need make separate query for get total count of rows.
For example:
CriteriaQuery<Long> countQuery = cb.createQuery(Long.class);
countQuery.select(cb.count(countQuery.from(Foo.class)));
if (Objects.nonNull(filters)) {
countQuery.where(filters);
}
return new PageImpl<>(result, pageable, em.createQuery(countQuery).getSingleResult());
where filters is equal to your adding predicate for match value expression.
Also, you may use a TupleQuery with custom SQL function for calculate count of rows in one select query.
Like this:
public class SqlFunctionsMetadataBuilderContributor implements MetadataBuilderContributor {
#Override
public void contribute(MetadataBuilder metadataBuilder) {
metadataBuilder.applySqlFunction(
"count_over",
new SQLFunctionTemplate(
StandardBasicTypes.LONG,
"(count(?1) over())"
)
);
}
}
and Criteria:
public Page<Foo> findAll(Specification<Foo> specification, Pageable pageable) {
CriteriaQuery<Tuple> cq = cb.createTupleQuery();
Root<Foo.class> fooRoot = cq.from(Foo.class);
cq.select(cb.tuple(fooRoot, cb.function("count_over", Long.class, fooRoot.get("id"))));
Predicate filters = specification.toPredicate(fooRoot, cq, cb);
if (Objects.nonNull(filters)) {
cq.where(filters);
}
TypedQuery<Tuple> query = em.createQuery(cq);
query.setFirstResult((int) pageable.getOffset());
query.setMaxResults(pageable.getPageSize());
List<Tuple> result = query.getResultList();
if (result.isEmpty()) {
return new PageImpl<>(List.of());
}
return new PageImpl<>(
result.stream().map(tuple -> (Foo) tuple.get(0)).collect(toUnmodifiableList()),
pageable,
(long) result.get(0).get(1)
);
}
See more about SQLFunction: https://vladmihalcea.com/hibernate-sql-function-jpql-criteria-api-query/ and Custom SQL for Order in JPA Criteria API

In spring data mongodb how to achieve pagination for aggregation

In spring data mongodb using mongotemplate or mongorepository, how to achieve pagination for aggregateion
This is an answer to an old post, but I'll provide an answer in case anyone else comes along while searching for something like this.
Building on the previous solution by Fırat KÜÇÜK, giving the results.size() as the value for the "total" field in the PageImpl constructor will not making paging work the way, well, you expect paging to work. It sets the total size to the page size every time, so instead, you need to find out the actual total number of results that your query would return:
public Page<UserListItemView> list(final Pageable pageable) {
long total = getCount(<your property name>, <your property value>);
final Aggregation agg = newAggregation(
skip(pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize())
);
final List<UserListItemView> results = mongoTemplate
.aggregate(agg, User.class, UserListItemView.class)
.getMappedResults();
return new PageImpl<>(results, pageable, total);
}
Now, then, the best way to get the total number of results is another question, and it is one that I am currently trying to figure out. The method that I tried (and it worked) was to almost run the same aggregation twice, (once to get the total count, and again to get the actual results for paging) but using only the MatchOperation followed by a GroupOperation to get the count:
private long getCount(String propertyName, String propertyValue) {
MatchOperation matchOperation = match(Criteria.where(propertyName).is(propertyValue));
GroupOperation groupOperation = group(propertyName).count().as("count");
Aggregation aggregation = newAggregation(matchOperation, groupOperation);
return mongoTemplate.aggregate(aggregation, Foo.class, NumberOfResults.class).getMappedResults().get(0).getCount();
}
private class NumberOfResults {
private int count;
public int getCount() {
return count;
}
public void setCount(int count) {
this.count = count;
}
}
It seems kind of inefficient to run nearly the same query twice, but if you are going to page results, the pageable object must know the total number of results if you really want it to behave like paging. If anyone can improve on my method to get the total count of results, that would be awesome!
Edit: This will also provide the count, and it is simpler because you do not need a wrapper object to hold the result, so you can replace the entire previous code block with this one:
private long getCount(String propertyName, String propertyValue) {
Query countQuery = new Query(Criteria.where(propertyName).is(propertyValue));
return mongoTemplate.count(countQuery, Foo.class);
}
In addition to ssouris solution you can use Pageable classes for the results.
public Page<UserListItemView> list(final Pageable pageable) {
final Aggregation agg = newAggregation(
skip(pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize())
);
final List<UserListItemView> results = mongoTemplate
.aggregate(agg, User.class, UserListItemView.class)
.getMappedResults();
return new PageImpl<>(results, pageable, results.size())
}
You can use MongoTemplate
org.spring.framework.data.mongodb.core.aggregation.Aggregation#skip
and
org.springframework.data.mongodb.core.aggregation.Aggregation#limit
Aggregation agg = newAggregation(
project("tags"),
skip(10),
limit(10)
);
AggregationResults<TagCount> results = mongoTemplate.aggregate(agg, "tags", TagCount.class);
List<TagCount> tagCount = results.getMappedResults();
As per the answer https://stackoverflow.com/a/39784851/4546949 I wrote code for Java.
Use aggregation group to get count and array of data with other paging information.
AggregationOperation group = Aggregation.group().count().as("total")
.addToSet(pageable.getPageNumber()).as("pageNumber")
.addToSet(pageable.getPageSize()).as("pageSize")
.addToSet(pageable.getOffset()).as("offset")
.push("$$ROOT").as("data");
Use Aggregation project to slice as per the paging information.
AggregationOperation project = Aggregation.project()
.andInclude("pageSize", "pageNumber", "total", "offset")
.and(ArrayOperators.Slice.sliceArrayOf("data").offset((int) pageable.getOffset()).itemCount(pageable.getPageSize()))
.as("data");
Use mongo template to aggregate.
Aggregation aggr = newAggregation(group, project);
CustomPage page = mongoTemplate.aggregate(aggregation, Foo.class, CustomPage.class).getUniqueMappedResult();
Create a CustomPage.
public class CustomPage {
private long pageSize;
private long pageNumber;
private long offset;
private long total;
private List<Foo> data;
}
Here is my generic solution:
public Page<ResultObject> list(Pageable pageable) {
// build your main stages
List<AggregationOperation> mainStages = Arrays.asList(match(....), group(....));
return pageAggregation(pageable, mainStages, "target-collection", ResultObject.class);
}
public <T> Page<T> pageAggregation(
final Pageable pageable,
final List<AggregationOperation> mainStages,
final String collection,
final Class<T> clazz) {
final List<AggregationOperation> stagesWithCount = new ArrayList<>(mainStages);
stagesWithCount.add(count().as("count"));
final Aggregation countAgg = newAggregation(stagesWithCount);
final Long count = Optional
.ofNullable(mongoTemplate.aggregate(countAgg, collection, Document.class).getUniqueMappedResult())
.map(doc -> ((Integer) doc.get("count")).longValue())
.orElse(0L);
final List<AggregationOperation> stagesWithPaging = new ArrayList<>(mainStages);
stagesWithPaging.add(sort(pageable.getSort()));
stagesWithPaging.add(skip(pageable.getOffset()));
stagesWithPaging.add(limit(pageable.getPageSize()));
final Aggregation resultAgg = newAggregation(stagesWithPaging);
final List<T> result = mongoTemplate.aggregate(resultAgg, collection, clazz).getMappedResults();
return new PageImpl<>(result, pageable, count);
}
To return a Paged Object with correct value of pageable object , I find this is the best and simple way.
Aggregation aggregation = Aggregation.newAggregation(Aggregation.match(Criteria.where("type").is("project")),
Aggregation.group("id").last("id").as("id"), Aggregation.project("id"),
Aggregation.skip(pageable.getPageNumber() * pageable.getPageSize()),
Aggregation.limit(pageable.getPageSize()));
PageableExecutionUtils.getPage(mongoTemplate.aggregate(aggregation, Draft.class, Draft.class).getMappedResults(), pageable,() -> mongoTemplate.count(Query.of(query).limit(-1).skip(-1), Draft.class));
Another approach would be to extend the PagingAndSortingRepository<T, ID> interface. Then, you can create an #Aggregation query method like this:
#Aggregation(pipeline = {
"{ $match: { someField: ?0 } }",
"{ $project: { _id: 0, someField: 1} }"
})
List<StuffAggregateModel> aggregateStuff(final String somePropertyName, final Pageable pageable);
Just call this from your business logic service class and construct the Pageable (which also contains sort options, if desired) and call the repo method. I like this approach because of the simplicity and the sheer minimization of the amount of code that you have to write. If your query (aggregation pipeline) is simple enough, this is probably the best solution. Maintenance coding for this approach is nearly effortless.
My answer with MongoDB $facet
// User(_id, first name, etc), Car (user_id, brand, etc..)
LookupOperation lookupStageCar = Aggregation.lookup(‘cars ’, ‘user_id’, ‘_id’, ‘car’);
MatchOperation matchStage = Aggregation.match(Criteria.where(‘car.user_id ‘).exists(true));
CountOperation countOperation = Aggregation.count().as("total");
AddFieldsOperation addFieldsOperation = Aggregation.addFields().addFieldWithValue("page", pageable.getPageNumber()).build();
SkipOperation skipOperation = Aggregation.skip(Long.valueOf(pageable.getPageNumber() * pageable.getPageSize()));
LimitOperation limitOperation = Aggregation.limit(pageable.getPageSize());
// here the magic
FacetOperation facetOperation = Aggregation.facet( countOperation, addFieldsOperation).as("metadata")
.and(skipOperation, limitOperation).as("data");
// users with car
List<AggrigationResults> map = mongoTemplate.aggregate(Aggregation.newAggregation( lookupStageCar, matchStage, facetOperation), "User", AggrigationResults.class).getMappedResults();
———————————————————————————
public class AggrigationResults {
private List<Metadata> metadata;
private List<User> data;
}
public class Metadata {
private long total;
private long page;
}
———————————————————————————
output:
{
"metadata" : [
{
"total" : 300,
"page" : 3
}
],
"data" : [
{
... original document ...
},
{
... another document ...
},
{
... etc up to 10 docs ...
}
]
}
see : How to use MongoDB aggregation for pagination?

How to set a Http Session parameter in a JPA named query

I want to set a parameter in a named query (JPA 2.0), so my dataTable would render the respective dataSet. The parameter is obtained remotely and injected in a AbstractFacade class.
I've tried to achieve this through the code above, but it's not working.
Can someone help me?
AbstractFacade (main code):
private String prefDep;
public List<T> findByPrefDep() {
prefDep= FacesContext.getCurrentInstance().getExternalContext().getSessionMap().get("xPrefDep");
javax.persistence.criteria.CriteriaQuery cq = getEntityManager().getCriteriaBuilder().createQuery();
cq.select(cq.from(entityClass));
return getEntityManager().createQuery(cq).setParameter("prefDep", prefDep).getResultList();
}
The Entity class (main code):
#NamedQuery(name = "Capacitacao.findByPrefDep", query = "SELECT c FROM Capacitacao c WHERE c.prefDep = :prefDep"),
The AbstractController:
public Collection<T> getItems() {
if (items == null) {
items = this.ejbFacade.findByPrefDep();
}
return items;
}
There is no exception launched, but the dataSet rendered corresponds to a findAll named query.
Thanks in advance.
Your code doesn't use your named query at all. A named query has a name, and your code doesn't use that name anywhere.
Use
getEntityManager().createNamedQuery("Capacitacao.findByPrefDep", Capacitacao.class)
.setParameter("prefDep", prefDep)
.getResultList();
You could have found that yourself by simply reading the EntityManager javadoc.

Resources