Frameworks: Spring 4.0.7 and Hibernate 4.3.6
We are having problems with a PagingAndSortingRepository #Query taking an excessive amount of time to complete as it appears to be running through recordset data and building structures before factoring in page data limits.
In non-joined scenarios, the fetch first n rows appears as it should, and the resultset is much smaller making the selection very snappy on the first batch of pages.
The initial problem we ran into was that we were unable to do a #Query with an inner join fetch (to force the join) on detail records. Removing the fetch obviously gets us off the ground, but creates a nested SQL loop which is not desirable.
To get around this, we added countQuery which got us the proper count back, but now appears to load the entire structure in memory before returning the paged results. Turning on debugging shows the records being loaded in. The fetch first n rows has not been added.
Having nested SQLs running in a loop is not an option for us, so removing the fetch does not help.
#Query(value = "select e from InvoiceHistoryHeader e inner join fetch e.details f where e.company = ?1 and e.division = ?2 and e.customerNumber = ?3)", countQuery = "select count(e) from InvoiceHistoryHeader e where e.company = ?1 and e.division = ?2 and e.customerNumber = ?3)")
Page<InvoiceHistoryHeader> findByCustomerNumber(String company, String division, String customerNumber, Pageable pageable);
Can anyone provide a solution to this problem, or give a workaround that may help? I can provide more detail as needed.
EDIT (11/11/2014)
Hibernate output (no fetch):
Hibernate: select count(trim(invoicehis0_.HH_TID)) as col_0_0_ from invhh invoicehis0_ where trim(invoicehis0_.HH_CO)=? and trim(invoicehis0_.HH_DIV)=? and trim(invoicehis0_.HH_CUS)=? and trim(invoicehis0_.HH_BR)=?
Hibernate: select invoicehis0_.hh_tid as hh_tid1_7_, trim(invoicehis0_.HH_CO) as hh_co2_7_, trim(invoicehis0_.HH_CUS) as hh_cus3_7_, trim(invoicehis0_.HH_DTI) as hh_dti4_7_, trim(invoicehis0_.HH_DIV) as hh_div5_7_, trim(invoicehis0_.HH_ORD) as hh_ord6_7_, trim(invoicehis0_.HH_BR) as hh_br7_7_ from invhh invoicehis0_ where trim(invoicehis0_.HH_CO)=? and trim(invoicehis0_.HH_DIV)=? and trim(invoicehis0_.HH_CUS)=? and trim(invoicehis0_.HH_BR)=? fetch first 20 rows only
Hibernate: select details0_.hd_tid as hd_tid2_7_0_, trim(details0_.HD_SEQ) as hd_seq1_6_0_, trim(details0_.HD_TID) as hd_tid2_6_0_, details0_.hd_seq as hd_seq1_6_1_, details0_.hd_tid as hd_tid2_6_1_, trim(details0_.HD_DTL) as hd_dtl3_6_1_, trim(details0_.HD_LIN) as hd_lin4_6_1_ from invhd details0_ where details0_.hd_tid=?
Hibernate: select details0_.hd_tid as hd_tid2_7_0_, trim(details0_.HD_SEQ) as hd_seq1_6_0_, trim(details0_.HD_TID) as hd_tid2_6_0_, details0_.hd_seq as hd_seq1_6_1_, details0_.hd_tid as hd_tid2_6_1_, trim(details0_.HD_DTL) as hd_dtl3_6_1_, trim(details0_.HD_LIN) as hd_lin4_6_1_ from invhd details0_ where details0_.hd_tid=?
<above two SQL repeated 18 more times>
Hibernate output (fetch and countQuery):
Hibernate: select invoicehis0_.hh_tid as hh_tid1_7_0_, details1_.hd_seq as hd_seq1_6_1_, details1_.hd_tid as hd_tid2_6_1_, trim(invoicehis0_.HH_CO) as hh_co2_7_0_, trim(invoicehis0_.HH_CUS) as hh_cus3_7_0_, trim(invoicehis0_.HH_DTI) as hh_dti4_7_0_, trim(invoicehis0_.HH_DIV) as hh_div5_7_0_, trim(invoicehis0_.HH_ORD) as hh_ord6_7_0_, trim(invoicehis0_.HH_BR) as hh_br7_7_0_, trim(details1_.HD_DTL) as hd_dtl3_6_1_, trim(details1_.HD_LIN) as hd_lin4_6_1_, details1_.hd_tid as hd_tid2_7_0__, trim(details1_.HD_SEQ) as hd_seq1_6_0__, trim(details1_.HD_TID) as hd_tid2_6_0__ from invhh invoicehis0_ inner join invhd details1_ on trim(invoicehis0_.HH_TID)=details1_.hd_tid where trim(invoicehis0_.HH_CO)=? and trim(invoicehis0_.HH_DIV)=? and trim(invoicehis0_.HH_CUS)=? and trim(invoicehis0_.HH_BR)=?
When using fetch in join and paging Hibernate does in-memory pagination, which means it loads all records in memory.
Do not use fetch joins with paging.
Related
I am trying to do the mappings and trying to write a non-native HQL query by joining the 4 tables. The logic is written in stored procedure, but we want to migrate it to hibernate/JPA. I am unable to do proper mappings and write a query to create same logic
FROM
[dbo].[vwInstitutionUser]AS IU
INNER JOIN [dbo].[vwCommonCourse] AS C
ON C.CustomerID=CAST(IU.InstitutionID AS VARCHAR(10))
INNER JOIN [dbo].[vwCommonCourses] AS CM
ON C.CourseID = CM.CourseID
INNER JOIN dbo.vwProductMaster AS PM
ON CM.LearningActivityID = PM.ResourceID
OR CM.AssessmentID = PM.AssessmentID
WHERE
IU.UserID = #UserID
AND PM.IsCert = CASE WHEN #SubReportID='MACS' THEN PM.IsCert ELSE
#IsCert END
ORDER BY PM.IsCert, PM.ResourceName
Please let me know if anybody has same use case, Thanks!
I m not sure if I undrestand you correctly, but i would recommend you the following actions:
Each inner join can be mapped as a relation within JPA (#OneToMany,
#OneToOne, #ManyToOne,#ManyToMany)
Ordering can be implemented with Comparable interface on entity level
For the computed values i would suggest to use sql computed value or service layer within your business logic which will provide you those values (#Transient field)
You can also write NativeQuery for you model
Hope that i answered at least some of your questions :)
I try to do the following inside a Spring Boot application : create a native query and page it so it can returns a page of a given number of elements from a #RestController.
Here's the snippet of my code, where em is the #PersistanceContext EntityManager, and the repository method is the following, knowing that queryString is the native query :
Query searchQuery = em.createNativeQuery(this.queryString, MyEntity.class);
List<MyEntity> resultsList = searchQuery.getResultList();
return new PageImpl<>(resultsList, PageRequest.of(index,size), resultsList.size());
My problem is that the Page returned has a content of the complete query result, not a content of the size of size parameter inside the PageRequest.of.
Has anybody faced the same issue and could give a working example on how to paginate a nativeQuery please ?
Thanks for your help
You are mixing Spring Data JPA (Pageable) with JPA EntityManager. You can't do that. If you are already using a native query then simply put the pagination in the query. You can use what your database supports, for example the standard:
SELECT [a_bunch_of_columns]
FROM dbo.[some_table]
ORDER BY [some_column_or_columns]
OFFSET #PageSize * (#PageNumber - 1) ROWS
FETCH NEXT #PageSize ROWS ONLY;
this is example of using native query with pagination:
#Query("SELECT c FROM Customer As c INNER JOIN Offer as f on f.id=c.specialOffer.id inner join User As u on u.id=f.user.id where u.id=?1 And c.status=?2")
Page<Customer> getAllCustomerToShop(Integer shopId,String status,Pageable pageable)
and then you can call it as:
getAllCustomerToShop(shopId,"status",PageRequest.of(index, PAGE_SIZE));
Modify your code as follows
Query searchQuery = em.createNativeQuery(this.queryString, MyEntity.class)
.setFirstResult(pageable.getPageNumber() * pageable.getPageSize())
.setMaxResults(pageable.getPageSize());
I am trying to convert a "raw" PostGIS SQL query into a Rails ActiveRecord query. My goal is to convert two sequential ActiveRecord queries (each taking ~1ms) into a single ActiveRecord query taking (~1ms). Using the SQL below with ActiveRecord::Base.connection.execute I was able to validate the reduction in time.
Thus, my direct request is to help me to convert this query into an ActiveRecord query (and the best way to execute it).
SELECT COUNT(*)
FROM "users"
INNER JOIN (
SELECT "centroid"
FROM "zip_caches"
WHERE "zip_caches"."postalcode" = '<postalcode>'
) AS "sub" ON ST_Intersects("users"."vendor_coverage", "sub"."centroid")
WHERE "users"."active" = 1;
NOTE that the value <postalcode> is the only variable data in this query. Obviously, there are two models here User and ZipCache. User has no direct relation to ZipCache.
The current two step ActiveRecord query looks like this.
zip = ZipCache.select(:centroid).where(postalcode: '<postalcode>').limit(1).first
User.where{st_intersects(vendor_coverage, zip.centroid)}.count
Disclamer: I've never used PostGIS
First in your final request, it seems like you've missed the WHERE "users"."active" = 1; part.
Here is what I'd do:
First add a active scope on user (for reusability)
scope :active, -> { User.where(active: 1) }
Then for the actual query, You can have the sub query without executing it and use it in a joins on the User model, such as:
subquery = ZipCache.select(:centroid).where(postalcode: '<postalcode>')
User.active
.joins("INNER JOIN (#{subquery.to_sql}) sub ON ST_Intersects(users.vendor_coverage, sub.centroid)")
.count
This allow minimal raw SQL, while keeping only one query.
In any case, check the actual sql request in your console/log by setting the logger level to debug.
The amazing tool scuttle.io is perfect for converting these sorts of queries:
User.select(Arel.star.count).where(User.arel_table[:active].eq(1)).joins(
User.arel_table.join(ZipCach.arel_table).on(
Arel::Nodes::NamedFunction.new(
'ST_Intersects', [
User.arel_table[:vendor_coverage], Sub.arel_table[:centroid]
]
)
).join_sources
)
These are a couple of question, should Spring Roo through reverse engineering adding FetchType.LAZY for Set fields in .aj files or should I do it manually?
If FetchType.LAZY is not present at .aj files, I could make it through the queries " SELECT st1 FROM parentTable t1 JOIN FETCH t1.childTable st1" into the select, right ?
The point here is that I can add FetchType.LAZY to the files manually (Refactor .aj file > Push In..) and then on .java file (high risk if I want Roo to keep the control on my domain classes).
About to make it through queries, I can't do this because I'm getting:
> query specified join fetching, but the owner of the fetched association was not present in the select list [FromElement{explicit,not a collection join,fetch join,fetch non-lazy properties,classAlias=ii,role=...domain.Industry.i18nIndustries,
tableName=..I18nIndustry,
tableAlias=i18nindust3_,
origin=..Industry industry2_,
columns={industry2_.IdIndustry ,className=..domain.I18nIndustry}}]
[select u.idUserName, u.isActiveInd, u.employeeNbr, u.name, c.name as CompanyName, ii.name as IndustryName from Users u JOIN u.idCompany c JOIN c.idIndustry i JOIN FETCH i.i18nIndustries ii WHERE u.idUserName = :username ]
Here the .aj files generated by Roo:
privileged aspect Users_Roo_DbManaged {
...
#ManyToOne
#JoinColumn(name = "IdCompany", referencedColumnName = "IdCompany", nullable = false)
private Company Users.idCompany; ...
..
privileged aspect Company_Roo_DbManaged {
...
#ManyToOne
#JoinColumn(name = "IdCity", referencedColumnName = "IdCity", nullable = false)
private City Company.idCity;
#ManyToOne
#JoinColumn(name = "IdIndustry", referencedColumnName = "IdIndustry", nullable = false)
private Industry Company.idIndustry; ..
..
privileged aspect Industry_Roo_DbManaged { ..
#OneToMany(mappedBy = "idIndustry")
private Set<I18nIndustry> Industry.i18nIndustries; ...
Can you please give me a clue, about what is happening here ?
Spring MVC 3.2,
Roo 1.2.4,
MSSql database
Thanks folks!
--JR
About
These are a couple of question, should Spring Roo through reverse engineering adding FetchType.LAZY for Set fields in .aj files or should I do it manually?
This is not needed because this is the default for this kind of relationship (see https://stackoverflow.com/a/13765381/2295657).
If FetchType.LAZY is not present at .aj files, I could make it through the queries " SELECT st1 FROM parentTable t1 JOIN FETCH t1.childTable st1" into the select, right ?
This makes a eager loading of the relationship data. I think you are in a mistake about the meaning of LAZY and EAGER modes:
EAGER: Requires to load value when instance is loaded into memory (default for basic and no-collections-relationship properties)
LAZY: when instance is loaded a proxy is stored in property and, when anything tries to get data, loads it form DB (default for collections-relationship properties)
On the other hand, to load a LAZY property requires a DB Connection alive when anything tries to get the data, so, you must execute this code inside a no-static, #Transactional annotated, method (read #Transactional JavaDoc for more info).
I advice you to make a look in the JPA specification. In my experience, I so useful.
I found whats happening, This is your HQL
[select u.idUserName, u.isActiveInd, u.employeeNbr, u.name, c.name as CompanyName, ii.name as IndustryName from Users u JOIN u.idCompany c JOIN c.idIndustry i JOIN FETCH i.i18nIndustries ii WHERE u.idUserName = :username ]
What i understand from this hql is u want to fetch list of i18nIndustries ii along with that perticuler idIndustry i correct?
But when you use JOIN FETCH on any object you have to select the main object to hold child object, in your case main object is idIndustry. JOIN FETCH i.i18nIndustries ii means "when fetching idIndustry, also fetch the i18nIndustries, linked to the idIndustry". But your query doesn't fetch idIndustry So the fetch makes no sense.
In select either u have to fetch i also so that the owner (i) of the fetched association (i.i18nIndustries ii) will present in the select list. as your error is
query specified join fetching, but the owner of the fetched
association was not present in the select list
or remove JOIN FETCH just use JOIN
So keeping in mind u want list of i18nIndustries ii, Your query will become :(added i in select)
select u.idUserName, u.isActiveInd, u.employeeNbr, u.name, c.name as
CompanyName, i,ii.name as IndustryName from Users u JOIN u.idCompany c
JOIN c.idIndustry i JOIN FETCH i.i18nIndustries ii WHERE u.idUserName
= :username
I am using NHibernate Session.Query() method to get some data from a sql db. Recently i noticed the query is taking over 1000ms! Using a profiler I found out this time is mostly spent in NHibernate.Linq.DefaultQueryProvider.PrepareQuery() (70%) and the actual query only takes 300ms. The query looks like this
var q = session.Query<Answer>().
Where(a => a.User.IsExpert);
And the resulting sql is this:
select answer0_.ID as ID0_,
answer0_.TotalAnswer as TotalAns2_0_,
answer0_.Rating0 as Rating3_0_,
answer0_.Rating1 as Rating4_0_,
answer0_.Rating2 as Rating5_0_,
answer0_.Rating3 as Rating6_0_,
answer0_.caseID as caseID0_,
answer0_.userid as userid0_
from Answer answer0_
inner join Users user1_
on answer0_.userid = user1_.ID
where user1_.IsExpert = 1
Any ideas how to speed up the PrepareQuery call?