I am using NHibernate Session.Query() method to get some data from a sql db. Recently i noticed the query is taking over 1000ms! Using a profiler I found out this time is mostly spent in NHibernate.Linq.DefaultQueryProvider.PrepareQuery() (70%) and the actual query only takes 300ms. The query looks like this
var q = session.Query<Answer>().
Where(a => a.User.IsExpert);
And the resulting sql is this:
select answer0_.ID as ID0_,
answer0_.TotalAnswer as TotalAns2_0_,
answer0_.Rating0 as Rating3_0_,
answer0_.Rating1 as Rating4_0_,
answer0_.Rating2 as Rating5_0_,
answer0_.Rating3 as Rating6_0_,
answer0_.caseID as caseID0_,
answer0_.userid as userid0_
from Answer answer0_
inner join Users user1_
on answer0_.userid = user1_.ID
where user1_.IsExpert = 1
Any ideas how to speed up the PrepareQuery call?
Related
I am trying to convert a "raw" PostGIS SQL query into a Rails ActiveRecord query. My goal is to convert two sequential ActiveRecord queries (each taking ~1ms) into a single ActiveRecord query taking (~1ms). Using the SQL below with ActiveRecord::Base.connection.execute I was able to validate the reduction in time.
Thus, my direct request is to help me to convert this query into an ActiveRecord query (and the best way to execute it).
SELECT COUNT(*)
FROM "users"
INNER JOIN (
SELECT "centroid"
FROM "zip_caches"
WHERE "zip_caches"."postalcode" = '<postalcode>'
) AS "sub" ON ST_Intersects("users"."vendor_coverage", "sub"."centroid")
WHERE "users"."active" = 1;
NOTE that the value <postalcode> is the only variable data in this query. Obviously, there are two models here User and ZipCache. User has no direct relation to ZipCache.
The current two step ActiveRecord query looks like this.
zip = ZipCache.select(:centroid).where(postalcode: '<postalcode>').limit(1).first
User.where{st_intersects(vendor_coverage, zip.centroid)}.count
Disclamer: I've never used PostGIS
First in your final request, it seems like you've missed the WHERE "users"."active" = 1; part.
Here is what I'd do:
First add a active scope on user (for reusability)
scope :active, -> { User.where(active: 1) }
Then for the actual query, You can have the sub query without executing it and use it in a joins on the User model, such as:
subquery = ZipCache.select(:centroid).where(postalcode: '<postalcode>')
User.active
.joins("INNER JOIN (#{subquery.to_sql}) sub ON ST_Intersects(users.vendor_coverage, sub.centroid)")
.count
This allow minimal raw SQL, while keeping only one query.
In any case, check the actual sql request in your console/log by setting the logger level to debug.
The amazing tool scuttle.io is perfect for converting these sorts of queries:
User.select(Arel.star.count).where(User.arel_table[:active].eq(1)).joins(
User.arel_table.join(ZipCach.arel_table).on(
Arel::Nodes::NamedFunction.new(
'ST_Intersects', [
User.arel_table[:vendor_coverage], Sub.arel_table[:centroid]
]
)
).join_sources
)
Frameworks: Spring 4.0.7 and Hibernate 4.3.6
We are having problems with a PagingAndSortingRepository #Query taking an excessive amount of time to complete as it appears to be running through recordset data and building structures before factoring in page data limits.
In non-joined scenarios, the fetch first n rows appears as it should, and the resultset is much smaller making the selection very snappy on the first batch of pages.
The initial problem we ran into was that we were unable to do a #Query with an inner join fetch (to force the join) on detail records. Removing the fetch obviously gets us off the ground, but creates a nested SQL loop which is not desirable.
To get around this, we added countQuery which got us the proper count back, but now appears to load the entire structure in memory before returning the paged results. Turning on debugging shows the records being loaded in. The fetch first n rows has not been added.
Having nested SQLs running in a loop is not an option for us, so removing the fetch does not help.
#Query(value = "select e from InvoiceHistoryHeader e inner join fetch e.details f where e.company = ?1 and e.division = ?2 and e.customerNumber = ?3)", countQuery = "select count(e) from InvoiceHistoryHeader e where e.company = ?1 and e.division = ?2 and e.customerNumber = ?3)")
Page<InvoiceHistoryHeader> findByCustomerNumber(String company, String division, String customerNumber, Pageable pageable);
Can anyone provide a solution to this problem, or give a workaround that may help? I can provide more detail as needed.
EDIT (11/11/2014)
Hibernate output (no fetch):
Hibernate: select count(trim(invoicehis0_.HH_TID)) as col_0_0_ from invhh invoicehis0_ where trim(invoicehis0_.HH_CO)=? and trim(invoicehis0_.HH_DIV)=? and trim(invoicehis0_.HH_CUS)=? and trim(invoicehis0_.HH_BR)=?
Hibernate: select invoicehis0_.hh_tid as hh_tid1_7_, trim(invoicehis0_.HH_CO) as hh_co2_7_, trim(invoicehis0_.HH_CUS) as hh_cus3_7_, trim(invoicehis0_.HH_DTI) as hh_dti4_7_, trim(invoicehis0_.HH_DIV) as hh_div5_7_, trim(invoicehis0_.HH_ORD) as hh_ord6_7_, trim(invoicehis0_.HH_BR) as hh_br7_7_ from invhh invoicehis0_ where trim(invoicehis0_.HH_CO)=? and trim(invoicehis0_.HH_DIV)=? and trim(invoicehis0_.HH_CUS)=? and trim(invoicehis0_.HH_BR)=? fetch first 20 rows only
Hibernate: select details0_.hd_tid as hd_tid2_7_0_, trim(details0_.HD_SEQ) as hd_seq1_6_0_, trim(details0_.HD_TID) as hd_tid2_6_0_, details0_.hd_seq as hd_seq1_6_1_, details0_.hd_tid as hd_tid2_6_1_, trim(details0_.HD_DTL) as hd_dtl3_6_1_, trim(details0_.HD_LIN) as hd_lin4_6_1_ from invhd details0_ where details0_.hd_tid=?
Hibernate: select details0_.hd_tid as hd_tid2_7_0_, trim(details0_.HD_SEQ) as hd_seq1_6_0_, trim(details0_.HD_TID) as hd_tid2_6_0_, details0_.hd_seq as hd_seq1_6_1_, details0_.hd_tid as hd_tid2_6_1_, trim(details0_.HD_DTL) as hd_dtl3_6_1_, trim(details0_.HD_LIN) as hd_lin4_6_1_ from invhd details0_ where details0_.hd_tid=?
<above two SQL repeated 18 more times>
Hibernate output (fetch and countQuery):
Hibernate: select invoicehis0_.hh_tid as hh_tid1_7_0_, details1_.hd_seq as hd_seq1_6_1_, details1_.hd_tid as hd_tid2_6_1_, trim(invoicehis0_.HH_CO) as hh_co2_7_0_, trim(invoicehis0_.HH_CUS) as hh_cus3_7_0_, trim(invoicehis0_.HH_DTI) as hh_dti4_7_0_, trim(invoicehis0_.HH_DIV) as hh_div5_7_0_, trim(invoicehis0_.HH_ORD) as hh_ord6_7_0_, trim(invoicehis0_.HH_BR) as hh_br7_7_0_, trim(details1_.HD_DTL) as hd_dtl3_6_1_, trim(details1_.HD_LIN) as hd_lin4_6_1_, details1_.hd_tid as hd_tid2_7_0__, trim(details1_.HD_SEQ) as hd_seq1_6_0__, trim(details1_.HD_TID) as hd_tid2_6_0__ from invhh invoicehis0_ inner join invhd details1_ on trim(invoicehis0_.HH_TID)=details1_.hd_tid where trim(invoicehis0_.HH_CO)=? and trim(invoicehis0_.HH_DIV)=? and trim(invoicehis0_.HH_CUS)=? and trim(invoicehis0_.HH_BR)=?
When using fetch in join and paging Hibernate does in-memory pagination, which means it loads all records in memory.
Do not use fetch joins with paging.
I have a simple table (SQL server and EF6) Myvalues, with columns Id & Value (double)
I'm trying to get the sum of the natural log of all values in this table. My LINQ statement is:
var sum = db.Myvalues.Select(x => Math.Log(x.Value)).Sum();
It compiles fine, but I'm getting a RTE:
LINQ to Entities does not recognize the method 'Double Log(Double)' method, and this method cannot be translated into a store expression.
What am I doing wrong/how can I fix this?
FWIW, I can execute the following SQL query directly against the database which gives me the correct answer:
select exp(sum(LogCol)) from
(select log(Myvalues.Value) as LogCol From Myvalues
) results
LINQ tries to translate Math.Log into a SQL command so it is executed against the DB.
This is not supported.
The first solution (for SQL Server) is to use one of the existing SqlFunctions. More specifically, SqlFunctions.Log.
The other solution is to retrieve all your items from your DB using .ToList(), and execute Math.Log with LINQ to Objects (not LINQ to Entities).
As EF cannot translate Math.Log() you could get your data in memory and execute the function form your client:
var sum = db.Myvalues.ToList().Select(x => Math.Log(x.Value)).Sum();
My understanding is that the use of scalar or conversion functions causes immediate execution of a LINQ query. It is also my understanding that subqueries are executed upon demand of the outer query which would typically be once per element. For the following example would I be right in saying that the inner query is executed immediately? If so, as this would produce a scalar value how would this affect how the outer query operates?
IEnumerable<string> outerQuery = names.Where ( n => n.Length == names
.OrderBy(n2 => n2.Length).Select(n2 => n2.Length).First());
I would expect the above query to operate in a similar way as below, ie as if there wasn't a subquery.
int val = names.OrderBy(n2 => n2.Length).Select(n2 => n2.Length).First();
IEnumerable<string> outerQuery = names.Where ( n => n.Length == val );
This example was taken from Joseph and Ben Albahari's C# 4.0 in a Nutshell (Chp 8 P331/332) and my confusion stems from the accompanying diagram which appears to show that the subquery is being evaluated each time the outer query iterates through the elements of names.
Could someone clarify how LINQ works in this setup? Any help would be appreciated!
For the following example would I be right in saying that the inner query is executed immediately?
No, the inner query will be executed for each item in names when the outer query is enumerated. If you want it to be executed only once, use the second code sample.
EDIT: as LukeH pointed out, this is only true of Linq to Objects. Other Linq providers (Linq to SQL, Entity Framework...) might be able to optimize this automatically
What is names? If it's collection (and you use LINQ to Objects) then "subquery" will be executed for each outer query item. If it's actually query object then result depends on actual IQueryable.Provider. For example, for LINQ to SQL you will give SQL query with scalar subquery. And in the most cases subquery actually will be executed only once.
I've been developing a webapp using Linq to NHibernate for the past few months, but haven't profiled the SQL it generates until now. Using NH Profiler, it now seems that the following chunk of code hits the DB more than 3,000 times when the Linq expression is executed.
var activeCaseList = from c in UserRepository.GetCasesByProjectManagerID(consultantId)
where c.CompletionDate == null
select new { c.PropertyID, c.Reference, c.Property.Address, DaysOld = DateTime.Now.Subtract(c.CreationDate).Days, JobValue = String.Format("£{0:0,0}", c.JobValue), c.CurrentStatus };
Where the Repository method looks like:
public IEnumerable<Case> GetCasesByProjectManagerID(int projectManagerId)
{
return from c in Session.Linq<Case>()
where c.ProjectManagerID == projectManagerId
select c;
}
It appears to run the initial Repository query first, then iterates through all of the results checking to see if the CompletionDate is null, but issuing a query to get c.Property.Address first.
So if the initial query returns 2,000 records, even if only five of them have no CompletionDate, it still fires off an SQL query to bring back the address details for the 2,000 records.
The way I had imagined this would work, is that it would evaluate all of the WHERE and SELECT clauses and simply amalgamate them, so the inital query would be like:
SELECT ... WHERE ProjectManager = #p1 AND CompleteDate IS NOT NULL
Which would yield 5 records, and then it could fire the further 5 queries to obtain the addresses. Am I expecting too much here, or am I simply doing something wrong?
Anthony
Change the declaration of GetCasesByProjectManagerID:
public IQueryable<Case> GetCasesByProjectManagerID(int projectManagerId)
You can't compose queries with IEnumerable<T> - they're just sequences. IQueryable<T> is specifically designed for composition like this.
Since I can't add a comment yet. Jon Skeet is right you'll want to use IQueryable, this is allows the Linq provider to Lazily construct the SQL. IEnumerable is the eager version.