Linq to Entities performance problem with many columns - performance

I am having an issue with getting linq to entities to perform well. The query I have (not mine, maintaining someone's code :-)), has several includes that I've determined are all necessary for the WPF screen that consumes the results of this query.
Now, the SQL generated executes very fast and only returns one row of data. But it is returning 570 columns, and i think the performance hit is in the overhead of creating all the objects and all of those fields.
I've tried using lazy loading, but that doesn't seem to have any effect on performance.
I've tried removing any of the "include" statements that aren't necessary, but it appears that they all are needed.
here's the linq query:
var myQuery =
from appt in ctx.Appointments
.Include("ScheduleColumnProfile")
.Include("EncounterReason")
.Include("Visit")
.Include("Visit.Patient")
.Include("Visit.Patient.PatientInsurances")
.Include("Visit.Patient.PatientInsurances.InsuranceType")
.Include("Visit.Patient.PatientInsurances.InsuranceCarrier")
.Include("MasterLookup")
.Include("User1")
.Include("User2")
.Include("Site")
.Include("Visit.Patient_CoPay")
.Include("Visit.Patient_CoPay.User")
.Include("Visit.VisitInstructions.InstructionSheet")
where appt.VisitId == visitId
&& appt.MasterLookup.LookupDescription.ToUpper() != Rescheduled
&& appt.Site.PracticeID == practiceId
&& appt.MasterLookup.LookupDescription.ToUpper() != Cancelled
orderby appt.AppointmentId descending
select appt;
The SQL generate is 4000 lines long with 570 columns in the select statment and 3 or 4 Union ALLs, so I'm not going to paste it here unless someone REALLY wants to see it. Basically, i'm looking for a way to get rid of the unions if possible, and trim down the columns to only what's needed.
Help!
:-)

if anyone is keeping track, this is the solution that ended up working for me. Thanks to everyone who commented and made suggestions... it eventually lead me to what i have below.
ctx.ContextOptions.LazyLoadingEnabled = true;
var myQuery =
from appt in ctx.Appointments
where appt.VisitId == visitId
&& appt.MasterLookup.LookupDescription.ToUpper() != Rescheduled
&& appt.Site.PracticeID == practiceId
&& appt.MasterLookup.LookupDescription.ToUpper() != Cancelled
orderby appt.AppointmentId descending
select appt;
var myAppt = myQuery.FirstOrDefault();
ctx.LoadProperty(myAppt, a => a.EncounterReason);
ctx.LoadProperty(myAppt, a => a.ScheduleColumnProfile);
ctx.LoadProperty(myAppt, a => a.Visit);
ctx.LoadProperty(myAppt, a => a.MasterLookup);
ctx.LoadProperty(myAppt, a => a.User1);
ctx.LoadProperty(myAppt, a => a.User2);
ctx.LoadProperty(myAppt, a => a.PatientReferredProvider);
var myVisit = myAppt.Visit;
ctx.LoadProperty(myVisit, v => v.Patient);
ctx.LoadProperty(myVisit, v => v.Patient_CoPay);
ctx.LoadProperty(myVisit, v => v.VisitInstructions);
ctx.LoadProperty(myVisit, v => v.EligibilityChecks);
var pat = myVisit.Patient;
ctx.LoadProperty(pat, p => p.PatientInsurances);
//load child insurances
foreach (PatientInsurance patIns in myAppt.Visit.Patient.PatientInsurances)
{
ctx.LoadProperty(patIns, p => p.InsuranceType);
ctx.LoadProperty(patIns, p => p.InsuranceCarrier);
}
//load child instruction sheets
foreach (VisitInstruction vi in myAppt.Visit.VisitInstructions)
{
ctx.LoadProperty(vi, i => i.InstructionSheet);
}
//load child copays
foreach (Patient_CoPay coPay in myAppt.Visit.Patient_CoPay)
{
ctx.LoadProperty(coPay, c => c.User);
}
//load child eligibility checks
foreach (EligibilityCheck ec in myAppt.Visit.EligibilityChecks)
{
ctx.LoadProperty(ec, e => ec.MasterLookup);
ctx.LoadProperty(ec, e => ec.EligibilityResponse);
}

I would recommend creating a new Class that contains only the properties that you need to display. When you project to a new type you don't need to have Include statements, but you can still access the navigation properties of the entity.
var myQuery = from appt in ctx.Appointments
where appt.VisitId == visitId
&& appt.MasterLookup.LookupDescription.ToUpper() != Rescheduled
&& appt.Site.PracticeID == practiceId
&& appt.MasterLookup.LookupDescription.ToUpper() != Cancelled
orderby appt.AppointmentId descending
select new DisplayClass
{
Property1 = appt.Prop1,
Proeprty2 = appt.Visit.Prop1,
.
.
.
};

Related

how to set value in List with out using foreach loop

I am having two lists and filter based on a group of values.
var UserIdList = response.Users.Select(p => p.Id).ToList();
var filteredRecords =
(from o in om.Organizations join u in om.Users on o.Id equals u.OrganizationId where UserIdList.Contains(u.Id)
select new { Enabled = o.Enabled, Id = u.Id }).ToList();
Now i want to set 'Exists' property in 'response.Users' to true if 'Id' exists in filteredRecords.
Please let me know how can I set value with out using foreach loop.
I have tried with
response.Users.Where(x => x.Exists = (filteredRecords .Any(z => z.Id == x.Id))).ToList();
but could not succeed as it is giving only filter results.
I want full records which are matched and which are not
Linq doesn't really support update scenarios as it's for querying data.
For lists however there's a ForEach extension method:
UserList
.ToList()
.ForEach(item=> item.Exists = filteredRecords.Any(f=> f.Id == item.Id));
Is this clearer to read and easier to understand than a foreach loop...
Edit (after question updates)
// You can't use this... it won't even compile,
// x.Exists = ... is an assignment not a condition
response.Users.Where(x => x.Exists = (filteredRecords .Any(z => z.Id == x.Id))).ToList();
What you want to do is to work with a subset of users and update the values.
response
.Users
.Where(x=> filteredRecords.Any(z => z.Id == x.Id))
.ToList() // This materialises your IEnumerable/IQueryable to allow the ForEach extension method
.ForEach(x => x.Exists = true);
you could of course use:
var usersToEnable = response.Users.Where(x=> filteredRecords.Any(z => z.Id == x.Id);
foreach(var user in usersToEnable)
user.Enabled = true;
As a direct response to:
Please let me know how can I set value with out using foreach loop. I have tried with
response.Users.Where(x => x.Exists = (filteredRecords .Any(z => z.Id == x.Id))).ToList(); but could not succeed as it is giving only
filter results. I want full records which are matched and which are
not
What you've done there is selected a collection of response user items you wish to have Exists set to true.
Now you need to set that bool in this filtered collection, and then return the full response collection afterwards, instead of returning the filtered collection.
I think this is where you're getting confused.

multiple where conditions with linq and list

I am attempting to use the following code to do an update on a table with entity framework. The where statement will only work if I remove the and, either side of the and will bring results, but the and results in a null. I know that the value I am searching for exists.
foreach (fdd element in FddList)
{
var slist = context.ResidenceFDDs.ToList<ResidenceFDD>();
ResidenceFDD fddtoupdate = slist
.Where(s =>
s.StName.Contains("Adrienne") &&
s.StNum == element.addressnumb.ToString())
.FirstOrDefault<ResidenceFDD>();
fddtoupdate.Comments = "Comment newly added.";
context.SaveChanges();
}
if you are using .toString() then use as below
ResidenceFDD fddtoupdate = slist.Where(
s =>
s.StName.Contains("Adrienne") &&
s.StNum == element.addressnumb.ToString())
.FirstOrDefault<ResidenceFDD>().AsEnumerable();

linq to sql fetching all the records category wise in the list<> and then looping

i am fetching all the records from the database with the help of this query organization wise. they become about 30-40 records
List<PagesRef> paages = (from pagess in pagerepository.GetAllPages()
join pagesref in pagerepository.GetAllPageRef()
on pagess.int_PageId equals pagesref.int_PageId
where (pagess.int_PostStatusId != 3 && pagess.int_OrganizationId == Authorization.OrganizationID)
&& pagesref.int_PageRefId == pagesref.Pages.PagesRefs.FirstOrDefault(m => m.int_PageId == pagess.int_PageId && m.bit_Active == true && (m.vcr_PageTitle != null && m.vcr_PageTitle != "")).int_PageRefId
select pagesref).ToList();
next the next step what i want to do is to loop through the above list as linq to object query without going to the database to generate 3 level hierarchical record. can some one give me some insight or idea how can i do it?
edit
var parentrecord = paages.Where(n => n.Pages.int_PageParent == 0).OrderBy(m => m.Pages.int_SortOrder == null).OrderBy(m => m.int_PageId);
foreach (var secondlevel in parentrecord) // if parentrecord found
{
var seclevel = paages.Where(m => m.Pages.int_PageParent == secondlevel.Pages.int_PageId).OrderBy(m => m.Pages.int_SortOrder == null).OrderBy(m => m.Pages.int_SortOrder);
secondlevel.vcr_PageTitle = "parent";
pagesreff.Add(secondlevel); // if parentrecord found then loop and add in there
foreach (var thdlevel in seclevel)
{
var thirdlevel = paages.Where(m => m.Pages.int_PageParent == thdlevel.Pages.int_PageId).OrderBy(m => m.Pages.int_SortOrder == null).OrderBy(m => m.int_PageId).OrderBy(m => m.Pages.int_SortOrder);
thdlevel.vcr_PageTitle = "child";
pagesreff.Add(thdlevel); // if parentrecord child found then loop and add in there
foreach (var thd in thirdlevel)
{
thd.vcr_PageTitle = "subchild";
pagesreff.Add(thd); // if parentrecord child found then loop and add in there
}
}
}
After ToList(); linq-to-sql go to database and get rows. After that, you have collection of objects and can do what you want with linq to objects:
var filteredList = paages.Where(someFilter);
there will be no new sql requests.
Update
Your problem is that you filter in navigation property, so you should load your navigation property with your first query. I'm not sure (linq-to-sql was many years ago:)), but this should help you (I assume that m.Pages is of type Page):
List<PagesRef> paages = (from pagess in pagerepository.GetAllPages()
join pagesref in pagerepository.GetAllPageRef()
on pagess.int_PageId equals pagesref.int_PageId
where (pagess.int_PostStatusId != 3 && pagess.int_OrganizationId == Authorization.OrganizationID)
&& pagesref.int_PageRefId == pagesref.Pages.PagesRefs.FirstOrDefault(m => m.int_PageId == pagess.int_PageId && m.bit_Active == true && (m.vcr_PageTitle != null && m.vcr_PageTitle != "")).int_PageRefId
select pagesref).AssociateWith<Page>.ToList();

Restrict records returned from nested LINQ statement

how would I restrict the "Charges" returned in this linq query, to a specified date range:
var dte = DateTime.Parse("2012-01-01");
var dte2 = DateTime.Parse("2012-02-01");
var meetingrooms = tblMeetingRoom
.Where(r => r.building_id==1)
.GroupBy(p => p.tblType)
.Select(g => new
{
TypeName = g.Key.room_type,
TypeID = g.Key.type_id,
TypeCount = g.Count(),
charges =
from rt in charges
where (rt.type_id == g.Key.type_id)
select new {
rt.chargedate,
rt.people,
rt.charge
}
});
meetingrooms.Dump();
I think it needs to go inbetween here somehwhere:
from rt in charges
where (rt.type_id == g.Key.type_id) (EG) && rt.chargedate>=dte and rt.chargedate <dte2
select new {
Thanks for any help,
Mark
from rt in charges
where rt.type_id == g.Key.type_id && (rt.chargedate >= dte && rt.chargedate <= dte2)
select new {
.....
#Maarten - I've been trying this for ages - LinqPad kept giving errors, so I was trying all different ways I could think of - for whatever reason, I must have made some typos, that I didn't make above!
Sorry for wasting everyone's time - the above works "as-is"!
Mark

How do I add a condition to a LINQ query that uses groups?

Here is my current, working, query:
var lsFooterRow = from i in _context.Inventory
where i.ClaimId == claimID
&& i.Taxable == false
group i by new { i.ClaimId }
into grp
select new
{
SumOfReplValue = grp.Sum(i => i.Price),
SumOfACV = grp.Sum(i => i.ACV),
SumOfReplCost = grp.Sum(i => i.ReplacementCost)
};
What I'd like to add, to make it conditional, is something like this so that it adds a filter to the base query along with ClaimID and Taxable:
if (reportType == "R")
lsFooterRow = lsFooterRow.Where(i => i.ReplCost > 0);
This is not working because is does not recognize ReplCost, only SumOfReplValue, SumOfACV and SumOfReplCost.
Can someone please tell me, without doing the query in two steps, a way to add this condition? If there is no way to do it, a two step approach would be greatly appreciated :-)
Thanks in Advance!
If I understand correctly, you should probably break out intial query into multiple pieces.
var lsFooterRow = from i in _context.Inventory
where i.ClaimId == claimID
&& i.Taxable == false
select i;
// conditional where
if (reportType == "R")
lsFooterRow = lsFooterRow.Where(i => i.ReplacementCost > 0);
var aggregateFooterRow = from i in lsFooterRow
group i by new { i.ClaimId }
into grp
select new
{
SumOfReplValue = grp.Sum(i => i.Price),
SumOfACV = grp.Sum(i => i.ACV),
SumOfReplCost = grp.Sum(i => i.ReplacementCost)
};
That way you are filtering on the replacement cost before it gets aggregated, which sounds like what you want to do.
You did express concerns about a two part query but that shouldn't really pose a problem. The nice thing about this is it will not compose and execute the sql until you enumerate over the final version of the query. The entity framework engine is smart enough to simply add the second where as another condition in the final where statement in SQL. That means that your contitional where will neatly be part of the query rather than an afterthought. You can break up the query and add conditional things as much as you want.
The ability to break up the query into multiple pieces and conditionally compose the query is a tremendous benefit that LINQ has over SQL.
#Devin's answer is probably cleanest and makes an important pedant point about the delayed execution of linq and how 2 steps doesn't mean two queries. That being said, if you want to do it in one query, you could write the first query to include the extra condition like so:
var lsFooterRow = from i in _context.Inventory
where i.ClaimId == claimID
&& i.Taxable == false
&& (i.ReplacementCost > 0 || reportType != "R")
group i by new { i.ClaimId }
into grp
select new
{
SumOfReplValue = grp.Sum(i => i.Price),
SumOfACV = grp.Sum(i => i.ACV),
SumOfReplCost = grp.Sum(i => i.ReplacementCost)
};
EDIT:
Hmm, the only thing I can think of that would make this fail, and #Devin's work, is if you're changing the value of reportType between this declaration and where ever the actual enumeration of lsFooterRow is occurring. If this is happening, you can always just .ToList() it immediately. Or, less resource intensively, copy the reportType to a temporary variable that is never changed, and reference that within your query instead.
string _reportType = reportType //only set here, nowhere else
var lsFooterRow = from i in _context.Inventory
where i.ClaimId == claimID
&& i.Taxable == false
&& (i.ReplacementCost > 0 || _reportType != "R")
group i by new { i.ClaimId }
into grp
select new
{
SumOfReplValue = grp.Sum(i => i.Price),
SumOfACV = grp.Sum(i => i.ACV),
SumOfReplCost = grp.Sum(i => i.ReplacementCost)
};
But now it's no longer as clean, and captures the _reportType variable unnecessarily within the closure.
if you know what sql query needs to be executed at the DB...you can try http://www.linqpad.net/
which I feel is pretty handy when u work with linq...

Resources