LINQ Performance Issue on only a few hundred records - linq

in the following code i've commented on the line that SLOWS my page right down. I did some speed test to reveal the CONTAINS LINQ expression is the problem.
Does anyone know how to change this one line to be more efficient using something else instead. I'm also curious as to why its so slow.
Any ideas (thanks in advance):
var allWaste = _securityRepository.FindAllWaste(userId, SystemType.W);
var allWasteIndicatorItems = _securityRepository.FindAllWasteIndicatorItems();
// First get all WASTE RECORDS
var searchResults = (from s in allWaste
join x in allWasteIndicatorItems on s.WasteId equals x.WasteId
where (s.Description.Contains(searchText)
&& s.Site.SiteDescription.EndsWith(searchTextSite)
&& (s.CollectedDate >= startDate && s.CollectedDate <= endDate))
&& x.EWC.EndsWith(searchTextEWC)
select s).Distinct();
var results = searchResults.AsEnumerable();
if (hazardous != "-1")
{
// User has requested to filter on Hazardous or Non Hazardous only rather than Show All
var HazardousBoolFiltered = (from we in _db.WasteIndicatorItems
.Join(_db.WasteIndicators, wii => wii.WasteIndicatorId, wi => wi.WasteIndicatorId, (wii, wi) => new { wasteid = wii.WasteId, wasteindicatorid = wii.WasteIndicatorId, hazardtypeid = wi.HazardTypeId })
.Join(_db.HazardTypes, w => w.hazardtypeid, h => h.HazardTypeId, (w, h) => new { wasteid = w.wasteid, hazardous = h.Hazardous })
.GroupBy(g => new { g.wasteid, g.hazardous })
.Where(g => g.Key.hazardous == true && g.Count() >= 1)
select we).AsEnumerable(); // THIS IS FAST
// Now join the 2 object to eliminate all the keys that do not apply
if (bHazardous)
results = (from r in results join x in HazardousBoolFiltered on r.WasteId equals x.Key.wasteid select r).AsEnumerable(); //This is FAST
else
results = (from r in results.Where(x => !HazardousBoolFiltered
.Select(y => y.Key.wasteid).Contains(x.WasteId)) select r).AsEnumerable(); // This is DOG SLOW 10-15 seconds !--- THIS IS SLOWING EXECUTION by 10 times --!
}
return results.AsQueryable();

I suggest using a logging / tracing framework like smart inspect or log4net combined with a debug text writer.
http://www.codesprouts.com/post/View-LINQ-To-SQL-Statements-Using-A-Debug-TextWriter.aspx
another possibility is to use the sql server profiler and see what sql linq2sql produces.
also a very nice way is to use the mvc mini profiler in combination with the Profiled DB Connection and the SqlFormatters.SqlServerFormatter.

Try Any (MSDN)
Try this:
results = (from r in results
.Where(x => !HazardousBoolFiltered
.Any(y => y.Key.wasteid == r.WasteId)))
.AsEnumerable()
Or Count:
results = (from r in results
.Where(x => HazardousBoolFiltered
.Count(y => y.Key.wasteid == r.WasteId) == 0))
.AsEnumerable()

Related

Get Group sum not using group.Sum() in linq

The following query works, but I want to get the same result without using grp.Sum(). Can we do it?
from item in (await VehicleReplaceCostDataAsync())
group item by (item.type, item.size, item.ADA, item.eseq) into grp
orderby (grp.Key.eseq, grp.Key.size, grp.Key.ADA)
select new VehicleReplacementCost
{
type = grp.Key.type,
size = grp.Key.size,
ADA = grp.Key.ADA,
count = grp.Sum(x => x.count),
cost = grp.Sum(x => x.cost),
Fcount = grp.Sum(x => x.Fcount),
Fcost = grp.Sum(x => x.Fcost),
eseq = grp.Key.eseq,
}).ToList();
Perhaps by using .Aggregate()? [docs]
count = grp.Aggregate(0, (a, b) => a + b.count)
Thanks for the answer from Astrid. It looks like a good one, but I didn't test it. My colleague gave this solution instead by using yield:
var groups = costs
.GroupBy(type => (type.SystemId, type.Type, type.Size, type.ADA, type.Eseq))
.OrderBy(group => (group.Key.SystemId, group.Key.Eseq, group.Key.Size, group.Key.ADA));
foreach (var group in groups)
{
var result = new ProgramGuideVehicleCostRow
{
SystemId = group.Key.SystemId,
Type = group.Key.Type,
Size = group.Key.Size,
ADA = group.Key.ADA,
};
foreach (var row in group)
{
result.Cost += row.Cost;
result.Fcost += row.Fcost;
result.Count += row.Count;
result.Fcount += row.Fcount;
}
yield return result;
}

How do I optimize this LINQ query? It runs localhost but it doesn't run on Azure

I have this LINQ query and am getting results I need. However it takes 5-6 seconds to show results on localhost, and I can't even run this on Azure.
I'm new to LINQ, and I'm sure that I'm doing something inefficient.
Could someone direct me to optimize?
var joblist = (from t in db.Tracking
group t by t.JobNumber into j
let id = j.Max(x => x.ScanDate)
select new
{
jn = j.Key,
ti = j.FirstOrDefault(y => y.ScanDate == id).TrackingId,
sd = j.FirstOrDefault(y => y.ScanDate == id).ScanDate,
lc = j.FirstOrDefault(y => y.ScanDate == id).LocationId
}).Where(z => z.lc == lid).Where(z => z.jn != null);
jfilter = (from tr in joblist
join lc in db.Location on tr.lc equals lc.LocationId
join lt in db.LocType on lc.LocationType equals lt.LocationType
select new ScanMod
{
TrackingId = tr.ti,
LocationName = lc.LocationName,
JobNumber = tr.jn,
LocationTypeName = lt.LocationTypeName,
ScanDate = tr.sd,
StoneId = ""
}).OrderByDescending(z => z.ScanDate);
UPDATE:
This query runs on Azure(s1) but it takes 30 seconds. This table has 500,000 rows and I assume that OrderByDescending or FirstOrDefault is killing it...
var joblist = db.Tracking
.GroupBy(j => j.JobNumber)
.Select(g => g.OrderByDescending(j => j.ScanDate).FirstOrDefault());
jfilter = (from tr in joblist
join lc in db.Location on tr.LocationId equals lc.LocationId
join lt in db.LocType on lc.LocationType equals lt.LocationType
where tr.LocationId == lid
select new ScanMod
{
TrackingId = tr.TrackingId,
LocationName = lc.LocationName,
JobNumber = tr.JobNumber,
LocationTypeName = lt.LocationTypeName,
ScanDate = tr.ScanDate,
StoneId = ""
}).OrderByDescending(z => z.ScanDate);

EF6.1: Delay between query completed and connection close

We are using EF6.1 and dotConnect for Oracle.
Trying to debug why our development performs poorly in one of customers server, I turned on the DatabaseLogger Interceptor feature to get the queries executed in DB and try to improve them. One thing I've found is that in some queries there is some delay of seconds between the query has been executed and the connection has been closed. For example:
Connection opened at 17/06/2014 9:47:42 +02:00
SELECT
"top". (...)
FROM ( SELECT
"Project1". (...)
FROM ( SELECT
"Extent1". (...)
"Extent2". (...)
FROM (...)
WHERE (...)
) "Project1"
ORDER BY (...)
) "top"
WHERE ROWNUM <= 1
-- p__linq__0: '589' (Type = Int32, IsNullable = false)
-- Executing in 17/06/2014 9:47:43 +02:00
-- Completed in 288 ms with result: aj
Connection closed at 17/06/2014 9:47:51 +02:00
As you can see, there's a delay of many seconds. This query is runned many times, but the delay does not appear in every instance, others just have 2 or 3 seconds of delay, and others just none.
I don't really know where to begin to investigate. Is it something related to EF or to dotConnect Oracle? What job is done after the query has been executed and the connection closed?
EDIT: This is the code I'm using to retrieve the info from DB:
var users = db.USUARIO
.Where(x => x.COORDENADA.Any(c => c.LATITUD != 0 && c.LONGITUD != 0))
.OrderBy(x => x.NOMBRE).AsQueryable();
(...)
var list = users.ToList()
.Select(x => new
{
Usuario = x,
LastCoord = db.COORDENADA
.Include(c => c.TIPO)
.Where(c => c.USUARIOID == x.USUARIOID && c.LATITUD != 0 && c.LONGITUD != 0)
.OrderByDescending(c => c.FECHAHORAPDA)
.ThenBy(c => c.TIPO.RUTA).FirstOrDefault()
})
.Select(x => new ListItem
{
ID = x.LastCoord.COORDENADAID,
Marcadores = new List<COORDENADA>(new[] { x.LastCoord }),
Principal = x.Usuario.NOMBRE.ToTitleCase(),
Inferior1 = x.LastCoord.FECHAHORAPDA.ToString("dd/MM HH:mm"),
Color = x.LastCoord.TIPO.COLOR
});
The db.COORDENADA... is the query I exposed before.

Linq All with a condition

DB:
I'm trying to bring back data only when All of the ReviewItems meet the condition of ReviewItemStatus==3. This works.
Problem: But then I want to narrow the scope of All to all ReviewItems where ReviewerID==1000
// I want ALL groupAccountLinks only for ReviewerID==1000 and AccountID
// 0) (and thus ReviewItems) for Account Charlie have ReviewItemStatusID==3
var xx = Accounts.Where(acc => acc.GroupAccountLinks.All(gal =>
// do ANY of the (1) associated reviewItems contain ri.ReviewItemStatusID == 3
gal.ReviewItems.Any(ri => ri.ReviewItemStatusID == 3)
// This doesn't work
//&& ri.Review.ReviewerID == 1000
)
&& acc.AccountID == 1002 // Charlie
);
Will be going against EF4.1 Currently testing using Linqpad and LinqToSQL test db.
You have && Condition .So, I think your table not have all AccountId=1002 and All ReviewItemStatusID =3 . So Change && condition to || condition.
The first thing I would do would be to take the && acc.AccountID == 1002 off and make sure that you're getting the entire unfiltered set of Accounts that have a ReviewItem in status 3. If that looks good try doing your filter this way:
var xx = Accounts
.Where(acc => acc.GroupAccountLinks
.All(gal => gal.ReviewItems
.Any(ri => ri.ReviewItemStatusID == 3))
.FirstOrDefault(acc => acc.AccountID == 1002);
Thanks to Jesse, Slauma and RemeshRams which lead me to this:
var xxx = Accounts.Where(
// do All Accounts satisfy condition
u => u.Accounts.All(
// do All GroupAccountLinks (and thus ReviewItems) for Account meet the condition of ReviewItemStatusID==3
acc => acc.GroupAccountLinks.All(
// for ReviewItems where ReviewerID==1000, do they All have ReviewItemStatusID==3
gal => gal.ReviewItems.Where(
ri => ri.Review.ReviewerID == 1000)
.All(
ri => ri.ReviewItemStatusID == 3
)
// Make sure there are some ReviewItems
&& gal.ReviewItems.Any()
)
)
);

Error trying to exclude records with a JOIN to another object

In my code below, is there any way I can use the results in the object 'WasteRecordsExcluded' to join with searchResults, essentially excluding the WasteId's I don't want.
If I debug to the last line I get the error :
base {System.SystemException} = {"The query contains references to items defined on a different data context."}
Or if joining is impossible then i could change bHazardous from TRUE to FALSE and FALSE to TRUE and do some kind of 'NOT IN' comparison.
Going bananas with this one, anyone help? Kind Regards :
var allWaste = _securityRepository.FindAllWaste(userId, SystemType.W);
var allWasteIndicatorItems = _securityRepository.FindAllWasteIndicatorItems();
// First get all WASTE RECORDS
var searchResults = (from s in allWaste
join x in allWasteIndicatorItems on s.WasteId equals x.WasteId
where (s.Description.Contains(searchText)
&& s.Site.SiteDescription.EndsWith(searchTextSite)
&& (s.CollectedDate >= startDate && s.CollectedDate <= endDate))
&& x.EWC.EndsWith(searchTextEWC)
select s).Distinct();
var results = searchResults;
if (hazardous != "-1")
{
// User has requested to filter on Hazardous or Non Hazardous only rather than Show All
var WasteRecordsExcluded = (from we in _db.WasteIndicatorItems
.Join(_db.WasteIndicators, wii => wii.WasteIndicatorId, wi => wi.WasteIndicatorId, (wii, wi) => new { wasteid = wii.WasteId, wasteindicatorid = wii.WasteIndicatorId, hazardtypeid = wi.HazardTypeId })
.Join(_db.HazardTypes, w => w.hazardtypeid, h => h.HazardTypeId, (w, h) => new { wasteid = w.wasteid, hazardous = h.Hazardous })
.GroupBy(g => new { g.wasteid, g.hazardous })
.Where(g => g.Key.hazardous == bHazardous && g.Count() >= 1)
select we);
// Now join the 2 object to eliminate all the keys that do not apply
results = results.Where(n => WasteRecordsExcluded.All(t2 => n.WasteId == t2.Key.wasteid));
}
return results;
Maybe something like this:
.....
var results = searchResults.ToList();
.....
.....
.Where(g => g.Key.hazardous == bHazardous && g.Count() >= 1)
select we).ToList();
.....

Resources