Optimizing LINQ to Sharepoint - linq

I have three lists on Sharepoint 2010 and I have working code that gets the lists and relates them. My problem is that it takes around 15 seconds to load my page. I am a rank beginner with LINQ to Sharepoint and LINQ in general. MY question is: Is there a way to make this code run faster?
SeatingChartContext dc = new SeatingChartContext(SPContext.Current.Web.Url);
EntityList<Seating_chartItem> seatCharts = dc.GetList<Seating_chartItem>("seating_chart");
EntityList<UsersItem> users = dc.GetList<UsersItem>("users");
EntityList<Excluded_usersItem> exusers = dc.GetList<Excluded_usersItem>("excluded_users");
// EntityList<LogsItem> logs = dc.GetList<LogsItem>("logs");
List<Seating_chartItem> seatList = (from seat in seatCharts where seat.Room == 0 where seat.Floor == floor select seat).ToList();
List <UsersItem> usersList = (from user in users select user).ToList();
List <Excluded_usersItem> xusersList = (from xuser in exusers select xuser).ToList();
var results = from seat in seatList
join user in usersList on
seat.User_id equals user.User_id
where seat.Room == 0
where seat.Floor == floor
where !(from xuser in xusersList select xuser.User_id).Contains(user.User_id)
select new
{
sid = seat.Seat_id,
icon = seat.Icon,
topCoord = seat.Top_coord,
leftCoord = seat.Left_coord,
name = user.Name,
phone = user.Phone,
mobile = user.Mobile,
content = seat.Content
};
The time this code takes is frustrating, to say the least.
Thanks.

One immediate thing: You are re-querying xusersList everytime within your join:
where !(from xuser in xusersList select xuser.User_id).Contains(user.User_id)
Instead just first extract the user ids only (since that is the only thing you need)
var xusersList = (from xuser in exusers select xuser.User_id).ToList();
then use it directly:
where !xusersList.Contains(user.User_id)
Even better - determine the valid users before your query:
usersList = usersList.Where( user => !xusersList.Contains(user.User_id))
.ToList();
Now you can just completely remove this where condition from your query.
Also these where conditions seem to be unneeded:
where seat.Room == 0
where seat.Floor == floor
since you have filtered your seatList this way already.
Having said that you should log some performance data to see what actually takes the most time - is it acquiring the inital lists or your actual join/linq query?

Related

Building LINQ Expression Getting Ignored

I'm trying to build a linq query coming from a table grid from the client side, so im expecting page offset, page start, order and the traditional paging parameters. I have the following code:
[Route("api/settings/logs")]
public Rest.DatatablesResponse GetLogs(int draw, int start, int length) {
var query_string = Request.GetQueryNameValuePairs().ToDictionary(x => x.Key, x => x.Value);
var search = query_string["search.value"];
int order_column = int.Parse(query_string["order[0].column"]);
var order_direction = query_string["order[0].dir"];
var count = db.Logs.Count(q => q.Mode == 2);
var logs = (from l in db.Logs
where l.Mode == 2
select new {
id = l.ID,
mode = l.Mode,
phase_id = l.Phase.ID,
created = l.Created,
user = l.User.Name,
blender_name = l.Blender.Name,
oil_name = l.Oil,
oil_quantity = l.OilQuantity,
production_cycle_name = l.ProductionCycle.Name
});
if (order_direction == "asc") {
if (order_column == 0) logs.OrderBy(q => q.created);
else if (order_column == 2) logs.OrderBy(q => q.production_cycle_name);
} else {
if (order_column == 0) logs.OrderByDescending(q => q.created);
else if (order_column == 2) logs.OrderByDescending(q => q.production_cycle_name);
};
if (!string.IsNullOrEmpty(search)) {
logs.Where(q => q.blender_name.Contains(search) ||
q.oil_name.Contains(search) ||
SqlFunctions.StringConvert((decimal)q.id).Contains(search));
}
logs.Skip(start).Take(length);
DateTime dtDateTime = new DateTime(1970,1,1,0,0,0,0,System.DateTimeKind.Utc);
var steps = from l in logs.ToList()
select new {
id = l.id,
message = StringHelpers.FormatWith(_tpl_message[l.phase_id.ToString() + l.mode.ToString() ], l) ,
created = dtDateTime.AddSeconds(l.created).ToString("h:mmtt - MMMM d, yyyy"),
production_cycle_name = l.production_cycle_name
};
return new Rest.DatatablesResponse {
draw = draw,
recordsTotal = count,
recordsFiltered = count,
data = steps.ToArray()
};
}
My problem is the skip and take and orderby expressions are getting ignored for some reason, and this is the SQL code generated just before converting my linq expressions to a list. From my understanding, the query should not be executed or evaluated until my logs.ToList() call, so the ordering and take/skip should be taken into account, but it's not:
{SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Mode] AS [Mode],
[Extent1].[Phase_ID] AS [Phase_ID],
[Extent1].[Created] AS [Created],
[Extent2].[Name] AS [Name],
[Extent3].[Name] AS [Name1],
[Extent1].[Oil] AS [Oil],
[Extent1].[OilQuantity] AS [OilQuantity],
[Extent4].[Name] AS [Name2]
FROM [dbo].[Steps] AS [Extent1]
LEFT OUTER JOIN [dbo].[Users] AS [Extent2] ON [Extent1].[User_Id] = [Extent2].[Id]
LEFT OUTER JOIN [dbo].[Blenders] AS [Extent3] ON [Extent1].[Blender_ID] = [Extent3].[ID]
LEFT OUTER JOIN [dbo].[ProductionCycles] AS [Extent4] ON [Extent1].[ProductionCycle_ID] = [Extent4].[ID]
WHERE 2 = [Extent1].[Mode]}
Irrelevant P.S. I'm using the not so clever ifs for building the order expression instead of using DynamicLINQ since i have only two sortable columns.
logs.Skip(start).Take(length);
Creates a IQueryable<T> where T is the same anonymous type of which logs is an IQueryable<T> but with start items skipped. Then from that it creates a similarly typed IQueryable<T> where lenght items are the most that will be taken.
Then it throws that away and lets it be garbage collected. (Or ideally the compiler or jitter steps will realise it's thrown away and cut out the whole thing).
Then logs.ToList() goes back to the logs you still have and creates a list from it.
You should replace the Skip and Take line with:
logs = logs.Skip(start).Take(length);
So that you are actually making use of this skipping and taking.
I'm using the not so clever ifs for building the order expression instead of using DynamicLINQ since i have only two sortable columns.
There's nothing particularly not-clever about that, except that you make the same mistake; apply the OrderBy and then throwing away the result instead of using it. Likewise with the Where. You need logs = logs.OrderBy(...) etc.
I'd also question from l in logs.ToList() select new {…} here.
It might be the best approach, if obtaining that list in one step has some advantage. However otherwise:
from l in logs select new {…}
Do the select work on the database, retrieving just what you need.
from l in logs.AsEnumerable() select new {…}
Do the select work in the application, appropriate if part of it cannot be converted to database work, but do it as it comes rather than loading it all into memory first.
from l in await logs.ToListAsync() select new {…}
Has the downside of ToList() but in asynchronous uses, then (assuming your provider has a ToListAsync() method) allows for awaiting.
ToList() is rarely the best option here.

Entity Framework returns wrong data after execution of two similar queries

I have two similar queries, the first one:
var activatedSerialNumbers = (from activation in entities.Activations
where !canceledActivationsIds.Contains(activation.Id)
where activation.CustomerId == customerId
join licenseConfiguration in entities.LicenseConfigurations
on activation.Id equals licenseConfiguration.ActivationId
where licenseConfiguration.ProductId == productId
join activatedSerialNumber in entities.ActivatedSerialNumbers
on activation.Id equals activatedSerialNumber.ActivationId
where deactivatedSams.All(dsn => dsn.ToLower() !=
activatedSerialNumber.Name.ToLower())
select new SamWithLicense
{
Name = activatedSerialNumber.Name,
Features = licenseConfiguration.LicenseFeatures
}).ToList();
The second:
var activationsForSam = (from activation in entities.Activations
where !canceledActivationsIds.Contains(activation.Id)
where activation.CustomerId == customerId
let activatedSerialNumbers = activation.ActivatedSerialNumbers
.Select(sn => sn.Name.ToLower())
where activatedSerialNumbers.Contains(loweredSn)
join licenseConfiguration in entities.LicenseConfigurations
on activation.Id equals activatedProduct.ActivationId
select new SamWithLicense
{
Name = selectedSerialNumber,
Features = licenseConfiguration.LicenseFeatures
}).ToList();
In some situations I execute them one after another and in most cases it works fine, but somethimes - not. In the result of second query Counter takes from another row:
Visual Studio - Quick watch
SQL Management Studio
I guess it's a matter of a EF cache or smth, but don't know how to fix it properly.
In your first query you are joining the Activation Id (PK) to LicenseConfigurations ActivationId (FK)
join licenseConfiguration in entities.LicenseConfigurations
on activation.Id equals licenseConfiguration.ActivationId
in your second query, it looks like you are joining on a value defined outside of the query "activatedProduct"
join licenseConfiguration in entities.LicenseConfigurations
on activation.Id equals activatedProduct.ActivationId

Speed up LINQ query - EF5

I have the following LINQ query using EF5 and generic repository, unit of work patterns to a SQL Server 2008 db
var countriesArr = GetIdsFromDelimStr(countries);
var competitionsArr = GetIdsFromDelimStr(competitions);
var filterTeamName = string.Empty;
if (teamName != null)
{
filterTeamName = teamName.ToUpper();
}
using (var unitOfWork = new FootballUnitOfWork(ConnFooty))
{
// give us our selection of teams
var teams =
(from team in
unitOfWork.TeamRepository.Find()
where ((string.IsNullOrEmpty(filterTeamName) || team.Name.ToUpper().Contains(filterTeamName)) &&
(countriesArr.Contains(team.Venue.Country.Id) || countriesArr.Count() == 0))
select new
{
tId = team.Id
}).Distinct();
// give us our selection of contests
var conts = (
from cont in
unitOfWork.ContestRepository.Find(
c =>
((c.ContestType == ContestType.League && competitionsArr.Count() == 0) ||
(competitionsArr.Contains(c.Competition.Id) && competitionsArr.Count() == 0)))
select new
{
contId = cont.Id
}
).Distinct();
// get selection of home teams based on contest
var homecomps = (from fixt in unitOfWork.FixtureDetailsRepository.Find()
where
teams.Any(t => t.tId == fixt.HomeTeam.Id) &&
conts.Any(c => c.contId == fixt.Contest.Id)
select new
{
teamId = fixt.HomeTeam.Id,
teamName = fixt.HomeTeam.Name,
countryId = fixt.HomeTeam.Venue.Country.Id != null ? fixt.HomeTeam.Venue.Country.Id : 0,
countryName = fixt.HomeTeam.Venue.Country.Id != null ? fixt.HomeTeam.Venue.Country.Name : string.Empty,
compId = fixt.Contest.Competition.Id,
compDesc = fixt.Contest.Competition.Description
}).Distinct();
// get selection of away teams based on contest
var awaycomps = (from fixt in unitOfWork.FixtureDetailsRepository.Find()
where
teams.Any(t => t.tId == fixt.AwayTeam.Id) &&
conts.Any(c => c.contId == fixt.Contest.Id)
select new
{
teamId = fixt.AwayTeam.Id,
teamName = fixt.AwayTeam.Name,
countryId = fixt.AwayTeam.Venue.Country.Id != null ? fixt.AwayTeam.Venue.Country.Id : 0,
countryName = fixt.AwayTeam.Venue.Country.Id != null ? fixt.AwayTeam.Venue.Country.Name : string.Empty,
compId = fixt.Contest.Competition.Id,
compDesc = fixt.Contest.Competition.Description
}).Distinct();
// ensure that we return the max competition based on id for home teams
var homemax = (from t in homecomps
group t by t.teamId
into grp
let maxcomp = grp.Max(g => g.compId)
from g in grp
where g.compId == maxcomp
select g).Distinct();
// ensure that we return the max competition based on id for away teams
var awaymax = (from t in awaycomps
group t by t.teamId
into grp
let maxcomp = grp.Max(g => g.compId)
from g in grp
where g.compId == maxcomp
select g).Distinct();
var filteredteams = homemax.Union(awaymax).OrderBy(t => t.teamName).AsQueryable();
As you can see we want to return the following format which is passed across to a WebAPI so we cast the results to types we can relate to in the UI.
Essentially what we are trying to do is get the home and away teams from a fixture, these fixtures have a contest which relates to a competition. We then get the highest competition id from the grouping and then this is returned with that team. The country is related to the team based on the venue id, when I was originally doing this i had problems figuring out how to do OR joins in linq which is why i split it down to getting home teams and away team and then grouping them based on competition then unioning them together.
An idea of current table size is fixtures has 7840 rows, teams has 8581 rows, contests has 337 rows and competitions has 96 rows. The table that is likely to increase rapidly is the fixture table as this is related to football.
The output we want to end up with is
Team Id, Team Name, Country Id, Country Name, Competition Id, Competition Name
Using no filtering this query takes on average around 5 secs, just wondering if anybody has any ideas/pointers on how to make it quicker.
thanks in advance Mark
I can't judge whether it will speed up things, but your homemax and awaymax queries could be
var homemax = from t in homecomps
group t by t.teamId into grp
select grp.OrderByDescending(x => x.compId).FirstOrDefault();
var awaymax = from t in awaycomps
group t by t.teamId into grp
select grp.OrderByDescending(x => x.compId).FirstOrDefault();
Further, as you are composing one very large query it may perform better when you cut it up in a few smaller queries that fetch intermediary results. Sometimes a few more roundtrips to the database perform better than one very large query for which the database engine can't find a good execution plan.
Another thing is all these Distinct()s. Do you always need them? I think you can do without because you are always fetching data from one table without joining a child collection. Removing them may save a bunch.
Yet another optimization could be to remove the ToUpper. The comparison is done by the database engine in SQL and chances are that the database has a case-insensitive collation. If so, the comparison is never case sensitive even if you'd want it to be! Constructs like Name.ToUpper cancel the use of any index on Name (it is not sargable).

Entity Framework SQL Selecting 600+ Columns

I have a query generated by entity framework running against oracle that's too slow. It runs in about 4 seconds.
This is the main portion of my query
var query = from x in db.BUILDINGs
join pro_co in db.PROFILE_COMMUNITY on x.COMMUNITY_ID equals pro_co.COMMUNITY_ID
join co in db.COMMUNITies on x.COMMUNITY_ID equals co.COMMUNITY_ID
join st in db.STATE_PROFILE on co.STATE_CD equals st.STATE_CD
where pro_co.PROFILE_NM == authorizedUser.ProfileName
select new
{
COMMUNITY_ID = x.COMMUNITY_ID,
COUNTY_ID = x.COUNTY_ID,
REALTOR_GROUP_NM = x.REALTOR_GROUP_NM,
BUILDING_NAME_TX = x.BUILDING_NAME_TX,
ACTIVE_FL = x.ACTIVE_FL,
CONSTR_SQFT_AVAIL_NB = x.CONSTR_SQFT_AVAIL_NB,
TRANS_RAIL_FL = x.TRANS_RAIL_FL,
LAST_UPDATED_DT = x.LAST_UPDATED_DT,
CREATED_DATE = x.CREATED_DATE,
BUILDING_ADDRESS_TX = x.BUILDING_ADDRESS_TX,
BUILDING_ID = x.BUILDING_ID,
COMMUNITY_NM = co.COMMUNITY_NM,
IMAGECOUNT = x.BUILDING_IMAGE2.Count(),
StateCode = st.STATE_NM,
BuildingTypeItems = x.BUILDING_TYPE_ITEM,
BuildingZoningItems = x.BUILDING_ZONING_ITEM,
BuildingSpecFeatures = x.BUILDING_SPEC_FEATURE_ITEM,
buildingHide = x.BUILDING_HIDE,
buildinghideSort = x.BUILDING_HIDE.Count(y => y.PROFILE_NM == ProfileName) > 0 ? 1 : 0,
BUILDING_CITY_TX = x.BUILDING_CITY_TX,
BUILDING_ZIP_TX = x.BUILDING_ZIP_TX,
LPF_GENERAL_DS = x.LPF_GENERAL_DS,
CONSTR_SQFT_TOTAL_NB = x.CONSTR_SQFT_TOTAL_NB,
CONSTR_STORIES_NB = x.CONSTR_STORIES_NB,
CONSTR_CEILING_CENTER_NB = x.CONSTR_CEILING_CENTER_NB,
CONSTR_CEILING_EAVES_NB = x.CONSTR_CEILING_EAVES_NB,
DESCR_EXPANDABLE_FL = x.DESCR_EXPANDABLE_FL,
CONSTR_MATERIAL_TYPE_TX = x.CONSTR_MATERIAL_TYPE_TX,
SITE_ACRES_SALE_NB = x.SITE_ACRES_SALE_NB,
DESCR_PREVIOUS_USE_TX = x.DESCR_PREVIOUS_USE_TX,
CONSTR_YEAR_BUILT_TX = x.CONSTR_YEAR_BUILT_TX,
DESCR_SUBDIVIDE_FL = x.DESCR_SUBDIVIDE_FL,
LOCATION_CITY_LIMITS_FL = x.LOCATION_CITY_LIMITS_FL,
TRANS_INTERSTATE_NEAREST_TX = x.TRANS_INTERSTATE_NEAREST_TX,
TRANS_INTERSTATE_MILES_NB = x.TRANS_INTERSTATE_MILES_NB,
TRANS_HIGHWAY_NAME_TX = x.TRANS_HIGHWAY_NAME_TX,
TRANS_HIGHWAY_MILES_NB = x.TRANS_HIGHWAY_MILES_NB,
TRANS_AIRPORT_COM_NAME_TX = x.TRANS_AIRPORT_COM_NAME_TX,
TRANS_AIRPORT_COM_MILES_NB = x.TRANS_AIRPORT_COM_MILES_NB,
UTIL_ELEC_SUPPLIER_TX = x.UTIL_ELEC_SUPPLIER_TX,
UTIL_GAS_SUPPLIER_TX = x.UTIL_GAS_SUPPLIER_TX,
UTIL_WATER_SUPPLIER_TX = x.UTIL_WATER_SUPPLIER_TX,
UTIL_SEWER_SUPPLIER_TX = x.UTIL_SEWER_SUPPLIER_TX,
UTIL_PHONE_SVC_PVD_TX = x.UTIL_PHONE_SVC_PVD_TX,
CONTACT_ORGANIZATION_TX = x.CONTACT_ORGANIZATION_TX,
CONTACT_PHONE_TX = x.CONTACT_PHONE_TX,
CONTACT_EMAIL_TX = x.CONTACT_EMAIL_TX,
TERMS_SALE_PRICE_TX = x.TERMS_SALE_PRICE_TX,
TERMS_LEASE_SQFT_NB = x.TERMS_LEASE_SQFT_NB
};
There is a section of code that tacks on dynamic where and sort clauses to the query but I've left those out. The query takes about 4 seconds to run no matter what is in the where and sort.
I dropped the generated SQL in Oracle and an explain plan didn't appear to show anything that screamed fix me. Cost is 1554
If this isn't allowed I apologize but I can't seem to find a good way to share this information. I've uploaded the explain plan generated by Sql Developer here: http://www.123server.org/files/explainPlanzip-e1d291efcd.html
Table Layout
Building
--------------------
- BuildingID
- CommunityId
- Lots of other columns
Profile_Community
-----------------------
- CommunityId
- ProfileNM
- lots of other columns
state_profile
---------------------
- StateCD
- ProfileNm
- lots of other columns
Profile
---------------------
- Profile-NM
- a few other columns
All of the tables with allot of columns have 120-150 columns each. It seems like entity is generating a select statement that pulls every column from every table instead of just the ones I want.
The thing that's bugging me and I think might be my issue is that in my LINQ I've selected 50 items, but the generated sql is returning 677 columns. I think returning so many columns is the source of my slowness possibly.
Any ideas why I am getting so many columns returned in SQL or how to speed my query?
I have a suspicion some of the performance is being impacted by your object creation. Try running the query without just a basic "select x" and see if it's the SQL query taking time or the object creation.
Also if the query being generated is too complicated you could try separating it out into smaller sub-queries which gradually enrich your object rather than trying to query everything at once.
I ended up creating a view and having the view only select the columns I wanted and joining on things that needed to be left-joined in linq.
It's pretty annoying that EF selects every column from every table you're trying to join across. But I guess I only noticed this because I am joining a bunch of tables with 150+ columns in them.

LINQ SubCollection SubSelect

Hello – I’m trying to get a where condition to apply to a sub collection. I can get the criteria to return the proper parents. However, I want the sub collection to be limited to the criteria as well.
In my example code, I only want people with “LINK” skills; also, I only want the skills for each person to equal “LINK.” That is, each person should only have “LINK” for their skills.
Thanks in advance.
List<Skill> skills = new List<Skill>();
skills.Add(new Skill(){SkillName="ASP.NET"});
skills.Add(new Skill(){SkillName="C#"});
Person p1 = new Person(){ Name="Me", Skills=skills} ;
List<Skill> skills2 = new List<Skill>();
skills2.Add(new Skill(){SkillName="ASP.NET"});
skills2.Add(new Skill(){SkillName="C#"});
skills2.Add(new Skill(){SkillName="LINQ"});
Person p2 = new Person(){ Name="You", Skills=skills2} ;
List<Person> People = new List<Person>();
People.Add(p1);
People.Add(p2);
var oResult = (from item in People
from sk in item.Skills
where sk.SkillName == "LINQ"
select item
).ToList();
When I run this. I get p2 (correct), but I want the skills of P2 to only equal LINQ
Do this:
var oResult = (from item in People
where item.Skills.Count() == 1 &&
item.Skills.Any(s => s.SkillName == "LINQ")
select item
).ToList();
This query will return nothing because p2 (You) has other skills in addition to LINQ.
You can do what you want this way:
foreach (var person in oResult)
{
person.Skills.RemoveAll(s => !s.SkillName.Equals("LINQ"));
}
Note: while using LINQ you're only filtering your data. To post process it you use something like the foreach I show you above.
Try this:
var oResult = (from item in People
where item.Skills != null
where item.Skills.Count() > 0
where item.Skills.All(s => s.SkillName == "LINQ")
select item
).ToList();
Even though your example shows that the Skills collection has a value, you want to make sure that your code doesn't blow up if the Skills property is null.
Also, the All predicate returns true if your list is empty so you need to ensure that it is not empty. The above query reads better, but depending on the implementation of the Skills property calling Count() may cause the entire collection to be iterated. You could use Any() instead to ensure that the collection is not empty.
var oResult = (from item in People
where item.Skills != null
where item.Skills.Any()
where item.Skills.All(s => s.SkillName == "LINQ")
select item
).ToList();

Resources