I have the following LINQ query using EF5 and generic repository, unit of work patterns to a SQL Server 2008 db
var countriesArr = GetIdsFromDelimStr(countries);
var competitionsArr = GetIdsFromDelimStr(competitions);
var filterTeamName = string.Empty;
if (teamName != null)
{
filterTeamName = teamName.ToUpper();
}
using (var unitOfWork = new FootballUnitOfWork(ConnFooty))
{
// give us our selection of teams
var teams =
(from team in
unitOfWork.TeamRepository.Find()
where ((string.IsNullOrEmpty(filterTeamName) || team.Name.ToUpper().Contains(filterTeamName)) &&
(countriesArr.Contains(team.Venue.Country.Id) || countriesArr.Count() == 0))
select new
{
tId = team.Id
}).Distinct();
// give us our selection of contests
var conts = (
from cont in
unitOfWork.ContestRepository.Find(
c =>
((c.ContestType == ContestType.League && competitionsArr.Count() == 0) ||
(competitionsArr.Contains(c.Competition.Id) && competitionsArr.Count() == 0)))
select new
{
contId = cont.Id
}
).Distinct();
// get selection of home teams based on contest
var homecomps = (from fixt in unitOfWork.FixtureDetailsRepository.Find()
where
teams.Any(t => t.tId == fixt.HomeTeam.Id) &&
conts.Any(c => c.contId == fixt.Contest.Id)
select new
{
teamId = fixt.HomeTeam.Id,
teamName = fixt.HomeTeam.Name,
countryId = fixt.HomeTeam.Venue.Country.Id != null ? fixt.HomeTeam.Venue.Country.Id : 0,
countryName = fixt.HomeTeam.Venue.Country.Id != null ? fixt.HomeTeam.Venue.Country.Name : string.Empty,
compId = fixt.Contest.Competition.Id,
compDesc = fixt.Contest.Competition.Description
}).Distinct();
// get selection of away teams based on contest
var awaycomps = (from fixt in unitOfWork.FixtureDetailsRepository.Find()
where
teams.Any(t => t.tId == fixt.AwayTeam.Id) &&
conts.Any(c => c.contId == fixt.Contest.Id)
select new
{
teamId = fixt.AwayTeam.Id,
teamName = fixt.AwayTeam.Name,
countryId = fixt.AwayTeam.Venue.Country.Id != null ? fixt.AwayTeam.Venue.Country.Id : 0,
countryName = fixt.AwayTeam.Venue.Country.Id != null ? fixt.AwayTeam.Venue.Country.Name : string.Empty,
compId = fixt.Contest.Competition.Id,
compDesc = fixt.Contest.Competition.Description
}).Distinct();
// ensure that we return the max competition based on id for home teams
var homemax = (from t in homecomps
group t by t.teamId
into grp
let maxcomp = grp.Max(g => g.compId)
from g in grp
where g.compId == maxcomp
select g).Distinct();
// ensure that we return the max competition based on id for away teams
var awaymax = (from t in awaycomps
group t by t.teamId
into grp
let maxcomp = grp.Max(g => g.compId)
from g in grp
where g.compId == maxcomp
select g).Distinct();
var filteredteams = homemax.Union(awaymax).OrderBy(t => t.teamName).AsQueryable();
As you can see we want to return the following format which is passed across to a WebAPI so we cast the results to types we can relate to in the UI.
Essentially what we are trying to do is get the home and away teams from a fixture, these fixtures have a contest which relates to a competition. We then get the highest competition id from the grouping and then this is returned with that team. The country is related to the team based on the venue id, when I was originally doing this i had problems figuring out how to do OR joins in linq which is why i split it down to getting home teams and away team and then grouping them based on competition then unioning them together.
An idea of current table size is fixtures has 7840 rows, teams has 8581 rows, contests has 337 rows and competitions has 96 rows. The table that is likely to increase rapidly is the fixture table as this is related to football.
The output we want to end up with is
Team Id, Team Name, Country Id, Country Name, Competition Id, Competition Name
Using no filtering this query takes on average around 5 secs, just wondering if anybody has any ideas/pointers on how to make it quicker.
thanks in advance Mark
I can't judge whether it will speed up things, but your homemax and awaymax queries could be
var homemax = from t in homecomps
group t by t.teamId into grp
select grp.OrderByDescending(x => x.compId).FirstOrDefault();
var awaymax = from t in awaycomps
group t by t.teamId into grp
select grp.OrderByDescending(x => x.compId).FirstOrDefault();
Further, as you are composing one very large query it may perform better when you cut it up in a few smaller queries that fetch intermediary results. Sometimes a few more roundtrips to the database perform better than one very large query for which the database engine can't find a good execution plan.
Another thing is all these Distinct()s. Do you always need them? I think you can do without because you are always fetching data from one table without joining a child collection. Removing them may save a bunch.
Yet another optimization could be to remove the ToUpper. The comparison is done by the database engine in SQL and chances are that the database has a case-insensitive collation. If so, the comparison is never case sensitive even if you'd want it to be! Constructs like Name.ToUpper cancel the use of any index on Name (it is not sargable).
Related
I want to change code below to be sql translateable because now i get exception.
Basicallly i want list of customers from certain localisation and there could be more than one customer with the same CustomerNumber so i want to take the one that was most recently added.
In other words - distinct list of customers from localisation where "distinct algorithm" works by taking the most recently added customer if there is conflict.
The code below works only if it is client side. I could move Group By and Select after ToListAsync but i want to avoid taking unnecessary data from database (there is include which includes list that is pretty big for every customer).
var someData = await DbContext.Set<Customer>()
.Where(o => o.Metadata.Localisation == localisation)
.Include(nameof(Customer.SomeLongList))
.GroupBy(x => x.CustomerNumber)
.Select(gr => gr.OrderByDescending(x => x.Metadata.DateAdded).FirstOrDefault())
.ToListAsync();
Short answer:
No way. GroupBy has limitation: after grouping only Key and Aggregation result can be selected. And you are trying to select SomeLongList and full entity Customer.
Best answer:
It can be done by the SQL and ROW_NUMBER Window function but without SomeLongList
Workaround:
It is because it is not effective
var groupingQuery =
from c in DbContext.Set<Customer>()
group c by new { c.CustomerNumber } into g
select new
{
g.Key.CustomerNumber,
DateAdded = g.Max(x => x.DateAdded)
};
var query =
from c in DbContext.Set<Customer>().Include(x => x.SomeLongList)
join g in groupingQuery on new { c.CustomerNumber, c.DateAdded } equals
new { g.CustomerNumber, g.DateAdded }
select c;
var result = await query.ToListAsync();
My code is as below:
var conntionRecord1Id = (from connectionBase in orgServiceContext.CreateQuery("connection")
where connectionBase["record1roleid"] == null
select new { OpportunityId = connectionBase["record1id"] }).Distinct().ToList();
var query =
from opportunity in orgServiceContext.CreateQuery("opportunity")
orderby opportunity["createdon"] ascending
select new
{
Topic = opportunity.Attributes.Contains("name") == true ? opportunity["name"] : null,
OpportunityId = opportunity.Attributes.Contains("opportunityid") == true ? opportunity["opportunityid"] : null,
PostalCode = opportunity.Attributes.Contains("new_address_postalcode") == true ? opportunity["new_address_postalcode"] : null,
};
var result = (from f in query.ToList() where conntionRecord1Id.Contains(f.OpportunityId) select f).ToList();
But in above query where i am using Contains it giving count 0. even though I have common records in the list
The Contains Linq extension method does not know how to compare complex objects.
Comparing f.OpportunityId to a list of Guids instead of a list of an anonymous type will work:
var conntionRecord1Id = (from connectionBase in orgServiceContext
.CreateQuery("connection")
where connectionBase["record1roleid"] == null
select connectionBase.GetAttributeValue<Guid?>("record1id"))
.Distinct()
.ToList();
Implement IEqualityComparer in a custom comparer class to compare complex objects: https://stackoverflow.com/a/6694563/1817350
Keep in mind that calling .ToList() brings down all the records into memory and will cause performance problems with large amounts of records.
I have three lists on Sharepoint 2010 and I have working code that gets the lists and relates them. My problem is that it takes around 15 seconds to load my page. I am a rank beginner with LINQ to Sharepoint and LINQ in general. MY question is: Is there a way to make this code run faster?
SeatingChartContext dc = new SeatingChartContext(SPContext.Current.Web.Url);
EntityList<Seating_chartItem> seatCharts = dc.GetList<Seating_chartItem>("seating_chart");
EntityList<UsersItem> users = dc.GetList<UsersItem>("users");
EntityList<Excluded_usersItem> exusers = dc.GetList<Excluded_usersItem>("excluded_users");
// EntityList<LogsItem> logs = dc.GetList<LogsItem>("logs");
List<Seating_chartItem> seatList = (from seat in seatCharts where seat.Room == 0 where seat.Floor == floor select seat).ToList();
List <UsersItem> usersList = (from user in users select user).ToList();
List <Excluded_usersItem> xusersList = (from xuser in exusers select xuser).ToList();
var results = from seat in seatList
join user in usersList on
seat.User_id equals user.User_id
where seat.Room == 0
where seat.Floor == floor
where !(from xuser in xusersList select xuser.User_id).Contains(user.User_id)
select new
{
sid = seat.Seat_id,
icon = seat.Icon,
topCoord = seat.Top_coord,
leftCoord = seat.Left_coord,
name = user.Name,
phone = user.Phone,
mobile = user.Mobile,
content = seat.Content
};
The time this code takes is frustrating, to say the least.
Thanks.
One immediate thing: You are re-querying xusersList everytime within your join:
where !(from xuser in xusersList select xuser.User_id).Contains(user.User_id)
Instead just first extract the user ids only (since that is the only thing you need)
var xusersList = (from xuser in exusers select xuser.User_id).ToList();
then use it directly:
where !xusersList.Contains(user.User_id)
Even better - determine the valid users before your query:
usersList = usersList.Where( user => !xusersList.Contains(user.User_id))
.ToList();
Now you can just completely remove this where condition from your query.
Also these where conditions seem to be unneeded:
where seat.Room == 0
where seat.Floor == floor
since you have filtered your seatList this way already.
Having said that you should log some performance data to see what actually takes the most time - is it acquiring the inital lists or your actual join/linq query?
Given a date, what is the most efficient way to query the last record before that date, any record that equals that date, and the next one after that date.
It should be functionally equivalent to a query like this:
from asset in Assets
where asset.Id == assetId
select new {
Previous = (from a in a.Orders where a.Date < myDate orderby a.Date descending select a).FirstOrDefault(),
Current = (from a in a.Orders where a.Date == myDate select a).SingleOrDefault(),
Next = (from a in a.Orders where a.Date > myDate orderby a.Date select a).FirstOrDefault()
}
As is, this query runs three queries, and presumably has to sort the dataset by myDate three times to do it.
Some similar questions:
How do I get 5 records before AND after a record with a specific ID? (just uses two queries)
How do I get records before and after given one? Not in Linq, and therefore hard for me to take advantage of (my team will get annoyed).
To provide the "most efficient" query depends on what you mean by efficient.
If you want a single query to the database, a single sort of orders by date and finally fast look-ups by date then I suggest the following might be the most efficient. :-)
var orders =
(from a in Assets
where a.Id == assetId
from o in a.Orders
orderby o.Date
select o).ToArray();
var previous = orders.LastOrDefault(o => o.Date < myDate);
var current = orders.SingleOrDefault(o => o.Date == myDate);
var next = orders.FirstOrDefault(o => o.Date > myDate);
This should query the database once for the orders associated with the required asset Id, sort them by date, and return them as an array in memory. Since this is in memory it is now blindingly fast to look for the current, previous & next records for the specified date.
Does your Orders table have a sequential ID field? If so, you might be able to do it with:
from asset in Assets
where asset.Id == assetID
let current = asset.Orders.Where(x => x.Date == myDate).FirstOrDefault()
where current != null
let previous = asset.Orders.Where(x => x.id == current.id - 1).FirstOrDefault()
let next = asset.Orders.Where(x => x.id == current.id + 1).FirstOrDefault()
select new {
Previous = previous,
Current = current,
Next = next
};
If it doesn't, then it'd be a bit more code:
from asset in Assets
where asset.Id == assetID
let current = asset.Orders.Where(x => x.Date == myDate).FirstOrDefault()
where current != null
let previous = asset.Orders.Where(x => x.Date < current.Date).OrderByDescending(x => x.Date).FirstOrDefault()
let next = asset.Orders.Where(x => x.Date > current.Date).OrderBy(x => x.Date).FirstOrDefault()
select new {
Previous = previous,
Current = current,
Next = next
};
That should get compiled into a single SQL query that utilizes sub-queries. IE: the database server will execute multiple queries, but your client program is only submitting one.
Edit One other idea that would work if your Order table had sequential IDs:
var sample = (from asset in Assets
where asset.Id == assetID
let current = asset.Orders.Where(x => x.Date == myDate).FirstOrDefault()
where current != null
from order in asset.Orders
where order.Id == current.id - 1
select order)
.Take(3)
.ToArray();
var Previous = sample[0];
var Current = sample[1];
var Next = sample[2];
Other Answers, for example, SkipWhile etc. very very slow. Good luck ^^
//Current Record
var query
= (from item in db.Employee
where item.UserName.Equals(_username)
select item).SingleOrDefault();
//Next Record
var query
= (from item in db.Employee
where item.UserName.CompareTo(_username) > 0
select item).FirstOrDefault();
//Previous Record
var query
= (from item in db.Employee
where item.UserName.CompareTo(_username) < 0
orderby item.UserName Descending
select item).FirstOrDefault();
Almost the same, but the SQL query plan might be different.
var q =
from asset in Assets
where asset.Id == assetID
select new
{
Previous = asset.Orders.where(a => a.Date == asset.Orders.Where(x => x.Date < myDate).Max(x => x.Date)).FirstOrDefault(),
Current = asset.Orders.Where(x => x.Date == myDate).FirstOrDefault(),
Next = asset.Orders.where(a => a.Date == asset.Orders.Where(x => x.Date > myDate).Min(x => x.Date)).FirstOrDefault()
};
Assuming that we have the following table:
Person:
PersonID,
Name,
Age,
Gender
And we are providing a search function that allows users to search the table according to the name and/or the age.
The tricky part in writing the SQL ( or LINQ) query is that the users can choose to search for both field, or any one field, or no field. If he wants to search for all then he would just have to leave the textbox blank.
The logic to do this can be written as follows:
var p;
if(Name_TextBox=='')
{
p=from row in person
select row ;
}
else
{
p= from row in person
where row.Name=Name_TextBox
select row ;
}
// repeat the same for age
Now after a while the code gets very long and messy... How can I compress the above into a single query with no if-else?
Try code like this
string personName = txtPersonName.Text;
int personAge = Convert.ToInt32(txtAge.Text);
var opportunites = from p in this.DataContext.Persons
select new
{
p.PersonID,
p.Name,
p.Age,
p.Gender
};
if (personsID != 0)
opportunites = opportunites.Where(p => p.PersonID == personID);
if (personName != string.Empty)
opportunites = opportunites.Where(p => p.Name.StartsWith(personName));
if (personAge != 0)
opportunites = opportunites.Where(p => p.Age == personAge);
This will work fine. If personName is not given it will be not add to where, and if given then it will added.
One alternative which I have used in SQL which could be implemented in Linq too is
var p = from p in Person
where p.Name == Name_TextBox || Name_TextBox == String.Empty
select p;
(Note that your 'linq' is using SQL syntax, which won't compile. Also you can't declare a var as you are doing without directly assigning a value)
why not use the null coalescing operator? eg.
var products = from a in context.products
where a.ID == (productID ?? a.ID)
select a;
This works really well on my system