I have this linq query (using linq 4.3.0):
return ExecuteODataQuery(Db.AdvisorFees, f => f.Advisor);
Implemented as follows:
protected IQueryable<TType> ExecuteODataQuery<TType>(IQueryable<TType> queryFunc, Func<TType, Advisor> advisorFunc) where TType : Entity
{
var systemAccount = GetSystemAccountThrow();
var results = queryFunc.Where(x => x.Active).AsEnumerable().Where(x => CheckAdvisorOrChannel(systemAccount, advisorFunc(x)));
return results;
}
This is my custom method:
public static bool CheckAdvisorOrChannel(SystemAccount systemAccount, Advisor advisor)
{
return (systemAccount.IsChannelSystemAccount ? systemAccount.Channel.Code == advisor.Channel.Code : systemAccount.Advisor.Code == advisor.Code);
}
This currently works. If I remove the AsEnumerable() I get a LinqToEntities not supported exception on my custom method. I cannot use AsEnumerable() as I have too many records and do not want to do the rest of the filtering in memory. Is it possible to call a custom method on IQueryable instead? I've been playin with extension methods and expressions but have not managed to get this working yet. Any pointers?
I would prefer to not use expression trees as much as possible.
I'm looking for an elegant way to execute a Contains() statement in a scalable way. Please allow me to give some background before I come to the actual question.
The IN statement
In Entity Framework and LINQ to SQL the Contains statement is translated as a SQL IN statement. For instance, from this statement:
var ids = Enumerable.Range(1,10);
var courses = Courses.Where(c => ids.Contains(c.CourseID)).ToList();
Entity Framework will generate
SELECT
[Extent1].[CourseID] AS [CourseID],
[Extent1].[Title] AS [Title],
[Extent1].[Credits] AS [Credits],
[Extent1].[DepartmentID] AS [DepartmentID]
FROM [dbo].[Course] AS [Extent1]
WHERE [Extent1].[CourseID] IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
Unfortunately, the In statement is not scalable. As per MSDN:
Including an extremely large number of values (many thousands) in an IN clause can consume resources and return errors 8623 or 8632
which has to do with running out of resources or exceeding expression limits.
But before these errors occur, the IN statement becomes increasingly slow with growing numbers of items. I can't find documentation about its growth rate, but it performs well up to a few thousands of items, but beyond that it gets dramatically slow. (Based on SQL Server experiences).
Scalable
We can't always avoid this statement. A JOIN with the source data in stead would generally perform much better, but that's only possible when the source data is in the same context. Here I'm dealing with data coming from a client in a disconnected scenario. So I have been looking for a scalable solution. A satisfactory approach turned out to be cutting the operation into chunks:
var courses = ids.ToChunks(1000)
.Select(chunk => Courses.Where(c => chunk.Contains(c.CourseID)))
.SelectMany(x => x).ToList();
(where ToChunks is this little extension method).
This executes the query in chunks of 1000 that all perform well enough. With e.g. 5000 items, 5 queries will run that together are likely to be faster than one query with 5000 items.
But not DRY
But of course I don't want to scatter this construct all over my code. I am looking for an extension method by which any IQueryable<T> can be transformed into a chunky executing statement. Ideally something like this:
var courses = Courses.Where(c => ids.Contains(c.CourseID))
.AsChunky(1000)
.ToList();
But maybe this
var courses = Courses.ChunkyContains(c => c.CourseID, ids, 1000)
.ToList();
I've given the latter solution a first shot:
public static IEnumerable<TEntity> ChunkyContains<TEntity, TContains>(
this IQueryable<TEntity> query,
Expression<Func<TEntity,TContains>> match,
IEnumerable<TContains> containList,
int chunkSize = 500)
{
return containList.ToChunks(chunkSize)
.Select (chunk => query.Where(x => chunk.Contains(match)))
.SelectMany(x => x);
}
Obviously, the part x => chunk.Contains(match) doesn't compile. But I don't know how to manipulate the match expression into a Contains expression.
Maybe someone can help me make this solution work. And of course I'm open to other approaches to make this statement scalable.
I’ve solved this problem with a little different approach a view month ago. Maybe it’s a good solution for you too.
I didn’t want my solution to change the query itself. So a ids.ChunkContains(p.Id) or a special WhereContains method was unfeasible. Also should the solution be able to combine a Contains with another filter as well as using the same collection multiple times.
db.TestEntities.Where(p => (ids.Contains(p.Id) || ids.Contains(p.ParentId)) && p.Name.StartsWith("Test"))
So I tried to encapsulate the logic in a special ToList method that could rewrite the Expression for a specified collection to be queried in chunks.
var ids = Enumerable.Range(1, 11);
var result = db.TestEntities.Where(p => Ids.Contains(p.Id) && p.Name.StartsWith ("Test"))
.ToChunkedList(ids,4);
To rewrite the expression tree I discovered all Contains Method calls from local collections in the query with a view helping classes.
private class ContainsExpression
{
public ContainsExpression(MethodCallExpression methodCall)
{
this.MethodCall = methodCall;
}
public MethodCallExpression MethodCall { get; private set; }
public object GetValue()
{
var parent = MethodCall.Object ?? MethodCall.Arguments.FirstOrDefault();
return Expression.Lambda<Func<object>>(parent).Compile()();
}
public bool IsLocalList()
{
Expression parent = MethodCall.Object ?? MethodCall.Arguments.FirstOrDefault();
while (parent != null) {
if (parent is ConstantExpression)
return true;
var member = parent as MemberExpression;
if (member != null) {
parent = member.Expression;
} else {
parent = null;
}
}
return false;
}
}
private class FindExpressionVisitor<T> : ExpressionVisitor where T : Expression
{
public List<T> FoundItems { get; private set; }
public FindExpressionVisitor()
{
this.FoundItems = new List<T>();
}
public override Expression Visit(Expression node)
{
var found = node as T;
if (found != null) {
this.FoundItems.Add(found);
}
return base.Visit(node);
}
}
public static List<T> ToChunkedList<T, TValue>(this IQueryable<T> query, IEnumerable<TValue> list, int chunkSize)
{
var finder = new FindExpressionVisitor<MethodCallExpression>();
finder.Visit(query.Expression);
var methodCalls = finder.FoundItems.Where(p => p.Method.Name == "Contains").Select(p => new ContainsExpression(p)).Where(p => p.IsLocalList()).ToList();
var localLists = methodCalls.Where(p => p.GetValue() == list).ToList();
If the local collection passed in the ToChunkedList method was found in the query expression, I replace the Contains call to the original list with a new call to a temporary list containing the ids for one batch.
if (localLists.Any()) {
var result = new List<T>();
var valueList = new List<TValue>();
var containsMethod = typeof(Enumerable).GetMethods(BindingFlags.Static | BindingFlags.Public)
.Single(p => p.Name == "Contains" && p.GetParameters().Count() == 2)
.MakeGenericMethod(typeof(TValue));
var queryExpression = query.Expression;
foreach (var item in localLists) {
var parameter = new List<Expression>();
parameter.Add(Expression.Constant(valueList));
if (item.MethodCall.Object == null) {
parameter.AddRange(item.MethodCall.Arguments.Skip(1));
} else {
parameter.AddRange(item.MethodCall.Arguments);
}
var call = Expression.Call(containsMethod, parameter.ToArray());
var replacer = new ExpressionReplacer(item.MethodCall,call);
queryExpression = replacer.Visit(queryExpression);
}
var chunkQuery = query.Provider.CreateQuery<T>(queryExpression);
for (int i = 0; i < Math.Ceiling((decimal)list.Count() / chunkSize); i++) {
valueList.Clear();
valueList.AddRange(list.Skip(i * chunkSize).Take(chunkSize));
result.AddRange(chunkQuery.ToList());
}
return result;
}
// if the collection was not found return query.ToList()
return query.ToList();
Expression Replacer:
private class ExpressionReplacer : ExpressionVisitor {
private Expression find, replace;
public ExpressionReplacer(Expression find, Expression replace)
{
this.find = find;
this.replace = replace;
}
public override Expression Visit(Expression node)
{
if (node == this.find)
return this.replace;
return base.Visit(node);
}
}
Please allow me to provide an alternative to the Chunky approach.
The technique involving Contains in your predicate works well for:
A constant list of values (no volatile).
A small list of values.
Contains will do great if your local data has those two characteristics because these small set of values will be hardcoded in the final SQL query.
The problem begins when your list of values has entropy (non-constant). As of this writing, Entity Framework (Classic and Core) do not try to parameterize these values in any way, this forces SQL Server to generate a query plan every time it sees a new combination of values in your query. This operation is expensive and gets aggravated by the overall complexity of your query (e.g. many tables, a lot of values in the list, etc.).
The Chunky approach still suffers from this SQL Server query plan cache pollution problem, because it does not parametrizes the query, it just moves the cost of creating a big execution plan into smaller ones that are more easy to compute (and discard) by SQL Server, furthermore, every chunk adds an additional round-trip to the database, which increases the time needed to resolve the query.
An Efficient Solution for EF Core
🎉 NEW! QueryableValues EF6 Edition has arrived!
For EF Core keep reading below.
Wouldn't it be nice to have a way of composing local data in your query in a way that's SQL Server friendly? Enter QueryableValues.
I designed this library with these two main goals:
It MUST solve the SQL Server's query plan cache pollution problem ✅
It MUST be fast! ⚡
It has a flexible API that allows you to compose local data provided by an IEnumerable<T> and you get back an IQueryable<T>; just use it as if it were another entity of your DbContext (really), e.g.:
// Sample values.
IEnumerable<int> values = Enumerable.Range(1, 1000);
// Using a Join (query syntax).
var query1 =
from e in dbContext.MyEntities
join v in dbContext.AsQueryableValues(values) on e.Id equals v
select new
{
e.Id,
e.Name
};
// Using Contains (method syntax)
var query2 = dbContext.MyEntities
.Where(e => dbContext.AsQueryableValues(values).Contains(e.Id))
.Select(e => new
{
e.Id,
e.Name
});
You can also compose complex types!
It goes without saying that the provided IEnumerable<T> is only enumerated at the time that your query is materialized (not before), preserving the same behavior of EF Core in this regard.
How Does It Works?
Internally QueryableValues creates a parameterized query and provides your values in a serialized format that is natively understood by SQL Server. This allows your query to be resolved with a single round-trip to the database and avoids creating a new query plan on subsequent executions due to the parameterized nature of it.
Useful Links
Nuget Package
GitHub Repository
Benchmarks
SQL Server Cache Pollution Problem
QueryableValues is distributed under the MIT license
Linqkit to the rescue! Might be a better way that does it directly, but this seems to work fine and makes it pretty clear what's being done. The addition being AsExpandable(), which lets you use the Invoke extension.
using LinqKit;
public static IEnumerable<TEntity> ChunkyContains<TEntity, TContains>(
this IQueryable<TEntity> query,
Expression<Func<TEntity,TContains>> match,
IEnumerable<TContains> containList,
int chunkSize = 500)
{
return containList
.ToChunks(chunkSize)
.Select (chunk => query.AsExpandable()
.Where(x => chunk.Contains(match.Invoke(x))))
.SelectMany(x => x);
}
You might also want to do this:
containsList.Distinct()
.ToChunks(chunkSize)
...or something similar so you don't get duplicate results if something this occurs:
query.ChunkyContains(x => x.Id, new List<int> { 1, 1 }, 1);
Another way would be to build the predicate this way (of course, some parts should be improved, just giving the idea).
public static Expression<Func<TEntity, bool>> ContainsPredicate<TEntity, TContains>(this IEnumerable<TContains> chunk, Expression<Func<TEntity, TContains>> match)
{
return Expression.Lambda<Func<TEntity, bool>>(Expression.Call(
typeof (Enumerable),
"Contains",
new[]
{
typeof (TContains)
},
Expression.Constant(chunk, typeof(IEnumerable<TContains>)), match.Body),
match.Parameters);
}
which you could call in your ChunkContains method
return containList.ToChunks(chunkSize)
.Select(chunk => query.Where(ContainsPredicate(chunk, match)))
.SelectMany(x => x);
Using a stored procedure with a table valued parameter could also work well. You in effect write a joint In the stored procedure between your table / view and the table valued parameter.
https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/table-valued-parameters
I am working with code first approach in EDM and facing an error for which I can't the solution.Pls help me
LINQ to Entities does not recognize the method 'Boolean
CheckMeetingSettings(Int64, Int64)' method, and this method cannot be
translated into a store expression.
My code is following(this is the query which I have written
from per in obj.tempPersonConferenceDbSet
where per.Conference.Id == 2
select new PersonDetials
{
Id = per.Person.Id,
JobTitle = per.Person.JobTitle,
CanSendMeetingRequest = CheckMeetingSettings(6327,per.Person.Id)
}
public bool CheckMeetingSettings(int,int)
{
///code I have written.
}
Please help me out of this.
EF can not convert custom code to SQL. Try iterating the result set and assigning the property outside the LINQ query.
var people = (from per in obj.tempPersonConferenceDbSet
where per.Conference.Id == 2
order by /**/
select new PersonDetials
{
Id = per.Person.Id,
JobTitle = per.Person.JobTitle,
}).Skip(/*records count to skip*/)
.Take(/*records count to retrieve*/)
.ToList();
people.ForEach(p => p.CanSendMeetingRequest = CheckMeetingSettings(6327, p.Id));
With Entity Framework, you cannot mix code that runs on the database server with code that runs inside the application. The only way you could write a query like this, is if you defined a function inside SQL Server to implement the code that you've written.
More information on how to expose that function to LINQ to Entities can be found here.
Alternatively, you would have to call CheckMeetingSettings outside the initial query, as Eranga demonstrated.
Try:
var personDetails = obj.tempPersonConferenceDbSet.Where(p=>p.ConferenceId == 2).AsEnumerable().Select(p=> new PersonDetials
{
Id = per.Person.Id,
JobTitle = per.Person.JobTitle,
CanSendMeetingRequest = CheckMeetingSettings(6327,per.Person.Id)
});
public bool CheckMeetingSettings(int,int)
{
///code I have written.
}
You must use AsEnumerable() so you can preform CheckMeetingSettings.
Linq to Entities can't translate your custom code into a SQL query.
You might consider first selecting only the database columns, then add a .ToList() to force the query to resolve. After you have those results you van do another select where you add the information from your CheckMeetingSettings method.
I'm more comfortable with the fluid syntax so I've used that in the following example.
var query = obj.tempPersonConferenceDbSet
.Where(per => per.Conference.Id == 2).Select(per => new { Id = per.Person.Id, JobTitle = per.Person.JobTitle })
.ToList()
.Select(per => new PersonDetails { Id = per.Id,
JobTitle = per.JobTitle,
CanSendMeetingRequest = CheckMeetingSettings(6327, per.Person.Id) })
If your CheckMeetingSettings method also accesses the database you might want to consider not using a seperate method to prevent a SELECT N+1 scenario and try to express the logic as part of the query in terms that the database can understand.
I am trying to send a Linq query as a string to a method to be used in a where clause. Since IEnumerable wouldn't work for this, I have converted my IEnumerable to IQueryable and still it throws error. The following is the code:
public static void FilterData(string Query)
{
if((List<MemberMaintenanceData>)HttpContext.Current.Session["Allmembers"] != null)
{
//Get the IEnumerable object colection from session
var data = (List<MemberMaintenanceData>) HttpContext.Current.Session["Allmembers"];
//Convert it to IQueryable
IQueryable<MemberMaintenanceData> queryData = data.AsQueryable();
//This line doesn't compile!!
queryData = queryData.Where(Query);
HttpContext.Current.Session["Allmembers"] = queryData.AsEnumerable().ToList();
}
}
I intended passing "a => a.AccountId == 1000" as Query
There is a free (and open source) library, provided by Microsoft for parsing strings into Lambda expressions that can then be used in Linq queries. It also contains versions of the standard query operators such as Where() that take a string parameter. You can find it described in Scott Guthries blog post on Dynamic Linq.
For example, you can do queries like this (adapted from a snippet from the Scott guthrie link)
// imagine these have come from a drop down box or some other user input...
string thingToSelectBy = "City";
string citySelectedByUser = "London";
int minNumberOfOrders = 10;
string whereClause = String.Format("{0} = #0 and Orders.Count >= #1", thingToSelectBy);
var query = db.Customers
.Where(whereClause, citySelectedByUser, minNumberOfOrders)
.OrderBy("CompanyName")
.Select("new(CompanyName as Name, Phone");
The Where clause in thisw code snippet shows how you create a where clause using a parameterised string and then dynamically inject values for the parameters at run time, for example, based on user input. This works for parameters of any type.
In your example, the where clause would be
whereClause = "AccountId = 1000";
So in effect you would be doing something like
var newFilteredQueryData = queryData.Where("AccountId = 1000");
That link also contains the location where you can download the source code and a comprehensive document describing the dynamic query API and expression language.
Given a class such as:
public class foo
{
public int AccountID {get;set;}
}
You should be able to do something like this:
Expression<Func<foo, bool>> filter = f => f.AccountID == 1000;
And then pass that as your query. If it is really needed as a string you can do this:
filter.ToString();
//By Using this library
using System.Linq.Dynamic.Core;
InventoryList = Repository.GetAll(); // IQueryable
string filterString = "UnitPrice > 10 And Qty>100 OR Description.Contains("Dairy")";
var filteredGenericList = InventoryList.Where(filterString);
Is it possible ...??? I have 4 DropDownLists on my main page and the
user may select from any, all or some of
the DropDownLists. I am capturing their selection (or non-selection) using a SESSION
variable. What I would like to be able to do is pass the session
variable values to my Data Access Layer and build a WHERE clause
(maybe using StringBuilder) and then place that variable SOMEHOW into
my query expression. Is that possible??? Sorry, I'm a newbie. Thanks ~susan~
public class DLgetRestaurants
{
FVTCEntities db = new FVTCEntities();
public List<RESTAURANT> getRestaurants(string cuisineName, string priceName, string cityName)
[Build a string based on the values passed to the function]
{
var cuisineID = db.CUISINEs.First(s => s.CUISINE_NAME == cuisineName).CUISINE_ID;
List<RESTAURANT> result = (from RESTAURANT in db.RESTAURANTs.Include("CITY").Include("CUISINE").Include("Price")
where **[USE STRINGBUIDER EXPRSSION HERE]**
select RESTAURANT).ToList();
return result;
}
}
You can compose Where conditions which are linked by a logical AND relatively easy in LINQ extension method syntax:
var query = db.RESTAURANTs.Include("CITY").Include("CUISINE").Include("Price");
if (userHasSelectedInDDL1)
query = query.Where(r => r.PropertyForDDL1 == ValueFromDDL1);
if (userHasSelectedInDDL2)
query = query.Where(r => r.PropertyForDDL2 == ValueFromDDL2);
if (userHasSelectedInDDL3)
query = query.Where(r => r.PropertyForDDL3 == ValueFromDDL3);
if (userHasSelectedInDDL4)
query = query.Where(r => r.PropertyForDDL4 == ValueFromDDL4);
List<RESTAURANT> result = query.ToList();
For a much more flexible solution to build queries dynamically the Dynamic LINQ Library recommended by boca is probably the better choice.
I have done this in the past using the Dynamic Linq Library.