I'm looking for an elegant way to execute a Contains() statement in a scalable way. Please allow me to give some background before I come to the actual question.
The IN statement
In Entity Framework and LINQ to SQL the Contains statement is translated as a SQL IN statement. For instance, from this statement:
var ids = Enumerable.Range(1,10);
var courses = Courses.Where(c => ids.Contains(c.CourseID)).ToList();
Entity Framework will generate
SELECT
[Extent1].[CourseID] AS [CourseID],
[Extent1].[Title] AS [Title],
[Extent1].[Credits] AS [Credits],
[Extent1].[DepartmentID] AS [DepartmentID]
FROM [dbo].[Course] AS [Extent1]
WHERE [Extent1].[CourseID] IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
Unfortunately, the In statement is not scalable. As per MSDN:
Including an extremely large number of values (many thousands) in an IN clause can consume resources and return errors 8623 or 8632
which has to do with running out of resources or exceeding expression limits.
But before these errors occur, the IN statement becomes increasingly slow with growing numbers of items. I can't find documentation about its growth rate, but it performs well up to a few thousands of items, but beyond that it gets dramatically slow. (Based on SQL Server experiences).
Scalable
We can't always avoid this statement. A JOIN with the source data in stead would generally perform much better, but that's only possible when the source data is in the same context. Here I'm dealing with data coming from a client in a disconnected scenario. So I have been looking for a scalable solution. A satisfactory approach turned out to be cutting the operation into chunks:
var courses = ids.ToChunks(1000)
.Select(chunk => Courses.Where(c => chunk.Contains(c.CourseID)))
.SelectMany(x => x).ToList();
(where ToChunks is this little extension method).
This executes the query in chunks of 1000 that all perform well enough. With e.g. 5000 items, 5 queries will run that together are likely to be faster than one query with 5000 items.
But not DRY
But of course I don't want to scatter this construct all over my code. I am looking for an extension method by which any IQueryable<T> can be transformed into a chunky executing statement. Ideally something like this:
var courses = Courses.Where(c => ids.Contains(c.CourseID))
.AsChunky(1000)
.ToList();
But maybe this
var courses = Courses.ChunkyContains(c => c.CourseID, ids, 1000)
.ToList();
I've given the latter solution a first shot:
public static IEnumerable<TEntity> ChunkyContains<TEntity, TContains>(
this IQueryable<TEntity> query,
Expression<Func<TEntity,TContains>> match,
IEnumerable<TContains> containList,
int chunkSize = 500)
{
return containList.ToChunks(chunkSize)
.Select (chunk => query.Where(x => chunk.Contains(match)))
.SelectMany(x => x);
}
Obviously, the part x => chunk.Contains(match) doesn't compile. But I don't know how to manipulate the match expression into a Contains expression.
Maybe someone can help me make this solution work. And of course I'm open to other approaches to make this statement scalable.
I’ve solved this problem with a little different approach a view month ago. Maybe it’s a good solution for you too.
I didn’t want my solution to change the query itself. So a ids.ChunkContains(p.Id) or a special WhereContains method was unfeasible. Also should the solution be able to combine a Contains with another filter as well as using the same collection multiple times.
db.TestEntities.Where(p => (ids.Contains(p.Id) || ids.Contains(p.ParentId)) && p.Name.StartsWith("Test"))
So I tried to encapsulate the logic in a special ToList method that could rewrite the Expression for a specified collection to be queried in chunks.
var ids = Enumerable.Range(1, 11);
var result = db.TestEntities.Where(p => Ids.Contains(p.Id) && p.Name.StartsWith ("Test"))
.ToChunkedList(ids,4);
To rewrite the expression tree I discovered all Contains Method calls from local collections in the query with a view helping classes.
private class ContainsExpression
{
public ContainsExpression(MethodCallExpression methodCall)
{
this.MethodCall = methodCall;
}
public MethodCallExpression MethodCall { get; private set; }
public object GetValue()
{
var parent = MethodCall.Object ?? MethodCall.Arguments.FirstOrDefault();
return Expression.Lambda<Func<object>>(parent).Compile()();
}
public bool IsLocalList()
{
Expression parent = MethodCall.Object ?? MethodCall.Arguments.FirstOrDefault();
while (parent != null) {
if (parent is ConstantExpression)
return true;
var member = parent as MemberExpression;
if (member != null) {
parent = member.Expression;
} else {
parent = null;
}
}
return false;
}
}
private class FindExpressionVisitor<T> : ExpressionVisitor where T : Expression
{
public List<T> FoundItems { get; private set; }
public FindExpressionVisitor()
{
this.FoundItems = new List<T>();
}
public override Expression Visit(Expression node)
{
var found = node as T;
if (found != null) {
this.FoundItems.Add(found);
}
return base.Visit(node);
}
}
public static List<T> ToChunkedList<T, TValue>(this IQueryable<T> query, IEnumerable<TValue> list, int chunkSize)
{
var finder = new FindExpressionVisitor<MethodCallExpression>();
finder.Visit(query.Expression);
var methodCalls = finder.FoundItems.Where(p => p.Method.Name == "Contains").Select(p => new ContainsExpression(p)).Where(p => p.IsLocalList()).ToList();
var localLists = methodCalls.Where(p => p.GetValue() == list).ToList();
If the local collection passed in the ToChunkedList method was found in the query expression, I replace the Contains call to the original list with a new call to a temporary list containing the ids for one batch.
if (localLists.Any()) {
var result = new List<T>();
var valueList = new List<TValue>();
var containsMethod = typeof(Enumerable).GetMethods(BindingFlags.Static | BindingFlags.Public)
.Single(p => p.Name == "Contains" && p.GetParameters().Count() == 2)
.MakeGenericMethod(typeof(TValue));
var queryExpression = query.Expression;
foreach (var item in localLists) {
var parameter = new List<Expression>();
parameter.Add(Expression.Constant(valueList));
if (item.MethodCall.Object == null) {
parameter.AddRange(item.MethodCall.Arguments.Skip(1));
} else {
parameter.AddRange(item.MethodCall.Arguments);
}
var call = Expression.Call(containsMethod, parameter.ToArray());
var replacer = new ExpressionReplacer(item.MethodCall,call);
queryExpression = replacer.Visit(queryExpression);
}
var chunkQuery = query.Provider.CreateQuery<T>(queryExpression);
for (int i = 0; i < Math.Ceiling((decimal)list.Count() / chunkSize); i++) {
valueList.Clear();
valueList.AddRange(list.Skip(i * chunkSize).Take(chunkSize));
result.AddRange(chunkQuery.ToList());
}
return result;
}
// if the collection was not found return query.ToList()
return query.ToList();
Expression Replacer:
private class ExpressionReplacer : ExpressionVisitor {
private Expression find, replace;
public ExpressionReplacer(Expression find, Expression replace)
{
this.find = find;
this.replace = replace;
}
public override Expression Visit(Expression node)
{
if (node == this.find)
return this.replace;
return base.Visit(node);
}
}
Please allow me to provide an alternative to the Chunky approach.
The technique involving Contains in your predicate works well for:
A constant list of values (no volatile).
A small list of values.
Contains will do great if your local data has those two characteristics because these small set of values will be hardcoded in the final SQL query.
The problem begins when your list of values has entropy (non-constant). As of this writing, Entity Framework (Classic and Core) do not try to parameterize these values in any way, this forces SQL Server to generate a query plan every time it sees a new combination of values in your query. This operation is expensive and gets aggravated by the overall complexity of your query (e.g. many tables, a lot of values in the list, etc.).
The Chunky approach still suffers from this SQL Server query plan cache pollution problem, because it does not parametrizes the query, it just moves the cost of creating a big execution plan into smaller ones that are more easy to compute (and discard) by SQL Server, furthermore, every chunk adds an additional round-trip to the database, which increases the time needed to resolve the query.
An Efficient Solution for EF Core
🎉 NEW! QueryableValues EF6 Edition has arrived!
For EF Core keep reading below.
Wouldn't it be nice to have a way of composing local data in your query in a way that's SQL Server friendly? Enter QueryableValues.
I designed this library with these two main goals:
It MUST solve the SQL Server's query plan cache pollution problem ✅
It MUST be fast! ⚡
It has a flexible API that allows you to compose local data provided by an IEnumerable<T> and you get back an IQueryable<T>; just use it as if it were another entity of your DbContext (really), e.g.:
// Sample values.
IEnumerable<int> values = Enumerable.Range(1, 1000);
// Using a Join (query syntax).
var query1 =
from e in dbContext.MyEntities
join v in dbContext.AsQueryableValues(values) on e.Id equals v
select new
{
e.Id,
e.Name
};
// Using Contains (method syntax)
var query2 = dbContext.MyEntities
.Where(e => dbContext.AsQueryableValues(values).Contains(e.Id))
.Select(e => new
{
e.Id,
e.Name
});
You can also compose complex types!
It goes without saying that the provided IEnumerable<T> is only enumerated at the time that your query is materialized (not before), preserving the same behavior of EF Core in this regard.
How Does It Works?
Internally QueryableValues creates a parameterized query and provides your values in a serialized format that is natively understood by SQL Server. This allows your query to be resolved with a single round-trip to the database and avoids creating a new query plan on subsequent executions due to the parameterized nature of it.
Useful Links
Nuget Package
GitHub Repository
Benchmarks
SQL Server Cache Pollution Problem
QueryableValues is distributed under the MIT license
Linqkit to the rescue! Might be a better way that does it directly, but this seems to work fine and makes it pretty clear what's being done. The addition being AsExpandable(), which lets you use the Invoke extension.
using LinqKit;
public static IEnumerable<TEntity> ChunkyContains<TEntity, TContains>(
this IQueryable<TEntity> query,
Expression<Func<TEntity,TContains>> match,
IEnumerable<TContains> containList,
int chunkSize = 500)
{
return containList
.ToChunks(chunkSize)
.Select (chunk => query.AsExpandable()
.Where(x => chunk.Contains(match.Invoke(x))))
.SelectMany(x => x);
}
You might also want to do this:
containsList.Distinct()
.ToChunks(chunkSize)
...or something similar so you don't get duplicate results if something this occurs:
query.ChunkyContains(x => x.Id, new List<int> { 1, 1 }, 1);
Another way would be to build the predicate this way (of course, some parts should be improved, just giving the idea).
public static Expression<Func<TEntity, bool>> ContainsPredicate<TEntity, TContains>(this IEnumerable<TContains> chunk, Expression<Func<TEntity, TContains>> match)
{
return Expression.Lambda<Func<TEntity, bool>>(Expression.Call(
typeof (Enumerable),
"Contains",
new[]
{
typeof (TContains)
},
Expression.Constant(chunk, typeof(IEnumerable<TContains>)), match.Body),
match.Parameters);
}
which you could call in your ChunkContains method
return containList.ToChunks(chunkSize)
.Select(chunk => query.Where(ContainsPredicate(chunk, match)))
.SelectMany(x => x);
Using a stored procedure with a table valued parameter could also work well. You in effect write a joint In the stored procedure between your table / view and the table valued parameter.
https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/table-valued-parameters
In my Edit Controller Action, I post the object to update.
[HttpPost]
public virtual ActionResult Edit(Case myCase){
var currentDocuments = db.CaseDocuments.Where(p => p.idCase == myCase.idCase);
foreach (CaseDocument docInDB in currentDocuments )
{
var deleteDoc = true;
foreach (CaseDocument docNew in myCase.CaseDocuments )
{
if (docNew.idDocument == docInDB.idDocument)
deleteDoc = false;
}
if (deleteDoc )
db.CaseDocuments.Remove(docInDB);
}
foreach (CaseDocument pc in myCase.CaseDocuments)
{
if (pc.idDocument == 0)
db.CaseDocuments.Add(pc);
else
db.Entry(pc).State = EntityState.Modified;
}
*** **db.Entry(myCase).State = EntityState.Modified;** //THIS LINE
db.SaveChanges();
}
The Case model has a collection of Documents, and they are posted along with the Case Model.
As soon I enter the action, I can count the number of documents in the collection, and lets say there are 3.
Then, in order to see if I need to delete documents from database (as the user deleted one from UI), I need to get the Documents for that case from database in this way:
var currentDocuments = db.CaseDocuments.Where(p => p.idCase == myCase.idCase);
And here starts the weird thing: as soon I executa that statement, the myCase.Documents is loaded with what it is in database (lets say there are 4)!! So, I'm not able to compare the 2 collections (to detect if a document was deleted and remove it from db).
What I need is during the Edit Action of my Case model, I need to create/update/modify its documents. Do I need to see this from other angle? What I'm doing is wrong?
EDIT:
After the comments, I realized that the line where I marked myCase as Modified, was at the begining, and I suppose that this was the reason for that behaviour.
Now, moving that line to just before the db.SaveChanges(), fixed that problem, but at the db.Entry(myCase).State = EntityState.Modified; says "There is already an object with the same key in ObjectStateManager. "
What am I doing wrong here? This code looks bad!
Try it this way:
[HttpPost]
public virtual ActionResult Edit(Case myCase){
var currentDocumentIds = db.CaseDocuments
.Where(p => p.idCase == myCase.idCase)
.Select(p => p.idDocument);
foreach (int idInDb in currentDocumentsIds.Where(i => !myCase.CaseDocuments
.Any(ci => ci.idDocumnet == i))
{
var docToDelete = new CaseDocument { idDocument = idInDb };
db.CaseDocuments.Remove(docToDelete);
}
foreach (CaseDocument pc in myCase.CaseDocuments)
{
if (pc.idDocument == 0)
db.CaseDocuments.Add(pc);
else
db.Entry(pc).State = EntityState.Modified;
}
db.Entry(myCase).State = EntityState.Modified;
db.SaveChanges();
}
Edit: The difference between this code and your code is the way how it works with existing documents. It doesn't load them - it loads just their ids. This way you will save some data transfer from database but it should also help you avoiding that exception. When you load the document from the database you have it already attached in the context but if you try to call this:
db.Entry(pc).State = EntityState.Modified;
you will try to attach another instance of the document with the same key to the context. That is not allowed - context can have attached only single instance with unique key.
I´m still having a hard time with Linq.
I need to write a Update Function tat receives an object that has a list. Actually, A region has a list of cities. I want to pass an object "Region" that has a name filed and a list of cities. The problem, is the city objects came from another context and I am unable to attach them to this context. I have been trying several functions, and always get an error like "EntitySet was modified during enumeration" or other. I am tring to make the code below work, but if anyone has a different approach please help.
public int Updateregion(region E)
{
try
{
using (var ctx = new AppDataDataContext())
{
var R =
(from edt in ctx.regiaos
where edt.ID == E.ID
select edt).SingleOrDefault();
if (R != null)
{
R.name = R.name;
R.description = E.description;
}
R.cities = null;
R.cities.AddRange(Edited.Cities);
ctx.SubmitChanges();
return 0 //OK!
}
}
catch (Exception e)
{
......
}
You can't attach objects retrieved from one datacontext to another, it's not supported by Linq-to-SQL. You need to somehow dettach the objects from their original context, but this isn't supported either. One can wonder why a dettach method isn't available, but at least you can fake it by mapping the list to new objects:
var cities = Edited.Cities.Select(city => new City {
ID = city.ID,
Name = city.Name,
/* etc */
});
The key here is to remember to map the primary key and NOT map any of the relation properties. They must be set to null. After this, you should be able to attach the new cities list, and have it work as expected.
I'm trying to project parts of a Display and its list of locations from a WCF Data service into a custom type. Is this doable in WCF Data Services in a Silverlight client? There is some help here, but it doesn't show getting a list back as well as simple strings.
Currently I'm getting "NotSupportedException: Constructing or initializing instances of the type UserQuery+Info with the expression d.Base.Title is not supported.".
It would be a bonus if you could tell me how to do Expand on Locations in this syntax (I know about Displays.Expand("Locations")) or if I need it.
LINQPad snippet
var displays = from d in Displays.Where(d => d.Id == 3136)
select new Info
{
Name = d.Base.Title,
};
displays.Dump();
}
public class Info
{
private string name;
public string Name
{
get
{
return this.name;
}
set
{
this.name = value;
}
}
public IEnumerable<Location> locations;
public IEnumerable<Location> Locations
{
get{ return this.locations;}
set{ this.locations = value;}
}
The problem is that you are effectively asking your WCF server to construct some type it has no knowledge about. Since it is unable to do so, you have to it yourself on your computer:
Displays
.Where(d => d.Id == 3136)
.AsEnumerable()
.Select(d => new Info { Name = d.Base.Title })
This will run the Where() on the server, but the Select() on your computer.
As already noted by svick you can't ask the server for types it doesn't understand (at least not using OData that is). But you can still only ask for properties you want and nothing more.
Since I don't have your service available the below sample uses the demo service on odata.org:
DemoService ctx = new DemoService(new Uri("http://services.odata.org/OData/OData.svc/"));
var q =
ctx.Products
.Where(p => p.ID == 1)
.Select(p =>
new Product
{
Category = new Category
{
Name = p.Category.Name
}
});
var r =
q.AsEnumerable()
.Select(p =>
new
{
CategoryName = p.Category.Name
});
The first query "q" will run compoletely on server (except for creation of the client side objects) and it will only get the Name of the category (and metadata about all the entities in question). It will translate to URL like /Products(1)?$expand=Category&$select=Category/Name.
The second query starts with the AsEnumerable, which effectively executes the first query and then it just performs a simple transform into an anonymous type. This is done completely on the client (no server interaction).
I have a couple of tables with similar relationship structure to the standard Order, OrderLine tables.
When creating a data context, it gives the Order class an OrderLines property that should be populated with OrderLine objects for that particular Order object.
Sure, by default it will delay load the stuff in the OrderLine property but that should be fairly transparent right?
Ok, here is the problem I have: I'm getting an empty list when I go MyOrder.OrderLines but when I go myDataContext.OrderLines.Where(line => line.OrderId == 1) I get the right list.
public void B()
{
var dbContext = new Adis.CA.Repository.Database.CaDataContext(
"<connectionString>");
dbContext.Connection.Open();
dbContext.Transaction = dbContext.Connection.BeginTransaction();
try
{
//!!!Edit: Imortant to note that the order with orderID=1 already exists
//!!!in the database
//just add some new order lines to make sure there are some
var NewOrderLines = new List<OrderLines>()
{
new OrderLine() { OrderID=1, LineID=300 },
new OrderLine() { OrderID=1, LineID=301 },
new OrderLine() { OrderID=1, LineID=302 },
new OrderLine() { OrderID=1, LineID=303 }
};
dbContext.OrderLines.InsertAllOnSubmit(NewOrderLines);
dbContext.SubmitChanges();
//this will give me the 4 rows I just inserted
var orderLinesDirect = dbContext.OrderLines
.Where(orderLine => orderLine.OrderID == 1);
var order = dbContext.Orders.Where(order => order.OrderID == 1);
//this will be an empty list
var orderLinesThroughOrder = order.OrderLines;
}
catch (System.Data.SqlClient.SqlException e)
{
dbContext.Transaction.Rollback();
throw;
}
finally
{
dbContext.Transaction.Rollback();
dbContext.Dispose();
dbContext = null;
}
}
So as far as I can see, I'm not doing anything particularly strange but I would think that orderLinesDirect and orderLinesThroughOrder would give me the same result set.
Can anyone tell me why it doesn't?
You're just adding OrderLines; not any actual Orders. So the Where on dbContext.Orders returns an empty list.
How you can still find the property OrderLines on order I don't understand, so I may be goofing up here.
[Edit]
Could you update the example to show actual types, especially of the order variable? Imo, it shoud be an IQueryable<Order>, but it's strange that you can .OrderLines into that. Try adding a First() or FirstOrDefault() after the Where.