Using SelectMany on IQueryable vs IEnumerable - linq

I wanted to do a little experiment to see the cost of operations if I process the data at CosmosDB and return it, vs. if I read the whole document and process it in-memory.
Suppose you have a collection of States in CosmosDB. Each State is one document. The State has some info like Capital, Chief Minister and a list of cities. One state will have multiple cities under it.
The requirement is to fetch the list of all cities in our collection.
Scenario 1
Created an IQueryable which reads all documents from container, applies SelectMany on the queryable and feeds the filtered response to FeedIterator:
var states = container.GetItemLinqQueryable<State>();
var iterator = states.SelectMany(s => s.Cities).ToFeedIterator();
while (iterator.HasMoreResults)
{
var cityList = await iterator.ReadNextAsync();
RUs += cityList.RequestCharge;
foreach (var city in cityList)
{
response.Add(city);
}
}
Result:
Scenario 2
Read all state documents from collection and pulled them in-memory. Applied SelectMany for cities in-memory.
public async Task<List<Workload>> GetAllItemsAsync()
{
var response = new List<State>();
var iterator = container.GetItemLinqQueryable<State>().ToFeedIterator();
while (iterator.HasMoreResults)
{
var states = await iterator.ReadNextAsync();
RUs += states.RequestCharge;
foreach (var state in states)
{
response.Add(state);
}
}
return response;
}
var states = await GetAllItemsAsync();
var cities = states.SelectMany(w => w.Cities).ToList();
Result:
This, understandably consumes much fewer RUs because there's no processing to be done by CosmosDB.
In this scenario, is it better to perform SelectMany on the IEnumerable instead of having CosmosDB do the processing? Is the drawback of not fetching all the documents to avoid the unnecessary data transfer?
Scenario 3
I tried another method, where I fetched the list of cities from each document and then merged them -
var cities = container.GetItemLinqQueryable<State>()
.Select(w => w.Cities);
var iterator = cities.ToFeedIterator();
while (iterator.HasMoreResults)
{
var citiesList = await iterator.ReadNextAsync();
RUs += scenarioList.RequestCharge;
foreach (var cities in citiesList)
{
response.AddRange(cities);
}
}
Result:
This operation consumed the most RUs. Can someone explain this?
I would assume that SelectMany would require more processing that simply selecting the value of the property from each document.

Related

CRM Linq find all parents that have 0 children

How can I find (preferably using CRM Linq) parent entities that have 0 children. For example how can I find all accounts that have 0 contacts.
If you are going to use the query expression route, which I would recommend then the following code will be useful
var entityAlias = "con";
var query = new QueryExpression
{
EntityName = "account",
ColumnSet = new ColumnSet(true),
Criteria =
{
FilterOperator = LogicalOperator.And,
Conditions =
{
new ConditionExpression(entityAlias, "contactid",ConditionOperator.Null)
}
}
LinkEntities =
{
new LinkEntity
{
EntityAlias = entityAlias,
LinkFromEntityName = "account",
LinkFromAttributeName = "accountid",
LinkToEntityName = "contact",
LinkToAttributeName = "parentcustomerid",
Columns = new ColumnSet("parentcustomerid", "contactid"),
JoinOperator = JoinOperator.LeftOuter,
}
},
};
var response = service.RetrieveMultiple(query);
var accounts = response.Entities;
In this code I have not limited the columns, this will reduce performance and you should only return the columns needed.
If there is the case for more than 5000 records are going to be returned then you will need to use paging and loop the query to find all the entities,
This can be found here:
https://msdn.microsoft.com/en-us/library/gg327917.aspx
However if you are certain you want to use LINQ then you can use the following code:
public static IEnumerable<Account> FindAccountsWithNoContacts()
{
var contactRelationship = new Relationship("contact_customer_accounts");
foreach(var account in XrmContext.AccountSet)
{
XrmContext.LoadProperty(contactRelationship);
if(!account.RelatedEntities.ContainsKey(contactRelationship)
yield return account;
}
}
My problem with the LINQ code is that all the enities, both the account and contact entities, will be loaded into memory. With large entity sets this can cause OutOfMemoryException, whereas the query expression route will pass the query to the Dynamics server to execute; which should make the execution of the code faster.
The thing you are looking for is left outer join. Which is unfortunately not possible in CRM using LINQ. However you can do it using query expression or FetchXML.
Here is a link that can help you:
https://community.dynamics.com/crm/b/gonzaloruiz/archive/2014/02/23/all-about-outer-join-queries-in-crm-2011-and-crm-2013

Scalable Contains method for LINQ against a SQL backend

I'm looking for an elegant way to execute a Contains() statement in a scalable way. Please allow me to give some background before I come to the actual question.
The IN statement
In Entity Framework and LINQ to SQL the Contains statement is translated as a SQL IN statement. For instance, from this statement:
var ids = Enumerable.Range(1,10);
var courses = Courses.Where(c => ids.Contains(c.CourseID)).ToList();
Entity Framework will generate
SELECT
[Extent1].[CourseID] AS [CourseID],
[Extent1].[Title] AS [Title],
[Extent1].[Credits] AS [Credits],
[Extent1].[DepartmentID] AS [DepartmentID]
FROM [dbo].[Course] AS [Extent1]
WHERE [Extent1].[CourseID] IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
Unfortunately, the In statement is not scalable. As per MSDN:
Including an extremely large number of values (many thousands) in an IN clause can consume resources and return errors 8623 or 8632
which has to do with running out of resources or exceeding expression limits.
But before these errors occur, the IN statement becomes increasingly slow with growing numbers of items. I can't find documentation about its growth rate, but it performs well up to a few thousands of items, but beyond that it gets dramatically slow. (Based on SQL Server experiences).
Scalable
We can't always avoid this statement. A JOIN with the source data in stead would generally perform much better, but that's only possible when the source data is in the same context. Here I'm dealing with data coming from a client in a disconnected scenario. So I have been looking for a scalable solution. A satisfactory approach turned out to be cutting the operation into chunks:
var courses = ids.ToChunks(1000)
.Select(chunk => Courses.Where(c => chunk.Contains(c.CourseID)))
.SelectMany(x => x).ToList();
(where ToChunks is this little extension method).
This executes the query in chunks of 1000 that all perform well enough. With e.g. 5000 items, 5 queries will run that together are likely to be faster than one query with 5000 items.
But not DRY
But of course I don't want to scatter this construct all over my code. I am looking for an extension method by which any IQueryable<T> can be transformed into a chunky executing statement. Ideally something like this:
var courses = Courses.Where(c => ids.Contains(c.CourseID))
.AsChunky(1000)
.ToList();
But maybe this
var courses = Courses.ChunkyContains(c => c.CourseID, ids, 1000)
.ToList();
I've given the latter solution a first shot:
public static IEnumerable<TEntity> ChunkyContains<TEntity, TContains>(
this IQueryable<TEntity> query,
Expression<Func<TEntity,TContains>> match,
IEnumerable<TContains> containList,
int chunkSize = 500)
{
return containList.ToChunks(chunkSize)
.Select (chunk => query.Where(x => chunk.Contains(match)))
.SelectMany(x => x);
}
Obviously, the part x => chunk.Contains(match) doesn't compile. But I don't know how to manipulate the match expression into a Contains expression.
Maybe someone can help me make this solution work. And of course I'm open to other approaches to make this statement scalable.
I’ve solved this problem with a little different approach a view month ago. Maybe it’s a good solution for you too.
I didn’t want my solution to change the query itself. So a ids.ChunkContains(p.Id) or a special WhereContains method was unfeasible. Also should the solution be able to combine a Contains with another filter as well as using the same collection multiple times.
db.TestEntities.Where(p => (ids.Contains(p.Id) || ids.Contains(p.ParentId)) && p.Name.StartsWith("Test"))
So I tried to encapsulate the logic in a special ToList method that could rewrite the Expression for a specified collection to be queried in chunks.
var ids = Enumerable.Range(1, 11);
var result = db.TestEntities.Where(p => Ids.Contains(p.Id) && p.Name.StartsWith ("Test"))
.ToChunkedList(ids,4);
To rewrite the expression tree I discovered all Contains Method calls from local collections in the query with a view helping classes.
private class ContainsExpression
{
public ContainsExpression(MethodCallExpression methodCall)
{
this.MethodCall = methodCall;
}
public MethodCallExpression MethodCall { get; private set; }
public object GetValue()
{
var parent = MethodCall.Object ?? MethodCall.Arguments.FirstOrDefault();
return Expression.Lambda<Func<object>>(parent).Compile()();
}
public bool IsLocalList()
{
Expression parent = MethodCall.Object ?? MethodCall.Arguments.FirstOrDefault();
while (parent != null) {
if (parent is ConstantExpression)
return true;
var member = parent as MemberExpression;
if (member != null) {
parent = member.Expression;
} else {
parent = null;
}
}
return false;
}
}
private class FindExpressionVisitor<T> : ExpressionVisitor where T : Expression
{
public List<T> FoundItems { get; private set; }
public FindExpressionVisitor()
{
this.FoundItems = new List<T>();
}
public override Expression Visit(Expression node)
{
var found = node as T;
if (found != null) {
this.FoundItems.Add(found);
}
return base.Visit(node);
}
}
public static List<T> ToChunkedList<T, TValue>(this IQueryable<T> query, IEnumerable<TValue> list, int chunkSize)
{
var finder = new FindExpressionVisitor<MethodCallExpression>();
finder.Visit(query.Expression);
var methodCalls = finder.FoundItems.Where(p => p.Method.Name == "Contains").Select(p => new ContainsExpression(p)).Where(p => p.IsLocalList()).ToList();
var localLists = methodCalls.Where(p => p.GetValue() == list).ToList();
If the local collection passed in the ToChunkedList method was found in the query expression, I replace the Contains call to the original list with a new call to a temporary list containing the ids for one batch.
if (localLists.Any()) {
var result = new List<T>();
var valueList = new List<TValue>();
var containsMethod = typeof(Enumerable).GetMethods(BindingFlags.Static | BindingFlags.Public)
.Single(p => p.Name == "Contains" && p.GetParameters().Count() == 2)
.MakeGenericMethod(typeof(TValue));
var queryExpression = query.Expression;
foreach (var item in localLists) {
var parameter = new List<Expression>();
parameter.Add(Expression.Constant(valueList));
if (item.MethodCall.Object == null) {
parameter.AddRange(item.MethodCall.Arguments.Skip(1));
} else {
parameter.AddRange(item.MethodCall.Arguments);
}
var call = Expression.Call(containsMethod, parameter.ToArray());
var replacer = new ExpressionReplacer(item.MethodCall,call);
queryExpression = replacer.Visit(queryExpression);
}
var chunkQuery = query.Provider.CreateQuery<T>(queryExpression);
for (int i = 0; i < Math.Ceiling((decimal)list.Count() / chunkSize); i++) {
valueList.Clear();
valueList.AddRange(list.Skip(i * chunkSize).Take(chunkSize));
result.AddRange(chunkQuery.ToList());
}
return result;
}
// if the collection was not found return query.ToList()
return query.ToList();
Expression Replacer:
private class ExpressionReplacer : ExpressionVisitor {
private Expression find, replace;
public ExpressionReplacer(Expression find, Expression replace)
{
this.find = find;
this.replace = replace;
}
public override Expression Visit(Expression node)
{
if (node == this.find)
return this.replace;
return base.Visit(node);
}
}
Please allow me to provide an alternative to the Chunky approach.
The technique involving Contains in your predicate works well for:
A constant list of values (no volatile).
A small list of values.
Contains will do great if your local data has those two characteristics because these small set of values will be hardcoded in the final SQL query.
The problem begins when your list of values has entropy (non-constant). As of this writing, Entity Framework (Classic and Core) do not try to parameterize these values in any way, this forces SQL Server to generate a query plan every time it sees a new combination of values in your query. This operation is expensive and gets aggravated by the overall complexity of your query (e.g. many tables, a lot of values in the list, etc.).
The Chunky approach still suffers from this SQL Server query plan cache pollution problem, because it does not parametrizes the query, it just moves the cost of creating a big execution plan into smaller ones that are more easy to compute (and discard) by SQL Server, furthermore, every chunk adds an additional round-trip to the database, which increases the time needed to resolve the query.
An Efficient Solution for EF Core
🎉 NEW! QueryableValues EF6 Edition has arrived!
For EF Core keep reading below.
Wouldn't it be nice to have a way of composing local data in your query in a way that's SQL Server friendly? Enter QueryableValues.
I designed this library with these two main goals:
It MUST solve the SQL Server's query plan cache pollution problem ✅
It MUST be fast! ⚡
It has a flexible API that allows you to compose local data provided by an IEnumerable<T> and you get back an IQueryable<T>; just use it as if it were another entity of your DbContext (really), e.g.:
// Sample values.
IEnumerable<int> values = Enumerable.Range(1, 1000);
// Using a Join (query syntax).
var query1 =
from e in dbContext.MyEntities
join v in dbContext.AsQueryableValues(values) on e.Id equals v
select new
{
e.Id,
e.Name
};
// Using Contains (method syntax)
var query2 = dbContext.MyEntities
.Where(e => dbContext.AsQueryableValues(values).Contains(e.Id))
.Select(e => new
{
e.Id,
e.Name
});
You can also compose complex types!
It goes without saying that the provided IEnumerable<T> is only enumerated at the time that your query is materialized (not before), preserving the same behavior of EF Core in this regard.
How Does It Works?
Internally QueryableValues creates a parameterized query and provides your values in a serialized format that is natively understood by SQL Server. This allows your query to be resolved with a single round-trip to the database and avoids creating a new query plan on subsequent executions due to the parameterized nature of it.
Useful Links
Nuget Package
GitHub Repository
Benchmarks
SQL Server Cache Pollution Problem
QueryableValues is distributed under the MIT license
Linqkit to the rescue! Might be a better way that does it directly, but this seems to work fine and makes it pretty clear what's being done. The addition being AsExpandable(), which lets you use the Invoke extension.
using LinqKit;
public static IEnumerable<TEntity> ChunkyContains<TEntity, TContains>(
this IQueryable<TEntity> query,
Expression<Func<TEntity,TContains>> match,
IEnumerable<TContains> containList,
int chunkSize = 500)
{
return containList
.ToChunks(chunkSize)
.Select (chunk => query.AsExpandable()
.Where(x => chunk.Contains(match.Invoke(x))))
.SelectMany(x => x);
}
You might also want to do this:
containsList.Distinct()
.ToChunks(chunkSize)
...or something similar so you don't get duplicate results if something this occurs:
query.ChunkyContains(x => x.Id, new List<int> { 1, 1 }, 1);
Another way would be to build the predicate this way (of course, some parts should be improved, just giving the idea).
public static Expression<Func<TEntity, bool>> ContainsPredicate<TEntity, TContains>(this IEnumerable<TContains> chunk, Expression<Func<TEntity, TContains>> match)
{
return Expression.Lambda<Func<TEntity, bool>>(Expression.Call(
typeof (Enumerable),
"Contains",
new[]
{
typeof (TContains)
},
Expression.Constant(chunk, typeof(IEnumerable<TContains>)), match.Body),
match.Parameters);
}
which you could call in your ChunkContains method
return containList.ToChunks(chunkSize)
.Select(chunk => query.Where(ContainsPredicate(chunk, match)))
.SelectMany(x => x);
Using a stored procedure with a table valued parameter could also work well. You in effect write a joint In the stored procedure between your table / view and the table valued parameter.
https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/table-valued-parameters

How many times the query will hit the database?

Can any one plz optimize the below linq query.It should hit the database only once.
List<LearningItem> items = this.learningitemRepository.GetAll().ToList();
var model = new List<StatementViewerModel>();
foreach (var statement in subjects)
{
var mi = new StatementViewerModel();
mi.UserName = statement.UserName;
mi.SubjectName = statement.Name;
**int nofItems = items.Where(x => x.SubjectId == statement.SubjectId).Count();**
double ratio = (double)statement.AttendedItems / (double)nofItems;
int subjectprogress = (int)(ratio * 100);
mi.Progress = subjectprogress;
model.Add(mi);
}
From what you have posted, your Database should only be being accessed through the GetAll() method of your learmingitemRepository. So if you are worried that you are querying the Database more than once, the GetAll() function is where you should look (assuming your example is not in a function that is being repeatedly called itself).
The rest of your code is iterating over your subjects collection and adding objects to your model collection. There does not appear to be anything in the foreach loop that is accessing your Database.

Complex foreach loop possible to shorten to linq?

I have a cluttery piece of code that I would like to shorten using Linq. It's about the part in the foreach() loop that performs an additional grouping on the result set and builds a nested Dictionary.
Is this possible using a shorter Linq syntax?
var q = from entity in this.Context.Entities
join text in this.Context.Texts on new { ObjectType = 1, ObjectId = entity.EntityId} equals new { ObjectType = text.ObjectType, ObjectId = text.ObjectId}
into texts
select new {entity, texts};
foreach (var result in q)
{
//Can this grouping be performed in the LINQ query above?
var grouped = from tx in result.texts
group tx by tx.Language
into langGroup
select new
{
langGroup.Key,
langGroup
};
//End grouping
var byLanguage = grouped.ToDictionary(x => x.Key, x => x.langGroup.ToDictionary(y => y.PropertyName, y => y.Text));
result.f.Apply(x => x.Texts = byLanguage);
}
return q.Select(x => x.entity);
Sideinfo:
What basically happens is that "texts" for every language and for every property for a certain objecttype (in this case hardcoded 1) are selected and grouped by language. A dictionary of dictionaries is created for every language and then for every property.
Entities have a property called Texts (the dictionary of dictionaries). Apply is a custom extension method which looks like this:
public static T Apply<T>(this T subject, Action<T> action)
{
action(subject);
return subject;
}
isn't this far simpler?
foreach(var entity in Context.Entities)
{
// Create the result dictionary.
entity.Texts = new Dictionary<Language,Dictionary<PropertyName,Text>>();
// loop through each text we want to classify
foreach(var text in Context.Texts.Where(t => t.ObjectType == 1
&& t.ObjectId == entity.ObjectId))
{
var language = text.Language;
var property = text.PropertyName;
// Create the sub-level dictionary, if required
if (!entity.Texts.ContainsKey(language))
entity.Texts[language] = new Dictionary<PropertyName,Text>();
entity.Texts[language][property] = text;
}
}
Sometimes good old foreach loops do the job much better.
Language, PropertyName and Text have no type in your code, so I named my types after the names...

Add an entity from another DataContex in linq

I am trying to merge data between two identical schema databases using Linq-to-sql:
List<Contact> contacts = (from c in oldDb.Contact
select c).ToList();
contacts.ForEach(c => c.CreatedByID = 0);
newDb.Contact.InsertAllOnSubmit(contacts);
newDb.SubmitChanges();
Merely throws an "An attempt has been made to Attach or Add an entity that is not new, perhaps having been loaded from another DataContext. This is not supported." exception.
Other than doing the following, how else can this be done generically (in reasonable execution time):
List<Contact> contacts = (from c in oldDb.Contact
select c).ToList();
contacts.ForEach(c => { c.CreatedByID = 0; newDb.Contact.InsertAllOnSubmit(contacts); });
newDb.SubmitChanges();
along with:
private t GetNewObject<t>(t oldObj)
{
t newObj = (t)System.Reflection.Assembly.GetExecutingAssembly().CreateInstance(typeof(t).Name);
PropertyInfo[] props = typeof(t).GetProperties();
foreach (PropertyInfo _prop in props)
{
_prop.SetValue(newObj, _prop.GetValue(oldObj, null), null);
}
return newObj;
}
The problem is this method is rather slow when there's only 11 objects and 75 properties, I need to do this for a couple hundred thousand objects so any performance gains I can get at this end would greatly reduce overall run time.
Basically, is there any Detach or similar call I could do that will disconnect the existing objects from the old DataContext and connect them to the new DataContext. Without having to create all new objects for each and every one of the returned rows.
I didnt got the
contacts.ForEach(c => { c.CreatedByID = 0; newDb.Contact.InsertAllOnSubmit(contacts); });
shouldnt be something like
contacts.ForEach(c => {
Contact c2 = GetNewObject<Contact>(c);
c2.CreatedByID = 0;
newDb.Contact.InsertOnSubmit(c2);
});
also, here's a way of detaching the object from the old database: http://omaralzabir.com/linq_to_sql__how_to_attach_object_to_a_different_data_context/

Resources