var result=(from refridgerators in context.A
group refridgerators by new { refridgerators.Id, refridgerators.Name } into gr
select new DAO<String>
{
Key = gr.Key.Name,
Count = gr.Count()
}).OrderByDescending(k => k.Count).Take(numberOfRecords).ToList();
This is my linq to sql query this is working perfectly fine.
this shows top 5 records (sorted by their count) if i pass numberOfRecords =5.
now my problem is i don`t want to modify query. so what should i do in above query to show all records. This is in relation with my requirement i want to use same query to show all refridgerators and Top 5 , top 10 refridgerators.
I am not sure if it is possible using LINQ. but i guess there must be something related to this.
I would simply don't add the Take to the query but to the code where you consume this query.
So for example:
public static IEnumerable<DAO> GetRefrigerators()
{
var query = from refridgerators in context.A
group refridgerators by new { refridgerators.Id, refridgerators.Name } into gr
select new DAO<String>
{
Key = gr.Key.Name,
Count = gr.Count()
};
return query.OrderByDescending(k => k.Count);
}
Now you can either use:
var refrigerators = GetRefrigerators().Take(5).ToList();
or
var refrigerators = GetRefrigerators().Take(10).ToList();
or
var refrigerators = GetRefrigerators().ToList();
I'd make numberOfRecords a int? and if the value is null you should not call Take(numberOfRecords)
var result = (from refridgerators in context.A
group refridgerators by new { refridgerators.Id, refridgerators.Name } into gr
select new DAO<String>
{
Key = gr.Key.Name,
Count = gr.Count()
}).OrderByDescending(k => k.Count);
if(numberOfRecords.HasValue)
result = result.Take(numberOfRecords.Value);
return result.ToList();
I know it changes your query a little bit but I believe it is pretty acceptable, adding a numberOfRecords of a super high value adds an overheard to your query which isn't useful in your case.
So this post is very old, I know, but my solution is not provided (maybe someone wants to achieve the same after 2 years, like me):
You could create your own extension-method that also works with linq 2 sql. Example:
public static class LinqExtensions
{
public static IEnumerable<TResult> TakeIfNotNull<TResult>(this IEnumerable<TResult> source, int? count)
{
return !count.HasValue ? source : source.Take(count.Value);
}
}
this takes an int? instead of int and returns either the source-enumerable if count is null else it returns the result of default Take-method.
So you could write like:
.OrderByDescending(k => k.Count).TakeIfNotNull(numberOfRecords).ToList();
if numberOfCounts is null, it'll take all records.
As the right solution has already been posted by Tim, still I want your attention to the simpler solution, if suits to your requirement.
Add one more if condition at the top of this query.
if(allRecordRequired)
{
numberOfRecords = 2000000;
}
Leave your query as it is.
Pass a huge number to your method.A number that is equal (at least) or bigger than the items count.That should give you all records.
Related
I'm looking for an elegant way to execute a Contains() statement in a scalable way. Please allow me to give some background before I come to the actual question.
The IN statement
In Entity Framework and LINQ to SQL the Contains statement is translated as a SQL IN statement. For instance, from this statement:
var ids = Enumerable.Range(1,10);
var courses = Courses.Where(c => ids.Contains(c.CourseID)).ToList();
Entity Framework will generate
SELECT
[Extent1].[CourseID] AS [CourseID],
[Extent1].[Title] AS [Title],
[Extent1].[Credits] AS [Credits],
[Extent1].[DepartmentID] AS [DepartmentID]
FROM [dbo].[Course] AS [Extent1]
WHERE [Extent1].[CourseID] IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
Unfortunately, the In statement is not scalable. As per MSDN:
Including an extremely large number of values (many thousands) in an IN clause can consume resources and return errors 8623 or 8632
which has to do with running out of resources or exceeding expression limits.
But before these errors occur, the IN statement becomes increasingly slow with growing numbers of items. I can't find documentation about its growth rate, but it performs well up to a few thousands of items, but beyond that it gets dramatically slow. (Based on SQL Server experiences).
Scalable
We can't always avoid this statement. A JOIN with the source data in stead would generally perform much better, but that's only possible when the source data is in the same context. Here I'm dealing with data coming from a client in a disconnected scenario. So I have been looking for a scalable solution. A satisfactory approach turned out to be cutting the operation into chunks:
var courses = ids.ToChunks(1000)
.Select(chunk => Courses.Where(c => chunk.Contains(c.CourseID)))
.SelectMany(x => x).ToList();
(where ToChunks is this little extension method).
This executes the query in chunks of 1000 that all perform well enough. With e.g. 5000 items, 5 queries will run that together are likely to be faster than one query with 5000 items.
But not DRY
But of course I don't want to scatter this construct all over my code. I am looking for an extension method by which any IQueryable<T> can be transformed into a chunky executing statement. Ideally something like this:
var courses = Courses.Where(c => ids.Contains(c.CourseID))
.AsChunky(1000)
.ToList();
But maybe this
var courses = Courses.ChunkyContains(c => c.CourseID, ids, 1000)
.ToList();
I've given the latter solution a first shot:
public static IEnumerable<TEntity> ChunkyContains<TEntity, TContains>(
this IQueryable<TEntity> query,
Expression<Func<TEntity,TContains>> match,
IEnumerable<TContains> containList,
int chunkSize = 500)
{
return containList.ToChunks(chunkSize)
.Select (chunk => query.Where(x => chunk.Contains(match)))
.SelectMany(x => x);
}
Obviously, the part x => chunk.Contains(match) doesn't compile. But I don't know how to manipulate the match expression into a Contains expression.
Maybe someone can help me make this solution work. And of course I'm open to other approaches to make this statement scalable.
I’ve solved this problem with a little different approach a view month ago. Maybe it’s a good solution for you too.
I didn’t want my solution to change the query itself. So a ids.ChunkContains(p.Id) or a special WhereContains method was unfeasible. Also should the solution be able to combine a Contains with another filter as well as using the same collection multiple times.
db.TestEntities.Where(p => (ids.Contains(p.Id) || ids.Contains(p.ParentId)) && p.Name.StartsWith("Test"))
So I tried to encapsulate the logic in a special ToList method that could rewrite the Expression for a specified collection to be queried in chunks.
var ids = Enumerable.Range(1, 11);
var result = db.TestEntities.Where(p => Ids.Contains(p.Id) && p.Name.StartsWith ("Test"))
.ToChunkedList(ids,4);
To rewrite the expression tree I discovered all Contains Method calls from local collections in the query with a view helping classes.
private class ContainsExpression
{
public ContainsExpression(MethodCallExpression methodCall)
{
this.MethodCall = methodCall;
}
public MethodCallExpression MethodCall { get; private set; }
public object GetValue()
{
var parent = MethodCall.Object ?? MethodCall.Arguments.FirstOrDefault();
return Expression.Lambda<Func<object>>(parent).Compile()();
}
public bool IsLocalList()
{
Expression parent = MethodCall.Object ?? MethodCall.Arguments.FirstOrDefault();
while (parent != null) {
if (parent is ConstantExpression)
return true;
var member = parent as MemberExpression;
if (member != null) {
parent = member.Expression;
} else {
parent = null;
}
}
return false;
}
}
private class FindExpressionVisitor<T> : ExpressionVisitor where T : Expression
{
public List<T> FoundItems { get; private set; }
public FindExpressionVisitor()
{
this.FoundItems = new List<T>();
}
public override Expression Visit(Expression node)
{
var found = node as T;
if (found != null) {
this.FoundItems.Add(found);
}
return base.Visit(node);
}
}
public static List<T> ToChunkedList<T, TValue>(this IQueryable<T> query, IEnumerable<TValue> list, int chunkSize)
{
var finder = new FindExpressionVisitor<MethodCallExpression>();
finder.Visit(query.Expression);
var methodCalls = finder.FoundItems.Where(p => p.Method.Name == "Contains").Select(p => new ContainsExpression(p)).Where(p => p.IsLocalList()).ToList();
var localLists = methodCalls.Where(p => p.GetValue() == list).ToList();
If the local collection passed in the ToChunkedList method was found in the query expression, I replace the Contains call to the original list with a new call to a temporary list containing the ids for one batch.
if (localLists.Any()) {
var result = new List<T>();
var valueList = new List<TValue>();
var containsMethod = typeof(Enumerable).GetMethods(BindingFlags.Static | BindingFlags.Public)
.Single(p => p.Name == "Contains" && p.GetParameters().Count() == 2)
.MakeGenericMethod(typeof(TValue));
var queryExpression = query.Expression;
foreach (var item in localLists) {
var parameter = new List<Expression>();
parameter.Add(Expression.Constant(valueList));
if (item.MethodCall.Object == null) {
parameter.AddRange(item.MethodCall.Arguments.Skip(1));
} else {
parameter.AddRange(item.MethodCall.Arguments);
}
var call = Expression.Call(containsMethod, parameter.ToArray());
var replacer = new ExpressionReplacer(item.MethodCall,call);
queryExpression = replacer.Visit(queryExpression);
}
var chunkQuery = query.Provider.CreateQuery<T>(queryExpression);
for (int i = 0; i < Math.Ceiling((decimal)list.Count() / chunkSize); i++) {
valueList.Clear();
valueList.AddRange(list.Skip(i * chunkSize).Take(chunkSize));
result.AddRange(chunkQuery.ToList());
}
return result;
}
// if the collection was not found return query.ToList()
return query.ToList();
Expression Replacer:
private class ExpressionReplacer : ExpressionVisitor {
private Expression find, replace;
public ExpressionReplacer(Expression find, Expression replace)
{
this.find = find;
this.replace = replace;
}
public override Expression Visit(Expression node)
{
if (node == this.find)
return this.replace;
return base.Visit(node);
}
}
Please allow me to provide an alternative to the Chunky approach.
The technique involving Contains in your predicate works well for:
A constant list of values (no volatile).
A small list of values.
Contains will do great if your local data has those two characteristics because these small set of values will be hardcoded in the final SQL query.
The problem begins when your list of values has entropy (non-constant). As of this writing, Entity Framework (Classic and Core) do not try to parameterize these values in any way, this forces SQL Server to generate a query plan every time it sees a new combination of values in your query. This operation is expensive and gets aggravated by the overall complexity of your query (e.g. many tables, a lot of values in the list, etc.).
The Chunky approach still suffers from this SQL Server query plan cache pollution problem, because it does not parametrizes the query, it just moves the cost of creating a big execution plan into smaller ones that are more easy to compute (and discard) by SQL Server, furthermore, every chunk adds an additional round-trip to the database, which increases the time needed to resolve the query.
An Efficient Solution for EF Core
🎉 NEW! QueryableValues EF6 Edition has arrived!
For EF Core keep reading below.
Wouldn't it be nice to have a way of composing local data in your query in a way that's SQL Server friendly? Enter QueryableValues.
I designed this library with these two main goals:
It MUST solve the SQL Server's query plan cache pollution problem ✅
It MUST be fast! ⚡
It has a flexible API that allows you to compose local data provided by an IEnumerable<T> and you get back an IQueryable<T>; just use it as if it were another entity of your DbContext (really), e.g.:
// Sample values.
IEnumerable<int> values = Enumerable.Range(1, 1000);
// Using a Join (query syntax).
var query1 =
from e in dbContext.MyEntities
join v in dbContext.AsQueryableValues(values) on e.Id equals v
select new
{
e.Id,
e.Name
};
// Using Contains (method syntax)
var query2 = dbContext.MyEntities
.Where(e => dbContext.AsQueryableValues(values).Contains(e.Id))
.Select(e => new
{
e.Id,
e.Name
});
You can also compose complex types!
It goes without saying that the provided IEnumerable<T> is only enumerated at the time that your query is materialized (not before), preserving the same behavior of EF Core in this regard.
How Does It Works?
Internally QueryableValues creates a parameterized query and provides your values in a serialized format that is natively understood by SQL Server. This allows your query to be resolved with a single round-trip to the database and avoids creating a new query plan on subsequent executions due to the parameterized nature of it.
Useful Links
Nuget Package
GitHub Repository
Benchmarks
SQL Server Cache Pollution Problem
QueryableValues is distributed under the MIT license
Linqkit to the rescue! Might be a better way that does it directly, but this seems to work fine and makes it pretty clear what's being done. The addition being AsExpandable(), which lets you use the Invoke extension.
using LinqKit;
public static IEnumerable<TEntity> ChunkyContains<TEntity, TContains>(
this IQueryable<TEntity> query,
Expression<Func<TEntity,TContains>> match,
IEnumerable<TContains> containList,
int chunkSize = 500)
{
return containList
.ToChunks(chunkSize)
.Select (chunk => query.AsExpandable()
.Where(x => chunk.Contains(match.Invoke(x))))
.SelectMany(x => x);
}
You might also want to do this:
containsList.Distinct()
.ToChunks(chunkSize)
...or something similar so you don't get duplicate results if something this occurs:
query.ChunkyContains(x => x.Id, new List<int> { 1, 1 }, 1);
Another way would be to build the predicate this way (of course, some parts should be improved, just giving the idea).
public static Expression<Func<TEntity, bool>> ContainsPredicate<TEntity, TContains>(this IEnumerable<TContains> chunk, Expression<Func<TEntity, TContains>> match)
{
return Expression.Lambda<Func<TEntity, bool>>(Expression.Call(
typeof (Enumerable),
"Contains",
new[]
{
typeof (TContains)
},
Expression.Constant(chunk, typeof(IEnumerable<TContains>)), match.Body),
match.Parameters);
}
which you could call in your ChunkContains method
return containList.ToChunks(chunkSize)
.Select(chunk => query.Where(ContainsPredicate(chunk, match)))
.SelectMany(x => x);
Using a stored procedure with a table valued parameter could also work well. You in effect write a joint In the stored procedure between your table / view and the table valued parameter.
https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/table-valued-parameters
I had the following query using normal linq and it was working great (using anonymous type),
var result = from s in Items
group s by s.StartTime into groupedItems
select new {groupedItems.Key, Items= groupedItems.OrderBy(x => x.Name) };
But using Dynamic Linq I cannot get it to order by within the groupby.
result = Items.GroupBy("StartTime", "it").OrderBy("Name");
It states the Name isn't available. It is worth noting that if I take my OrderBy off, everything works great but items inside each "Key" are not ordered.
This is a good question!
I simulated your situation by creating a class called Item.
public class Item
{
public DateTime StartTime { get; set; }
public string Name { get; set; }
}
and then created a basic list of items to do the groupby.
List<Item> Items = new List<Item>()
{
new Item() { StartTime = DateTime.Today, Name = "item2"},
new Item() { StartTime = DateTime.Today, Name = "item1"},
new Item() { StartTime = DateTime.Today.AddDays(-1), Name = "item3"},
};
Now the big difference in the 2 queries is where the order by is being performed. In the first query, when you perform groupedItems.OrderBy(x => x.Name) its being performed on a IGrouping<DateTime,Item> or a single entry as it iterates through all the groupings.
In the second query, the orderby is being performed after the fact. This means you're doing an orderby on a IEnumerable<IGrouping<DateTime,Item>> because the iterations have already happened.
Since Microsoft was nice they added something to help deal with this for expressions. This overload allows you to specify the item returned as it iterates through the collection. Here's an example of the code:
var expressionResult = Items.GroupBy(x => x.StartTime,
(key, grpItems) => new { key, Items = grpItems.OrderBy(y => y.Name) });
The second part of the GroupBy you can specify a lambda expression that takes a key and a grouping of items under that key and return an entry that you specify, which is the same as you're doing in the original query.
Hope this helps!
I have a function inside a class that will run a Linq to Entities query (or any type of Linq query actually), and it's gonna return 2 columns in the resultset. I would like to return an object to whoever is calling my function that will allow Intellisense to know what I have returned.
Let me explain. If I have a function like this:
public static IQueryable GetInfo(MyEntityModel oEntityModel)
{
var query =
(from t in oEntityModel.Table1
from u in t.Table2
where t.Status == true &&
u.Status == true
select new
{
t.Column1,
u.Column2
})
return query;
}
What can (should) I put instead of IQueryable so that whoever calls my GetInfo function, will get Intellisense from the resultset, and show that it has a Column1 and Column2?
var linqresult = ClsLinqTeste.GetInfo(oEntityModel);
if (linqresult.Column1 == 1)
{
foreach (var oItem in linqresult)
{
.. do stuff...
}
}
Tks
You cannot return an anonymous type from a function, they are strictly "inline" classes. When you return it, the foreach loop will only be able to interpret the result as an plain object. I guess you could use reflection to query the property names and values, however it seems much more straight forward to define a data transfer type to hold the results.
See this question, and this blog post.
So you could create a simple struct or class:
public class MyDataResult
{
public object Column1 { get; set; }
public object Column2 { get; set; }
}
Then modify your query in the function:
public static IQueryable<MyDataResult> GetInfo(MyEntityModel oEntityModel)
{
var query =
(from t in oEntityModel.Table1
from u in t.Table2
where t.Status == true &&
u.Status == true
select new MyDataResult
{
Column1 = t.Column1,
Column2 = u.Column2
})
return query;
}
Something like that should work. Note that I used "object" for the properties in MyDataResult. I don't know the types of the columns you are returning, you should use the actual types in order to get full intellisense.
You are returning a collection of anonymous types, they will be casted to objects, so when you try to iterate over them, altough they will be your objects (and they will contain your properties) at compile time they will be casted to objects:
foreach (var x in ClsLinqTeste.GetInfo(oEntityModel))
{
//x is an Object
}
You can read more about it here.
If you want to have intellisense, I suggest you create a custom class they will hold your properties and return not an anonymous type (using new {}) but object of your class (new MyClass(prop1, prop2)). You also need to change signature of your method, so it returns IQueryable<YourClass> and not just plain non-generic IQueryable.
As others have said, creating a new type to hold the two columns is usually the best option.
But if, for some reason, you don't want to do that and you are using .Net 4.0, you can use Tuple:
public static IQueryable<Tuple<Column1Type, Column2Type>>
GetInfo(MyEntityModel oEntityModel)
{
return from …
select Tuple.Create(t.Column1, u.Column2);
}
var linqresult = ClsLinqTeste.GetInfo(oEntityModel);
foreach (var oItem in linqresult)
Console.WriteLIne(oItem.Item1, oItem.Item2);
When you return your resultset AsQueryable, the app is already able to give you intellisense, however in your example, you must specify either .FirstOrDefault if you know your collection will only have a single row, or iterate over your collection to get the items from it, like so:
This is what you're doing:
var linqresult = ClsLinqTeste.GetInfo(oEntityModel);
if (linqresult.Column1 == 1)
{
..do stuff...
}
This is how you should do it:
var linqresult = ClsLinqTeste.GetInfo(oEntityModel);
foreach(var item in linqresult)
{
if (item.Column1 == 1)
{
..do stuff...
}
}
You must iterate over linqresult because when you query with link, it returns a result set, even if it just has one column. As with any collection, your data columns aren't available on the whole result set, only with individual items.
If you want to strongly typed enumerate a non-generic IEnumerable (IEnumerable.GetEnumerator() instead of IEnumerable<T>.GetEnumerator<T>()) you can use the Cast<>() extension, like so
var myquery = GetQueryable();
for (var item in myquery.Cast<MyDataType>())
{
// use item.Column1 directly and strongly typed with intellisense
}
I have a couple of tables with similar relationship structure to the standard Order, OrderLine tables.
When creating a data context, it gives the Order class an OrderLines property that should be populated with OrderLine objects for that particular Order object.
Sure, by default it will delay load the stuff in the OrderLine property but that should be fairly transparent right?
Ok, here is the problem I have: I'm getting an empty list when I go MyOrder.OrderLines but when I go myDataContext.OrderLines.Where(line => line.OrderId == 1) I get the right list.
public void B()
{
var dbContext = new Adis.CA.Repository.Database.CaDataContext(
"<connectionString>");
dbContext.Connection.Open();
dbContext.Transaction = dbContext.Connection.BeginTransaction();
try
{
//!!!Edit: Imortant to note that the order with orderID=1 already exists
//!!!in the database
//just add some new order lines to make sure there are some
var NewOrderLines = new List<OrderLines>()
{
new OrderLine() { OrderID=1, LineID=300 },
new OrderLine() { OrderID=1, LineID=301 },
new OrderLine() { OrderID=1, LineID=302 },
new OrderLine() { OrderID=1, LineID=303 }
};
dbContext.OrderLines.InsertAllOnSubmit(NewOrderLines);
dbContext.SubmitChanges();
//this will give me the 4 rows I just inserted
var orderLinesDirect = dbContext.OrderLines
.Where(orderLine => orderLine.OrderID == 1);
var order = dbContext.Orders.Where(order => order.OrderID == 1);
//this will be an empty list
var orderLinesThroughOrder = order.OrderLines;
}
catch (System.Data.SqlClient.SqlException e)
{
dbContext.Transaction.Rollback();
throw;
}
finally
{
dbContext.Transaction.Rollback();
dbContext.Dispose();
dbContext = null;
}
}
So as far as I can see, I'm not doing anything particularly strange but I would think that orderLinesDirect and orderLinesThroughOrder would give me the same result set.
Can anyone tell me why it doesn't?
You're just adding OrderLines; not any actual Orders. So the Where on dbContext.Orders returns an empty list.
How you can still find the property OrderLines on order I don't understand, so I may be goofing up here.
[Edit]
Could you update the example to show actual types, especially of the order variable? Imo, it shoud be an IQueryable<Order>, but it's strange that you can .OrderLines into that. Try adding a First() or FirstOrDefault() after the Where.
how come this work
public IQueryable<Category> getCategories(int postId)
{
subnusMVCRepository<Categories> categories = new subnusMVCRepository<Categories>();
subnusMVCRepository<Post_Category_Map> postCategoryMap = new subnusMVCRepository<Post_Category_Map>();
var query = from c in categories.GetAll()
join pcm in postCategoryMap.GetAll() on c.CategoryId equals pcm.CategoryId
where pcm.PostId == 1
select new Category
{
Name = c.Name,
CategoryId = c.CategoryId
};
return query;
}
but this does not
public IQueryable<Category> getCategories(int postId)
{
subnusMVCRepository<Categories> categories = new subnusMVCRepository<Categories>();
subnusMVCRepository<Post_Category_Map> postCategoryMap = new subnusMVCRepository<Post_Category_Map>();
var query = from c in categories.GetAll()
join pcm in postCategoryMap.GetAll() on c.CategoryId equals pcm.CategoryId
where pcm.PostId == postId
select new Category
{
Name = c.Name,
CategoryId = c.CategoryId
};
return query;
}
The issue is most likely in the implementation of the query provider.
pcm.PostId == 1
and
pcm.PostId == postId
actually have a big difference. In the expression tree the first is generated as a ConstantExpression which doesnt need to be evaulated.
With the second, the compiler actually generates an inner class here (this is the _DisplayClassX that you see). This class will have a property (will most likely be the same name as your parameter) and the expression tree will create a MemberAccessExpression which points to the auto-generated DisplayClassX. When you query provider comes accross this you need to Compile() the Lambda expression and evaluate the delegate to get the value to use in your query.
Hope this helps.
cosullivan
The problem is not the linq itself,
you need to be sure that the context or provider object is able to fetch the data.
try testing the
subnusMVCRepository<Categories> categories = new subnusMVCRepository<Categories>();
subnusMVCRepository<Post_Category_Map> postCategoryMap = new subnusMVCRepository<Post_Category_Map>();
objects and see if they are populated or if they behaving as required.
you may want to search the generated code for c__DisplayClass1 and see what you can see there. some times the generated code dose some weird things.
when you step into you code check the locals and the variable values. this may also give you some clues.
Edit : Have you tried to return a List<> collection ? or an Enumerable type?
Edit : What is the real type of the item and query may not be iterable