If I were to select some rows based on certain criteria I can use ICriterion object in NHibernate.Criterion, such as this:
public List<T> GetByCriteria()
{
SimpleExpression newJobCriterion =
NHibernate.Criterion.Expression.Eq("LkpStatu", statusObject);
ICriteria criteria = Session.GetISession().CreateCriteria(typeof(T)).SetMaxResults(maxResults);
criteria.Add(newJobCriterion );
return criteria.List<T>();
}
Or I can use LINQ's where clause to filter what I want:
public List<T> GetByCriteria_LINQ()
{
ICriteria criteria = Session.GetISession().CreateCriteria(typeof(T)).SetMaxResults(maxResults);
return criteria.Where(item=>item.LkpStatu=statusObject).ToList();
}
I would prefer the second one, of course. Because
It gives me strong typing
I don't need to learn yet-another-syntax in the form of NHibernate
The issue is is there any performance advantage of the first one over the second one? From what I know, the first one will create SQL queries, so it will filter the data before pass into the memory. Is this kind of performance saving big enough to justify its use?
As usual it depends. First note that in your second snippet there is .List() missing right after return criteria And also note that you won't get the same results on both examples. The first one does where and then return top maxResults, the second one however first selects top maxResults and then does where.
If your expected result set is relatively small and you are likely to use some of the results in lazy loads then it's actually better to take the second approach. Because all entities loaded through a session will stay in its first level cache.
Usually however you don't do it this way and use the first approach.
Perhaps you wanted to use NHibernate.Linq (located in Contrib project ). Which does linq translation to Criteria for you.
I combine the two and made this:
var crit = _session.CreateCriteria(typeof (T)).SetMaxResults(100);
return (from x in _session.Linq<T>(crit) where x.field == <something> select x).ToList();
Related
What is the explanation for EF downloading all result rows when AsEnumerable() is used?
What I mean is that this code:
context.Logs.AsEnumerable().Where(x => x.Id % 2 == 0).Take(100).ToList();
will download all the rows from the table before passing any row to the Where() method and there could be millions of rows in the table.
What I would like it to do, is to download only enough to gather 100 rows that would satisfy the Id % 2 == 0 condition (most likely just around 200 rows).
Couldn't EF do on demand loading of rows like you can with plain ADO.NET using Read() method of SqlDataReader and save time and bandwidth?
I suppose that it does not work like that for a reason and I'd like to hear a good argument supporting that design decision.
NOTE: This is a completely contrived example and I know normally you should not use EF this way, but I found this in some existing code and was just surprised my assumptions turned out to be incorrect.
The short answer: The reason for the different behaviors is that, when you use IQueryable directly, a single SQL query can be formed for your entire LINQ query; but when you use IEnumerable, the entire table of data must be loaded.
The long answer: Consider the following code.
context.Logs.Where(x => x.Id % 2 == 0)
context.Logs is of type IQueryable<Log>. IQueryable<Log>.Where is taking an Expression<Func<Log, bool>> as the predicate. The Expression represents an abstract syntax tree; that is, it's more than just code you can run. Think of it as being represented in memory, at runtime, like this:
Lambda (=>)
Parameters
Variable: x
Body
Equals (==)
Modulo (%)
PropertyAccess (.)
Variable: x
Property: Id
Constant: 2
Constant: 0
The LINQ-to-Entities engine can take context.Logs.Where(x => x.Id % 2 == 0) and mechanically convert it into a SQL query that looks something like this:
SELECT *
FROM "Logs"
WHERE "Logs"."Id" % 2 = 0;
If you change your code to context.Logs.Where(x => x.Id % 2 == 0).Take(100), the SQL query becomes something like this:
SELECT *
FROM "Logs"
WHERE "Logs"."Id" % 2 = 0
LIMIT 100;
This is entirely because the LINQ extension methods on IQueryable use Expression instead of just Func.
Now consider context.Logs.AsEnumerable().Where(x => x.Id % 2 == 0). The IEnumerable<Log>.Where extension method is taking a Func<Log, bool> as a predicate. That is only runnable code. It cannot be analyzed to determine its structure; it cannot be used to form a SQL query.
Entity Framework and Linq use lazy loading. It means (among other things) that they will not run the query until they need to enumerate the results: for instance using ToList() or AsEnumerable(), or if the result is used as an enumerator (in a foreach for instance).
Instead, it builds a query using predicates, and returns IQueryable objects to further "pre-filter" the results before actually returning them. You can find more infos here for instance. Entity framework will actually build a SQL query depending on the predicates you have passed it.
In your example:
context.Logs.AsEnumerable().Where(x => x.Id % 2 == 0).Take(100).ToList();
From the Logs table in the context, it fetches all, returns a IEnumerable with the results, then filters the result, takes the first 100, then lists the results as a List.
On the other hand, just removing the AsEnumerable solves your problem:
context.Logs.Where(x => x.Id % 2 == 0).Take(100).ToList();
Here it will build a query/filter on the result, then only once the ToList() is executed, query the database.
It also means that you can dynamically build a complex query without actually running it on the DB it until the end, for instance:
var logs = context.Logs.Where(a); // first filter
if (something) {
logs = logs.Where(b); // second filter
}
var results = logs.Take(100).ToList(); // only here is the query actually executed
Update
As mentionned in your comment, you seem to already know what I just wrote, and are just asking for a reason.
It's even simpler: since AsEnumerable casts the results to another type (a IQueryable<T> to IEnumerable<T> in this case), it has to convert all the results rows first, so it has to fetch the data first. It's basically a ToList in this case.
Clearly, you understand why it's better to avoid using AsEnumerable() the way you do in your question.
Also, some of the other answers have made it very clear why calling AsEnumerable() changes the way the query is performed and read. In short, it's because you are then invoking IEnumrable<T> extension methods rather than the IQueryable<T> extension methods, the latter allowing you to combine predicates before executing the query in the database.
However, I still feel that this doesn't answer your actual question, which is a legitimate question. You said (emphasis mine):
What I mean is that this code:
context.Logs.AsEnumerable().Where(x => x.Id % 2 == 0).Take(100).ToList();
will download all the rows from the table before passing any row to the Where() method and there could be millions of rows in the table.
My question to you is: what made you conclude that this is true?
I would argue that, because you are using IEnumrable<T> instead of IQueryable<T>, it's true that the query being performed in the database will be a simple:
select * from logs
... without any predicates, unlike what would have happened if you had used IQueryable<T> to invoke Where and Take.
However, the AsEnumerable() method call does not fetch all the rows at that moment, as other answers have implied. In fact, this is the implementation of the AsEnumerable() call:
public static IEnumerable<TSource> AsEnumerable<TSource>(this IEnumerable<TSource> source)
{
return source;
}
There is no fetching going on there. In fact, even the calls to IEnumerable<T>.Where() and IEnumerable<T>.Take() don't actually start fetching any rows at that moment. They simply setup wrapping IEnumerables that will filter results as they are iterated on. The fetching and iterating of the results really only begins when ToList() is called.
So when you say:
Couldn't EF do on demand loading of rows like you can with plain ADO.NET using Read() method of SqlDataReader and save time and bandwidth?
... again, my question to you would be: doesn't it do that already?
If your table had 1,000,000 rows, I would still expect your code snippet to only fetch up to 100 rows that satisfy your Where condition, and then stop fetching rows.
To prove the point, try running the following little program:
static void Main(string[] args)
{
var list = PretendImAOneMillionRecordTable().Where(i => i < 500).Take(10).ToList();
}
private static IEnumerable<int> PretendImAOneMillionRecordTable()
{
for (int i = 0; i < 1000000; i++)
{
Console.WriteLine("fetching {0}", i);
yield return i;
}
}
... when I run it, I only get the following 10 lines of output:
fetching 0
fetching 1
fetching 2
fetching 3
fetching 4
fetching 5
fetching 6
fetching 7
fetching 8
fetching 9
It doesn't iterate through the whole set of 1,000,000 "rows" even though I am chaining Where() and Take() calls on IEnumerable<T>.
Now, you do have to keep in mind that, for your little EF code snippet, if you test it using a very small table, it may actually fetch all the rows at once, if all the rows fit within the value for SqlConnection.PacketSize. This is normal. Every time SqlDataReader.Read() is called, it never only fetches a single row at a time. To reduce the amount of network call roundtrips, it will always try to fetch a batch of rows at a time. I wonder if this is what you observed, and this mislead you into thinking that AsEnumerable() was causing all rows to be fetched from the table.
Even though you will find that your example doesn't perform nearly as bad as you thought, this would not be a reason not to use IQueryable. Using IQueryable to construct more complex database queries will almost always provide better performance, because you can then benefit from database indexes, etc to fetch results more efficiently.
AsEnumerable() eagerly loads the DbSet<T> Logs
You probably want something like
context.Logs.Where(x => x.Id % 2 == 0).AsEnumerable();
The idea here is that you're applying a predicate filter to the collection before actually loading it from the database.
An impressive subset of the world of LINQ is supported by EF. It will translate your beautiful LINQ queries into SQL expressions behind the scenes.
I have come across this before.
The context command is not executed until a linq function is called, because you have done
context.Logs.AsEnumerable()
it has assumed you have finished with the query and therefore compiled it and returns all rows.
If you changed this to:
context.Logs.Where(x => x.Id % 2 == 0).AsEnumerable()
It would compile a SQL statement that would get only the rows where the id is modular 2.
Similarly if you did
context.Logs.Where(x => x.Id % 2 == 0).Take(100).ToList();
that would create a statement that would get the top 100...
I hope that helps.
LinQ to Entities has a store expression formed by all the Linq methods before It goes to an enumeration.
When you use AsEnumerable() and then Where() like this:
context.Logs.Where(...).AsEnumerable()
The Where() knows that the previous chain call has a store expression so he appends his predicate to It for lazy loading.
The overload of Where that is being called is different if you call this:
context.Logs.AsEnumerable().Where(...)
Here the Where() only knows that his previous method is an enumeration (it could be any kind of "enumerable" collection) and the only way that he can apply his condition is iterating over the collection with the IEnumerable implementation of the DbSet class, which must to retrieve the records from the database first.
I don't think you should ever use this:
context.Logs.AsEnumerable().Where(x => x.Id % 2 == 0).Take(100).ToList();
The correct way of doing things would be:
context.Logs.AsQueryable().Where(x => x.Id % 2 == 0).Take(100).ToList();
Answer with explanations here:
What's the difference(s) between .ToList(), .AsEnumerable(), AsQueryable()?
Why use AsQueryable() instead of List()?
I think I'm not undertanding LINQ well. I want to do:
foreach (MyObjetc myObject in myObjectCollection)
{
myObjet.MyProperty = newValue
}
Just change all values for a property in all elements of my collection.
Using LINQ wouldn't be this way?
myObjectCollection.Select(myObject => myObject.MyProperty = newValue)
It doesn't work. The property value is not changed. Why?
Edit:
Sorry, guys. Certainly, foreach is the right way. But,in my case, I must repeat the foreach in many collections, and I didn't want to repeat the loop. So, finally, I have found an 'inermediate' solution, the 'foreach' method, just similar to the 'Select':
myObjectCollection.ForEach(myObject => myObject.MyProperty = newValue)
Anyway may be it's not as clear as the more simple:
foreach (MyObjetc myObject in myObjectCollection) myObjet.MyProperty = newValue;
First off, this is not a good idea. See below for arguments against it.
It doesn't work. The property value is not changed. Why?
It doesn't work because Select() doesn't actually iterate through the collection until you enumerate it's results, and it requires an expression that evaluates to a value.
If you make your expression return a value, and add something that will fully evaluate the query, such as ToList(), to the end, then it will "work", ie:
myObjectCollection.Select(myObject => { myObject.MyProperty = newValue; return myObject;}).ToList();
That being said, ToList() has some disadvantages - mainly, it's doing a lot of extra work (to create a List<T>) that's not needed, which adds a large cost. Avoiding it would require enumerating the collection:
foreach(var obj in myObjectCollection.Select(myObject => { myObject.MyProperty = newValue; return myObject; }))
{ }
Again, I wouldn't recommend this. At this point, the Select option is far uglier, more typing, etc. It's also a violation of the expectations involved with LINQ - LINQ is about querying, which suggests there shouldn't be side effects when using LINQ, and the entire purpose here is to create side effects.
But then, at this point, you're better off (with less typing) doing it the "clear" way:
foreach (var obj in myObjectCollection)
{
obj.MyProperty = newValue;
}
This is shorter, very clear in its intent, and very clean.
Note that you can add a ForEach<T> extension method which performs an action on each object, but I would still recommend avoiding that. Eric Lippert wrote a great blog post about the subject that is worth a read: "foreach" vs "ForEach".
As sircodesalot mentioned, you can't use linq to do something like that. Remember that linq is a querying language, which means all you can do is query it. Doing changes must be done in other logic.
What you could do, if you don't want to do it the first way and if your collection is a list already (but not an IEnumerable) you can use the ForEach extension method in linq to do what you're asking.
One other point I should mention is that the Select method does a projection of some specific information to return to an IEnumerable. So, if you wanted to grab only a specific property from a collection you would use that. That's all it does.
I am new to LINQ queries and to EF too, I usually work with MySQL and I can't guess how to write really simples queries.
I'd like to select all results from a table. So, I used like this:
ZXContainer db = new ZXContainer();
ViewBag.ZXproperties = db.ZXproperties.All();
But I see that I have to write something inside All(---).
Could someone guide me in how could I do that? And if someone has any good link for references too, I thank so much.
All() is an boolean evaluation performed on all of the elements in a collection (though immediately returns false when it reaches an element where the evaluation is false), for example, you want to make sure that all of said ZXproperties have a certain field set as true:
bool isTrue = db.ZXproperties.All(z => z.SomeFieldName == true);
Which will either make isTrue true or false. LINQ is typically lazy-loading, so if you're calling db.ZXproperties directly, you have access to all of the objects as is, but it isn't quite what you're looking for. You can either load all of the objects at the variable assignment with an .ToList():
ViewBag.ZXproperties = db.ZXproperties.ToList();
or you can use the below expression:
ViewBag.ZXproperties = from s in db.ZXproperties
select s;
Which is really no different than saying:
ViewBag.ZXproperties = db.ZXproperties;
The advantage of .ToList() is that if you are wanting to do multiple calls on this ViewBag.ZXproperties, it will only require the initial database call when it is assigning the variable. Alternatively, if you do any form of queryable action on the data, such as .Where(), you'll have another query performed, which is less than ideal if you already have the data to work with.
To select everything, just skip the .All(...), as ZXproperties allready is a collection.
ZXContainer db = new ZXContainer();
ViewBag.ZXproperties = db.ZXproperties;
You might want (or sometimes even need) to call .ToList() on this collection before use...
You don't use All. Just type
ViewBag.ZXproperties = db.ZXproperties;
or
ViewBag.ZXproperties = db.ZXproperties.ToList();
The All method is used to determine if all items of collection match some condition.
If you just want all of the items, you can just use it directly:
ViewBag.ZXproperties = db.ZXproperties;
If you want this evaluated immediately, you can convert it to a list:
ViewBag.ZXproperties = db.ZXproperties.ToList();
This will force it to be pulled across the wire immediately.
You can use this:
var result = db.ZXproperties.ToList();
For more information on linq see 101 linq sample.
All is some checking on all items and argument in it, called lambda expression.
DynamicObject LINQ query with the List compiles fine:
List<string> list = new List<string>();
var query = (from dynamic d in list where d.FirstName == "John" select d);
With our own custom class that we use for the "usual" LINQ compiler reports the error "An expression tree may not contain a dynamic
operation":
DBclass db = new DBclass();
var query = (from dynamic d in db where d.FirstName == "John" select d);
What shall we add to handle DynamicObject LINQ?
Does DBClass implement IEnumerable? Perhaps there is a method on it you should be calling to return an IEnumerable collection?
You could add a type, against which to write the query.
I believe your problem is, that in the first expression, where you are using the List<>, everything is done in memory using IEnumerable & Link-to-Objects.
Apparently, your DBClass is an IQueryable using Linq-to-SQL. IQueryables use an expression tree to build an SQL statement to send to the database.
In other words, despite looking much alike, the two statements are doing radically different things, one of which is allowed & one which isn't. (Much in the way var y = x * 5; will either succeed or fail depending on if x is an int or a string).
Further, your first example may compile, but as far as I can tell, it will fail when you run it. That's not a particular good benchmark for success.
The only way I see this working is if the query using dynamic is made on IEnumerables using Link-to-Objects. (Load the full table into a List, and then query on the list)
I've been developing a webapp using Linq to NHibernate for the past few months, but haven't profiled the SQL it generates until now. Using NH Profiler, it now seems that the following chunk of code hits the DB more than 3,000 times when the Linq expression is executed.
var activeCaseList = from c in UserRepository.GetCasesByProjectManagerID(consultantId)
where c.CompletionDate == null
select new { c.PropertyID, c.Reference, c.Property.Address, DaysOld = DateTime.Now.Subtract(c.CreationDate).Days, JobValue = String.Format("£{0:0,0}", c.JobValue), c.CurrentStatus };
Where the Repository method looks like:
public IEnumerable<Case> GetCasesByProjectManagerID(int projectManagerId)
{
return from c in Session.Linq<Case>()
where c.ProjectManagerID == projectManagerId
select c;
}
It appears to run the initial Repository query first, then iterates through all of the results checking to see if the CompletionDate is null, but issuing a query to get c.Property.Address first.
So if the initial query returns 2,000 records, even if only five of them have no CompletionDate, it still fires off an SQL query to bring back the address details for the 2,000 records.
The way I had imagined this would work, is that it would evaluate all of the WHERE and SELECT clauses and simply amalgamate them, so the inital query would be like:
SELECT ... WHERE ProjectManager = #p1 AND CompleteDate IS NOT NULL
Which would yield 5 records, and then it could fire the further 5 queries to obtain the addresses. Am I expecting too much here, or am I simply doing something wrong?
Anthony
Change the declaration of GetCasesByProjectManagerID:
public IQueryable<Case> GetCasesByProjectManagerID(int projectManagerId)
You can't compose queries with IEnumerable<T> - they're just sequences. IQueryable<T> is specifically designed for composition like this.
Since I can't add a comment yet. Jon Skeet is right you'll want to use IQueryable, this is allows the Linq provider to Lazily construct the SQL. IEnumerable is the eager version.