Entity Framework is building a SQL query that takes less than 2 seconds. Even though the query is ugly, it's fast. In the EF SELECT statementI'm building a DTO based on my entites. The DTO has around 20 properties and it's plain. So, EF is running a query that returns these 20 properties I need. Everything seems fine but building the DTO takes almost 10 seconds. I have already told EF not to track changes (AsNoTracking() over the db sets) so I'm not sure what's happening between the query is executed and the DTO is built.
What's EF doing under the hood that makes it so slow? Again, the query is fine and returns really fast but building the DTO with those properties is really slow.
I replaced this logic by a stored procedure that returns the same 20 properties and it is really, really fast. The DTO is the same so there has to be something in the middle, between reading the results of the query and building the DTO that is different.
I hope someone can help me figure out what's going on.
EDIT:
I'm adding how I'm populating the DTO based on a comment.
.Select(p => new PMPanelOffersDTO()
{
OfferId = p.Id,
OfferDate = p.DateCreated,
UserId = p.UserId,
OfferAmountId = p.AmountId,
OfferAmountTypeId = p.Amount != null ? p.Amount.OfferType.Id : default(int?),
OfferAmountType = p.Amount != null ? p.Amount.OfferType.Name : null,
TowerCompanyName = p.Amount != null && p.Amount.Tower != null ? p.Amount.Tower.CompanyName : null,
... { the same for the other properties }
Do you apply paging? (I mean skip&take) And most importantly where exactly ToList() is used?
We have exactly the same problem, solved in a such a way that use ToList() after DTO construction.
Try using .ToList() on the resulting object. That's all it took in our case. I agree it made no sense. Apparently the IEnumerable<> was not a happy thing
Related
Below is a snippit of simplified version of a problem I am having with Entity Framework v4 where the first load seems to take around 30 seconds on a table with 36 rows!
After that it is very quick to load until you change the search params, then it takes 30 seconds again but once that combination of search params has been done once it is quick.
This is repeated each time a different combination of params is used.
IQueryable<User> result= GetAllUsers();
if (!String.IsNullOrWhiteSpace(firstNameSearchParam))
{
result = result.Where(u => u.firstname.contains(firstNameSearchParam))
}
if (!String.IsNullOrWhiteSpace(lastNameSearchParam))
{
result = result.Where(u => u.lastname.contains(lastNameSearchParam))
}
Var ret = result.ToArray();
Any ideas would be really appreciated.
I'm not sure if pre-compiling the views will help. I tried but couldn't get it to work.
how long it takes when you execute the query from the sql side?
you can use this idea mentioned by scott here on this link Dynamic Linq
and i think this will work with you also with Entity Framework, and there is another idea that you can use the entity framework metadata
hope that this will help you
regards
I started using compiled queries to increase the performance of some commonly executed linq to entities queries. In one scenario I only boiled the query down to it's most basic form and pre-compiled that, then I tack on additional where clauses based on user input.
I seem to be losing the performance benefit of compiled queries in this particular case. Can someone explain why?
Here's an example of what I'm doing...
IEnumerable<Task> tasks = compiledQuery.Invoke(context, userId);
if(status != null)
{
tasks = tasks.Where(x=x.Status == status);
}
if(category != null)
{
tasks = tasks.Where(x=x.Category == category);
}
return tasks;
I think it's important to understand how Compiled Queries in EF work.
When you execute a query Entity Framework will map your expression tree with the help of your mapping file (EDMX or with code first your model definitions) to a SQL query. This can be a complex and performance intensive task.
Precompiling stores the results of these mapping phase so the next time you hit the query it has the SQL already available and it only has to set the current parameters.
The problem is that a precompiled query will lose it's performance benefit as soon as you modifie the query. Let's say you have the following:
IQueryable query = GetCompiledQuery(); // => db.Tasks.Where(t => t.Id == myId);
var notModifiedResult = query.ToList(); // Fast
int ModifiedResult = query.Count(); // Slow
With the first query you will have all the benefits of precompiling because EF has the SQL already generated for you and can execute this immediatly.
The second query will lose the precompiling because it has to regenerate it's SQL.
If you would now execute a query on notModifiedResult this will be a Linq To Objects one because you have already executed your SQL to the database and fetched all the elements in memory.
You can however chain Compiled Queries (that is, use a compiled query in another compiled query).
But your code would require a series of compiled queries:
- The default
- One where status != null
- One where category != null
- One where both status and category != null
(Note: I haven't done any EF work for ages, and then it was just pottering. This is just an informed guess, really.)
This could be the culprit:
IEnumerable<Task> tasks = compiledQuery.Invoke(context, userId);
Any further querying will have to be done within the .NET process, not in SQL. All the possible results will have to be fetched from the database and filtered locally. Try this instead:
IQueryable<Task> tasks = compiledQuery.Invoke(context, userId);
(Assuming that's valid, of course.)
The compiled query can't be changed, only the parameters can be changed. What you are doing here is actually running the query, and THEN filtering the results.
.Invoke(context, userId); // returns all the results
.Where(....) // filters on that entire collection
You can see if there is a clever way to restate your query, so that the parameters can be included in all cases, but not have any effect. I haven't worked with compiled queries, sorry about that, but does this work (using -1 as the "ignore" value)?
// bunch of code to define the compiled query part, copied from [msdn][1]
(ctx, total) => from order in ctx.SalesOrderHeaders
where (total == -1 || order.TotalDue >= total)
select order);
In SQL, you do this by either using dynamic sql, or having a default value (or null) that you pass in which indicates that parameter should be ignored
select * from table t
where
(#age = 0 or t.age = #age) and
(#weight is null or t.weight = #weight)
Any way to make this more efficient?
internal Func<enntities, IQueryable<CategoryList>> GetCategoryListWithPostingCount =
CompiledQuery.Compile((entities entities) =>
from c in entities.Categories.Include("Postings_Categories")
where c.ParentCategoryID == null
orderby c.DisplayOrder
select new CategoryList
{
ParentCategoryName = c.CategoryName,
ParentCategoryID = c.CategoryID,
SubCategories = (
from s in entities.Categories
where s.ParentCategoryID == c.CategoryID
select new SubCategoryList
{
PostingCount = s.Postings_Categories.Count,
SubCategoryName = s.CategoryName,
SubCategoryID = s.CategoryID
})
});
Any suggestions for improvement would depend on us knowing a lot more than just the LINQ query. For example, if you have a lot of SubCategoryLists per Category, and if Category has a lot of scalar data attached to it (big description text, etc), it might be faster to pull the category list in a separate database round-trip. But it probably won't.
Looking at what tables are joined and adding the appropriate indices could make a difference, too.
But all in all, I'd say this is a case of premature optimization. The code looks clean, and I can tell what you're trying to do with it, so call that good for now. If you find that it's a slow point in your program, worry about it then.
PS--The question is tagged LINQ-to-SQL, but this looks like LINQ-to-Entities to me...
I am using Entity Framework for the first time and noticed that the entities object returns entity collections.
DBEntities db = new DBEntities();
db.Users; //Users is an ObjectSet<User>
User user = db.Users.Where(x => x.Username == "test").First(); //Is this getting executed in the SQL or in memory?
user.Posts; //Posts is an EntityCollection<Post>
Post post = user.Posts.Where(x => x.PostID == "123").First(); //Is this getting executed in the SQL or in memory?
Do both ObjectSet and EntityCollection implement IQueryable? I am hoping they do so that I know the queries are getting executed at the data source and not in memory.
EDIT: So apparently EntityCollection does not while ObjectSet does. Does that mean I would be better off using this code?
DBEntities db = new DBEntities();
User user = db.Users.Where(x => x.Username == "test").First(); //Is this getting executed in the SQL or in memory?
Post post = db.Posts.Where(x => (x.PostID == "123")&&(x.Username == user.Username)).First(); // Querying the object set instead of the entity collection.
Also, what is the difference between ObjectSet and EntityCollection? Shouldn't they be the same?
Thanks in advance!
EDIT: Sorry, I'm new to this. I'm trying to understand. Attached EntityCollections are lazy loaded, so if I access them then memory is populated with them. Rather than doing two querys to the object sets like in my last edit, I am curious if this query would be more what I was after:
DBEntities db = new DBEntities();
User user = (from x in db.Users
from y in x.Posts
where x.Username == "test"
where y.PostID == 123
select x).First();
ObjectSet<T> does implement IQueryable<T>, but EntityCollection<T> does not.
The difference is that ObjectSet<T> is meant to be used for querying directly (which is why it does implement the interface). EntityCollection<T>, on the other hand, is used for the "many" end of a result set, typically returned in a query done on an ObjectSet<T>. As such, it impelments IEnumerable<T>, but not IQueryable<T> (as it's already the populated results of a query).
I was almost ready to say yes, they both do. Luckily I check the documentation first.
EntityCollection does not implement IQueryable.
As for the difference, ObjectSet<TEntity> represents the the objects generated from a table in a database. EntityCollection<TEntity> represents a collection of entity objects on the 'Many' side of One to Many or Many to Many relationship.
I've been developing a webapp using Linq to NHibernate for the past few months, but haven't profiled the SQL it generates until now. Using NH Profiler, it now seems that the following chunk of code hits the DB more than 3,000 times when the Linq expression is executed.
var activeCaseList = from c in UserRepository.GetCasesByProjectManagerID(consultantId)
where c.CompletionDate == null
select new { c.PropertyID, c.Reference, c.Property.Address, DaysOld = DateTime.Now.Subtract(c.CreationDate).Days, JobValue = String.Format("£{0:0,0}", c.JobValue), c.CurrentStatus };
Where the Repository method looks like:
public IEnumerable<Case> GetCasesByProjectManagerID(int projectManagerId)
{
return from c in Session.Linq<Case>()
where c.ProjectManagerID == projectManagerId
select c;
}
It appears to run the initial Repository query first, then iterates through all of the results checking to see if the CompletionDate is null, but issuing a query to get c.Property.Address first.
So if the initial query returns 2,000 records, even if only five of them have no CompletionDate, it still fires off an SQL query to bring back the address details for the 2,000 records.
The way I had imagined this would work, is that it would evaluate all of the WHERE and SELECT clauses and simply amalgamate them, so the inital query would be like:
SELECT ... WHERE ProjectManager = #p1 AND CompleteDate IS NOT NULL
Which would yield 5 records, and then it could fire the further 5 queries to obtain the addresses. Am I expecting too much here, or am I simply doing something wrong?
Anthony
Change the declaration of GetCasesByProjectManagerID:
public IQueryable<Case> GetCasesByProjectManagerID(int projectManagerId)
You can't compose queries with IEnumerable<T> - they're just sequences. IQueryable<T> is specifically designed for composition like this.
Since I can't add a comment yet. Jon Skeet is right you'll want to use IQueryable, this is allows the Linq provider to Lazily construct the SQL. IEnumerable is the eager version.