Linq mapping EntityDb to Dto is very slow.
I get the only ONE record from table with join other tables by Id.
Example:
DataAccess da;
var res = from t1 in da.Table1
join t2 in da.Table2 on t1.rf_table2Id equals t2.Table2Id
join etc...
over 20 joins...
where t1.Table1Id == 20 /*example*/
select new MyDto
{
Id = t1.Table1Id,
Name = t1.Name,
Type = new ReferenceDto()
{
Id = t2.Table2Id,
Name = t2.Name
},
and etc...
over 50 fields
};
The problem in mapping. If I get the record without mapping all fields, only Id,
res.FirstOrDefault() executes in 100-500 milliseconds, fast.
But, if I map all fields the res.FirstOrDefault() takes 3 seconds to execute, which is too slow.
My DTO is structure view.
In SQL Server Profiler, the query runs very fast.
What can I do ?
I need to get more information at the one time, by Id the record.
Solution is:
1) I get the original sql-query from SQL Server Profiler.
2) Create plain DTO class (over 200 fields) for execute query by context.ExecuteStoreQuery.
3) Then, I've done mapping the result plain DTO to my structure DTO Model.
Conclusion is:
About 3 seconds to execute linq statement with mapping to big structure DTO with about 200 fields.
And ONLY 500-600 milliseconds with new solution, described above!!!
Improved by 5-6 times!!!
With large DTO structure linq statement is bad, when we want to get the large record data (with many tables) by Id.
P.S.
Structure DTO Model is class with sub-classes, example:
select new MyDto
{
Id = t1.Table1Id,
Name = t1.Name,
Type = new ReferenceDto()
{
Id = t2.Table2Id,
Name = t2.Name
},
and etc...
over 50 fields
};
Related
I'm trying to get specific rows from the table by ids, but what I have below is not working.
#Query(value = "SELECT * FROM row r where r.row_id = :row_ids", nativeQuery = true)
List<Object> temp(#Param("albumsIds") String row_ids);
row_ids is all the ids separated by an "or" - id:1 or id:2 or id:3
I'm just trying to do select * from row r where r.row_id = id:1, or r.row_id = id:2, or r.row_id = id:3
Does anyone have an idea what the problem is, or is there a better way to do it?
Simply use:
List<Object> findByIdIn(Collection<Integer> ids);
That will automatically setup the derived query from the query method name itself.
Alternatively, if you want to provide the query programmatically, then the query should be:
#Query(value = "SELECT * FROM row R WHERE R.row_id IN :ids", nativeQuery = true)
List<Object> temp(#Param("ids") Collection<Integer> ids);
MySQL lets use use the IN keyword in queries, which let's us provide a CSV for the IDs to parse and returns any records which ID is in the CSV. This obviously can be used on any data, just for this purpose we'll use the IDs as an example.
Using the method above should minimize the risk of SQL injection significantly, as the Java Collection type casting shouldn't allow a user to provide any values that can cause issues.
I have a function that first queries a function in the database that returns a list of employees that a user is allowed to see. This run in a couple ms. Using this list I query the employee table to get all the employees that this user is allowed to see. When I run the generated sql in query analyzer it takes only a few milliseconds. When it runs from entity framework it is taking over 8 seconds.
Initially I had a list of allowedEmployees but then read http://mikeinba.blogspot.ca/2009/09/speeding-up-linq-contains-queries-with.html and tried using a HashSet instead but the performace is still really bad. How can I get its performance similar to what it is in sql query analyzer? What am I doing wrong?
I am using SQL 2008 and EF5.0
public IQueryable<Employee> FindAllByWithPermissions(int eID, Expression<Func<Employee, bool>> predicate)
{
if (predicate != null)
{
HashSet<int> allowedEmployees = SecurityRepository.GetPermissableEmployees(eID);
return DataContext.Set<Employee>().Where(predicate).Where(p => allowedEmployees.Contains(p.EmployeeID)).AsQueryable<Employee>(); ;
}
throw new ArgumentNullException("Predicate value must be passed to FindBy<T,TKey>.");
}
It seems that when it writes the sql to have the IN clause it takes a long time. There are a few thousand permissable employees but why does generating the sql for it take so long?
Return an IQueryable instead of a HashSet, and then join it with your Employee IQueryable.
var availableEmployees = SecurityRepository.GetPermissableEmployees(eID);
var allEmployees= DataContext.Set<Employee>();
query = from item in allEmployees.Where(predicate)
join t in availableEmployees on item.EmployeeID equals t.EmployeeID
select item;
I have been trying for hours to get a Distinct to work for my code.
I am using EF 4.3, MVC3, Razor and trying to get a list downto product id and name.
When I run the Sql query against the DB, it's fine.
Sql Query is
SELECT DISTINCT [ProductId]
,[Product_Name]
FROM [dbo].[PRODUCT]
The only other column in that table is a country code so that's why a standard distinct() isn't working.
I have gone as far as creating an IEqualityComparer
Here is code:
public class DistinctProduct : IEqualityComparer<PRODUCT>
{
public bool Equals(PRODUCT x, PRODUCT y)
{
return x.ProductId.Equals(y.ProductId);
}
public int GetHashCode(PRODUCT obj)
{
return obj.ProductId.GetHashCode();
}
}
here is where I called it.
IEqualityComparer<PRODUCT> customComparer = new DistinctProduct();
IEnumerable<PRODUCT> y = db.PRODUCTs.Distinct(customComparer);
But when it hit's that Last line I get an error out of it stating...
LINQ to Entities does not recognize the method 'System.Linq.IQueryable`1[MyService.Models.PRODUCT] Distinct[PRODUCT](System.Linq.IQueryable`1[MyService.Models.PRODUCT], System.Collections.Generic.IEqualityComparer`1[MyService.Models.PRODUCT])' method, and this method cannot be translated into a store expression.
Can anyone tell me what I'm doing wrong?
Thanks,
David
Is there any reason you could just not use a distinct like the following?
var distinctProdcts = (from p in db.PRODUCTs
select new {
ProductId = p.ProductId,
Product_Name = p.ProductName
}).Distinct();
This would remove the country code from the query before you do the distinct.
Entity Framework is trying to translate your query to a SQL query. Obviously it does not know how to translate the IEqualityComparerer. I think the question is whether you want to do the Distinct in the datbase (in which case your client gets only filtered results) or you are OK with bringing all the data to the client and select distinct on the client. If you want the filtering to happen on the database side (which will make your app perform much better) and you want to be able to use different strategies for comparing you can come up with a code that builds distinct criteria on top of your query. If you are fine with bringing your data to the client (note that it can be a lot of data) you should be able just to do (.ToList() will trigger querying the database and materializing results):
IEnumerable<PRODUCT> y = db.PRODUCTs.ToList().Distinct(customComparer);
I have an Abstract type called Product, and five "Types" that inherit from Product in a table per type hierarchy fashion as below:
I want to get all of the information for all of the Products, including a smattering of properties from the different objects that inherit from products to project them into a new class for use in an MVC web page. My linq query is below:
//Return the required products
var model = from p in Product.Products
where p.archive == false && ((Prod_ID == 0) || (p.ID == Prod_ID))
select new SearchViewModel
{
ID = p.ID,
lend_name = p.Lender.lend_name,
pDes_rate = p.pDes_rate,
pDes_details = p.pDes_details,
pDes_totTerm = p.pDes_totTerm,
pDes_APR = p.pDes_APR,
pDes_revDesc = p.pDes_revDesc,
pMax_desc = p.pMax_desc,
dDipNeeded = p.dDipNeeded,
dAppNeeded = p.dAppNeeded,
CalcFields = new DAL.SearchCalcFields
{
pDes_type = p.pDes_type,
pDes_rate = p.pDes_rate,
pTFi_fixedRate = p.pTFi_fixedRate
}
}
The problem I have is accessing the p.pTFi_fixedRate, this is not returned with the Products collection of entities as it is in the super type of Fixed. How do I return the "super" type of Products (Fixed) properties using Linq and the Entity Framework. I actually need to return some fields from all the different supertypes (Disc, Track, etc) for use in calculations. Should I return these as separate Linq queries checking the type of "Product" that is returned?
This is a really good question. I've had a look in the Julie Lerman book and scouted around the internet and I can't see an elegant answer.
If it were me I would create a data transfer object will all the properties of the types and then have a separate query for each type and then union them all up. I would insert blanks into the DTO properies where the properties aren't relevant to that type. Then I would hope that the EF engine makes a reasonable stab at creating decent SQL.
Example
var results = (from p in context.Products.OfType<Disc>
select new ProductDTO {basefield1 = p.val1, discField=p.val2, fixedField=""})
.Union(
from p in context.Products.OfType<Fixed>
select new ProductDTO {basefield1 = p.val1, discField="", fixedField=p.val2});
But that can't be the best answer can it. Is there any others?
So Fixed is inherited from Product? If so, you should probably be querying for Fixed instead, and the Product properties will be pulled into it.
If you are just doing calculations and getting some totals or something, you might want to look at using a stored procedure. It will amount to fewer database calls and allow for much faster execution.
Well it depends on your model, but usually you need to do something like:
var model = from p in Product.Products.Include("SomeNavProperty")
.... (rest of query)
Where SomeNavProperty is the entity type that loads pTFi_fixedRate.
I want to create an Entity Object from a LinQ statement, but I don't want to load all its columns.
My ORDERS object has a lot of columns, but I just want to retrieve the REFERENCE and OPERATION columns so the SQL statement and result will be smaller.
This LinQ statement works properly and loads all my object attributes:
var orders = (from order in context.ORDERS
select order);
However the following statement fails to load only two properties of my object
var orders = (from order in context.ORDERS
select new ORDERS
{
REFERENCE = order.REFERENCE,
OPERATION = order.OPERATION
});
The error thrown is:
The entity or complex type
'ModelContextName.ORDERS' cannot be
constructed in a LINQ to Entities
query.
What is the problem? Isn't it possible to partially load an object this way?
Thank you in advance for your answers.
ANSWER
Ok I should thank you both Yakimych and Dean because I use both of your answers, and now I have:
var orders = (from order in context.ORDERS
select new
{
REFERENCE = order.REFERENCE,
OPERATION = order.OPERATION,
})
.AsEnumerable()
.Select(o =>
(ORDERS)new ORDERS
{
REFERENCE = o.REFERENCE,
OPERATION = o.OPERATION
}
).ToList().AsQueryable();
And I get exactly what I want, the SQL Statement is not perfect but it returns only the 2 columns I need (and another column which contains for every row "1" but I don't know why for the moment) –
I also tried to construct sub objects with this method and it works well.
No, you can't project onto a mapped object. You can use an anonymous type instead:
var orders = (from order in context.ORDERS
select new
{
REFERENCE = order.REFERENCE,
OPERATION = order.OPERATION
});
The problem with the above solution is that from the moment you call AsEnumerable(), the query will get executed on the database. In most of the cases, it will be fine. But if you work with some large database, fetching the whole table(or view) is probably not what you want. So, if we remove the AsEnumerable, we are back to square 1 with the following error:
The entity or complex type 'ModelContextName.ORDERS' cannot be constructed in a LINQ to Entities query.
I have been struggling with this problem for a whole day and here is what I found. I created an empty class inheriting from my entity class and performed the projection using this class.
public sealed class ProjectedORDERS : ORDERS {}
The projected query (using covariance feature):
IQueryable<ORDERS> orders = (from order in context.ORDERS
select new ProjectedORDERS
{
REFERENCE = order.REFERENCE,
OPERATION = order.OPERATION,
});
Voilà! You now have a projected query that will map to an entity and that will get executed only when you want to.
I think the issue is creating new entities within the query itself, so how about trying this:
context.ORDERS.ToList().Select(o => new ORDERS
{
REFERENCE = o.REFERENCE,
OPERATION = o.OPERATION
});