I am new to entity framework. I am using a linq query that will fetch many records (upto millions) from database. There are many filter parameters in where condition and on each request the parameters may change. So i wanted to know whether compiled queries will be effective in this case or will it be a new query on each request. Here is my query:
List<FarmerDetailsReport> fdr =
(from fp in mstfp join pd in personalDetails on fp.personDetails.Id equals pd.Id
join ic in identityCertificate on fp.identityCertificate.Id equals ic.Id
join pid in pacsInsuranceData on fp.pacsInsuranceData.Id equals pid.Id into temp
from pid in temp.DefaultIfEmpty()
join bd in bankDetails on fp.bankDetails.Id equals bd.Id
join cd in contactDetails on fp.contactDetails.Id equals cd.Id
join id in incomeDetails on fp.incomeDetails.Id equals id.Id into tmp
from id in tmp.DefaultIfEmpty()
join ua in userAttributes on fp.UserId equals ua.EmailID
where ((ua.CompanyName == companyName ) && (cd.District == model.DistrictForProfileMIS ) && (cd.Block == model.BlockForProfileMIS) && (bd.bankName == model.BankForProfileMIS ) && Status == "Active")
select new FarmerDetailsReport { .......... }).ToList();
Short answer:
Yes...... well, maybe.
Long answer:
This is hard to answer as you have no control over the actual SQL that gets generated.
We had perf problems with some queries like this as the optimizer would optimize for a certain wet of filter cases (like short circuits of clauses) then when a new query was made with a massive change in parameters it would take AGES.
What we did in the end:
Don't use a big LINQ query, create a stored proc or view where you have more control over the SQL generated.
Used things like OPTION(RECOMPILE) ... looks this up it was very useful.
Have a few overloads of the query for different parameters so that the DB can optimize them separately.
Obviously this is just what we did, it might not be perfect for you. I STRONGLY suggest getting the generated SQL for each different parametrized version and going over it with your DBA (if you have one) or your team and google if you don't.
Related
I am trying to query data in CRM 2011 and I need to join an ActivityParty. I can't seem to find any good documentation on it. Has anyone done this before. This is my query so far:
var linqQuery = (from r in gServiceContext.CreateQuery("campaignresponse")
join c in gServiceContext.CreateQuery("contact") on ((EntityReference)r["customer"]).Id equals c["contactid"] into opp
join a in gServiceContext,CreateQuery("lead") on ((EntityReference)r["customer"]).Id equals c["leadid"] into opp
from o in opp.DefaultIfEmpty()
where ((EntityReference)r["new_distributorid"]).Id.Equals(lProfileProperty.PropertyValue) && ((OptionSetValue)r["new_distributorstatus"]).Equals("100000002")
select new
{ }
So if you look at the query what I am trying to do is join the contact Entity and the lead Entity to CamapaignResponse via the customer field and the customer field is an ActivityParty field. Any ideas on how to do this? Thanks!
I'm having a hard time figuring out how your query is supposed to work, but I can tell you that you'll have an easier time if you generate Linq entities using the SDK. Then, instead of
from r in gServiceContext.CreateQuery("campaignresponse") where ...
you can just write
gServiceContext.CampaignResponseSet
.Where(cr => cr.property == value)
.Select(cr => new {cr, cr.childObject});
and so forth. You'll get strong typing and IntelliSense, too.
Is this linq query efficient?
var qry = ((from member in this.ObjectContext.TreeMembers.Where(m => m.UserId == userId && m.Birthdate == null)
select member.TreeMemberId).Except(from item in this.ObjectContext.FamilyEvents select item.TreeMemberId));
var mainQry = from mainMember in this.ObjectContext.TreeMembers
where qry.Contains(mainMember.TreeMemberId)
select mainMember;
Will this be translated into multiple sql calls or just one? Can it be optimised? Basically I have 2 tables, I want to select those records from table1 where datetime is null and that record should not exist in table2.
The easiest way to find out if the query will make multiple calls is to set the .Log property of the data context. I typically set it to write to a DebugOutputWriter. A good example for this kind of class can be found here.
For a general way of thinking about it however, if you use a property of your class that does not directly map to a database field in a where clause or a join clause, it will typically make multiple calls. From what you have provided, it looks like this is not the case for your scenario, but I can't absolutely certain and suggest using the method listed above.
Let's say I have an Order table which has a FirstSalesPersonId field and a SecondSalesPersonId field. Both of these are foreign keys that reference the SalesPerson table. For any given order, either one or two salespersons may be credited with the order. In other words, FirstSalesPersonId can never be NULL, but SecondSalesPersonId can be NULL.
When I drop my Order and SalesPerson tables onto the "Linq to SQL Classes" design surface, the class builder spots the two FK relationships from the Order table to the SalesPerson table, and so the generated Order class has a SalesPerson field and a SalesPerson1 field (which I can rename to SalesPerson1 and SalesPerson2 to avoid confusion).
Because I always want to have the salesperson data available whenever I process an order, I am using DataLoadOptions.LoadWith to specify that the two salesperson fields are populated when the order instance is populated, as follows:
dataLoadOptions.LoadWith<Order>(o => o.SalesPerson1);
dataLoadOptions.LoadWith<Order>(o => o.SalesPerson2);
The problem I'm having is that Linq to SQL is using something like the following SQL to load an order:
SELECT ...
FROM Order O
INNER JOIN SalesPerson SP1 ON SP1.salesPersonId = O.firstSalesPersonId
INNER JOIN SalesPerson SP2 ON SP2.salesPersonId = O.secondSalesPersonId
This would make sense if there were always two salesperson records, but because there is sometimes no second salesperson (secondSalesPersonId is NULL), the INNER JOIN causes the query to return no records in that case.
What I effectively want here is to change the second INNER JOIN into a LEFT OUTER JOIN. Is there a way to do that through the UI for the class generator? If not, how else can I achieve this?
(Note that because I'm using the generated classes almost exclusively, I'd rather not have something tacked on the side for this one case if I can avoid it).
Edit: per my comment reply, the SecondSalesPersonId field is nullable (in the DB, and in the generated classes).
The default behaviour actually is a LEFT JOIN, assuming you've set up the model correctly.
Here's a slightly anonymized example that I just tested on one of my own databases:
class Program
{
static void Main(string[] args)
{
using (TestDataContext context = new TestDataContext())
{
DataLoadOptions dlo = new DataLoadOptions();
dlo.LoadWith<Place>(p => p.Address);
context.LoadOptions = dlo;
var places = context.Places.Where(p => p.ID >= 100 && p.ID <= 200);
foreach (var place in places)
{
Console.WriteLine(p.ID, p.AddressID);
}
}
}
}
This is just a simple test that prints out a list of places and their address IDs. Here is the query text that appears in the profiler:
SELECT [t0].[ID], [t0].[Name], [t0].[AddressID], ...
FROM [dbo].[Places] AS [t0]
LEFT OUTER JOIN (
SELECT 1 AS [test], [t1].[AddressID],
[t1].[StreetLine1], [t1].[StreetLine2],
[t1].[City], [t1].[Region], [t1].[Country], [t1].[PostalCode]
FROM [dbo].[Addresses] AS [t1]
) AS [t2] ON [t2].[AddressID] = [t0].[AddressID]
WHERE ([t0].[PlaceID] >= #p0) AND ([t0].[PlaceID] <= #p1)
This isn't exactly a very pretty query (your guess is as good as mine as to what that 1 as [test] is all about), but it's definitively a LEFT JOIN and doesn't exhibit the problem you seem to be having. And this is just using the generated classes, I haven't made any changes.
Note that I also tested this on a dual relationship (i.e. a single Place having two Address references, one nullable, one not), and I get the exact same results. The first (non-nullable) gets turned into an INNER JOIN, and the second gets turned into a LEFT JOIN.
It has to be something in your model, like changing the nullability of the second reference. I know you say it's configured as nullable, but maybe you need to double-check? If it's definitely nullable then I suggest you post your full schema and DBML so somebody can try to reproduce the behaviour that you're seeing.
If you make the secondSalesPersonId field in the database table nullable, LINQ-to-SQL should properly construct the Association object so that the resulting SQL statement will do the LEFT OUTER JOIN.
UPDATE:
Since the field is nullable, your problem may be in explicitly declaring dataLoadOptions.LoadWith<>(). I'm running a similar situation in my current project where I have an Order, but the order goes through multiple stages. Each stage corresponds to a separate table with data related to that stage. I simply retrieve the Order, and the appropriate data follows along, if it exists. I don't use the dataLoadOptions at all, and it does what I need it to do. For example, if the Order has a purchase order record, but no invoice record, Order.PurchaseOrder will contain the purchase order data and Order.Invoice will be null. My query looks something like this:
DC.Orders.Where(a => a.Order_ID == id).SingleOrDefault();
I try not to micromanage LINQ-to-SQL...it does 95% of what I need straight out of the box.
UPDATE 2:
I found this post that discusses the use of DefaultIfEmpty() in order to populated child entities with null if they don't exist. I tried it out with LINQPad on my database and converted that example to lambda syntax (since that's what I use):
ParentTable.GroupJoin
(
ChildTable,
p => p.ParentTable_ID,
c => c.ChildTable_ID,
(p, aggregate) => new { p = p, aggregate = aggregate }
)
.SelectMany (a => a.aggregate.DefaultIfEmpty (),
(a, c) => new
{
ParentTableEntity = a.p,
ChildTableEntity = c
}
)
From what I can figure out from this statement, the GroupJoin expression relates the parent and child tables, while the SelectMany expression aggregates the related child records. The key appears to be the use of the DefaultIfEmpty, which forces the inclusion of the parent entity record even if there are no related child records. (Thanks for compelling me to dig into this further...I think I may have found some useful stuff to help with a pretty huge report I've got on my pipeline...)
UPDATE 3:
If the goal is to keep it simple, then it looks like you're going to have to reference those salesperson fields directly in your Select() expression. The reason you're having to use LoadWith<>() in the first place is because the tables are not being referenced anywhere in your query statement, so the LINQ engine won't automatically pull that information in.
As an example, given this structure:
MailingList ListCompany
=========== ===========
List_ID (PK) ListCompany_ID (PK)
ListCompany_ID (FK) FullName (string)
I want to get the name of the company associated with a particular mailing list:
MailingLists.Where(a => a.List_ID == 2).Select(a => a.ListCompany.FullName)
If that association has NOT been made, meaning that the ListCompany_ID field in the MailingList table for that record is equal to null, this is the resulting SQL generated by the LINQ engine:
SELECT [t1].[FullName]
FROM [MailingLists] AS [t0]
LEFT OUTER JOIN [ListCompanies] AS [t1] ON [t1].[ListCompany_ID] = [t0].[ListCompany_ID]
WHERE [t0].[List_ID] = #p0
I am in the process of writing something that will use Linq to combine results from my database, via Linq2Sql and an in-memory list of objects in order to find out which of my in-memory objects match something on the database.
I've come up with the query in both expression and query syntax.
Expression Syntax
var query = order.Items.Join(productNonCriticalityList,
i => i.ProductID,
p => p.ProductID,
(i, p) => i);
Query Syntax
var query =
from p in productNonCriticalityList
join i in order.Items
on p.ProductID equals i.ProductID
select i;
I realise that we have all the code completion goodness with expression syntax, and I do actually use that more. Mainly because it's easier to create re-usable chunks of filter code that can be chained together to form more complex filters.
But for a join the latter seems far more readable to me, but maybe that is because I am used to writing T-SQL.
So, am I missing a trick or is it just a matter of getting used to it?
I agree with the other responders that the exact question you're asking is simply a matter of preference. Personaly, I mix the two forms depending upon which is clearer for the specific query that I'm writing.
If I have one comment though, I would say that the query looks like it might load all of the items from the order. That might be fine for a single order one time, but if you're looping through lots of orders, it might be more efficient to load all of the items for all of the in one go (you might want to additionally filter by date or customer, or whatever though). If you do that, you might get better results by switching the query around:
var productIds = (from p in productNonCriticalityList
orderby p.productID
select p.ProductID).Distinct();
var orderItems = from i in dc.OrderItems
where productIds.Contains(i.ProductID)
&& // Additional filtering here.
select i;
It's a bit backwards at first glance, but it could save you from loading in all the order items and also from sending lots of queries. It works because the where productIds.Contains(...) call can be converted to where i.ProductID in (1, 2, 3, 4, 5) in SQL. Of course, you'd have to judge it based on the expected number of order items, and the number of product IDs.
It really all comes down to preference. Some people just hate the idea of query like syntax in their code. I for one appreciate the query syntax, it is declarative and quite readable. Like you said though, the chainability of the first example is a nice thing to have. I guess for my money I would keep it query until I felt I needed to begin chaining the call.
I used to feel the same way. Now I find query syntax easier to read and write, particularly when things get complicated. As much as it irked me to type it the first time, 'let' does wonderful things in ways that would not be readable in Expression Syntax.
I prefer the Query syntax when its complex and Expression syntax when its a simple query.
If a DBA were to read the C# code to see what SQL we are using, they would understand and digest the Query syntax easier.
Taking a simple example:
Query
var col = from o in orders
orderby o.Cost ascending
select o;
Expression
var col2 = orders.OrderBy(o => o.Cost);
To me, the Expression syntax is an easier choice to understand here.
Another example:
Query
var col9 = from o in orders
orderby o.CustomerID, o.Cost descending
select o;
Expression
var col6 = orders.OrderBy(o => o.CustomerID).
ThenByDescending(o => o.Cost);
Both are easy to read and understand, however if the query was
//returns same results as above
var col5 = from o in orders
orderby o.Cost descending
orderby o.CustomerID
select o;
//NOTE the ordering of the orderby's
That looks a little confusing to be as the fields are in a different order and it appears a little backwards.
For Joins
Query
var col = from c in customers
join o in orders on
c.CustomerID equals o.CustomerID
select new
{
c.CustomerID,
c.Name,
o.OrderID,
o.Cost
};
Expression:
var col2 = customers.Join(orders,
c => c.CustomerID,o => o.CustomerID,
(c, o) => new
{
c.CustomerID,
c.Name,
o.OrderID,
o.Cost
}
);
I find that Query is better.
My summary would be use whatever looks easiest and fastest to understand given the query at hand. There is no golden rule of which to use. However, if there are a lot of joins, I'd go with Query syntax.
Well, both statements are equivalent. So you could youse them both, depending on the surrounging code and what is more readable. In my project I make the decision which syntax to use dependent on those two conditions.
Personally I would write the expression syntax in one line, but this is a matter of taste.
I have a non-trivial Linq To Sql query that I'm trying to break down into pieces for the sake of readability / further filtering / reuse.
The refactored code looks like this, where ids is the subquery performed to grab the ids.
var results = from solution in context.csExtendedQAIncident_Docs
join solutionText in context.csNTexts
on solution.chIdNo equals solutionText.chIdNo
where solutionText.chColumnId == "Solution"
//this is a very complicated subquery that returns a short list of the ids we need
&& (ids).Select(s => s.chIdNo)
//the TOP query portion - applied to just the ids
.Take(count ?? Settings.Current.WCFServices().Output.HomePage.MaxRows)
.Contains(solution.chIdNo)
select solution;
The 'ids' is an IOrderedQueryable<csExtendedQAIncident_Docs> which itself has a number of criteria and nested subqueries (i.e. Contains which gets translated into EXISTS style queries on SQL server)
In any event, the issue here is that when the full-blown subquery is included in the results query above, the query works without a hitch.
When the query is pulled out into it's own variable, the query dies at runtime with a NullReferenceException at SqlFactory.Member (partial stack trace below)
System.NullReferenceException: Object reference not set to an instance of an object.
at System.Data.Linq.SqlClient.SqlFactory.Member(SqlExpression expr, MemberInfo member)
at System.Data.Linq.SqlClient.QueryConverter.VisitMemberAccess(MemberExpression ma)
at System.Data.Linq.SqlClient.QueryConverter.VisitInner(Expression node)
at System.Data.Linq.SqlClient.QueryConverter.Visit(Expression node)
at System.Data.Linq.SqlClient.QueryConverter.VisitExpression(Expression exp)
at System.Data.Linq.SqlClient.QueryConverter.VisitMethodCall(MethodCallExpression mc)
at System.Data.Linq.SqlClient.QueryConverter.VisitInner(Expression node)
at System.Data.Linq.SqlClient.QueryConverter.Visit(Expression node)
at System.Data.Linq.SqlClient.QueryConverter.VisitExpression(Expression exp)
at
I suspect this is a bug in Linq to Sql's query evaluation process -- but was wondering if anyone else might have come across such an issue?
I stepped into the Framework source code -- but of course the trouble with that is that while I can hit the exact source line of the exception, the Linq assembly I'm using has been optimized and I can't inspect variables, etc.
Note: Keep in mind that this query, despite being a little long is actually not a great candidate in this scenario for a SPROC (especially since we're stuck on SQL 2000 here and can't parameterize TOP :( )
Edit: Found one other reference to a very similar issue out there that was never resolved http://www.eggheadcafe.com/software/aspnet/31934404/linq-combined-query.aspx
Edit2: Ask and ye shall receive ;0 I figured I would keep the id subquery out of here because I don't think its actually relevant (and as mentioned its complicated, etc). Please don't blame the crap db design on me -- but know that I worked on the SQL for this query for a bit to get acceptable performance, and then translated to Linq To Sql.
var ids = from solutionIds in context.csExtendedQAIncident_Docs
where solutionIds.iIncidentTypeId == 102094
&& solutionIds.tiRecordStatus == 1
&& solutionIds.iLanguage == 102074
&& null != solutionIds.chIdNo
&& (from openTo in context.csOpenTos
where onyxIdentity.GetDocumentAccessLevels().Union(new[] { "BUSG5" }).ToArray().Contains(openTo.vchOpenTo)
select openTo.chIdNo
).Distinct().Contains(solutionIds.chIdNo)
&& (from solutionProductAssocation in context.csProductDocs
where (from allowedProduct in context.KB_User_Allowed_Products
where allowedProduct.UserId == userId
select allowedProduct.ModelCode
).Contains(solutionProductAssocation.chModelCd)
select solutionProductAssocation.chIdNo).Distinct().Contains(solutionIds.chIdNo)
orderby solutionIds.dtUpdateDate descending
select solutionIds;