LINQ generating SQL not supported by SQL Server 2008R2 - linq

I am using EF 6.1.3 in an ASP.NET MVC 5 project. Although my development environment is SQL Server 2012, QA and Production are on SQL Server 2008 R2.
I have a report that is to show the top 5 procedure codes utilized by a physician, so I have a simple entity with id, name, code1, code2,... code5.
There is a fair amount of logic in building the LINQ query, essentially I get a list of physician and a list of codes and join them. Here is the code that joins them and picks the top 5 codes:
var report =
from pr in providerRiskLevels
join nc in newCodes on pr.Provider.ProviderId equals nc.ProviderId
where pr.RiskCategoryId == RIskScoreIds.VisibleRisk
&& (filterRiskLevelNums.Contains(pr.RiskLevelNum))
&& (filterSpecialtyId == 0 || pr.Provider.Specialty.SpecialtyId == filterSpecialtyId)
select new ReportNewCodesEntity
{
ProviderId = pr.Provider.ProviderId,
ProviderName = pr.Provider.Name,
ProviderCode = pr.Provider.ProviderCode,
SpecialtyName = pr.Provider.Specialty.Name,
SpecialtyCode = pr.Provider.Specialty.SpecialtyCode,
RiskScore = pr.RiskScore,
RiskLevelName = pr.RiskLevelName,
RiskLevelNum = pr.RiskLevelNum,
NewCode1 = nc.Codes.OrderByDescending(c => c.Volume).FirstOrDefault().ProcedureCode,
NewCode2 = nc.Codes.OrderByDescending(c => c.Volume).Skip(1).FirstOrDefault().ProcedureCode,
NewCode3 = nc.Codes.OrderByDescending(c => c.Volume).Skip(2).FirstOrDefault().ProcedureCode,
NewCode4 = nc.Codes.OrderByDescending(c => c.Volume).Skip(3).FirstOrDefault().ProcedureCode,
NewCode5 = nc.Codes.OrderByDescending(c => c.Volume).Skip(4).FirstOrDefault().ProcedureCode
};
return report;
The problem is that the four Skip().FirstOrDefault() generate SQL with syntax not available on SQL Server 2008 R2:
OFFSET x ROWS FETCH NEXT 1 ROWS ONLY
where x = 1 to 4
I'm sure I can move this to a stored procedure (much better at T-SQL than LINQ), but since I have been trying to get a grip on LINQ, I was wondering if there was either:
A directive to tell LINQ the version of the target database so it would generate compliant code
Another method to extract these top 5 codes.
I do already have the codes ordered correctly in newCodes (see below) but was unable to pick them without the SKIP (which required it's own sort clause).
var newCodes =
from cc in codeCounts
group cc by
new {cc.ProviderId}
into g
select new ProviderNewCodesEntity
{
ProviderId = g.Key.ProviderId,
Codes = g.OrderByDescending (x => x.Volume).Take(5).ToList()
};
Best,
Scott

In order to get LINQ to generate compatible code, we simply edited the generated EDMX file (don't shriek yet , keep reading).
We found (on line 7)
<Schema Namespace="ComplianceRiskModel.Store" ProviderManifestToken="2012" Provider="System.Data.SqlClient" ...>
and changed it to
<Schema Namespace="ComplianceRiskModel.Store" ProviderManifestToken="2008" Provider="System.Data.SqlClient" ...>
To prevent losing this setting every time we refresh the model from the database, we are adding a build task to automatically update the EDMX file in a BeforeBuild target, as described at http://www.programmingmotility.com/2011/05/setting-providermanifesttoken-for-sql.html
Best,
Scott

Related

Entity Framework Core + Count with Group By

I have a table which contains ~600k records and 33 columns. In my project I am using EF Core (2.0.1) to retrieve data from database. I am having issues with below code:
var theCounter = (from f in _context.tblData.Take(100000)
group f by f.TypeId into data
select new DataDto { ID = data.Key, Count = data.Count() }).ToList();
This code is a part of REST API and when I am testing it from SOAP UI, I am gettin timeout error. When I tested the code for
Take(1000)
There are around 300 unique TypeIds.
it works fine. Any ideas how I can make it work?
-- EDIT 1:
Here is what I see when debugging the code:
Microsoft.EntityFrameworkCore.Query:Warning: Query: '(from TblData <generated>_1 in DbSet<TblData> select [<generated>_1]).Take(__p_0)' uses a row limiting operation (Skip/Take) without OrderBy which may lead to unpredictable results.
Microsoft.EntityFrameworkCore.Query:Warning: Query: '(from TblData <generated>_1 in DbSet<TblData> select [<generated>_1]).Take(__p_0)' uses a row limiting operation (Skip/Take) without OrderBy which may lead to unpredictable results.
Microsoft.EntityFrameworkCore.Query:Warning: The LINQ expression 'GroupBy([f].TypeId, [f])' could not be translated and will be evaluated locally.
Microsoft.EntityFrameworkCore.Query:Warning: The LINQ expression 'GroupBy([f].TypeId, [f])' could not be translated and will be evaluated locally.
Microsoft.EntityFrameworkCore.Query:Warning: The LINQ expression 'Count()' could not be translated and will be evaluated locally.
Microsoft.EntityFrameworkCore.Database.Command:Information: Executed DbCommand (131ms) [Parameters=[#__p_0='?'], CommandType='Text', CommandTimeout='30']
SELECT [t2].[Id], [t2].[at], [t2].[add], [t2].[AddDate], [t2].[aftc], [t2].[aftcd], [t2].[aid], [t2].[afl], [t2].[prdid], [t2].[cid], [t2].[TypeId], [t2].[env], [t2].[ext], [t2].[extddcode], [t2].[fn], [t2].[fn], [t2].[fic], [t2].[gid], [t2].[grp], [t2].[hnm], [t2].[IP], [t2].[icid], [t2].[ln], [t2].[lg], [t2].[pcid], [t2].[ret], [t2].[rts], [t2].[rnam], [t2].[sled], [t2].[seq], [t2].[sid], [t2].[styp]
FROM (
SELECT TOP(#__p_0) [t1].[Id], [t1].[at], [t1].[add], [t1].[AddDate], [t1].[aftc], [t1].[aftcd], [t1].[aid], [t1].[afl], [t1].[prdid], [t1].[cid], [t1].[TypeId], [t1].[env], [t1].[ext], [t1].[extddcode], [t1].[fn], [t1].[fn], [t1].[fic], [t1].[gid], [t1].[grp], [t1].[hnm], [t1].[IP], [t1].[icid], [t1].[ln], [t1].[lg], [t1].[pcid], [t1].[ret], [t1].[rts], [t1].[rnam], [t1].[sled], [t1].[seq], [t1].[sid], [t1].[styp]
FROM [TblData] AS [t1]
) AS [t2]
WHERE [t2].[TypeId] IS NOT NULL
ORDER BY [t2].[TypeId]
I think it is not translated properly. Any ideas why?
-- EDIT 2:
I have changed my queries to:
var query = _context.TblData
.Select(a => new {ID = a.Id, TypeId= a.TypeId})
.Distinct();
var q1 = query.GroupBy(p => p.TypeId)
.Select(g => new DataDto {TypeId= g.Key, Count = g.Count()});
return await q1.ToListAsync();
But it was translated to:
SELECT DISTINCT [a0].[Id], [a0].[TypeId] AS [TypeId]
FROM [tblData] AS [a0]
ORDER BY [a0].[TypeId]
When I checked directly in the database this query takes 14 seconds to execute. Any idea why it was not translated to something like:
SELECT DISTINCT [a0].[Id], COUNT([TypeId]) AS [TypeId]
FROM [tblData] AS [a0]
GROUP BY COUNT([a0].[Id])
ORDER BY [a0].[TypeId]
I had to upgrade EF Core version to 2.1 and LINQ is now translated properly into SQL.

Entity Framework SQL Selecting 600+ Columns

I have a query generated by entity framework running against oracle that's too slow. It runs in about 4 seconds.
This is the main portion of my query
var query = from x in db.BUILDINGs
join pro_co in db.PROFILE_COMMUNITY on x.COMMUNITY_ID equals pro_co.COMMUNITY_ID
join co in db.COMMUNITies on x.COMMUNITY_ID equals co.COMMUNITY_ID
join st in db.STATE_PROFILE on co.STATE_CD equals st.STATE_CD
where pro_co.PROFILE_NM == authorizedUser.ProfileName
select new
{
COMMUNITY_ID = x.COMMUNITY_ID,
COUNTY_ID = x.COUNTY_ID,
REALTOR_GROUP_NM = x.REALTOR_GROUP_NM,
BUILDING_NAME_TX = x.BUILDING_NAME_TX,
ACTIVE_FL = x.ACTIVE_FL,
CONSTR_SQFT_AVAIL_NB = x.CONSTR_SQFT_AVAIL_NB,
TRANS_RAIL_FL = x.TRANS_RAIL_FL,
LAST_UPDATED_DT = x.LAST_UPDATED_DT,
CREATED_DATE = x.CREATED_DATE,
BUILDING_ADDRESS_TX = x.BUILDING_ADDRESS_TX,
BUILDING_ID = x.BUILDING_ID,
COMMUNITY_NM = co.COMMUNITY_NM,
IMAGECOUNT = x.BUILDING_IMAGE2.Count(),
StateCode = st.STATE_NM,
BuildingTypeItems = x.BUILDING_TYPE_ITEM,
BuildingZoningItems = x.BUILDING_ZONING_ITEM,
BuildingSpecFeatures = x.BUILDING_SPEC_FEATURE_ITEM,
buildingHide = x.BUILDING_HIDE,
buildinghideSort = x.BUILDING_HIDE.Count(y => y.PROFILE_NM == ProfileName) > 0 ? 1 : 0,
BUILDING_CITY_TX = x.BUILDING_CITY_TX,
BUILDING_ZIP_TX = x.BUILDING_ZIP_TX,
LPF_GENERAL_DS = x.LPF_GENERAL_DS,
CONSTR_SQFT_TOTAL_NB = x.CONSTR_SQFT_TOTAL_NB,
CONSTR_STORIES_NB = x.CONSTR_STORIES_NB,
CONSTR_CEILING_CENTER_NB = x.CONSTR_CEILING_CENTER_NB,
CONSTR_CEILING_EAVES_NB = x.CONSTR_CEILING_EAVES_NB,
DESCR_EXPANDABLE_FL = x.DESCR_EXPANDABLE_FL,
CONSTR_MATERIAL_TYPE_TX = x.CONSTR_MATERIAL_TYPE_TX,
SITE_ACRES_SALE_NB = x.SITE_ACRES_SALE_NB,
DESCR_PREVIOUS_USE_TX = x.DESCR_PREVIOUS_USE_TX,
CONSTR_YEAR_BUILT_TX = x.CONSTR_YEAR_BUILT_TX,
DESCR_SUBDIVIDE_FL = x.DESCR_SUBDIVIDE_FL,
LOCATION_CITY_LIMITS_FL = x.LOCATION_CITY_LIMITS_FL,
TRANS_INTERSTATE_NEAREST_TX = x.TRANS_INTERSTATE_NEAREST_TX,
TRANS_INTERSTATE_MILES_NB = x.TRANS_INTERSTATE_MILES_NB,
TRANS_HIGHWAY_NAME_TX = x.TRANS_HIGHWAY_NAME_TX,
TRANS_HIGHWAY_MILES_NB = x.TRANS_HIGHWAY_MILES_NB,
TRANS_AIRPORT_COM_NAME_TX = x.TRANS_AIRPORT_COM_NAME_TX,
TRANS_AIRPORT_COM_MILES_NB = x.TRANS_AIRPORT_COM_MILES_NB,
UTIL_ELEC_SUPPLIER_TX = x.UTIL_ELEC_SUPPLIER_TX,
UTIL_GAS_SUPPLIER_TX = x.UTIL_GAS_SUPPLIER_TX,
UTIL_WATER_SUPPLIER_TX = x.UTIL_WATER_SUPPLIER_TX,
UTIL_SEWER_SUPPLIER_TX = x.UTIL_SEWER_SUPPLIER_TX,
UTIL_PHONE_SVC_PVD_TX = x.UTIL_PHONE_SVC_PVD_TX,
CONTACT_ORGANIZATION_TX = x.CONTACT_ORGANIZATION_TX,
CONTACT_PHONE_TX = x.CONTACT_PHONE_TX,
CONTACT_EMAIL_TX = x.CONTACT_EMAIL_TX,
TERMS_SALE_PRICE_TX = x.TERMS_SALE_PRICE_TX,
TERMS_LEASE_SQFT_NB = x.TERMS_LEASE_SQFT_NB
};
There is a section of code that tacks on dynamic where and sort clauses to the query but I've left those out. The query takes about 4 seconds to run no matter what is in the where and sort.
I dropped the generated SQL in Oracle and an explain plan didn't appear to show anything that screamed fix me. Cost is 1554
If this isn't allowed I apologize but I can't seem to find a good way to share this information. I've uploaded the explain plan generated by Sql Developer here: http://www.123server.org/files/explainPlanzip-e1d291efcd.html
Table Layout
Building
--------------------
- BuildingID
- CommunityId
- Lots of other columns
Profile_Community
-----------------------
- CommunityId
- ProfileNM
- lots of other columns
state_profile
---------------------
- StateCD
- ProfileNm
- lots of other columns
Profile
---------------------
- Profile-NM
- a few other columns
All of the tables with allot of columns have 120-150 columns each. It seems like entity is generating a select statement that pulls every column from every table instead of just the ones I want.
The thing that's bugging me and I think might be my issue is that in my LINQ I've selected 50 items, but the generated sql is returning 677 columns. I think returning so many columns is the source of my slowness possibly.
Any ideas why I am getting so many columns returned in SQL or how to speed my query?
I have a suspicion some of the performance is being impacted by your object creation. Try running the query without just a basic "select x" and see if it's the SQL query taking time or the object creation.
Also if the query being generated is too complicated you could try separating it out into smaller sub-queries which gradually enrich your object rather than trying to query everything at once.
I ended up creating a view and having the view only select the columns I wanted and joining on things that needed to be left-joined in linq.
It's pretty annoying that EF selects every column from every table you're trying to join across. But I guess I only noticed this because I am joining a bunch of tables with 150+ columns in them.

EntityFramework - SaveChanges not saving but SQL seen in Profiler

I have read through most every post on EF-SaveChanges and do not believe my answer lies in any of those posts.
I am using C#, .NET 4, EF 4.3.1, SQL Server 2008R2, VS 2k11 Beta, AutoMapper.
Here is my code:
using (Model.AnimalRescueEntities context = new Model.AnimalRescueEntities())
{
using (TransactionScope transaction = new TransactionScope())
{
context.Connection.Open();
//Retrieve the event
eventDB = context.Events.Single(e => e.ID == eventRegVM.EventID);
eventOrgBaseDB = context.Entity_Base.Single(b => b.ID == eventDB.Entity_Organisation.ID);
eventRegVM.Event = Mapper.Map<Model.Event, EventsViewModel>(eventDB);
eventRegVM.Event.Entity_Organisation.Entity_Base = Mapper.Map<Model.Entity_Base, Entity_BaseViewModel>(eventOrgBaseDB);
//saves Event_Registration
eventRegDB = Mapper.Map<Event_RegistrationViewModel, Model.Event_Registration>(eventRegVM);
eventRegDB.Event = eventDB;
eventRegDB.EventID = eventDB.ID;
eventRegDB.Event.Entity_Organisation = context.Entity_Organisation.Single(o => o.ID == eventOrgBaseDB.ID);
eventRegDB.Event.Entity_Organisation.Entity_Base = eventOrgBaseDB;
//Add the link between EVENT and REGISTRATION
context.Event_Registration.AddObject(eventRegDB);
int numChanges = context.SaveChanges();
var regs = context.Event_Registration.Where(r => r.ID != null).ToList();
}
}
I have SQL Profiler running in the background and when SaveChanges is called I see this SQL code (numChanges is 1):
exec sp_executesql N'declare #generated_keys table([ID] uniqueidentifier)
insert [dbo].[Event_Registration]([EventID], [DateSubmitted], [HasPaid], [PaymentMethod], [Comments], [AmountPaid])
output inserted.[ID] into #generated_keys
values (#0, #1, null, #2, null, #3)
select t.[ID]
from #generated_keys as g join [dbo].[Event_Registration] as t on g.[ID] = t.[ID]
where ##ROWCOUNT > 0',N'#0 uniqueidentifier,#1 datetime2(7),#2 int,#3 decimal(19,4)',#0='1D841F75-AEA1-4ED1-B3F0-4E3994D7FC0D',#1='2012-07-04 14:59:45.5239309',#2=0,#3=0
regs will contain three exist rows and my new row. However, I cannot run a SELECT statement in SQL Server and see my new row. Running this many times will get me the same result - three existing rows in the database and a new fourth row that will never make it to the database.
eventRegDB also contains the GUID created for the primary key, ID; I assume SQL server does this but not 100% sure of that.
I have taken the above TSQL and run it in a query window against my database - I get new rows in my Event_Registration table after that - that is how the three existing rows were created.
I see now exceptions or other errors generated and cannot find any reason this would not save to the database. Any ideas? If you want to see the schema for the SQL, how to recreate the database, the code (any or all), then ask - this is all hosted on http://animalrescue.codeplex.com/ but I haven't stored this code, yet.
You are missing transaction.Complete() so your transaction is never committed. When the using block for TransactionScope completes your transaction is rolled back.

Failed to batch insert in Subsonic3 with error "Must declare the scalar variable..."

I have met a problem about inserting multiple rows in a batch with Subsonic3. My development environment includes:
1. Visual Studio 2010, but use .NET 3.5
2. Active Record Mode in SubSonic 3.0.0.4
3. SQL Server 2005 express
4. Northwind sample database
I am using Active Reecord mode to insert mutiple "Product" into table "Products". If I insert the rows one by one, either call "aProduct.Add()" or call "Insert.Execute()" mutiple times (just like the codes below), it works fine.
private static Product[] CreateProducts(int count)
{
Product[] products = new Product[count];
for (int index = 0; index < products.Length; ++index)
{
products[index] = new Product
{
ProductName = string.Format("cheka-test-{0}", index.ToString()),
Discontinued = (index % 2 == 0),
};
}
return products;
}
private static void SucceedByMultiExecuteInsert()
{
Product[] products = CreateProducts(2);
// -------------------------------- prepare batch
NorthwindDB db = new NorthwindDB();
var inserts = from prod in products
select db.Insert.Into<Product>(x => x.ProductName, x => x.Discontinued).Values(prod.ProductName, prod.Discontinued);
// -------------------------------- batch insert
var selectAll = Product.All();
Console.WriteLine("--- before total rows = {0}", selectAll.Count().ToString());
foreach (Insert insert in inserts)
insert.Execute();
Console.WriteLine("+++ after inserting {0} rows, now total rows = {1}",
products.Length.ToString(), selectAll.Count().ToString());
}
but if I use "BatchQuery" like the codes below,
private static void FailByBatchInsert()
{
Product[] products = CreateProducts(2);
// -------------------------------- prepare batch
NorthwindDB db = new NorthwindDB();
BatchQuery batchquery = new BatchQuery(db.Provider, db.QueryProvider);
var inserts = from prod in products
select db.Insert.Into<Product>(x => x.ProductName, x => x.Discontinued).Values(prod.ProductName, prod.Discontinued);
foreach (Insert insert in inserts)
batchquery.Queue(insert);
// -------------------------------- batch insert
var selectAll = Product.All();
Console.WriteLine("--- before total rows = {0}", selectAll.Count().ToString());
batchquery.Execute();
Console.WriteLine("+++ after inserting {0} rows, now total rows = {1}",
products.Length.ToString(), selectAll.Count().ToString());
}
then it failed with the exception :
"
Unhandled Exception: System.Data.SqlClient.SqlException: Must declare the scalar variable "#ins_ProductName".
Must declare the scalar variable "#ins_ProductName".
"
Please give me some help to solve this problem. Many thanks.
I ran into this problem as well. If you look at the query it's attempting to run, you'll see it doing something like this (this isn't actual code but you'll get the point):
exec_sql N'insert into MyTable (SomeField) Values (#ins_SomeField)',N'#0 varchar(32)','#0=SomeValue'
For some reason it defines the parameters in the query with "#ins_"+FieldName but then passes the parameters as ordinals. I have yet to determine the pattern for why/when it does this but I've lost enough time during this dev cycle futzing with SubSonic to try and diagnose the problem properly.
The work-around I implemented will involve you downloading the 3.0.0.4 source from github and making a change on line 179 of Insert.cs.
Where it reads
ParameterName = _provider.ParameterPrefix + "ins_" + columnName.ToAlphaNumericOnly(),
Changing it to
ParameterName = _provider.ParameterPrefix + Inserts.Count.ToString(),
seemed to do the trick for me. I make no warranties about this solution for you, expressed or implied. It did work for me but your mileage may vary.
I should also note that there's similar logic around the "update" statements as well in Update.cs on lines 181 and 194 but I haven't had these give me problems... yet.
Honestly, I don't think SubSonic is ready for primetime and that's a shame because I really like how Rob set it up. That said, it's in my product for better or worse now so you make the best with what you got.

SQLiteException and SQLite error near "(": syntax error with Subsonic ActiveRecord

I ran into an interesting error with the following LiNQ query using LiNQPad and when using Subsonic 3.0.x w/ActiveRecord within my project and wanted to share the error and resolution for anyone else who runs into it.
The linq statement below is meant to group entries in the tblSystemsValues collection into their appropriate system and then extract the system with the highest ID.
from ksf in KeySafetyFunction where ksf.Unit == 2 && ksf.Condition_ID == 1
join sys in tblSystems on ksf.ID equals sys.KeySafetyFunction
join xval in (from t in tblSystemsValues
group t by t.tblSystems_ID into groupedT
select new
{
sysId = groupedT.Key,
MaxID = groupedT.Max(g=>g.ID),
MaxText = groupedT.First(gt2 => gt2.ID ==
groupedT.Max(g=>g.ID)).TextValue,
MaxChecked = groupedT.First(gt2 => gt2.ID ==
groupedT.Max(g=>g.ID)).Checked
}) on sys.ID equals xval.sysId
select new {KSFDesc=ksf.Description, sys.Description, xval.MaxText, xval.MaxChecked}
On its own, the subquery for grouping into groupedT works perfectly and the query to match up KeySafetyFunctions with their System in tblSystems also works perfectly on its own.
However, when trying to run the completed query in linqpad or within my project I kept running into a SQLiteException SQLite Error Near "("
First I tried splitting the queries up within my project because I knew that I could just run a foreach loop over the results if necessary. However, I continued to receive the same exception!
I eventually separated the query into three separate parts before I realized that it was the lazy execution of the queries that was killing me. It then became clear that adding the .ToList() specifier after the myProtectedSystem query below was the key to avoiding the lazy execution after combining and optimizing the query and being able to get my results despite the problems I encountered with the SQLite driver.
// determine the max Text/Checked values for each system in tblSystemsValue
var myProtectedValue = from t in tblSystemsValue.All()
group t by t.tblSystems_ID into groupedT
select new {
sysId = groupedT.Key,
MaxID = groupedT.Max(g => g.ID),
MaxText = groupedT.First(gt2 => gt2.ID ==groupedT.Max(g => g.ID)).TextValue,
MaxChecked = groupedT.First(gt2 => gt2.ID ==groupedT.Max(g => g.ID)).Checked};
// get the system description information and filter by Unit/Condition ID
var myProtectedSystem = (from ksf in KeySafetyFunction.All()
where ksf.Unit == 2 && ksf.Condition_ID == 1
join sys in tblSystem.All() on ksf.ID equals sys.KeySafetyFunction
select new {KSFDesc = ksf.Description, sys.Description, sys.ID}).ToList();
// finally join everything together AFTER forcing execution with .ToList()
var joined = from protectedSys in myProtectedSystem
join protectedVal in myProtectedValue on protectedSys.ID equals protectedVal.sysId
select new {protectedSys.KSFDesc, protectedSys.Description, protectedVal.MaxChecked, protectedVal.MaxText};
// print the gratifying debug results
foreach(var protectedItem in joined)
{
System.Diagnostics.Debug.WriteLine(protectedItem.Description + ", " + protectedItem.KSFDesc + ", " + protectedItem.MaxText + ", " + protectedItem.MaxChecked);
}
Avoid lazy evaluation by forcing an early execution with .ToList() on one of the components of the final query. The results will go into memory so try to make sure you are choosing a small set of data and don't force an unbounded query or gigantic query into a list.

Resources