SQLiteException and SQLite error near "(": syntax error with Subsonic ActiveRecord - linq

I ran into an interesting error with the following LiNQ query using LiNQPad and when using Subsonic 3.0.x w/ActiveRecord within my project and wanted to share the error and resolution for anyone else who runs into it.
The linq statement below is meant to group entries in the tblSystemsValues collection into their appropriate system and then extract the system with the highest ID.
from ksf in KeySafetyFunction where ksf.Unit == 2 && ksf.Condition_ID == 1
join sys in tblSystems on ksf.ID equals sys.KeySafetyFunction
join xval in (from t in tblSystemsValues
group t by t.tblSystems_ID into groupedT
select new
{
sysId = groupedT.Key,
MaxID = groupedT.Max(g=>g.ID),
MaxText = groupedT.First(gt2 => gt2.ID ==
groupedT.Max(g=>g.ID)).TextValue,
MaxChecked = groupedT.First(gt2 => gt2.ID ==
groupedT.Max(g=>g.ID)).Checked
}) on sys.ID equals xval.sysId
select new {KSFDesc=ksf.Description, sys.Description, xval.MaxText, xval.MaxChecked}
On its own, the subquery for grouping into groupedT works perfectly and the query to match up KeySafetyFunctions with their System in tblSystems also works perfectly on its own.
However, when trying to run the completed query in linqpad or within my project I kept running into a SQLiteException SQLite Error Near "("
First I tried splitting the queries up within my project because I knew that I could just run a foreach loop over the results if necessary. However, I continued to receive the same exception!
I eventually separated the query into three separate parts before I realized that it was the lazy execution of the queries that was killing me. It then became clear that adding the .ToList() specifier after the myProtectedSystem query below was the key to avoiding the lazy execution after combining and optimizing the query and being able to get my results despite the problems I encountered with the SQLite driver.
// determine the max Text/Checked values for each system in tblSystemsValue
var myProtectedValue = from t in tblSystemsValue.All()
group t by t.tblSystems_ID into groupedT
select new {
sysId = groupedT.Key,
MaxID = groupedT.Max(g => g.ID),
MaxText = groupedT.First(gt2 => gt2.ID ==groupedT.Max(g => g.ID)).TextValue,
MaxChecked = groupedT.First(gt2 => gt2.ID ==groupedT.Max(g => g.ID)).Checked};
// get the system description information and filter by Unit/Condition ID
var myProtectedSystem = (from ksf in KeySafetyFunction.All()
where ksf.Unit == 2 && ksf.Condition_ID == 1
join sys in tblSystem.All() on ksf.ID equals sys.KeySafetyFunction
select new {KSFDesc = ksf.Description, sys.Description, sys.ID}).ToList();
// finally join everything together AFTER forcing execution with .ToList()
var joined = from protectedSys in myProtectedSystem
join protectedVal in myProtectedValue on protectedSys.ID equals protectedVal.sysId
select new {protectedSys.KSFDesc, protectedSys.Description, protectedVal.MaxChecked, protectedVal.MaxText};
// print the gratifying debug results
foreach(var protectedItem in joined)
{
System.Diagnostics.Debug.WriteLine(protectedItem.Description + ", " + protectedItem.KSFDesc + ", " + protectedItem.MaxText + ", " + protectedItem.MaxChecked);
}

Avoid lazy evaluation by forcing an early execution with .ToList() on one of the components of the final query. The results will go into memory so try to make sure you are choosing a small set of data and don't force an unbounded query or gigantic query into a list.

Related

Entity Framework Core + Count with Group By

I have a table which contains ~600k records and 33 columns. In my project I am using EF Core (2.0.1) to retrieve data from database. I am having issues with below code:
var theCounter = (from f in _context.tblData.Take(100000)
group f by f.TypeId into data
select new DataDto { ID = data.Key, Count = data.Count() }).ToList();
This code is a part of REST API and when I am testing it from SOAP UI, I am gettin timeout error. When I tested the code for
Take(1000)
There are around 300 unique TypeIds.
it works fine. Any ideas how I can make it work?
-- EDIT 1:
Here is what I see when debugging the code:
Microsoft.EntityFrameworkCore.Query:Warning: Query: '(from TblData <generated>_1 in DbSet<TblData> select [<generated>_1]).Take(__p_0)' uses a row limiting operation (Skip/Take) without OrderBy which may lead to unpredictable results.
Microsoft.EntityFrameworkCore.Query:Warning: Query: '(from TblData <generated>_1 in DbSet<TblData> select [<generated>_1]).Take(__p_0)' uses a row limiting operation (Skip/Take) without OrderBy which may lead to unpredictable results.
Microsoft.EntityFrameworkCore.Query:Warning: The LINQ expression 'GroupBy([f].TypeId, [f])' could not be translated and will be evaluated locally.
Microsoft.EntityFrameworkCore.Query:Warning: The LINQ expression 'GroupBy([f].TypeId, [f])' could not be translated and will be evaluated locally.
Microsoft.EntityFrameworkCore.Query:Warning: The LINQ expression 'Count()' could not be translated and will be evaluated locally.
Microsoft.EntityFrameworkCore.Database.Command:Information: Executed DbCommand (131ms) [Parameters=[#__p_0='?'], CommandType='Text', CommandTimeout='30']
SELECT [t2].[Id], [t2].[at], [t2].[add], [t2].[AddDate], [t2].[aftc], [t2].[aftcd], [t2].[aid], [t2].[afl], [t2].[prdid], [t2].[cid], [t2].[TypeId], [t2].[env], [t2].[ext], [t2].[extddcode], [t2].[fn], [t2].[fn], [t2].[fic], [t2].[gid], [t2].[grp], [t2].[hnm], [t2].[IP], [t2].[icid], [t2].[ln], [t2].[lg], [t2].[pcid], [t2].[ret], [t2].[rts], [t2].[rnam], [t2].[sled], [t2].[seq], [t2].[sid], [t2].[styp]
FROM (
SELECT TOP(#__p_0) [t1].[Id], [t1].[at], [t1].[add], [t1].[AddDate], [t1].[aftc], [t1].[aftcd], [t1].[aid], [t1].[afl], [t1].[prdid], [t1].[cid], [t1].[TypeId], [t1].[env], [t1].[ext], [t1].[extddcode], [t1].[fn], [t1].[fn], [t1].[fic], [t1].[gid], [t1].[grp], [t1].[hnm], [t1].[IP], [t1].[icid], [t1].[ln], [t1].[lg], [t1].[pcid], [t1].[ret], [t1].[rts], [t1].[rnam], [t1].[sled], [t1].[seq], [t1].[sid], [t1].[styp]
FROM [TblData] AS [t1]
) AS [t2]
WHERE [t2].[TypeId] IS NOT NULL
ORDER BY [t2].[TypeId]
I think it is not translated properly. Any ideas why?
-- EDIT 2:
I have changed my queries to:
var query = _context.TblData
.Select(a => new {ID = a.Id, TypeId= a.TypeId})
.Distinct();
var q1 = query.GroupBy(p => p.TypeId)
.Select(g => new DataDto {TypeId= g.Key, Count = g.Count()});
return await q1.ToListAsync();
But it was translated to:
SELECT DISTINCT [a0].[Id], [a0].[TypeId] AS [TypeId]
FROM [tblData] AS [a0]
ORDER BY [a0].[TypeId]
When I checked directly in the database this query takes 14 seconds to execute. Any idea why it was not translated to something like:
SELECT DISTINCT [a0].[Id], COUNT([TypeId]) AS [TypeId]
FROM [tblData] AS [a0]
GROUP BY COUNT([a0].[Id])
ORDER BY [a0].[TypeId]
I had to upgrade EF Core version to 2.1 and LINQ is now translated properly into SQL.

Linq: Select Most Recent Record of Each Group

I want to get the latest record of each group from a SQL Server table using Linq.
Table Example:
I want to get this result:
My Linq query returns one record for each company, but it doesn't return the most recent ones:
var query = from p in db.Payments
where p.Status == false
&& DateTime.Compare(DateTime.Now, p.NextPaymentDate.Value) == 1
group p by p.CompanyID into op
select op.OrderByDescending(nd => nd.NextPaymentDate.Value).FirstOrDefault();
What am i missing here? Why isn't the NextPaymentDate being ordered correctly?
!!UPDATE!!
My query is working as expected. After analysing #Gilang and #JonSkeet comments i ran further tests and found that i wasn't getting the intended results due to a column that wasn't being updated.
var query = from p in db.Payments
where p.Status == false
group p by p.CompanyID into op
select new {
CompanyID = op.Key,
NextPaymentDate = op.Max(x => x.NextPaymentDate),
Status = false
};
The reason your query is not being ordered correctly is that your query does not do proper grouping. You did correctly grouping by CompanyID, but then you have to retrieve the maximum NextPaymentDate by calling aggregate function.
Status can be assigned false because it is already filtered by Where clause in the early clauses.

Building LINQ Expression Getting Ignored

I'm trying to build a linq query coming from a table grid from the client side, so im expecting page offset, page start, order and the traditional paging parameters. I have the following code:
[Route("api/settings/logs")]
public Rest.DatatablesResponse GetLogs(int draw, int start, int length) {
var query_string = Request.GetQueryNameValuePairs().ToDictionary(x => x.Key, x => x.Value);
var search = query_string["search.value"];
int order_column = int.Parse(query_string["order[0].column"]);
var order_direction = query_string["order[0].dir"];
var count = db.Logs.Count(q => q.Mode == 2);
var logs = (from l in db.Logs
where l.Mode == 2
select new {
id = l.ID,
mode = l.Mode,
phase_id = l.Phase.ID,
created = l.Created,
user = l.User.Name,
blender_name = l.Blender.Name,
oil_name = l.Oil,
oil_quantity = l.OilQuantity,
production_cycle_name = l.ProductionCycle.Name
});
if (order_direction == "asc") {
if (order_column == 0) logs.OrderBy(q => q.created);
else if (order_column == 2) logs.OrderBy(q => q.production_cycle_name);
} else {
if (order_column == 0) logs.OrderByDescending(q => q.created);
else if (order_column == 2) logs.OrderByDescending(q => q.production_cycle_name);
};
if (!string.IsNullOrEmpty(search)) {
logs.Where(q => q.blender_name.Contains(search) ||
q.oil_name.Contains(search) ||
SqlFunctions.StringConvert((decimal)q.id).Contains(search));
}
logs.Skip(start).Take(length);
DateTime dtDateTime = new DateTime(1970,1,1,0,0,0,0,System.DateTimeKind.Utc);
var steps = from l in logs.ToList()
select new {
id = l.id,
message = StringHelpers.FormatWith(_tpl_message[l.phase_id.ToString() + l.mode.ToString() ], l) ,
created = dtDateTime.AddSeconds(l.created).ToString("h:mmtt - MMMM d, yyyy"),
production_cycle_name = l.production_cycle_name
};
return new Rest.DatatablesResponse {
draw = draw,
recordsTotal = count,
recordsFiltered = count,
data = steps.ToArray()
};
}
My problem is the skip and take and orderby expressions are getting ignored for some reason, and this is the SQL code generated just before converting my linq expressions to a list. From my understanding, the query should not be executed or evaluated until my logs.ToList() call, so the ordering and take/skip should be taken into account, but it's not:
{SELECT
[Extent1].[ID] AS [ID],
[Extent1].[Mode] AS [Mode],
[Extent1].[Phase_ID] AS [Phase_ID],
[Extent1].[Created] AS [Created],
[Extent2].[Name] AS [Name],
[Extent3].[Name] AS [Name1],
[Extent1].[Oil] AS [Oil],
[Extent1].[OilQuantity] AS [OilQuantity],
[Extent4].[Name] AS [Name2]
FROM [dbo].[Steps] AS [Extent1]
LEFT OUTER JOIN [dbo].[Users] AS [Extent2] ON [Extent1].[User_Id] = [Extent2].[Id]
LEFT OUTER JOIN [dbo].[Blenders] AS [Extent3] ON [Extent1].[Blender_ID] = [Extent3].[ID]
LEFT OUTER JOIN [dbo].[ProductionCycles] AS [Extent4] ON [Extent1].[ProductionCycle_ID] = [Extent4].[ID]
WHERE 2 = [Extent1].[Mode]}
Irrelevant P.S. I'm using the not so clever ifs for building the order expression instead of using DynamicLINQ since i have only two sortable columns.
logs.Skip(start).Take(length);
Creates a IQueryable<T> where T is the same anonymous type of which logs is an IQueryable<T> but with start items skipped. Then from that it creates a similarly typed IQueryable<T> where lenght items are the most that will be taken.
Then it throws that away and lets it be garbage collected. (Or ideally the compiler or jitter steps will realise it's thrown away and cut out the whole thing).
Then logs.ToList() goes back to the logs you still have and creates a list from it.
You should replace the Skip and Take line with:
logs = logs.Skip(start).Take(length);
So that you are actually making use of this skipping and taking.
I'm using the not so clever ifs for building the order expression instead of using DynamicLINQ since i have only two sortable columns.
There's nothing particularly not-clever about that, except that you make the same mistake; apply the OrderBy and then throwing away the result instead of using it. Likewise with the Where. You need logs = logs.OrderBy(...) etc.
I'd also question from l in logs.ToList() select new {…} here.
It might be the best approach, if obtaining that list in one step has some advantage. However otherwise:
from l in logs select new {…}
Do the select work on the database, retrieving just what you need.
from l in logs.AsEnumerable() select new {…}
Do the select work in the application, appropriate if part of it cannot be converted to database work, but do it as it comes rather than loading it all into memory first.
from l in await logs.ToListAsync() select new {…}
Has the downside of ToList() but in asynchronous uses, then (assuming your provider has a ToListAsync() method) allows for awaiting.
ToList() is rarely the best option here.

Wait for DomainContext.Load<t> from an entityquery with joins to complete (returning new type via 'select new')

My app consolidates data from other DBs for reporting purposes. We can't link the databases, so all the data processing has to be done in code - this is fine as we want to allow manual validation during the imports.
Certain users will be able to start an update through the Silverlight 4 front end.
I have 3 tables in database x that are fed from one EF4 Model (ModelX). I want to join those tables together, select specific columns and return the result as a new entity that exists in a different EF4 Model (ModelY). I'm using this query:
var myQuery = from i in DBx.table1 from it in DBx.table2 from h in DBx.table3 where (i.id==it.id && h.otherid == i.otherid) select new ModelYServer {Name = i.name,Thing = it.thing, Stuff = h.stuff};
The bit i'm stuck on, is how to execute that query, and wait until the Asynchronous call has completed. Normally, i'd use:
DomainContext.Load<T>(myQuery).Completed += (sender,args) =>
{List<T> myList = ((LoadOperation<T>)sender.Entities.ToList();};
but I can't pass myQuery (an IEnumerable) into the DomainContext.Load() as that expects an EntityQuery. The dataset is very large, and is taking up to 30 seconds to return, so I definitely need to wait before continuing.
So can anyone tell me how I can wait for the IEnumerable query to complete, or suggest a better way of doing this (there very likely is one).
Thanks
Mick
One simple way is just to force it to evaluate by calling ToList:
var query = from i in DBx.table1
join it in DBx.table2 on i.id equals it.id
join h in DBx.table3 on i.otherid equals h.otherid
select new ModelYServer {
Name = i.name,
Thing = it.thing,
Stuff = h.stuff
};
// This will block until the results have been fetched
var results = query.ToList();
// Now use results...
(I've changed your where clause into joins on the earlier tables, as that's what you were effectively doing and this is more idiomatic, IMO.)

Getting Last rows from the result of Linq to Sql statement

I couldn't get last articles of every writers in this statement.
List<Editor> lstEditors = dataContext.GetTable<Editor>().Where(t => t.M_Active).Select(t => t).ToList();
var lstArticles = from article in DAO.context.GetTable<Article>().ToList()
join editor in lstEditors on article.RefEditorId equals editor.EditorId
select
new
{
article.M_ArticleId,
article.M_Subject,
article.M_Text,
editor.M_EditorId,
editor.M_Member.M_EditorPicture,
M_NameSurname = editor.M_Member.M_Fname + " " + editor.M_Member.M_Lname
};
Be careful, your query is fetching all the contents of both the Editor and the Yazi tables and then performs Linq-to-Objects on it.
I'm not sure what you ask exactly either, do you want to obtain the list of all writers (editors) along with the last article of each one of these writers?
Do you want to get the writers that did not write any articles yet also?
Edit:
explanation of methods causing an immediate query
Any time you call one of the methods listed below on an IQueryable object (tables or other queries), it performs the actual query to SQL server:
ToList(), ToArray(), ToLookup(), ToDictionay()
Count(), Sum(), Avg(), Aggregate(), Min(), Max()
First(), FirstOrDefault(), Last(), LastOrDefault()
getting last article written by each writer
//create a subquery that returns an editor and its last article date
var editorLastArticleDates =
from article in DAO.context.GetTable<Article>()
group article by article.RefEditor into g
let lastArticleDate= g.Max(x => x.Date)
select new
{
Editor = g.Key,
LastArticleDate = lastArticleDate,
};
//Note: We did not do a ToList() here so the query is not executed
// The editorLastArticleDates object is a IQueryable<>
var query =
from article in DAO.context.GetTable<Article>()
join editorLastArticleDate in editorLastArticleDates
on new { article.Editor, article.Date } // 1
equals new { editorLastArticleDate.Editor, // 2
Date = editorLastArticleDate.LastArticleDate } // 3
select new
{
article.M_ArticleId,
article.M_Subject,
article.M_Text,
article.RefEditor.M_EditorId,
article.RefEditor.M_Member.M_EditorPicture,
M_NameSurname = article.RefEditor.M_Member.M_Fname + " "
+ article.RefEditor.M_Member.M_Lname,
};
//Note: We did not do a ToList() yet so the query is not executed
// The query object is a IQueryable<>
Console.WriteLine(query.ToString()); //Displays SQL query on the console
var results = query.ToList(); // SQL query is executed on this line.
In the code above, I left some remarks on things I had problems with:
When using join, the section between new and equals access only variables declared before the join keyword while the section after the equals keyword has access to the variable defined between join and in.
When writing your join condition, make sure you use equals and not ==.
When using new { XXX, YYY } syntax in your join condition, you declare anonymous types. If the property names are not identical on both sides, it will not compile. In order to have identical property names in this sample, I added the Date = before my value.
By the way, you should use LinqPad to test your queries, it is really a nice tool.

Resources