How to Filter Records After GroupBy with LINQ + EF Core 6 - linq

Given this simple Contract entity and assuming that a customer can have many Contracts with different start dates and statuses.
public class Contract {
[Key]
int ContractId { get; set; }
int CustomerId { get; set; }
string Status { get; set; }
DateTime ContractStartDate { get; set; }
}
I'm trying to write a query to find that latest contract for each customer and then filter out certain statuses. I need to filter the status after finding the latest contract, to ensure I'm getting their current status.
ContractID
CustomerID
ContractStartDate
Status
1
1
2022-01-01
Active
2
1
2022-31-05
Inactive
3
2
2022-01-03
Active
4
2
2022-31-07
Inactive
From the above data set, I would expect to get contracts 2 and 4 in the results. This is why I can't filter on status before grouping, because then the latest inactive row would be eliminated before I group them by customer to find the latest row.
I've tried something like, this:
var latestContracts = Query<Contract>()
.GroupBy(grp => grp.CustomerId)
.Select(s => s.OrderByDescending(s => s.ContractStartDate).First())
.Where(w => w.Status == "Active");
but once I execute the query by calling ToListAsync(), etc. I get an error like this:
'The LINQ expression 'DbSet<Customer>()
.GroupBy(c => c.CustomerId)
.Select(g => g
.AsQueryable()
.OrderByDescending(e => e.
I CC ContractStartDate)
.First())
.Where(e0 => e0.AccountContractTermStatus == "Active")' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.'
If I remove the Where call, the query works fine and gives the intended newest results. I know that the support for translating LINQ to SQL is still not fully complete, even in EF Core 6, but this seems like something that should be possible. Am I missing something simple or is this just not yet doable via LINQ?
Happy to provide more context, as I've greatly simplified a production query into something I could share publicly.

Try the following emulation of query which EF Core cannot translate. It is almost direct translation what EF Core should do in your case.
var dataQuery = Query<Contract>();
var latestContracts =
from d in dataQuery.Select(d => new { d.CustomerId }).Distinct()
from c in dataQuery
.Where(c => d.CustomerId == c.CustomerId)
.OrderByDescending(c => c.ContractStartDate)
.Take(1)
where c.Status == "Active"
select c;
Or using Method Chain syntax:
var dataQuery = Query<Contract>();
var latestContracts = dataQuery
.Select(d => new { d.CustomerId })
.Distinct()
.SelectMany(d => dataQuery
.Where(c => d.CustomerId == c.CustomerId)
.OrderByDescending(c => c.ContractStartDate)
.Take(1))
.Where(c => c.Status == "Active");

Related

LINQ include and Projection

I have some classes defining entities with relationships
Account
has many Conversations [IEnumerable<Conversation> Conversations]
Conversation
has many Participants [IEnumerable<Account> Participants]
has many Messages [IEnumerable<Message> Messages]
Message
has one Sender [Account Sender]
has one Conversation [Conversation Conversation]
I'm trying to write a LINQ query that returns a list of Conversation ordered by date and including related participants and messages.
public async Task<List<Conversation>> FindAllByAccountIdAsync(Int32 id)
{
return await _Db.Conversations
.Where(c => c.Participants.Any(p => p.AccountId == id))
.Include(c => c.Participants)
.Include(c => c.Messages)
.ToListAsync();
}
This do the work but includes to much data i do not really need.
public async Task<List<Conversation>> FindAllByAccountIdAsync(Int32 id)
{
return await _Db.Conversations
.Where(c => c.Participants.Any(a => a.AccountId == id))
.Include(c => c.Participants.Select(a=> new
{
AccountId = a.AccountId,
Profile = new { FullName = a.Profile.FullName,
Email = a.Profile.Email
}
}))
// Only return the last message in
// Eventually I would not return an array with a single object but just the single object inside associated with the property LastMessageIn
.Include(c => c.Messages.OrderBy(m => m.Date).Select(m=> new
{
Body = m.Body,
SenderId = m.Sender.AccountId
}).Last())
.ToListAsync();
}
This script returns a mile long exception
{"message":"An error has occurred.","exceptionMessage":"The Include path expression must refer to a navigation property defined on the type. Use dotted paths for reference navigation properties and the Select operator for collection navigation properties........}
My mind resist understanding and learning LINQ I do not know if its just me but as soon requirements exceeds basic querying and projection it break out of my control
Someone have some hints?
I'm not sure if I understand your question, but I believe you want something like this:
public async Task<List<Conversation>> FindAllByAccountIdAsync(Int32 id)
{
return await _Db.Conversations
.Where(c => c.Participants.Any(p => p.AccountId == id))
.Include(c => c.Participants)
.Include(c => c.Messages)
.Select(c => new
{
Participants = c.Participants.Select(a=> new
{
AccountId = a.AccountId,
Profile = new { FullName = a.Profile.FullName,
Email = a.Profile.Email
}
},
//EDIT: using OrderByDescending and FirstOrDefault
Messages = c.Messages.OrderByDescending(m => m.Date).Select(m=> new
{
Body = m.Body,
SenderId = m.Sender.AccountId
}).FirstOrDefault())
//others properties here
}
.ToListAsync();
}
You cannot project on an Include. An include is simply Eager Loading. The output does not change in the C#. Only the amount of data that is originally loaded (ie performance) changes.
It seems you want a projection and not eager loading, which are completely incompatible concepts.
However I cannot understand what exactly what you are trying to achieve.
public async Task<List<Conversation>> FindAllByAccountIdAsync(Int32 id)
{
return await _Db.Conversations
.Where(c => c.Participants.Any(p => p.AccountId == id))
.Include(c => c.Participants.Select(_=>_))
.Include(c => c.Messages.Select(_=>_))
.ToListAsync();
}
Should be enough.

could not resolve property (complex properties)

I have a asp.net mvc application with NHibernate and I do not know how to resolve a problem to query some data. I have this query:
// create query
var query = session.QueryOVer<Laudo>().Fetch(x => x.Equipament).Eager;
// add some filters
if (idEquipament.HasValue)
query = query.And(x => x.Equipament.Id == idEquipament.Value);
//I got the error here...
if (idCompany.HasValue)
query = query.And(x => x.Equipament.Company.Id == idCompany.Value);
When I try to execute this query, I've got an exception with this message:
"could not resolve property: Equipament.Company.Id of: DomainModel.Laudo"
what can I do to fix this problem?
Thanks
You cannot use another entity property like that. NHibernate expects expression that can be evaluated to property of the current entity. You need to use JoinQueryOver or JoinAlias to join another entity, and perform where after that.
With JoinQueryOver:
// ...
query = query.JoinQueryOver(x => x.Equipment)
.JoinQueryOver(x => x.Company)
.Where(c => c.Id == idCompany.Value);
With JoinAlias:
Equipment equipment = null;
Company company = null;
// ...
query = query.JoinAlias(x => x.Equipment, () => equipment)
.JoinAlias(() => equipment.Company, () => company)
.Where(() => company.Id == idCompany.Value);
Some more info:
What is the difference between JoinQueryOver and JoinAlias?
What can be used as a NHibernate QueryOver alias?
Complex nHibernate QueryOver expression
The tags chosen for your question make me think you didn't want to use QueryOver, but LINQ.
This is achieved by using the extension method Query, in the NHibernate.Linq namespace:
var query = session.Query<Laudo>().Fetch(x => x.Equipament);
if (idEquipament.HasValue)
query = query.Where(x => x.Equipament.Id == idEquipament.Value);
if (idCompany.HasValue)
query = query.Where(x => x.Equipament.Company.Id == idCompany.Value);

EF 4.3 Code First - Querying a navigation property backwards

I am trying to query a one to many relationship but cannot figure out how to do this. The problem I have is that the ID of the field I want to filter by lives in the join table (not the main table)...
Its probably easier to illustrate rather than explain!!
The two classes I have are
public class DbUserClient
{
public virtual string UserId { get; set; }
public virtual int ClientId { get; set; }
public virtual DateTime AssignedOn { get; set; }
public virtual DateTime? ClearedOn { get; set; }
// navigation properties
public virtual DbUser User { get; set; }
public virtual DbClient Client { get; set; }
}
and
public class DbClient
{
public virtual int ClientId {get;set;}
public virtual string EntityName { get; set; }
public virtual bool Deleted { get; set; }
// navigation properties
public ICollection<DbUserClient> UserClients { get; set; }
}
In the program I have a repository that exposes the Clients i.e.
public ObservableCollection<DbClient> Clients
{
get { return context.Clients.Local; }
}
I am binding to this which is why I am keen on querying via the Client as this will refresh my "Local" collection. However I can't seem to figure out a way to include the UserClients as well as add the "where" clause.
I have tried something like
context.Clients.Include(c => c.UserClients.Where(uc => uc.UserId == "ME"));
But this results in the following exception
"The Include path expression must refer to a navigation property defined on the type. Use dotted paths for reference navigation properties and the Select operator for collection navigation properties.
Parameter name: path"
This works but unfortunately will not update my "Local" collection
from c in context.Clients
from uc in c.UserClients
where uc.ClientId == uc.ClientId && uc.UserId == "ME"
select new { c.ClientId, c.EntityName, uc.AssignedOn };
Any suggestions on where I have gone wrong?
Cheers
Abs
EDIT I : looking at the SQL Profiler the above query generates the following SQL
SELECT
[Extent1].[ClientId] AS [ClientId],
[Extent1].[EntityName] AS [EntityName],
[Extent2].[AssignedOn] AS [AssignedOn]
FROM [dbo].[Client] AS [Extent1]
INNER JOIN [dbo].[UserClient] AS [Extent2] ON [Extent1].[ClientId] = [Extent2]. [ClientId]
WHERE ([Extent2].[ClientId] = [Extent2].[ClientId]) AND (N'ME' = [Extent2].[UserId])
This is pretty simple and more or less along the lines of what I would have written myself if I was handcrafting the SQL
However although the suggested expression below works and as you pointed out populates the Local cache
context.Clients
.Where(c => c.UserClients.Any(uc => uc.UserId == userId))
.Select(c => new { DbClient = c, DbUser = c.UserClients.Where(uc => uc.UserId == userId).FirstOrDefault() }).ToList();
it produces the following SQL. This looks alot more complicated than it needs to be and I am assuming will have performance implications
exec sp_executesql N'SELECT
[Filter2].[ClientId] AS [ClientId],
[Filter2].[EntityName] AS [EntityName],
[Filter2].[Deleted] AS [Deleted],
[Limit1].[UserId] AS [UserId],
[Limit1].[ClientId] AS [ClientId1],
[Limit1].[AssignedOn] AS [AssignedOn],
[Limit1].[ClearedOn] AS [ClearedOn]
FROM (SELECT [Extent1].[ClientId] AS [ClientId], [Extent1].[EntityName] AS [EntityName], [Extent1].[Deleted] AS [Deleted]
FROM [dbo].[Client] AS [Extent1]
WHERE EXISTS (SELECT
1 AS [C1]
FROM [dbo].[UserClient] AS [Extent2]
WHERE ([Extent1].[ClientId] = [Extent2].[ClientId]) AND ([Extent2].[UserId] = #p__linq__0)
) ) AS [Filter2]
OUTER APPLY (SELECT TOP (1)
[Extent3].[UserId] AS [UserId],
[Extent3].[ClientId] AS [ClientId],
[Extent3].[AssignedOn] AS [AssignedOn],
[Extent3].[ClearedOn] AS [ClearedOn]
FROM [dbo].[UserClient] AS [Extent3]
WHERE ([Filter2].[ClientId] = [Extent3].[ClientId]) AND ([Extent3].[UserId] = #p__linq__1) ) AS [Limit1]',N'#p__linq__0 nvarchar(4000),#p__linq__1 nvarchar(4000)',#p__linq__0=N'ME',#p__linq__1=N'ME'
EDIT II : After playing around some more, I have found a solution that seems to fulfill my requirement. Looking at the SQL Profiler, I am happy with the generated SQL. This is similar to that of my orginal query.
exec sp_executesql N'SELECT
[Extent1].[ClientId] AS [ClientId],
[Extent1].[EntityName] AS [EntityName],
[Extent1].[Deleted] AS [Deleted],
[Extent2].[UserId] AS [UserId],
[Extent2].[ClientId] AS [ClientId1],
[Extent2].[AssignedOn] AS [AssignedOn],
[Extent2].[ClearedOn] AS [ClearedOn]
FROM [dbo].[Client] AS [Extent1]
INNER JOIN [dbo].[UserClient] AS [Extent2] ON [Extent1].[ClientId] = [Extent2].[ClientId]
WHERE [Extent2].[UserId] = #p__linq__0',N'#p__linq__0 nvarchar(4000)',#p__linq__0=N'ME'
I am assuming that there is no lazy loading involved here. If someone could confirm I would be grateful
context.Clients.Join
(
context.UserClients,
c => c.ClientId,
uc => uc.ClientId,
(user, usrclient) => new { DbClient = user, DbUserClient = usrclient }
).Where(uc => uc.DbUserClient.UserId == userId).Load();
You can load the clients which have at least one user with UserId = "ME":
var clients = context.Clients
.Where(c => c.UserClients.Any(uc => uc.UserId == "ME"))
.ToList();
This loads the correct clients but no user is included.
If you include the users...
var clients = context.Clients.Include(c => c.UserClients)
.Where(c => c.UserClients.Any(uc => uc.UserId == "ME"))
.ToList();
... you'll get the correctly filtered clients but it will include all users, not only the user "ME".
In order to get the users filtered as well your last approach, the projection, is the best way:
var clientsWithUser = context.Clients
.Where(c => c.UserClients.Any(uc => uc.UserId == "ME"))
.Select(c => new
{
Client = c,
User = c.UserClients.Where(uc => uc.UserId == "ME").FirstOrDefault()
})
.ToList();
This should also update the Local collection because you are loading full entities (Client and User) in the anonymous object list.
Edit
The last query in your question is fine, although it's not really the EF way to write a Join manually when you have navigation properties. The SQL and query result is most likely identical to:
context.UserClients.Include(uc => uc.Client)
.Where(uc => uc.UserId == userId)
.Load();
The Include in this query should translate into the same INNER JOIN that your hand-written LINQ Join produces.

Linq to NHibernate - select count problem

Given the classes A and B where
class A
{
string Name;
Ilist<B> BList;
}
class B
{
string Name;
}
With FluentNH mapping, relationship is many-to-many which is HasManyToMany(x => x.B) for A. B has no reference to A. NH version is 2.1.2.4000.
What should be the linq query to select the collection where each row contains B.Name and count of A's containing that B? Result must be the List of anonymous type who has 2 fields: Name and Count. Result also should include all B's, hence it should be outer join.
My intend is to get the result with minimum round-trips to database, possibly in one go.
If you want to do it in Linq in one hit in code, you could do this...
var result = Session.Linq<A>()
.SelectMany(a => a.BList, (a, b) => new { b.Name, A = a.Id })
.ToList()
.GroupBy(x => x.Name)
.Select(x => new { Name = x.Key, Count = x.Count() })
.ToList();
NHibernate.Linq (2.1.2.4000) can't handle a GroupBy after a SelectMany it seems, so the first ToList pulls all the data into memory. This is inefficient -- a SQL count would be better.
Alternatively, you could add a lazy loaded collection to your B class that goes back to A. If you're using a many-to-many table in the middle, that should be easy.
public class B
{
public virtual string Name { get; set; }
public virtual IList<A> AList { get; private set; }
}
Your query simply becomes...
var result = Session.Linq<B>()
.Where(b => b.AList.Count > 0)
.Select(b => new { b.Name, b.AList.Count }
.ToList();
Which produces very efficient SQL from Linq (using a count) and gives the same result.

Entity framework linq query Include() multiple children entities

This may be a really elementry question but whats a nice way to include multiple children entities when writing a query that spans THREE levels (or more)?
i.e. I have 4 tables: Company, Employee, Employee_Car and Employee_Country
Company has a 1:m relationship with Employee.
Employee has a 1:m relationship with both Employee_Car and Employee_Country.
If i want to write a query that returns the data from all 4 the tables, I am currently writing:
Company company = context.Companies
.Include("Employee.Employee_Car")
.Include("Employee.Employee_Country")
.FirstOrDefault(c => c.Id == companyID);
There has to be a more elegant way! This is long winded and generates horrendous SQL
I am using EF4 with VS 2010
Use extension methods.
Replace NameOfContext with the name of your object context.
public static class Extensions{
public static IQueryable<Company> CompleteCompanies(this NameOfContext context){
return context.Companies
.Include("Employee.Employee_Car")
.Include("Employee.Employee_Country") ;
}
public static Company CompanyById(this NameOfContext context, int companyID){
return context.Companies
.Include("Employee.Employee_Car")
.Include("Employee.Employee_Country")
.FirstOrDefault(c => c.Id == companyID) ;
}
}
Then your code becomes
Company company =
context.CompleteCompanies().FirstOrDefault(c => c.Id == companyID);
//or if you want even more
Company company =
context.CompanyById(companyID);
EF Core
For eager loading relationships more than one navigation away (e.g. grand child or grand parent relations), where the intermediate relation is a collection (i.e. 1 to many with the original 'subject'), EF Core has a new extension method, .ThenInclude(), and the syntax is slightly different to the older EF 4-6 syntax:
using Microsoft.EntityFrameworkCore;
...
var company = context.Companies
.Include(co => co.Employees)
.ThenInclude(emp => emp.Employee_Car)
.Include(co => co.Employees)
.ThenInclude(emp => emp.Employee_Country)
With some notes
As per above (Employees.Employee_Car and Employees.Employee_Country), if you need to include 2 or more child properties of an intermediate child collection, you'll need to repeat the .Include navigation for the collection for each child of the collection.
Personally, I would keep the extra 'indent' in the .ThenInclude to preserve your sanity.
For serialization of intermediaries which are 1:1 (or N:1) with the original subject, the dot syntax is also supported, e.g.
var company = context.Companies
.Include(co => co.City.Country);
This is functionally equivalent to:
var company = context.Companies
.Include(co => co.City)
.ThenInclude(ci => ci.Country);
However, in EFCore, the old EF4 / 6 syntax of using 'Select' to chain through an intermediary which is 1:N with the subject is not supported, i.e.
var company = context.Companies
.Include(co => co.Employee.Select(emp => emp.Address));
Will typically result in obscure errors like
Serialization and deserialization of 'System.IntPtr' instances are not supported
EF 4.1 to EF 6
There is a strongly typed .Include which allows the required depth of eager loading to be specified by providing Select expressions to the appropriate depth:
using System.Data.Entity; // NB!
var company = context.Companies
.Include(co => co.Employees.Select(emp => emp.Employee_Car))
.Include(co => co.Employees.Select(emp => emp.Employee_Country))
.FirstOrDefault(co => co.companyID == companyID);
The Sql generated is by no means intuitive, but seems performant enough. I've put a small example on GitHub here
You might find this article of interest which is available at codeplex.com.
Improving Entity Framework Query Performance Using Graph-Based Querying.
The article presents a new way of expressing queries that span multiple tables in the form of declarative graph shapes.
Moreover, the article contains a thorough performance comparison of this new approach with EF queries. This analysis shows that GBQ quickly outperforms EF queries.
How do you construct a LINQ to Entities query to load child objects directly, instead of calling a Reference property or Load()
There is no other way - except implementing lazy loading.
Or manual loading....
myobj = context.MyObjects.First();
myobj.ChildA.Load();
myobj.ChildB.Load();
...
Might be it will help someone, 4 level and 2 child's on each level
Library.Include(a => a.Library.Select(b => b.Library.Select(c => c.Library)))
.Include(d=>d.Book.)
.Include(g => g.Library.Select(h=>g.Book))
.Include(j => j.Library.Select(k => k.Library.Select(l=>l.Book)))
To doing this:
namespace Application.Test
{
using Utils.Extensions;
public class Test
{
public DbSet<User> Users { get; set; }
public DbSet<Room> Rooms { get; set; }
public DbSet<Post> Posts { get; set; }
public DbSet<Comment> Comments { get; set; }
public void Foo()
{
DB.Users.Include(x => x.Posts, x => x.Rooms, x => x.Members);
//OR
DB.Users.Include(x => x.Posts, x => x.Rooms, x => x.Members)
.ThenInclude(x => x.Posts, y => y.Owner, y => y.Comments);
}
}
}
this extension might be helpful:
namespace Utils.Extensions
{
using Microsoft.EntityFrameworkCore;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
public static partial class LinqExtension
{
public static IQueryable<TEntity> Include<TEntity>(
this IQueryable<TEntity> sources,
params Expression<Func<TEntity, object>>[] properties)
where TEntity : class
{
System.Text.RegularExpressions.Regex regex = new(#"^\w+[.]");
IQueryable<TEntity> _sources = sources;
foreach (var property in properties)
_sources = _sources.Include($"{regex.Replace(property.Body.ToString(), "")}");
return _sources;
}
public static IQueryable<TEntity> ThenInclude<TEntity, TProperty>(
this IQueryable<TEntity> sources,
Expression<Func<TEntity, IEnumerable<TProperty>>> predicate,
params Expression<Func<TProperty, object>>[] properties)
where TEntity : class
{
System.Text.RegularExpressions.Regex regex = new(#"^\w+[.]");
IQueryable<TEntity> _sources = sources;
foreach (var property in properties)
_sources = _sources.Include($"{regex.Replace(predicate.Body.ToString(), "")}.{regex.Replace(property.Body.ToString(), "")}");
return _sources;
}
}
}

Resources