Prior to one of the latest releases I could use a predicate like following:
(myEntity.myEnum, '<', 100) -
this no longer works as enums are treated like strings
how do we filter on an enum then?
You use the 'name' of the enumerated value. For example
On the server:
public enum RoleType {
Guest = 0,
Restricted = 1,
Standard = 2,
Admin = 3
}
To query:
var query = new EntityQuery("Roles").where("roleType", "==", 'Restricted');
Related
I am just not understanding the LINQ non-query syntax for GroupBy.
I have a collection of objects that I want to group by a single property. In this case Name
{ Id="1", Name="Bob", Age="23" }
{ Id="2", Name="Sally", Age="41" }
{ Id="3", Name="Bob", Age="73" }
{ Id="4", Name="Bob", Age="34" }
I would like to end up with a collection of all the unique names
{ Name="Bob" }
{ Name="Sally" }
Based on some examples I looked at I thought this would be the way to do it
var uniqueNameCollection = Persons.GroupBy(x => x.Name).Select(y => y.Key).ToList();
But I ended up with a collection with one item. So I though maybe I was over complicating things with the projection. I tried this
var uniqueNameCollection = Persons.GroupBy(x => x.Name).ToList();
Same result. I ended up with a single item in the collection. What am I doing wrong here? I am just looking to GroupBy the Name property.
var names = Persons.Select(p => p.Name).Distinct().ToList()
If you just want names
LINQ's GroupBy doesn't work the same way that SQL's GROUP BY does.
GroupBy takes a sequence and a function to find the field to group by as parameters, and return a sequence of IGroupings that each have a Key that is the field value that was grouped by and sequence of elements in that group.
IEnumerable<IGrouping<TSource>> GroupBy<TSource, TKey>(
IEnumerable<TSource> sequence,
Func<TSource, TKey> keySelector)
{ ... }
So if you start with a list like this:
class Person
{
public string Name;
}
var people = new List<Person> {
new Person { Name = "Adam" },
new Person { Name = "Eve" }
}
Grouping by name will look like this
IEnumerable<IGrouping<Person>> groups = people.GroupBy(person => person.Name);
You could then select the key from each group like this:
IEnumerable<string> names = groups.Select(group => group.Key);
names will be distinct because if there were multiple people with the same name, they would have been in the same group and there would only be one group with that name.
For what you need, it would probably be more efficient to just select the names and then use Distinct
var names = people.Select(p => p.Name).Distinct();
var uniqueNameCollection = Persons.GroupBy(x => x.Name).Select(y => y.Key).ToList();
Appears valid to me. .net Fiddle showing proper expected outcome: https://dotnetfiddle.net/2hqOvt
Using your data I ran the following code statement
var uniqueNameCollection = people.GroupBy(x => x.Name).Select(y => y.Key).ToList();
The return results were List
Bob
Sally
With 2 items in the List
run the following statement and your count should be 2.
people.GroupBy(x => x.Name).Select(y => y.Key).ToList().Count();
Works for me, download a nugget MoreLinq
using MoreLinq
var distinctitems = list.DistinctBy( u => u.Name);
I would like to define an enum-like structure in JS, but have two requirements:
The values be read-only, i.e. no users can assign to them.
The values (0, 1, 2, ...) can be mapped back into the names (as with Java's name method)
The methods I know to create enums like this typically meet one requirement or the other, not both.
I've tried:
const MyEnum = {
a: 0,
b: 1,
c: 2
};
The enum itself is constant, but the values are still mutable and I can't map values back to names efficiently.
When writing an enum in Typescript, it outputs:
var MyEnum;
(function (MyEnum) {
MyEnum[MyEnum["a"] = 0] = "a";
MyEnum[MyEnum["b"] = 1] = "b";
MyEnum[MyEnum["c"] = 2] = "c";
})(MyEnum || (MyEnum = {}));
This can map both ways, but still doesn't have constant values.
The only option I've found that meets both requirements would be using getters on a class:
class MyEnum {
get a() {
return 0;
}
...
}
This method dramatically restricts the legal names and has a lot of overhead, especially in browsers that don't inline getters well (or can't).
#Shmiddty suggested freezing an object:
const MyEnum = Object.freeze({
a: 0,
b: 1,
c: 2
});
This meets the constant requirement well, but doesn't provide a great way to map values back to names.
I could write a helper that builds the reverse mapping like:
function reverseEnum(enum) {
Object.keys(enum).forEach(k => {
enum[enum[k]] = k;
});
}
But any kind of programmatic solution to generate the reverse mapping will run into problems if the original object is frozen or otherwise actually constant.
Is there a clean, concise solution to this in JS?
This does a pretty good job, IMHO.
function Enum(a){
let i = Object
.keys(a)
.reduce((o,k)=>(o[a[k]]=k,o),{});
return Object.freeze(
Object.keys(a).reduce(
(o,k)=>(o[k]=a[k],o), v=>i[v]
)
);
} // y u so terse?
const FOO = Enum({
a: 0,
b: 1,
c: "banana"
});
console.log(FOO.a, FOO.b, FOO.c); // 0 1 banana
console.log(FOO(0), FOO(1), FOO("banana")); // a b c
try {
FOO.a = "nope";
}
catch (e){
console.log(e);
}
I'd use a Map so that your enum values can be any type, rather than having them coerced into strings.
function Enum(obj){
const keysByValue = new Map();
const EnumLookup = value => keysByValue.get(value);
for (const key of Object.keys(obj)){
EnumLookup[key] = obj[key];
keysByValue.set(EnumLookup[key], key);
}
// Return a function with all your enum properties attached.
// Calling the function with the value will return the key.
return Object.freeze(EnumLookup);
}
If your enum is all strings, I'd also probably change one line to:
EnumLookup[key] = Symbol(obj[key]);
to ensure that the enum values are being used properly. Using just a string, you have no guarantee that some code hasn't simply passed a normal string that happens to be the same as one of your enum values. If your values are always strings or symbols, you could also swap out the Map for a simple object.
Just recently implemented an Es6 version that works quite well:
const k_VALUES = {}
export class ErrorCode {
constructor(p_apiCode, p_httpCode){
this.apiCode = p_apiCode;
this.httpCode = p_httpCode;
k_VALUES[p_apiCode] = this;
}
static create(p_apiCode){
if(k_VALUES[p_apiCode]){
return k_VALUES[p_apiCode];
}
return ErrorCode.UNKNOWN;
}
}
ErrorCode.UNKNOWN = new ErrorCode(0, 500);
ErrorCode.NOT_FOUND = new ErrorCode(-1000, 404);
ErrorCode.NOT_FOUND_EMAIL = new ErrorCode(-1001, 404);
ErrorCode.BAD_REQUEST = new ErrorCode(-1010, 404);
I wanted to implement a similar pattern as what we do with Java enums. This enables me to use a constructor to pass values. The constructor then freezes the ErrorCode object - nice and convenient.
Usage: first import your enum class...
import {ErrorCode} from "../common/services/errors/ErrorCode";
Now, after importing the enum class, access it like so:
if( errCode.includes(ErrorCode.BAD_REQUEST.apiCode) ){...}
PS> This is used in conjunction with a Webpack setup using Babel to convert our ES6 classes down for browser compatibility.
I'm looking for an elegant way to execute a Contains() statement in a scalable way. Please allow me to give some background before I come to the actual question.
The IN statement
In Entity Framework and LINQ to SQL the Contains statement is translated as a SQL IN statement. For instance, from this statement:
var ids = Enumerable.Range(1,10);
var courses = Courses.Where(c => ids.Contains(c.CourseID)).ToList();
Entity Framework will generate
SELECT
[Extent1].[CourseID] AS [CourseID],
[Extent1].[Title] AS [Title],
[Extent1].[Credits] AS [Credits],
[Extent1].[DepartmentID] AS [DepartmentID]
FROM [dbo].[Course] AS [Extent1]
WHERE [Extent1].[CourseID] IN (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
Unfortunately, the In statement is not scalable. As per MSDN:
Including an extremely large number of values (many thousands) in an IN clause can consume resources and return errors 8623 or 8632
which has to do with running out of resources or exceeding expression limits.
But before these errors occur, the IN statement becomes increasingly slow with growing numbers of items. I can't find documentation about its growth rate, but it performs well up to a few thousands of items, but beyond that it gets dramatically slow. (Based on SQL Server experiences).
Scalable
We can't always avoid this statement. A JOIN with the source data in stead would generally perform much better, but that's only possible when the source data is in the same context. Here I'm dealing with data coming from a client in a disconnected scenario. So I have been looking for a scalable solution. A satisfactory approach turned out to be cutting the operation into chunks:
var courses = ids.ToChunks(1000)
.Select(chunk => Courses.Where(c => chunk.Contains(c.CourseID)))
.SelectMany(x => x).ToList();
(where ToChunks is this little extension method).
This executes the query in chunks of 1000 that all perform well enough. With e.g. 5000 items, 5 queries will run that together are likely to be faster than one query with 5000 items.
But not DRY
But of course I don't want to scatter this construct all over my code. I am looking for an extension method by which any IQueryable<T> can be transformed into a chunky executing statement. Ideally something like this:
var courses = Courses.Where(c => ids.Contains(c.CourseID))
.AsChunky(1000)
.ToList();
But maybe this
var courses = Courses.ChunkyContains(c => c.CourseID, ids, 1000)
.ToList();
I've given the latter solution a first shot:
public static IEnumerable<TEntity> ChunkyContains<TEntity, TContains>(
this IQueryable<TEntity> query,
Expression<Func<TEntity,TContains>> match,
IEnumerable<TContains> containList,
int chunkSize = 500)
{
return containList.ToChunks(chunkSize)
.Select (chunk => query.Where(x => chunk.Contains(match)))
.SelectMany(x => x);
}
Obviously, the part x => chunk.Contains(match) doesn't compile. But I don't know how to manipulate the match expression into a Contains expression.
Maybe someone can help me make this solution work. And of course I'm open to other approaches to make this statement scalable.
I’ve solved this problem with a little different approach a view month ago. Maybe it’s a good solution for you too.
I didn’t want my solution to change the query itself. So a ids.ChunkContains(p.Id) or a special WhereContains method was unfeasible. Also should the solution be able to combine a Contains with another filter as well as using the same collection multiple times.
db.TestEntities.Where(p => (ids.Contains(p.Id) || ids.Contains(p.ParentId)) && p.Name.StartsWith("Test"))
So I tried to encapsulate the logic in a special ToList method that could rewrite the Expression for a specified collection to be queried in chunks.
var ids = Enumerable.Range(1, 11);
var result = db.TestEntities.Where(p => Ids.Contains(p.Id) && p.Name.StartsWith ("Test"))
.ToChunkedList(ids,4);
To rewrite the expression tree I discovered all Contains Method calls from local collections in the query with a view helping classes.
private class ContainsExpression
{
public ContainsExpression(MethodCallExpression methodCall)
{
this.MethodCall = methodCall;
}
public MethodCallExpression MethodCall { get; private set; }
public object GetValue()
{
var parent = MethodCall.Object ?? MethodCall.Arguments.FirstOrDefault();
return Expression.Lambda<Func<object>>(parent).Compile()();
}
public bool IsLocalList()
{
Expression parent = MethodCall.Object ?? MethodCall.Arguments.FirstOrDefault();
while (parent != null) {
if (parent is ConstantExpression)
return true;
var member = parent as MemberExpression;
if (member != null) {
parent = member.Expression;
} else {
parent = null;
}
}
return false;
}
}
private class FindExpressionVisitor<T> : ExpressionVisitor where T : Expression
{
public List<T> FoundItems { get; private set; }
public FindExpressionVisitor()
{
this.FoundItems = new List<T>();
}
public override Expression Visit(Expression node)
{
var found = node as T;
if (found != null) {
this.FoundItems.Add(found);
}
return base.Visit(node);
}
}
public static List<T> ToChunkedList<T, TValue>(this IQueryable<T> query, IEnumerable<TValue> list, int chunkSize)
{
var finder = new FindExpressionVisitor<MethodCallExpression>();
finder.Visit(query.Expression);
var methodCalls = finder.FoundItems.Where(p => p.Method.Name == "Contains").Select(p => new ContainsExpression(p)).Where(p => p.IsLocalList()).ToList();
var localLists = methodCalls.Where(p => p.GetValue() == list).ToList();
If the local collection passed in the ToChunkedList method was found in the query expression, I replace the Contains call to the original list with a new call to a temporary list containing the ids for one batch.
if (localLists.Any()) {
var result = new List<T>();
var valueList = new List<TValue>();
var containsMethod = typeof(Enumerable).GetMethods(BindingFlags.Static | BindingFlags.Public)
.Single(p => p.Name == "Contains" && p.GetParameters().Count() == 2)
.MakeGenericMethod(typeof(TValue));
var queryExpression = query.Expression;
foreach (var item in localLists) {
var parameter = new List<Expression>();
parameter.Add(Expression.Constant(valueList));
if (item.MethodCall.Object == null) {
parameter.AddRange(item.MethodCall.Arguments.Skip(1));
} else {
parameter.AddRange(item.MethodCall.Arguments);
}
var call = Expression.Call(containsMethod, parameter.ToArray());
var replacer = new ExpressionReplacer(item.MethodCall,call);
queryExpression = replacer.Visit(queryExpression);
}
var chunkQuery = query.Provider.CreateQuery<T>(queryExpression);
for (int i = 0; i < Math.Ceiling((decimal)list.Count() / chunkSize); i++) {
valueList.Clear();
valueList.AddRange(list.Skip(i * chunkSize).Take(chunkSize));
result.AddRange(chunkQuery.ToList());
}
return result;
}
// if the collection was not found return query.ToList()
return query.ToList();
Expression Replacer:
private class ExpressionReplacer : ExpressionVisitor {
private Expression find, replace;
public ExpressionReplacer(Expression find, Expression replace)
{
this.find = find;
this.replace = replace;
}
public override Expression Visit(Expression node)
{
if (node == this.find)
return this.replace;
return base.Visit(node);
}
}
Please allow me to provide an alternative to the Chunky approach.
The technique involving Contains in your predicate works well for:
A constant list of values (no volatile).
A small list of values.
Contains will do great if your local data has those two characteristics because these small set of values will be hardcoded in the final SQL query.
The problem begins when your list of values has entropy (non-constant). As of this writing, Entity Framework (Classic and Core) do not try to parameterize these values in any way, this forces SQL Server to generate a query plan every time it sees a new combination of values in your query. This operation is expensive and gets aggravated by the overall complexity of your query (e.g. many tables, a lot of values in the list, etc.).
The Chunky approach still suffers from this SQL Server query plan cache pollution problem, because it does not parametrizes the query, it just moves the cost of creating a big execution plan into smaller ones that are more easy to compute (and discard) by SQL Server, furthermore, every chunk adds an additional round-trip to the database, which increases the time needed to resolve the query.
An Efficient Solution for EF Core
🎉 NEW! QueryableValues EF6 Edition has arrived!
For EF Core keep reading below.
Wouldn't it be nice to have a way of composing local data in your query in a way that's SQL Server friendly? Enter QueryableValues.
I designed this library with these two main goals:
It MUST solve the SQL Server's query plan cache pollution problem ✅
It MUST be fast! ⚡
It has a flexible API that allows you to compose local data provided by an IEnumerable<T> and you get back an IQueryable<T>; just use it as if it were another entity of your DbContext (really), e.g.:
// Sample values.
IEnumerable<int> values = Enumerable.Range(1, 1000);
// Using a Join (query syntax).
var query1 =
from e in dbContext.MyEntities
join v in dbContext.AsQueryableValues(values) on e.Id equals v
select new
{
e.Id,
e.Name
};
// Using Contains (method syntax)
var query2 = dbContext.MyEntities
.Where(e => dbContext.AsQueryableValues(values).Contains(e.Id))
.Select(e => new
{
e.Id,
e.Name
});
You can also compose complex types!
It goes without saying that the provided IEnumerable<T> is only enumerated at the time that your query is materialized (not before), preserving the same behavior of EF Core in this regard.
How Does It Works?
Internally QueryableValues creates a parameterized query and provides your values in a serialized format that is natively understood by SQL Server. This allows your query to be resolved with a single round-trip to the database and avoids creating a new query plan on subsequent executions due to the parameterized nature of it.
Useful Links
Nuget Package
GitHub Repository
Benchmarks
SQL Server Cache Pollution Problem
QueryableValues is distributed under the MIT license
Linqkit to the rescue! Might be a better way that does it directly, but this seems to work fine and makes it pretty clear what's being done. The addition being AsExpandable(), which lets you use the Invoke extension.
using LinqKit;
public static IEnumerable<TEntity> ChunkyContains<TEntity, TContains>(
this IQueryable<TEntity> query,
Expression<Func<TEntity,TContains>> match,
IEnumerable<TContains> containList,
int chunkSize = 500)
{
return containList
.ToChunks(chunkSize)
.Select (chunk => query.AsExpandable()
.Where(x => chunk.Contains(match.Invoke(x))))
.SelectMany(x => x);
}
You might also want to do this:
containsList.Distinct()
.ToChunks(chunkSize)
...or something similar so you don't get duplicate results if something this occurs:
query.ChunkyContains(x => x.Id, new List<int> { 1, 1 }, 1);
Another way would be to build the predicate this way (of course, some parts should be improved, just giving the idea).
public static Expression<Func<TEntity, bool>> ContainsPredicate<TEntity, TContains>(this IEnumerable<TContains> chunk, Expression<Func<TEntity, TContains>> match)
{
return Expression.Lambda<Func<TEntity, bool>>(Expression.Call(
typeof (Enumerable),
"Contains",
new[]
{
typeof (TContains)
},
Expression.Constant(chunk, typeof(IEnumerable<TContains>)), match.Body),
match.Parameters);
}
which you could call in your ChunkContains method
return containList.ToChunks(chunkSize)
.Select(chunk => query.Where(ContainsPredicate(chunk, match)))
.SelectMany(x => x);
Using a stored procedure with a table valued parameter could also work well. You in effect write a joint In the stored procedure between your table / view and the table valued parameter.
https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/table-valued-parameters
Here is simplified code
from oi in orderItems
group oiGrouped by ...
into orderItemsGroupedBySomething
select new
{
Key = orderItemsGroupedBySomething.Key,
Revenue = /*Here is some code that I want to extract to separate method, for example*/
orderItemsGroupedBySomething.Sum(x => x.UnitPrice * x.Quantity)
}
Actually it's more complex in my case. But I think it doesn't matter. I'm not able to extract to simple method calculation of orderItemsGroupedBySomething.Sum(x => x.UnitPrice * x.Quantity) because it's not known method for EntityFramework. I tried put it to expression but I get error "The LINQ expression node type 'Invoke' is not supported in LINQ to Entities." I compiled expression before use it in query, I think therefore I get error. How can I solve this problem?
I don't know how generic you need it to be, but something like this should work:
void Main()
{
OrderItems.GroupBy(oi => oi.SomeProp).Select(GetExpression());
}
public Expression<Func<IGrouping<KeyType, OrderItem>, dynamic>> GetExpression()
{
return (ig) => new { Key = ig.Key, Revenue = ig.Sum(x => x.UnitPrice * x.Quantity) };
}
edit: In the case of a grouping, I would probably return a Tuple in stead of an anonymous type.
I added method that return expression
public Expression<Func<OrderItem, decimal>> GetExpression()
{
return x => x.UnitPrice*x.Quantity;
}
Then I tried
from oi in orderItems
group oiGrouped by ...
into orderItemsGroupedBySomething
select new
{
Key = orderItemsGroupedBySomething.Key,
Revenue = orderItemsGroupedBySomething.Sum(GetExpression())
}
But it doesn't work such as with #LorentzVedeler answer. Because orderItemsGroupedBySomething is of type IGrouping that don't have Sum method with param of type Expression. Therefore I tried
orderItemsGroupedBySomething.AsQueryable.Sum(GetExpression())
But it causes Internal .NET Framework Data Provider error 1025. The issue was that I invoked method GetExpression() in linq. To solve this problem I put expression to local variable. The result
var expression = GetExpression();
from oi in orderItems
group oiGrouped by ...
into orderItemsGroupedBySomething
select new
{
Key = orderItemsGroupedBySomething.Key,
Revenue = orderItemsGroupedBySomething.AsQueryable.Sum(expression)
}
I have below code in c# 4, where I am trying to use linq for ordering, grouping.
IList<Component> components = Component.OrganizationalItem.OrganizationalItem.Components(true);
IEnumerable<Component> baggage = components.Where(x => x.IsBasedOnSchema(Constants.Schemas.BaggageAllowance.ToString()))
.OrderBy(x => x.ComponentValue("name").StringValue("Code"))
.GroupBy(x => x.ComponentValue("name").StringValue("Code"));
In above sample when I am trying to use GroupBy it is giving error, please see below:
Cannot implicitly convert type 'System.Collections.Generic.IEnumerable<System.Linq.IGrouping<string,Tridion.ContentManager.ContentManagement.Component>>' to 'System.Collections.Generic.IEnumerable<Tridion.ContentManager.ContentManagement.Component>'. An explicit conversion exists (are you missing a cast?)*
The result of GroupBy will be an IGrouping<string, Component> - it's a sequence of groups of components, rather than one sequence of components. That's the whole point of grouping. So this should be fine:
IEnumerable<IGrouping<string, Component>> baggage = ... query as before ...;
Or just use implicit typing:
var baggage = ...;
You can then iterate over the groups:
foreach (var group in baggage)
{
Console.WriteLine("Key: {0}", group.Key);
foreach (var component in group)
{
...
}
}