Generic Linq with specific where/order by clauses - linq

I want to create a generic method that allows me to search a Sitecore 7 index using Linq to Sitecore (.Net 4.5).
This will be used for various searches such as:
Get top 5 latest news pages
Get the last 20 event pages
Generic site search
Etc, etc, etc
I can create a very generic search method, which works for all types of page. It provides a generic template predicate for the Where clause that would be applicable for all types of search.
However, for the searches above I also need to add specific predicates for the Where clause, and specific expressions for Order By etc. The intention would be to create subclasses for each type of search, which would implement these specifics.
I've condensed some code which is shown below. In this I try to add specific functionality for a new page search.
All page classes derive from "Base".
public virtual ReadOnlyCollection<T> Search<T>() where T : Base, new()
{
List<T> results = new List<T>();
using (IProviderSearchContext context = ContentSearchManager.GetIndex("sitecore_web_index").CreateSearchContext())
{
Expression<Func<T, bool>> outerPredicate = PredicateBuilder.True<T>();
// Create a predicate for the template id.
Expression<Func<T, bool>> templatePredicate = PredicateBuilder.False<T>();
templatePredicate = templatePredicate.Or(baseItem => (baseItem.TemplateIdFromIndex.Equals("8b1fc00c76314d32b8e1bce93dd41ccd")));
// Create a predicate for a news page search.
Expression<Func<NewsPageBase, bool>> datePredicate = PredicateBuilder.False<NewsPageBase>();
datePredicate = datePredicate.And(newsPage => newsPage.ArticleDate < DateTime.Now);
// 1. outerPredicate = outerPredicate.And(datePredicate);
// 2. IQueryable<T> searchQuery = context.GetQueryable<T>().Where(outerPredicate).OrderByDescending(newsPage => newsPage.ArticleDate).Take(5);
IQueryable<T> searchQuery = context.GetQueryable<T>().Where(outerPredicate);
results = searchQuery.ToList();
}
return new ReadOnlyCollection<T>(results);
}
This code complies and runs.
However, if I uncomment the line marked as [1], a compiler error me from Anding the generic template predicate with the specific news page predicate.
The error is "The type arguments for method 'Sitecore.ContentSearch.Linq.Utilities.PredicateBuilder.And(System.Linq.Expressions.Expression>, System.Linq.Expressions.Expression>)' cannot be inferred from the usage".
It's a similar error if I uncomment the line marked as [2].
How do I create a generic method which has the specific functionality for each type of search?

Modifying your code as follows achieves what you need..
public virtual ReadOnlyCollection<T> Search<T>() where T : Base, new()
{
List<T> results = new List<T>();
using (IProviderSearchContext context = ContentSearchManager.GetIndex("sitecore_web_index").CreateSearchContext())
{
Expression<Func<T, bool>> outerPredicate = PredicateBuilder.True<T>();
// Create a predicate for a news page search.
Expression<Func<NewsPageBase, bool>> datePredicate = PredicateBuilder.True<NewsPageBase>();
datePredicate = datePredicate.And(newsPage => newsPage.ArticleDate < DateTime.Now);
//outerPredicate = outerPredicate.And((Expression<Func<T,bool>>)(object)datePredicate);
outerPredicate = outerPredicate.And((Expression<Func<T, bool>>)(object)datePredicate);
// 2. IQueryable<T> searchQuery = context.GetQueryable<T>().Where(outerPredicate).OrderByDescending(newsPage => newsPage.ArticleDate).Take(5);
IQueryable<T> searchQuery = context.GetQueryable<T>().Where(outerPredicate);
results = searchQuery.ToList();
}
return new ReadOnlyCollection<T>(results);
}
The changes I made were:
Removed the unused templatePredicate - I'm not sure if you had intended to use this or had forgotten to remove it when condensing your original code.
Create datePredicate using PredicateBuilder.True instead of PredicateBuilder.False - this is needed for use in .And() otherwise no results will be returned.
I (un)box datePredicate for use within outerPredicate.And(). The compiler complains if you try to cast datePredicate to Expression<Func<T, bool>> but casting it to object first solves this.

I have found a work around.
Instead of:
Expression<Func<NewsPageBase, bool>> datePredicate = PredicateBuilder.False<NewsPageBase>();
datePredicate = datePredicate.And(newsPage => newsPage.ArticleDate < DateTime.Now);
You can use:
Expression<Func<T, bool>> innerPredicate = PredicateBuilder.False<T>();
innerPredicate = innerPredicate.Or(item => ((DateTime)item[(ObjectIndexerKey)"article_date"] < DateTime.Now));
For the OrderBy clause, you can use:
searchQuery = searchQuery.OrderByDescending(item => item[(ObjectIndexerKey)"article_date"]);
This will work if your base class is subclassed from the Sitecore class "SearchResultItem".

Related

Unable to write Extension method to wrap a NEST client method

Basically, I was attempting to write the following extension method to avoid having to constantly write the .Suffix("keyword"). I dislike having string literals all over the place, and multiple properties of my ElasticSearch indices happen to require adding the keyword suffix to query properly.
public static class NestHelperExtensions
{
public static object UseKeywordSuffix(this object #object)
{
return #object.Suffix("keyword");
}
}
If I use the method presented above, it doesn't apply the suffix, but I'm not sure why it wouldn't work. I may be going about this the wrong way, and perhaps there is a way to add a model attribute or something to build a nestCilent.Search<T>, but when I attempted to use the KeywordAttribute, that didn't seem to work either.
Can anyone offer up an explanation as to why it wouldn't be this simple and if it should be possible, can you provide an example or workaround e.g. using attributes?
This won't work because the Suffix method is specifically handled when visiting the field expression
if (methodCall.Method.Name == nameof(SuffixExtensions.Suffix) && methodCall.Arguments.Any())
{
VisitConstantOrVariable(methodCall, _stack);
var callingMember = new ReadOnlyCollection<Expression>(
new List<Expression> { { methodCall.Arguments.First() } }
);
Visit(callingMember);
return methodCall;
}
So, an extension method like above would need to be called Suffix to begin and it would have to have at least one argument. You might think of supplying an optional parameter with a default value of "keyword" would work, but expression trees do not support this, so wouldn't work.
Another approach would be to utilise the AppendSuffix extension method on Expression<Func<T, object>> to build something; the nicest way to use this would be to pull the lambda expression out of the fluent call and into a variable
public static class NestHelperExtensions
{
public static Expression<Func<T, object>> KeywordSuffix<T>(this Expression<Func<T, object>> expression)
{
return expression.AppendSuffix("keyword");
}
}
var client = new ElasticClient();
Expression<Func<Person, object>> firstName = f => f.FirstName;
var searchResponse = client.Search<Person>(s => s
.Query(q => q
.Match(m => m
.Field(firstName.KeywordSuffix())
.Query("bar")
)
)
);
The not so nice way would be casting the lambda expression to Expression<Func<T, object>> inline
var searchResponse = client.Search<Person>(s => s
.Query(q => q
.Match(m => m
.Field(((Expression<Func<Person, object>>)(f => f.FirstName)).KeywordSuffix())
.Query("bar")
)
)
);
Another, perhaps simpler approach, would be to introduce a constant for the string "keyword", and use that in the Suffix extension method; it avoids using a string literal all over the place.

Linq query where there's a certain desired relationship between items in the result

A linq query Where clause can apply a func to an item in the original set and return a bool to include or not include the item based on the item's characteristics. Great stuff:
var q = myColl.Where(o => o.EffectiveDate = LastThursday);
But what if I want to find a set of items where each item is related to the last item in some way? Like:
var q = myColl.Where(o => o.EffectiveDate = thePreviousItem.ExpirationDate);
How do you make a Where (or other linq function) "jump out" of the current item?
Here's what I tried, trying to be clever. I made every item an array just so I can use the Aggregate function:
public IQueryable<T> CurrentVersions
{
get => AllVersions
.Select(vo => new T[] { vo })
.Aggregate((voa1, voa2) => voa1[0].BusinessExpirationDate.Value == voa2[0].BusinessEffectiveDate.Value ? voa1.Concat(voa2).ToArray() : voa1)
.SelectMany(vo => vo);
}
but that doesn't compile on the SelectMany:
The type arguments for method Enumerable.SelectMany<TSource,
TResult>(IEnumerable<TSource>, Func<TSource, IEnumerable<TResult>>)
cannot be inferred from the usage. Try specifying the type arguments
explicitly.
EDIT (SOLUTION)
As it turns out, I was on the right track, but was just confused about what SelectMany does. I didn't need it. I also needed to change IQueryable to IEnumerable because I'm using EF and you can't query after you let go of the DbContext. So, here is the actual solution.
public IEnumerable<T> CurrentVersions
{
get => AllVersions
.Select(vo => new T[] { vo })
.Aggregate((voa1, voa2) => voa1[0].BusinessExpirationDate.Value == voa2[0].BusinessEffectiveDate.Value ? voa1.Concat(voa2).ToArray() : voa1);
}
Linq queries are most effective when each item is processed in isolation. It doesn't work well when trying to relate items within the same collection, without having to process the same collection multiple times and standard linq operators.
The MoreLINQ library helps provide additional operators to fill in some of those gaps. I'm not sure what operators it provides that could be used in this instance, but I know it has a Pairwise() method that combines the current and previous items in the iteration.
In general, for situations like this, if you needed to roll out your own, it would be far easier to write it using a generator to generate your sequence. Either as a general purpose extension method:
public static IEnumerable<TSource> WhereWithPrevious<TSource>(
this IEnumerable<TSource> source,
Func<TSource, TSource, bool> predicate)
{
using (var iter = source.GetEnumerator())
{
if (!iter.MoveNext())
yield break;
var previous = iter.Current;
while (iter.MoveNext())
{
var current = iter.Current;
if (predicate(current, previous))
yield return current;
}
}
}
or one specifically for the problem you're trying to solve.
public static IEnumerable<MyType> GetVersions(IEnumerable<MyType> source)
{
using (var iter = source.GetEnumerator())
{
if (!iter.MoveNext())
yield break;
var previous = iter.Current;
while (iter.MoveNext())
{
var current = iter.Current;
if (current.EffectiveDate == previous.ExpirationDate)
yield return current;
}
}
}
An alternative approach which while standard practice in other languages but terribly inefficient here would be to zip the collection with itself offset by one.
var query = Collection.Skip(1).Zip(Collection, (c, p) => (current:c,previous:p))
.Where(x => x.current.EffectiveDate == x.previous.ExpirationDate)
...;
And with all of that said, using any of these options will most likely make your query incompatible with query providers. It's not something you would want expressed as a single query anyway.

Generic expression for where clause - "The LINQ expression node type 'Invoke' is not supported in LINQ to Entities."

I am trying to write a really generic way to load EF entities in batches, using the Contains method to generate a SQL IN statement. I've got it working if I pass the entire expression in, but when I try to build the expression dynamically, I am getting a "The LINQ expression node type 'Invoke' is not supported in LINQ to Entities." So I know this means that EF thinks I'm calling an arbitrary method and it can't translate it into SQL, but I can't figure out how to get it to understand the underlying expression.
So If I do something like this (just showing the relevant snippets):
Function declaration:
public static List<T> Load<T>(IQueryable<T> entityQuery, int[] entityIds, Func<T, int> entityKey, int batchSize = 500, Func<T, bool> postFilter = null) where T : EntityObject
{
var retList = new List<T>();
// Append a where clause to the query passed in, that will use a Contains expression, which generates a SQL IN statement. So our SQL looks something like
// WHERE [ItemTypeId] IN (1921,1920,1922)
// See http://rogeralsing.com/2009/05/21/entity-framework-4-where-entity-id-in-array/ for details
Func<int[], Expression<Func<T, bool>>> containsExpression = (entityArray => (expr => entityArray.Contains(entityKey(expr))));
// Build a new query with the current batch of IDs to retrieve and add it to the list we are returning
newQuery = entityQuery.Where<T>(containsExpression(entityIds));
retList.AddRange(newQuery.ToList());
return retList;
}
Call function:
var entities = BatchEntity.Load<ItemType>(from eItemType in dal.Context.InstanceContainer.ItemTypes
select eItemType
, itemTypeData
, (ek => ek.ItemTypeId)
);
I get "The LINQ expression node type 'Invoke' is not supported in LINQ to Entities."
But if I change it to be this:
Function declaration:
public static List<T> Load<T>(IQueryable<T> entityQuery, int[] entityIds, Func<int[], Expression<Func<T, bool>>> containsExpression, int batchSize = 500, Func<T, bool> postFilter = null) where T : EntityObject
{
var retList = new List<T>();
// Build a new query with the current batch of IDs to retrieve and add it to the list we are returning
newQuery = entityQuery.Where<T>(containsExpression(entityIds));
retList.AddRange(newQuery.ToList());
return retList;
}
Call function:
var entities = BatchEntity.Load<ItemType>(from eItemType in dal.Context.InstanceContainer.ItemTypes
select eItemType
, itemTypeData
, (entityArray => (ek => entityArray.Contains(ek.ItemTypeId)))
);
It works fine. Is there any way I can make EF understand the more generic version?
The problem, as you describe, is that the entityKey function in the first example is opaque since it is of type Func rather than Expression. However, you can get the behavior you want by implementing a Compose() method to combine two expressions. I posted the code to implement compose in this question: use Expression<Func<T,X>> in Linq contains extension.
With Compose() implemented, your function can be implemented as below:
public static List<T> Load<T>(this IQueryable<T> entityQuery,
int[] entityIds,
// note that this is an expression now
Expression<Func<T, int>> entityKey,
int batchSize = 500,
Expression<Func<T, bool>> postFilter = null)
where T : EntityObject
{
Expression<Func<int, bool>> containsExpression = id => entityIds.Contains(id);
Expression<Func<T, bool>> whereInEntityIdsExpression = containsExpression.Compose(entityKey);
IQueryable<T> filteredById = entityQuery.Where(whereInEntityIdsExpression);
// if your post filter is compilable to SQL, you might as well do the filtering
// in the database
if (postFilter != null) { filteredById = filteredById.Where(postFilter); }
// finally, pull into memory
return filteredById.ToList();
}

Dynamic LINQ OR Conditions

I'm looking to use LINQ to do multiple where conditions on a collection similar to
IEnumerable<Object> items;
items.Where(p => p.FirstName = "John");
items.Where(p => p.LastName = "Smith");
except for rather than having multiple AND conditions (as with this example), I'd like to have multiple OR conditions.
EDIT
Sorry, to clarify I don't know how many of these conditions I will have so
items.Where(p => p.FirstName = "John" || p => p.LastName = "Smith")
won't work.
Basically, here's what I'm trying to do:
foreach(var name in names)
{
items = items.Where(p => p.Name == name);
}
Use PredicateBuilder:
Suppose you want to write a LINQ to SQL or Entity Framework query that implements a keyword-style search. In other words, a query that returns rows whose description contains some or all of a given set of keywords...
The ideal approach is to dynamically construct a lambda expression tree that performs an or-based predicate.
Of all the things that will drive you to manually constructing expression trees, the need for dynamic predicates is the most common in a typical business application. Fortunately, it’s possible to write a set of simple and reusable extension methods that radically simplify this task. This is the role of our PredicateBuilder class...
It sounds like your whitelist of names is only known at runtime. Perhaps try this:
string[] names = new string[] {"John", "foo", "bar"};
var matching = items.Where(x => names.Contains(x.Name));
You can use .Union() to return results that satisfy any condition.
var results = items.Where(p => p.FirstName == "John")
.Union(items.Where(p => p.LastName == "Smith"));
This is inferior to using the || operator. It isn't clear from your edit why that wouldn't work.
public static Expression<Func<T, bool>> OrTheseFiltersTogether<T>(
this IEnumerable<Expression<Func<T, bool>>> filters)
{
Expression<Func<T, bool>> firstFilter = filters.FirstOrDefault();
if (firstFilter == null)
{
Expression<Func<T, bool>> alwaysTrue = x => true;
return alwaysTrue;
}
var body = firstFilter.Body;
var param = firstFilter.Parameters.ToArray();
foreach (var nextFilter in filters.Skip(1))
{
var nextBody = Expression.Invoke(nextFilter, param);
body = Expression.OrElse(body, nextBody);
}
Expression<Func<T, bool>> result = Expression.Lambda<Func<T, bool>>(body, param);
return result;
}
Then, later:
List<Expression<Func<Person, bool>>> filters = names
.Select<string, Expression<Func<Person, bool>>>(name =>
p => p.Name == name
).ToList();
Expression<Func<Person, bool>> filterOfOrs = filters.OrTheseFiltersTogether();
query = query.Where<Person>(filterOfOrs);
You can't make the Where clause dynamic, but you can dynamically create the Lambda Expression you pass to it. Create the right Expression, compile it and pass the resulting lambda expression as a parameter to the Where clause.
EDIT:
Okay, seems like you can skip the part where you have to manually create the Expression and can use PredicateBuilder for it, as already answered by AS-CII.

Generics and Database Access

I have the following method I can pass in a lambda expression to filter my result and then a callback method that will work on the list of results. This is just one particular table in my system, I will use this construct over and over. How can I build out a generic method, say DBget that takes a Table as a parameter(An ADO.NET dataservice entity to be fair) and pass in a filter (a lambda experssion).
public void getServiceDevelopmentPlan(Expression<Func<tblServiceDevelopmentPlan, bool>> filter, Action<List<tblServiceDevelopmentPlan>> callback)
{
var query = from employerSector in sdContext.tblServiceDevelopmentPlan.Where(filter)
select employerSector;
var DSQuery = (DataServiceQuery<tblServiceDevelopmentPlan>)query;
DSQuery.BeginExecute(result =>
{
callback(DSQuery.EndExecute(result).ToList<tblServiceDevelopmentPlan>());
}, null);
}
My first bash at this is:
public delegate Action<List<Table>> DBAccess<Table>(Expression<Func<Table, bool>> filter);
If you are using Linq to Ado.NET Dataservices or WCF Dataservices, your model will build you a lot of typed. Generally though you will be selecting and filtering. You need the following, then all your methods are just candy over the top of this:
Query Type 1 - One Filter, returns a list:
public void makeQuery<T>(string entity, Expression<Func<T, bool>> filter, Action<List<T>> callback)
{
IQueryable<T> query = plussContext.CreateQuery<T>(entity).Where(filter);
var DSQuery = (DataServiceQuery<T>)query;
DSQuery.BeginExecute(result =>
{
callback(DSQuery.EndExecute(result).ToList<T>());
}, null);
}
Query Type 2 - One Filter, returns a single entity:
public void makeQuery(string entity, Expression> filter, Action callback)
{
IQueryable<T> query = plussContext.CreateQuery<T>(entity).Where(filter);
var DSQuery = (DataServiceQuery<T>)query;
DSQuery.BeginExecute(result =>
{
callback(DSQuery.EndExecute(result).First<T>());
}, null);
}
What you need to do is overload these and swap out the filter for a simple array of filters
Expression<Func<T, bool>>[] filter
And repeat for single and list returns.
Bundle this into a singleton if you want one datacontext, or keep track of an array of contexts in some sort of hybrid factory/singleton and you are away. Let the constructor take a context or if non are supplied then use its own and you are away.
I then use this on a big line but all in one place:
GenericQuery.Instance.Create().makeQuery<tblAgencyBranches>("tblAgencyBranches", f => f.tblAgencies.agencyID == _agency.agencyID, res => { AgenciesBranch.ItemsSource = res; });
This may look complicated but it hides a lot of async magic, and in certain instances can be called straight from the button handlers. Not so much a 3 tier system, but a huge time saver.

Resources