Elasticsearch NEST - filter by multiple fields - elasticsearch

I have a query and want to do a search by multiple fields in case if no result by uuid need to do the same by parentUuid:
C# query:
return new NestedQuery
{
Path = Infer.Field<ElasticDocument>(t => t.KeywordFields),
Query = new BoolQuery
{
Filter = new List<QueryContainer>
{
new TermsQuery
{
Field = Infer.Field<ElasticDocument>(t => t.KeywordFields),
Terms = new List<string>
{
"term2"
},
Boost = 10
}
},
Should = new List<QueryContainer>
{
new TermQuery
{
Field = Infer.Field<ElasticDocument>(t => t.KeywordFields.First().UUID),
Value = filter.UUID
},
new TermQuery
{
Field = Infer.Field<ElasticDocument>(t => t.KeywordFields.First().ParentUUID),
Value = filter.UUID
},
}
}
};
example of documents:
"keywordFields": [
{
"value": "term1",
"uuid": "bf18ee9f-7592-488d-7985-2b9fe8b878ca",
"parentUUID": null
},
{
"value": "term2",
"uuid": "079205ed-30df-08f6-02a1-9caf093c3be0",
"parentUUID": "103d6061-cb99-4fba-8118-2ea501e4425d"
}
]
How to update the query to use or condition in case if no result by 'UUID' then let's do it by ParentUUID?

Try to combine bool query:
nested1 = new NestedQuery {
Path = Infer.Field<ElasticDocument>(t => t.KeywordFields),
Query = new BoolQuery {
Filter = new List<QueryContainer>
{
new TermsQuery {
Field = Infer.Field<ElasticDocument>(t => t.KeywordFields),
Terms = new List<string>
{
"term2"
},
Boost = 10
}
}
}
};
nested2 = new NestedQuery {
Path = Infer.Field<ElasticDocument>(t => t.KeywordFields),
Query = new BoolQuery {
Filter = new List<QueryContainer>
{
new TermsQuery {
Field = Infer.Field<ElasticDocument>(t => t.KeywordFields),
Terms = new List<string>
{
"term2"
},
Boost = 10
}
},
Should = new List<QueryContainer>
{
new BoolQuery {
MustNot = new List<QueryContainer> {
new TermQuery
{
Field = Infer.Field<ElasticDocument>(t => t.KeywordFields.First().UUID),
Value = filter.UUID
}
},
Must = new List<QueryContainer> {
new TermQuery
{
Field = Infer.Field<ElasticDocument>(t => t.KeywordFields.First().parentUUID),
Value = filter.UUID
}
}
}
}
}
};
And should nested1 with nested2
note: not IDE for this edit, the nested2 query could be simplify for sure.

Related

Build multiple filter in Elasticsearch

Can you suggest how do I build a query based on multiple filter.
Currently, I want to implement a search functionality using the following filters:
ProductName
Countries (array)
Cities (array)
The requirements is that, when counties or cities has no selected value, the query assumes that you are searching all countries and cities.
If there are selected counties and cities, then the result should be base on the selected counties and cities.
I have the query below to start with.
static void Main(string[] args)
{
var uri = new Uri("http://localhost:9200");
var connectionPool = new SingleNodeConnectionPool(uri);
var settings = new ConnectionSettings(connectionPool);
var client = new ElasticClient(settings);
if (counties.Count > 0)
{
foreach(string country in counties)
{
// Add a query to be added in client.Search
}
}
if (cities.Count > 0)
{
foreach(string city in cities)
{
// Add a query to be added in client.Search
}
}
client.Search<Product>(s => s
.Query(q => q
.Bool(b => b
.Must(mu => mu
.Match(m => m
.Field(f => f.ProductName)
.Query("some text")
),
.
)
)
)
);
}
I just answer my own question. Currently, I came with this approach.
var sd = new SearchDescriptor<object>();
var qc = new QueryContainer();
var qd = new QueryContainerDescriptor<object>();
sd.From(0);
sd.Size(100);
sd.Index("Products");
sd.Type("Product");
if (!string.IsNullOrEmpty(title))
{
qc = qd.Match(m1 => m1
.Field("title")
.Query(title)
);
}
if (countries.Count > 0)
{
qc = qd.Terms(t => t
.Field("country")
.Terms(countries.ToArray())
);
}
if (cities.Count > 0)
{
qc = qd.Terms(t => t
.Field("city")
.Terms(cities.ToArray())
);
}
sd.Query(q => q
.Bool(b => b
.Must(qc)
)
);
var result = client.Search<object>(s => s = sd);
You can use the terms query:
{
"query": {
"bool": {
"must": [
{ "match": { "field": "productName", "query": "some text"}},
// Only add this filter if the array is not empty
{ "terms": { "Country": ["Canada", "France"]}},
// Same here
{ "terms": { "City": ["Ottawa", "Paris"]}},
]
}
}
}

nest: build a QueryContainer request with must and a should constraint

I try to build an elasticsearch request with a must and a should request
my target request is this bellow:
POST /index/type/_search
{
"query": {
"bool" : {
"must" : {
"match":{"field1":{"query":"word1"}}
},
"should":{
"match":{"field2":{"query":"word2"}}
}
}
}}
I wrote:
var queryContainer = new QueryContainer();
queryContainer &= new MatchQuery() { Field = "field1", Query = "word1" };
queryContainer |= new MatchQuery() { Field = "field2", Query = "word2"};
var searchRequest = new SearchRequest<type>
{
Query = queryContainer
};
but it produces me :
POST /index/type/_search
{
"query":{
"bool":{
"should":[
{"match":{"field1":{"query":"word1"}}},
{"match":{"field2":{"query":"word2"}}}]
}
}
}
Do you know what should I write when building my QueryContainer?
Using operator overloading here is not that helpful; I think in this case, constructing the bool query is clearer
Object Initializer Syntax
var searchRequest = new SearchRequest<Document>()
{
Query = new BoolQuery
{
Must = new QueryContainer[] { new MatchQuery() { Field = "field1", Query = "word1" } },
Should = new QueryContainer[] { new MatchQuery() { Field = "field2", Query = "word2" } }
}
};
client.Search<Document>(searchRequest);
or Fluent API
client.Search<Document>(s => s
.Query(q => q
.Bool(b => b
.Must(fi => fi
.Match(m => m
.Field("field2")
.Query("word2")
)
)
.Should(sh => sh
.Match(m => m
.Field("field1")
.Query("word1")
)
)
)
)
);

Nest MultiMatch Field Boost

I'm trying to boost some fields over others in a multiMatch search.
Looking at the docs I see you can create a Field with boost by doing this
var titleField = Infer.Field<Page>(p => p.Title, 2);
I haven't been able to figure out how that translates to Fields though.
Something like this isn't right
var bodyField = Infer.Field<Page>(p => p.Body);
var titleField = Infer.Field<Page>(p => p.Title, 2);
var metaDescriptionField = Infer.Field<Page>(p => p.MetaDescription, 1.5);
var metaKeywordsField = Infer.Field<Page>(p => p.Keywords, 2);
MultiMatchQuery multiMatchQuery = new MultiMatchQuery()
{
Fields = Infer.Fields<Page>(bodyField, titleField, metaDescriptionField, metaKeywordsField),
Query = search.Term
};
Do I need to use the string names for the fields like
var titleFieldString = "Title^2";
and pass those into Infer.Fields
You can use the strongly typed Infer.Field<T>(); there is an implicit conversion from Field to Fields, and additional fields can be added with .And(). Here's an example
void Main()
{
var client = new ElasticClient();
Fields bodyField = Infer.Field<Page>(p => p.Body);
var titleField = Infer.Field<Page>(p => p.Title, 2);
var metaDescriptionField = Infer.Field<Page>(p => p.MetaDescription, 1.5);
var metaKeywordsField = Infer.Field<Page>(p => p.Keywords, 2);
var searchRequest = new SearchRequest<Page>()
{
Query = new MultiMatchQuery()
{
Fields = bodyField
.And(titleField)
.And(metaDescriptionField)
.And(metaKeywordsField),
Query = "multi match search term"
}
};
client.Search<Page>(searchRequest);
}
public class Page
{
public string Body { get; set; }
public string Title { get; set; }
public string MetaDescription { get; set; }
public string Keywords { get; set; }
}
this yields
{
"query": {
"multi_match": {
"query": "multi match search term",
"fields": [
"body",
"title^2",
"metaDescription^1.5",
"keywords^2"
]
}
}
}
You can also pass an array of Field which also implicitly converts to Fields
var searchRequest = new SearchRequest<Page>()
{
Query = new MultiMatchQuery()
{
Fields = new[] {
bodyField,
titleField,
metaDescriptionField,
metaKeywordsField
},
Query = "multi match search term"
}
};
As well as pass an array of strings
var searchRequest = new SearchRequest<Page>()
{
Query = new MultiMatchQuery()
{
Fields = new[] {
"body",
"title^2",
"metaDescription^1.5",
"keywords^2"
},
Query = "multi match search term"
}
};

Convert mongo query to Spring

How do I write the following mongoDB query in Spring?
db.reportData.aggregate([
{ $match : { iecode : {$in:["P110017","P111111"]} } } ,
{
"$group": {
"_id": "$iecode",
"treatmentArms": { "$first": "$evaluationDTOList" }
}
},
{ "$unwind": "$treatmentArms" },
{
"$group": {
"_id": null,
"Package": {
"$sum": {
"$cond": [
{ "$eq": [ "$treatmentArms.mechanismOrPkg", "Package" ] },
1, 0
]
}
},
"Behavioral-bias related mechanisms":{
"$sum": {
"$cond": [
{
"$and": [
{ "$eq": [ "$treatmentArms.mechanismOrPkg", "Mechanism" ] },
{ "$eq": [ "$treatmentArms.mechanismTested1", "Behavioral-bias related mechanisms" ] }
]
},
1,
0 ]
}
}
}
}
])
My Database name : MyProjects
Collection name : reportData
Please note that I am using spring MongoTemplate.
Edit
Here is my code which I tried
#SuppressWarnings("deprecation")
public void getIEQuestions(Map<String, List<String>> paramMap){
System.out.println("get ie questions-------------------------------------------");
DBCollection projects = getMongoOperations().getCollection("reportData");
BasicDBObject andQuery = new BasicDBObject();
List<BasicDBObject> filters = new ArrayList<BasicDBObject>();
for (Map.Entry<String, List<String>> entry : paramMap.entrySet()) {
ArrayList<String> vals = new ArrayList<String>();
boolean flag = false;
for (int i = 0; i < entry.getValue().size(); i++) {
if (entry.getValue().get(i) != null && entry.getValue().get(i) != "") {
System.out.println("entry.getValue().get(i): " + entry.getValue().get(i));
vals.add(entry.getValue().get(i));
flag = true;
}
}
if (flag) {
filters.add(new BasicDBObject(entry.getKey(), new BasicDBObject("$in", vals)));
}
}
andQuery.put("$and", filters);
//adding all the filters I want to apply
DBObject match = new BasicDBObject("$match", andQuery);
DBObject groupByIECode = new BasicDBObject("$group",
new BasicDBObject("_id","$iecode"))
.append("treatmentArms",new BasicDBObject("$first","$evaluationDTOList"));
DBObject unwind = new BasicDBObject("$unwind","$treatmentArms");
DBObject finalCalculation = new BasicDBObject("$group",new BasicDBObject("_id",null))
.append(
"Package", new BasicDBObject(
"$sum", new BasicDBObject(
"$cond", new Object[]{
new BasicDBObject(
"$eq", new Object[]{ "$treatmentArms.mechanismOrPkg", "Package"}
),
1,
0
}
)
)
);
final AggregationOutput output = projects.aggregate(match, groupByIECode, unwind, finalCalculation);
for (DBObject result1 : output.results()) {
String totalPackage = (String) result1.get("Package");
System.out.println("Total package: "+totalPackage);
}
}
It is giving me duplicateMongoKey exception.
Later I find out $cond operation is not yet supported by spring mongotemplate from here

Convert Lambda Expression into an Expression tree

I have that lambda:
var Ids = profileExample.CostCenters
.Where(CostCentre => CostCentre != null)
.Select(CostCentre => CostCentre.Id);
Then i convert to that expression tree
static IEnumerable<Int64> AboveLambdaConvertedToExpressionTree(Profile profileExample)
{
//Begin var Ids = profileExample.CostCenters.Where(CostCentre => CostCentre != null).Select(CostCentre => CostCentre.Id);
var property = profileExample.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance).Where(x => x.Name != "Id").First();
var collection = ((IEnumerable)property.GetValue(profileExample, null)).AsQueryable();
var collectionType = property.PropertyType.GetGenericArguments()[0];
var collectionTypeName = collectionType.Name;
var keyType = typeof(Int64);
var keyName = "Id";
//BeginWhere
var parameter = Expression.Parameter(collectionType, collectionTypeName);
var profileExampleWhere = Expression.Lambda(
Expression.NotEqual(parameter, Expression.Constant(null)),
parameter);
var profileExampleWhereCall = Expression.Call(typeof(Enumerable),
"Where",
new Type[] { collectionType },
collection.Expression,
profileExampleWhere);
//EndWhere
//BeginSelect
var profileExampleSelect = Expression.Lambda(Expression.PropertyOrField(parameter, keyName),
parameter);
var profileExampleSelectCall = Expression.Call(typeof(Enumerable),
"Select",
new Type[] { collectionType, keyType },
profileExampleWhereCall,
profileExampleSelect);
var Ids = Expression.Lambda(profileExampleSelectCall).Compile().DynamicInvoke();
//EndSelect
//End var Ids = profileExample.CostCenters.Where(CostCentre => CostCentre != null).Select(CostCentre => CostCentre.Id);
return ((IEnumerable)Ids).Cast<Int64>();
}
Now i want to do the same with bellow lambda
var result = Set.AsQueryable()
.Where(Profile => Profile.CostCenters.Select(CostCentre => CostCentre.Id)
.Any(Id => Ids.Contains(Id))).ToList();
But i stuck in .Any(Id => Ids.Contains(Id))....
var id = Expression.Parameter(typeof(long), "Id");
var costCentre = Expression.Parameter(typeof(CostCentre), "CostCentre");
var profile = Expression.Parameter(typeof(Profile), "Profile");
var selectLambda = Expression.Lambda(Expression.PropertyOrField(costCentre, "Id"), costCentre);
var selectCall = Expression.Call(typeof(Enumerable),
"Select",
new Type[] { typeof(CostCentre), typeof(long) },
Expression.PropertyOrField(profile, "CostCenters"),
selectLambda);
How can i call Any from selectCall and call Ids.Contains...
Full code to run as console application bellow:
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
namespace ExpressionTrees
{
class Program
{
static void Main(string[] args)
{
var Ids = profileExample.CostCenters.Where(CostCentre => CostCentre != null).Select(CostCentre => CostCentre.Id);
Ids = AboveLambdaConvertedToExpressionTree(profileExample);
var result = Set.AsQueryable().Where(Profile => Profile.CostCenters.Select(CostCentre => CostCentre.Id).Any(Id => Ids.Contains(Id))).ToList();
//Expression<Func<Profile, bool>> lambda = (Profile) => Profile.CostCenters.Select(CostCentre => CostCentre.Id).Any(Id => Ids.Contains(Id));
var id = Expression.Parameter(typeof(long), "Id");
var costCentre = Expression.Parameter(typeof(CostCentre), "CostCentre");
var profile = Expression.Parameter(typeof(Profile), "Profile");
var selectLambda = Expression.Lambda(Expression.PropertyOrField(costCentre, "Id"), costCentre);
var selectCall = Expression.Call(typeof(Enumerable),
"Select",
new Type[] { typeof(CostCentre), typeof(long) },
Expression.PropertyOrField(profile, "CostCenters"),
selectLambda);
}
static IEnumerable<Int64> AboveLambdaConvertedToExpressionTree(Profile profileExample)
{
// I show that as example of what i need to do
var keyType = typeof(Int64);
var keyName = "Id";
//Begin var Ids = profileExample.CostCenters.Where(CostCentre => CostCentre != null).Select(CostCentre => CostCentre.Id);
var property = profileExample.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance).Where(x => x.Name != keyName).First();
var collection = ((IEnumerable)property.GetValue(profileExample, null)).AsQueryable();
var collectionType = property.PropertyType.GetGenericArguments()[0];
var collectionTypeName = collectionType.Name;
//BeginWhere
var parameter = Expression.Parameter(collectionType, collectionTypeName);
var profileExampleWhere = Expression.Lambda(
Expression.NotEqual(parameter, Expression.Constant(null)),
parameter);
var profileExampleWhereCall = Expression.Call(typeof(Enumerable),
"Where",
new Type[] { collectionType },
collection.Expression,
profileExampleWhere);
//EndWhere
//BeginSelect
var profileExampleSelect = Expression.Lambda(Expression.PropertyOrField(parameter, keyName),
parameter);
var profileExampleSelectCall = Expression.Call(typeof(Enumerable),
"Select",
new Type[] { collectionType, keyType },
profileExampleWhereCall,
profileExampleSelect);
var Ids = Expression.Lambda(profileExampleSelectCall).Compile().DynamicInvoke();
//EndSelect
//End var Ids = profileExample.CostCenters.Where(CostCentre => CostCentre != null).Select(CostCentre => CostCentre.Id);
return ((IEnumerable)Ids).Cast<Int64>();
}
public partial class Profile
{
public virtual Int64 Id { get; set; }
public virtual ICollection<CostCentre> CostCenters { get; set; }
}
public partial class CostCentre
{
public virtual Int64 Id { get; set; }
}
public static Profile profileExample
{
get
{
return new Profile()
{
Id = 1,
CostCenters = new List<CostCentre>() { new CostCentre() { Id = 2 } }
};
}
}
public static IList<Profile> Set
{
get
{
return new List<Profile>() { new Profile() { Id = 1,
CostCenters = new List<CostCentre>() { new CostCentre() { Id = 1 },
new CostCentre() { Id = 2 } }
},
new Profile() { Id = 2,
CostCenters = new List<CostCentre>() { new CostCentre() { Id = 2 },
new CostCentre() { Id = 3 } }
},
new Profile() { Id = 3,
CostCenters = new List<CostCentre>() { new CostCentre() { Id = 3 } }
} };
}
}
}
}
Since Any is a Generic Method you need to create it for a specific type. The method below gets the Any<T> method from the Enumerable type.
public static MethodInfo GetAnyExtensionMethod(Type forType)
{
MethodInfo method =
typeof(Enumerable).GetMethods()
.First(m => m.Name.Equals("Any") &&
m.GetParameters().Count() == 2);
return method.MakeGenericMethod(new[] { forType });
}
Its solved with help of Mads from MS
class Program
{
static void Main(string[] args)
{
//var Ids = profileExample.CostCenters.Where(CostCentre => CostCentre != null).Select(CostCentre => CostCentre.Id);
var Ids = AboveLambdaConvertedToExpressionTree(profileExample);
//var result = Set.AsQueryable().Where(Profile => Profile.CostCenters.Select(CostCentre => CostCentre.Id).Any(Id => Ids.Contains(Id))).ToList();
var id = Expression.Parameter(typeof(long), "Id");
var costCentre = Expression.Parameter(typeof(CostCentre), "CostCentre");
var profile = Expression.Parameter(typeof(Profile), "Profile");
var selectLambda = Expression.Lambda(Expression.PropertyOrField(costCentre, "Id"), costCentre);
var selectCall = Expression.Call(typeof(Enumerable),
"Select",
new Type[] { typeof(CostCentre), typeof(long) },
Expression.PropertyOrField(profile, "CostCenters"),
selectLambda);
//var id2 = Expression.Parameter(typeof(long), "Id");
var containsCall = Expression.Call(typeof(Enumerable),
"Contains",
new Type[] { typeof(long) },
Expression.Constant(Ids),
id);
var anyLambda = Expression.Lambda(containsCall, id);
var anyCall = Expression.Call(typeof(Enumerable),
"Any",
new Type[] { typeof(long) },
selectCall,
anyLambda);
var whereLambda = Expression.Lambda(anyCall, profile);
var callExpression = Expression.Call(typeof(Queryable),
"Where",
new Type[] { typeof(Profile) },
Set.AsQueryable().Expression,
whereLambda);
var result = Expression.Lambda(callExpression).Compile().DynamicInvoke();
}

Resources