Elastic search recreating query in NEST not working - elasticsearch

Im trying to recreate query in from Kibana dev into NEST but it's not giving me the same results.
The query I run in Kibana works perfectly returning 1 result
Here is my Kibana query:
GET /cats/_doc/_search
{
"query":{
"bool" : {
"minimum_should_match" :3,
"should": [
{"term" : { "name" : "cats" }},
{"term" : { "name" : "are" }},
{"term" : { "name" : "craze" }}
]
}
}
}
When I create the query in NEST it returns no results except when I change the minimum_should_match to 1 (it then returns 2 results )
Here is my NEST query:
string[] tmp = "Cats are craze".ToLower().Split(new string[] { " " }, StringSplitOptions.None);
var cats = ElasticMain.Search<dynamic>(s => s.From(from).Size(20).Query(
q => q.Bool(
b => b.MinimumShouldMatch(tmp.Length).Should(
l => l.Terms(
t => t.Name("name").Field("name").Terms(tmp)))
)));
What am I doing wrong?

You are not building the same query in NEST as you have in Kibana; the former is using a terms query, whilst the latter is using three term queries in a bool query should clause. The semantics of these two queries in combination with minimum should match is different.
The same query in NEST is
var client = new ElasticClient();
string[] tmp = "Cats are craze".ToLower().Split(new string[] { " " }, StringSplitOptions.None);
var from = 0;
var searchResponse = client.Search<dynamic>(s => s
.From(from)
.Size(20)
.Query(q => q
.Bool(b =>
{
b.MinimumShouldMatch(tmp.Length);
var shouldQueries =
new List<Func<QueryContainerDescriptor<dynamic>, QueryContainer>>(tmp.Length);
for (var i = 0; i < tmp.Length; i++)
{
var value = tmp[i];
shouldQueries.Add(qc => qc.Term(t => t
.Field("name")
.Value(value)
));
}
b.Should(shouldQueries);
return b;
})
)
);
which builds the following query
{
"from": 0,
"query": {
"bool": {
"minimum_should_match": 3,
"should": [
{
"term": {
"name": {
"value": "cats"
}
}
},
{
"term": {
"name": {
"value": "are"
}
}
},
{
"term": {
"name": {
"value": "craze"
}
}
}
]
}
},
"size": 20
}
When the number of should clauses that must match is equal to minimum_should_match as in this example, it's effectively the same as saying that they are all must clauses (without the minimum_should_match)
var client = new ElasticClient();
string[] tmp = "Cats are craze".ToLower().Split(new string[] { " " }, StringSplitOptions.None);
var from = 0;
var searchResponse = client.Search<dynamic>(s => s
.From(from)
.Size(20)
.Query(q =>
tmp.Aggregate((QueryContainer)null, (qc, v) => qc && q.Term("name", v))
)
);
which takes advantage of operator overloading on NEST queries to && them together, to build the query
{
"from": 0,
"query": {
"bool": {
"must": [
{
"term": {
"name": {
"value": "cats"
}
}
},
{
"term": {
"name": {
"value": "are"
}
}
},
{
"term": {
"name": {
"value": "craze"
}
}
}
]
}
},
"size": 20
}

Related

Nest multiple terms in DSL syntax

I'm trying to convert following query into NEST DSL syntax:
{
"aggs": {
"color": {
"terms": {
"field": "Color"
}
},
"weight": {
"terms": {
"field": "Weight"
}
}
}
}
Each of the terms is coming from the list which holds name and fieldId. So far I've managed to do this:
var request2 = _client.Search<T>(s => s
.Aggregations(aggs =>
aggs.Terms("Weight", x => x.Field("Weight")).Terms("Color", x => x.Field("Weight"))));
Which works as expected however I need to be able to supply parameters Weight and Color dynamically as the method can be called with different set of parameters. Is there a way how to use something like:
aggs.Terms(x => x.field( myList.foreach(y.value))));
I guess this would work better with the Object initializer syntax, however I would rather get this working inside dsl.
Something like the following would work
var client = new ElasticClient();
var fields = new List<string>
{
"color",
"weight",
"foo",
"bar"
};
var response = client.Search<object>(s => s
.Aggregations(aggs =>
{
foreach (var field in fields)
{
aggs.Terms(field, t => t.Field(field));
}
return aggs;
})
);
which generates the following request
{
"aggs": {
"color": {
"terms": {
"field": "color"
}
},
"weight": {
"terms": {
"field": "weight"
}
},
"foo": {
"terms": {
"field": "foo"
}
},
"bar": {
"terms": {
"field": "bar"
}
}
}
}

Multi-term filter in ElasticSearch (NEST)

I am trying to query documents based on a given field having multiple possible values. For example, my documents have an "extension" property which is the extension type of a file like .docx, xls, .pdf, etc. I want to be able to filter my "extensions" property on any number of values, but cannot find the correct syntax needed to get this functionality. Here is my current query:
desc.Type("entity")
.Routing(serviceId)
.From(pageSize * pageOffset)
.Size(pageSize)
.Query(q => q
.Filtered(f => f
.Query(qq =>
qq.MultiMatch(m => m
.Query(query)
.OnFields(_searchFields)) ||
qq.Prefix(p1 => p1
.OnField("entityName")
.Value(query)) ||
qq.Prefix(p2 => p2
.OnField("friendlyUrl")
.Value(query))
)
.Filter(ff =>
ff.Term("serviceId", serviceId) &&
ff.Term("subscriptionId", subscriptionId) &&
ff.Term("subscriptionType", subscriptionType) &&
ff.Term("entityType", entityType)
)
)
);
P.S. It may be easier to think of it in the inverse, where I send up the file extensions I DON'T want and set up the query to get documents that DON'T have any of the extension values given.
After discussion, this should be a raw json query, that should work and can be translated to NEST quite easily:
POST /test/_search
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"term": {
"serviceId": "VALUE"
}
},
{
"term": {
"subscriptionId": "VALUE"
}
},
{
"term": {
"subscriptionType": "VALUE"
}
},
{
"term": {
"entityType": "VALUE"
}
}
],
"must_not": [
{
"terms": {
"extension": [
"docx",
"doc"
]
}
}
]
}
}
}
}
}
What had to be done:
In order to have clauses that have to exist and the ones, that need to be filtered out, bool query suited best.
Must query stores all clauses that are present in OPs query
Must_not query should store all extensions that need to be filtered out
If you want to return items that match ".doc" OR ".xls" then you want a TERMS query. Here is a sample:
var searchResult = ElasticClient
.Search<SomeESType>(s => s
.Query(q => q
.Filtered(fq => fq
.Filter(f => f
.Terms(t => t.Field123, new List<string> {".doc", ".xls"})
)
)
)
)

Elasticsearch - Term null?

I'm having trouble matching null terms in NEST.
I'm trying to get some details via a query.
It works ok, but one thing I can't understand -
For some reason I can't do a term that equals to null value.
What am I doing wrong?
My Code:
result = _mainManager.Client.Search<object>
(q => q
.Type("Mail")
.Query(c =>
c.Term("SentMail_Sender_Id", userId) &&
c.Term("SentMail_EmbedAccountId", null) &&
!c.Term("SentMail_Status", Status.REMOVED.ToString().ToLower()) &&
c.Range(v => v.OnField("SentMail_Upload_Files_Count").Greater(0)))
.Size(int.MaxValue)
.Sort(s => s.OnField("SentMail_Creation_Date").Descending()));
It works ok, no null term is found in my result Json:
{
"size": 2147483647,
"sort": [
{
"SentMail_Creation_Date": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"term": {
"SentMail_Sender_Id": {
"value": 7186
}
}
},
{
"range": {
"SentMail_Upload_Files_Count": {
"gt": "0"
}
}
}
],
"must_not": [
{
"term": {
"SentMail_Status": {
"value": "removed"
}
}
}
]
}
}
}
Found it!
result = _mainManager.Client.Search<object>
(q => q
.Type("Mail")
.Query(c =>
c.Term("SentMail_Sender_Id", userId) &&
!c.Term("SentMail_Status", Status.REMOVED.ToString().ToLower()) &&
c.Range(v => v.OnField("SentMail_Upload_Files_Count").Greater(0)))
.Filter(f => f.Missing("SentMail_EmbedAccountId"))
.Size(int.MaxValue)
.Sort(s => s.OnField("SentMail_Creation_Date").Descending()));

Elastic Search - Creating Query with NEST

Given the query:
GET places/_search
{
"query": {
"match_all": {}
},
"size": 0,
"aggs": {
"filtered_cells": {
"filter": {
"geo_bounding_box": {
"loc": {
"top_left": {
"lat": 53.480950,
"lon": -2.2374300
},
"bottom_right": {
"lat": 51.5085300,
"lon": -0.1257400
}
}
}
},
"aggs": {
"cells": {
"geohash_grid": {
"field": "loc",
"precision": "precision"
},
"aggs": {
"center_lat": {
"avg": {
"script": "doc['loc'].lat"
}
},
"center_lon": {
"avg": {
"script": "doc['loc'].lon"
}
}
}
}
}
}
}
}
I am trying to move this query into a piece of code I am developing, but I am having trouble structuring the query above using the NEST library.
Here is what I have so far:
var s = new SearchDescriptor<FacebookGraphResponse>()
.MatchAll()
.From(0)
.Size(10)
.Filter(filter => filter
.GeoBoundingBox(f => f.Loc, 53.480950, -2.2374300, 51.5085300, -0.1257400)
);
Is anyone able to provide any input on how I can translate this query to nest, as I am currently stumped.
Many Thanks
Solved it!
var s = new SearchDescriptor<FacebookGraphResponse>()
.MatchAll()
.Size(0)
.Aggregations(aggs => aggs
.Filter("filtered_cells", f => f
.Filter(f1 => f1
.GeoBoundingBox(bb => bb.Loc, -2.2374300, 53.480950, -0.1257400, 51.5085300))
.Aggregations(a => a
.GeoHash("cells", geo => geo
.GeoHashPrecision(GeoHashPrecision.Precision10)
.Field("loc")
.Aggregations(a1 => a1
.Average("center_lat", x => x.Script("doc['loc'].lat"))
.Average("center_lon", x => x.Script("doc['loc'].lon")))))));

ElasticSearch .NET Nest Sub Aggregation

Consider the following aggregation and sub aggregation:
.Aggregations(a => a
.Terms("ByCustomerByImpressionCount", tad => tad
.Field(o => o.CustomerName)
.OrderDescending("sumImpressionCount")
.Aggregations(ad => ad
.Sum("sumImpressionCount", sad => sad
.Field(o => o.ImpressionCount)
)
)
))
The query works fine, and I get results.
I can access the top aggregation with:
var agg = results.Aggs.Terms("ByCustomerByImpressionCount");
According to the docs, I should be able to get the sub agg with:
var subAgg = agg.Aggs.Sum("sumImpressionCount");
But Aggs isn't even a property of agg - any ideas?
EDIT:
This is the only way I've been able to get to the sub aggregation:
PopulateList(results.Aggs.Terms("CustomersByImpressionCount").Items, CustomersByImpressionCount);
private void PopulateList(IList<KeyItem> items, List<DataContainer> listToPopulate)
{
items.ForEach(item =>
{
listToPopulate
.Add(new DataContainer() { Name = item.Key, Sum = ((ValueMetric)item.Aggregations.First().Value).Value });
});
}
...but this seems to be overly complicated. Here is the ES query:
"query": {
"bool": {
"must": [
{ "range": { "requestedShipDate": {
"gte": "2014-11-26",
"lte": "2015-02-03"
} }
}
],
"must_not": [
{"terms": {
"status": [
"Order Shipped",
"Canceled",
"Completed"
]
}}
]
}
},
"aggs": {
"ByCustomerByImpressionCount": {
"terms": {
"field": "customerName",
"order": { "sumImpressionCount": "desc" }
},
"aggs": {
"sumImpressionCount": { "sum": { "field": "impressionCount" }}
}
}}
And here is partial response:
"ByCustomerByImpressionCount": {
"doc_count_error_upper_bound": -1,
"sum_other_doc_count": 26785,
"buckets": [
{
"key": "<One of our customers>",
"doc_count": 1722,
"sumImpressionCount": {
"value": 9583176
}
}
The documentation is wrong. We'll fix that ASAP.
Here is the correct way to access your aggregations:
var terms = results.Aggs.Terms("ByCustomerByImpressionCount");
foreach(var term in terms.Items)
{
var sum = term.Sum("sumImpressionCount");
}

Resources