Elasticsearch/Nest - Combining multiple range queries (OIC syntax) - elasticsearch

ES newbie here (transitioning over from the SolrNet world, so I am looking for pointers with a similar syntax)
Can anyone help with formulating a query that might look something like the code snippet below without the lambdas. Unfortunately there are very few examples in the plain old OI syntax and I have been stuck....
How would I formulate a set of boolean range queries, in solrnet I would do something like this to obtain items that fall in a certain price band:
List<ISolrQuery> queryList = new List<ISolrQuery>();
double[] priceList = GetPrices(..) //double array
for (int i = 1; i <= 10; i++)
{
queryList.Add(new solr.Query(new SolrQueryByRange<double>("price", priceList[i] * 0.95, priceList[i] * 1.15));
}
var results = _solr.Query(new SolrMultipleCriteriaQuery(queryList), new QueryOptions
{
Rows = 50,
Fields = new[] { "Item", "Created", "Price" },
});
Could anyone help with equivalent Elasticsearch/Nest query?

Something like the following
void Main()
{
var client = new ElasticClient(connection: new InMemoryConnection());
// prices from your GetPrices method...
var prices = new[] { 1d, 2d };
var searchRequest = new SearchRequest<Document>("index", "document")
{
Fields = new List<Nest.PropertyPathMarker>{ "Item", "Created", "Price" },
Size = 50,
Query = new BoolQuery
{
Should = prices.Select(p => new RangeQuery
{
Field = "price",
GreaterThanOrEqualTo = (p * 0.95).ToString(),
LowerThanOrEqualTo = (p * 1.15).ToString()
}.ToContainer())
}.ToContainer()
};
var result = client.Search<Document>(searchRequest);
Console.WriteLine(Encoding.UTF8.GetString(result.RequestInformation.Request));
}
public class Document
{
public string Item { get; set; }
public DateTime Created { get; set;}
public double Price {get; set;}
}
This searches for"document" types in index "index" (strongly typed to your Document POCO), using a bool query and should clauses of ranges constructed from the prices - if a document's price field falls into any one of the ranges, it will be considered a match.
The resulting query DSL from the NEST query above is
{
"size": 50,
"fields": [
"Item",
"Created",
"Price"
],
"query": {
"bool": {
"should": [
{
"range": {
"price": {
"gte": "0.95",
"lte": "1.15"
}
}
},
{
"range": {
"price": {
"gte": "1.9",
"lte": "2.3"
}
}
}
]
}
}
}

Related

Translate ElasticSearch query to Nest c#

I need some help in creating an AggregationDictionary from the following elasticsearch query
GET organisations/_search
{
"size": 0,
"aggs": {
"by_country": {
"nested": {
"path": "country"
},
"aggs": {
"by_country2": {
"filter": {
"bool": {
"must": [
{
"term": {
"country.isDisplayed": "true"
}
}
]
}
},
"aggs": {
"by_country3": {
"terms": {
"field": "country.displayName.keyword",
"size": 9999
}
}
}
}
}
}
}
}
I managed to write this horrible piece of code which I am pretty sure it is wrong, I am totally new to this.
AggregationDictionary aggs = new AggregationDictionary()
{
{
"countries_step1",
new NestedAggregation("countries_step1")
{
Path = "country",
Aggregations = new AggregationDictionary()
{
{
"countries_step2",
new FilterAggregation("countries_step2")
{
Filter = new BoolQuery
{
Must = new QueryContainer[] {
new NestedQuery
{
Query = new TermQuery
{
Field = "country.isDisplayed",
Value = true
}
}
}
},
Aggregations = new AggregationDictionary
{
{
"countries_step3",
new TermsAggregation("countries_step3")
{
Field = "country.displayName.keyword",
Size = 9999
}
}
}
}
}
}
}
}
};
Can someone tell me if I am in the correct direction? I am using Nest 6.6.0. Is there any tool that helps with these translations?
What you have so far is pretty solid, but when you try to execute this aggregation with the following call
var searchAsync = await client.SearchAsync<Document>(s => s.Size(0).Aggregations(aggs));
you will get this error
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "query malformed, empty clause found at [14:22]"
}
],
"type" : "illegal_argument_exception",
"reason" : "query malformed, empty clause found at [14:22]"
},
"status" : 400
}
Checking request which was sent to elasticsearch give us the answer why it happened
{
"aggs": {
"countries_step1": {
"aggs": {
"countries_step2": {
"aggs": {
"countries_step3": {
"terms": {
"field": "country.displayName.keyword",
"size": 9999
}
}
},
"filter": {}
}
},
"nested": {
"path": "country"
}
}
},
"size": 0
}
filter clause is empty, this is because you tried to used nested query but you didn't pass path parameter. We don't need nested query here (as shown in your example query), we can simplify the whole query to
var aggs = new AggregationDictionary()
{
{
"countries_step1",
new NestedAggregation("countries_step1")
{
Path = "country",
Aggregations = new AggregationDictionary()
{
{
"countries_step2",
new FilterAggregation("countries_step2")
{
Filter = new BoolQuery
{
Must = new QueryContainer[]
{
new TermQuery
{
Field = "country.isDisplayed",
Value = true
}
}
},
Aggregations = new AggregationDictionary
{
{
"countries_step3",
new TermsAggregation("countries_step3")
{
Field = "country.displayName.keyword",
Size = 9999
}
}
}
}
}
}
}
}
};
Now we have a valid request sent to elasticsearch.
There are a couple of things we can improve here:
1. Remove unnecessary bool query
Filter = new BoolQuery
{
Must = new QueryContainer[]
{
new TermQuery
{
Field = "country.isDisplayed",
Value = true
}
}
},
to
Filter =
new TermQuery
{
Field = "country.isDisplayed",
Value = true
},
2. Replace string field names
Usually, when doing calls from .Net there is some kind of POCO type which is helping us with writing strongly-typed requests to elasticsearch which helps us managing clean code and refactoring. With this, we can change field definition from
"country.displayName.keyword"
to
Infer.Field<Document>(f => f.Country.FirstOrDefault().DisplayName.Suffix("keyword"))
my types definition
public class Document
{
public int Id { get; set; }
[Nested]
public List<Country> Country { get; set; }
}
public class Country
{
public bool IsDisplayed { get; set; }
public string DisplayName { get; set; }
}
3. Consider using a fluent syntax
With NEST you can write queries in two ways: using object initializer syntax (which you did) or with help of fluent syntax. Have a look. Trying to write above query with the fluent syntax you will get something like
var searchResponse = await client.SearchAsync<Document>(s => s
.Size(0)
.Aggregations(a => a.Nested("by_country", n => n
.Path(p => p.Country)
.Aggregations(aa => aa
.Filter("by_country2", f => f
.Filter(q => q
.Term(t => t
.Field(field => field.Country.FirstOrDefault().IsDisplayed)
.Value(true)))
.Aggregations(aaa => aaa
.Terms("by_country3", t => t
.Field(field => field.Country.FirstOrDefault().DisplayName.Suffix("keyword"))
.Size(9999)
)))))));
which I find a little bit easier to follow and write, maybe it will be better for you as well.
As a final note, have a look into docs and check how you can debug your queries.
Hope that helps.

Can't nest terms aggregations more than two deep without further aggs being ignored?

I'm querying ElasticSearch using the Nest library for C#, to fetch graph data with multiple pivots. Each pivot is a nested TermsAggregation on a query, and everything works fine with one or two pivots. Once I get to three pivots, though, the SearchRequest object won't generate further aggregations.
The code to build the aggregations looks like this:
TermsAggregation topTermAgg = null;
TermsAggregation currentAgg = null;
foreach (var pivotName in activePivots)
{
newTermAgg = new TermsAggregation("pivot")
{
Field = pivot.ToString().ToLower()
};
if (topTermAgg == null)
{
topTermAgg = newTermAgg;
}
else
{
currentAgg.Aggregations = newTermAgg;
}
currentAgg = newTermAgg;
}
The SearchRequest itself is pretty straightforward:
var searchRequest = new SearchRequest(Indices.Index("a", "b", "c"))
{
Size = 0,
Aggregations = topTermAgg,
Query = query,
};
Unfortunately, the SearchRequest for 3 or more pivots, when converted to string, looks like this (via nestClient.Serializer.SerializeToString(searchRequest)):
{
"size": 0,
"query": {
"bool": <Fairly complex query, that works fine. It's the aggregation that has the problem.>
},
"aggs": {
"pivot": {
"terms": {
"field": "pivot1"
},
"aggs": {
"pivot": {
"terms": {
"field": "pivot2"
}
}
}
}
}
}
When I inspect the searchRequest object in the debugger, it quite definitely has 3 or more aggregations. What's going on here, and how can I get 3 or more nested terms aggregations to work properly?
I am using Nest version 5.01.
This must be related to the way in which you're building up the nested aggregations. Arbitrarily deep nested aggregations can be built with the client. Here's an example of a three deep nested aggregation
client.Search<Question>(s => s
.Aggregations(a => a
.Terms("top", ta => ta
.Field("top_field")
.Aggregations(aa => aa
.Terms("nested_1", nta => nta
.Field("nested_field_1")
.Aggregations(aaa => aaa
.Terms("nested_2", nnta => nnta
.Field("nested_field_3")
)
)
)
)
)
)
);
which serializes to
{
"aggs": {
"top": {
"terms": {
"field": "top_field"
},
"aggs": {
"nested_1": {
"terms": {
"field": "nested_field_1"
},
"aggs": {
"nested_2": {
"terms": {
"field": "nested_field_3"
}
}
}
}
}
}
}
}
You can also add values to AggregationDictionary directly
var request = new SearchRequest<Question>
{
Aggregations = new AggregationDictionary
{
{ "top", new TermsAggregation("top")
{
Field = "top_field",
Aggregations = new AggregationDictionary
{
{ "nested_1", new TermsAggregation("nested_1")
{
Field = "nested_field_1",
Aggregations = new AggregationDictionary
{
{ "nested_2", new TermsAggregation("nested_2")
{
Field = "nested_field_2"
}
}
}
}
}
}
}
}
}
};
client.Search<Question>(request);
is the same as the previous request. You can shorten this even further to
var request = new SearchRequest<Question>
{
Aggregations = new TermsAggregation("top")
{
Field = "top_field",
Aggregations = new TermsAggregation("nested_1")
{
Field = "nested_field_1",
Aggregations = new TermsAggregation("nested_2")
{
Field = "nested_field_2"
}
}
}
};
client.Search<Question>(request);
I got my code working by constructing the aggregation from the bottom-up, rather than from the top-down.
var terminalAggregation = <some aggregation. In my code, there's a lowest aggregation that's different from the rest. For the code I presented, you could just build the lowest pivot.>
TermsAggregation topTermAgg = null;
activePivots.Reverse();
foreach (var pivotName in activePivots)
{
newTermAgg = new TermsAggregation("pivot")
{
Field = pivot.ToString().ToLower(),
Aggregations = topTermAgg ?? terminalAggregation
};
topTermAgg = newTermAgg;
}
This looks like a bug in the Nest library; there are different classes like AggregationBase and BucketAggregationBase and AggregationDictionary that are all assignable to the "Aggregations" property, but it seems like there's some subtle flaw after the second assignment when you do this recursively.
The documentation is also not up-to-date: it claims that you can create an AggregationDictionary yourself, but since AggregationDictionary doesn't have a public Add() method, I really can't. Nor can I use the C#'s {}-after-insantiation syntax to populate its properties – again, because Add() is not public.

ElasticSearch - NEST equivalent for the "nested Filter Query" in Sense?

I am new to Elastic search,
I have object structure like,
class Student
{
int schoolId;
List<Subject> Subjects;
}
class Subject
{
string title;
string type;
}
I am trying to list down all the unique Subject titles. So far, I have the following Query that works in Sense and gives me desired result.
curl -XGET "http://localhost:9200/school-v0.1/student/_search?search_type=count&routing=5" -d '{
"filter":
{
"term": { "schoolId": 5 }
},
"aggs":
{
"SubjectsAggr":
{
"nested": { "path": "Subjects" },
"aggs":
{
"TitlesAggr":
{
"terms": { "field": "subject.title" }
}
}
}
}
}'
But I am not getting the NEST equivalent of the same. Refer below.
Am I missing something?
var result = this.ElasticClient.Search<Student>(q => q
.Index(this.ElasticClient.Index)
.Routing(schoolId)
.SearchType(SearchType.Count)
.Filter(q1 => q1.Term(a => a.schoolId, schoolId))
.Aggregations(student => student.Nested("SubjectsAggr", b => b.Path("subjects")
.Aggregations(sub => sub.Terms("TitlesAggr", s => s.Field("subject.title"))))));
//Below is what I hope to do, but the I get compilation errors because there seems to be something wrong with above NEST query that I have written
var subjectsAggregation = esResult.Aggs.Nested("SubjectsAggr");
var titlesAggregation = subjectsAggregation.Aggs.Terms("TitlesAggr");
var subjects = new List<string>();
foreach (var s in titlesAggregation.Items)
{
subjects.Add(s.Key);
}
Could you please help me find it? Thank you.
My NEST query was indeed correct. I was only not fetching the results out in appropriate way. I am now able to extract the needed information out of the search result as shown below.
var subjectsAggregation = result.Aggs.Nested("SubjectsAggr");
var titlesAggregation = subjectsAggregation.Aggregations["TitlesAggr"];
var titlesBucket = titlesAggregation as Bucket;
var titles = new List<string>();
if (titlesBucket != null)
{
foreach (var title in titlesBucket.Items)
{
titles.Add((((KeyItem) title).Key));
}
}

ElasticSearch NEST library to retrieve certain fields

Trying to achieve following,
Retrieve articles that match given id and genre
Retrieve selected fields for matching records
I have tried this with Sense(chrome plugin),
POST /d3acampaign/article/_search
{
"fields": ["title","genre"] ,
"query": {
"filtered": {
"filter": {
"bool": {
"must": [{
"term": {
"id": "6"
}
},
{
"term": {
"genre": "metal"
}
}]
}
}
}
}
}
For C# code i am trying to build the query using following construct,
FilterContainer fc = null;
TermFilter title = new TermFilter()
{
Field = "id",
Value = "6",
};
TermFilter genre = new TermFilter()
{
Field = "genre",
Value = "metal",
};
fc = title & genre;
QueryContainer qc = new FilteredQuery() { Filter = fc };
var searchRequest = new SearchRequest
{
SearchType = Elasticsearch.Net.SearchType.QueryAndFetch,
Query = qc,
Indices = new IndexNameMarker[] {"journal"},
Types = new TypeNameMarker[] { "article" },
};
var r = client.SearchAsync<Article>(searchRequest);
var l = (List<Article>) r.Result.Documents;
I am able to run this query and get matching records, but i amnt sure how to specify selected fields to retrieve. Let me know what can be changed in C# code to specify necessary fields.
Thanks in advance.
Based on this answer you can modify your request object as follow:
var searchRequest = new SearchRequest
{
...
Fields = new List<PropertyPathMarker>
{
Property.Path<Article>(p => p.YourField)
}
};
or if you will decide to use source filtering:
var searchRequest = new SearchRequest
{
...
Source = new SourceFilter
{
Include = new []
{
Property.Path<Article>(p => p.YourField)
}
}
};
Hope it helps.

Elasticsearch NEST - Filtering on multilevel nested types

I have a document model like this:
"customer": {
"properties": {
"id": { "type": "integer" },
"name": { "type": "string" },
"orders": {
"type": "nested",
"properties": {
"id": { "type": "integer" },
"orderDate" : { "type": "date", "format" : "YYYY-MM-dd" },
"orderLines": {
"type": "nested",
"properties": {
"seqno": { "type": "integer" },
"quantity": { "type": "integer" },
"articleId": { "type": "integer" }
}
}
}
}
}
}
A customer can have 0, 1 or multiple orders and an order can have 0, 1 or multiple orderLines
(this is a model I created for this question, as I think this is data everyone can understand, so if you spot any mistakes, please let me know, but don't let them distract you from my actual question)
I want to create a query with NEST which selects a (or all) customers with a specific value for customer.id, but only if they have at least one orderLine with a specific articleId.
I've looked at Need concrete documentation / examples of building complex index using NEST ElasticSearch library and Matching a complete complex nested collection item instead of separate members with Elastic Search, but was unable to create the query. Based upon the second question, I got to the point where I wrote
var results = client.Search<customer>(s => s
.From(0)
.Size(10)
.Types(typeof(customer))
.Query(q =>
q.Term(c => c.id, 12345)
&& q.Nested(n => n
.Path(c => c.order)
.Query(q2 => q2.Nested(n2 => n2
.Path(o => o.???))))
)
);
I expected the second Path to use order (orders is List) as generic type, but it is customer.
What is the code for the correct query?
In addition: is there more elaborate documentation of the search/query/filtering methods of NEST than the documentation on http://nest.azurewebsites.net/? In the first referenced question, both the links to the complex query tutorial (in question) and the unit test examples (accepted answer) do not work (timeout and 404 respectively).
Assuming we are modelling the customer to something on these lines
class customer
{
public int id { get; set; }
public string name { get; set;}
public class Orders {
public int id { get; set;}
public string orderData { get; set;}
public class OrderLines
{
public int seqno { get; set; }
public int quantity { get; set; }
public int articleId { get; set; }
}
[ElasticProperty(Type = FieldType.Nested)]
public List<OrderLines> orderLines { get; set; }
}
[ElasticProperty(Type = FieldType.Nested)]
public List<Orders> orders { get; set; }
};
The query in the above case would be :
var response = client.Search<customer>(
s => s.Index(<index_name_here>).Type("customer")
.Query(q => q.Term(p=>p.id, 1)
&&
q.Nested(n =>
n.Path("orders")
.Query(q2=> q2.Nested(
n2 => n2.Path("orders.orderLines")
.Query(q3 =>
q3.Term(c=>c.orders.First().orderLines.First().articleId, <article_id_here>)))))
));
As far as documentation the best I have come across is the same as the one you posted in the question and the resources linked there.

Resources