I have the below structure (small part of a very large elastic-search document)
sample: {
{
"md5sum":"4002cbda13066720513d1c9d55dba809",
"id":1,
"sha256sum":"1c6e77ec49413bf7043af2058f147fb147c4ee741fb478872f072d063f2338c5",
"sha1sum":"ba1e6e9a849fb4e13e92b33d023d40a0f105f908",
"created_at":"2016-02-02T14:25:19+00:00",
"updated_at":"2016-02-11T20:43:22+00:00",
"file_size":188416,
"type":{
"name":"EXE"
},
"tags":[
],
"sampleSources":[
{
"filename":"4002cbda13066720513d1c9d55dba809",
"source":{
"name":"default"
}
},
{
"filename":"4002cbda13066720332513d1c9d55dba809",
"source":{
"name":"default"
}
}
]
}
}
The filter I would like to use is to find by the 'name' contained within sample.sampleSources.source using elastic search.
I tried the below queries
curl -XGET "http://localhost:9200/app/sample/_search?pretty" -d {query}
where, {query} is
{
"query":{
"nested":{
"path":"sample.sampleSources",
"query":{
"nested":{
"path":"sample.sampleSources.source",
"query":{
"match":{
"sample.sampleSources.source.name":"default"
}
}
}
}
}
}
}
However, it is not returning me any results. I have certain cases in my document where the nesting is more deeper than this. Can someone please guide me as to how should I formulate this query so that it works for all cases?
EDIT 1
Mappings:
{
"app":{
"mappings":{
"sample":{
"sampleSources":{
"type":"nested",
"properties":{
"filename":{
"type":"string"
},
"source":{
"type":"nested",
"properties":{
"name":{
"type":"string"
}
}
}
}
}
}
EDIT 2
The solution posted by Waldemar Neto below works well for match query but not for a wild-card or neither for a regexp
Can you please guide? I need the wild-card and the regexp queries to be working for this.
i tried here using your examples and works fine.
Take a look in my data.
mapping:
PUT /app
{
"mappings": {
"sample": {
"properties": {
"sampleSources": {
"type": "nested",
"properties": {
"source": {
"type": "nested"
}
}
}
}
}
}
}
indexed data
POST /app/sample
{
"md5sum": "4002cbda13066720513d1c9d55dba809",
"id": 1,
"sha256sum": "1c6e77ec49413bf7043af2058f147fb147c4ee741fb478872f072d063f2338c5",
"sha1sum": "ba1e6e9a849fb4e13e92b33d023d40a0f105f908",
"created_at": "2016-02-02T14:25:19+00:00",
"updated_at": "2016-02-11T20:43:22+00:00",
"file_size": 188416,
"type": {
"name": "EXE"
},
"tags": [],
"sampleSources": [
{
"filename": "4002cbda13066720513d1c9d55dba809",
"source": {
"name": "default"
}
},
{
"filename": "4002cbda13066720332513d1c9d55dba809",
"source": {
"name": "default"
}
}
]
}
Search query
GET /app/sample/_search
{
"query": {
"nested": {
"path": "sampleSources.source",
"query": {
"match": {
"sampleSources.source.name": "default"
}
}
}
}
}
Example using wildcard
GET /app/sample/_search
{
"query": {
"nested": {
"path": "sampleSources.source",
"query": {
"wildcard": {
"sampleSources.source.name": {
"value": "*aul*"
}
}
}
}
}
}
The only thing that I saw some difference was in the path, you don't need to set the sample (type) in the nested path, only the inner objets.
Test and give me a feedback.
Related
I have the following query that works as expected:
GET <index_name>/_search
{
"sort": [
{
"irFileCreateTime": {
"order": "desc"
}
}
],
"query": {
"bool": {
"should": [
{
"match": {
"fileId": 46704
}
},
{
"match": {
"fileId": 46706
}
},
{
"match": {
"fileId": 46719
}
}
]
}
}
}
The problem is that I need to further filter the data, but the field I need to filter on is a text field. I have tried many different ways of putting a must match into my query but everything is either malformed or filters out all hits when I know it should only filter out half. How can I add a must match "irStatus":"COMPLETE" to this query? Thanks in advance.
What you're after is a term query on, preferably, the keyword of irStatus. That is to say:
GET index/_search
{
"sort": [
{
"irFileCreateTime": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"term": {
"irStatus.keyword": {
"value": "COMPLETE"
}
}
}
],
"should": [
{
"match": {
"fileId": 46704
}
},
{
"match": {
"fileId": 46706
}
},
{
"match": {
"fileId": 46719
}
}
]
}
}
}
Assuming your mapping looks something like this:
{
"mappings": {
"properties": {
"irFileCreateTime": {
"type": "date"
},
"fileId": {
"type": "integer"
},
"irStatus": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
The reason it's apparently failing on your end is that "COMPLETE" has been lowercased due to standard analyzer.
Alternatively, you could do:
{
"must":[
{
"query_string":{
"query":"irStatus:COMPLETE AND (fileId:(46704 OR 46706 OR 46719))"
}
}
]
}
I have a document with a nested structure the nested object has an assignment_name and a due_date:
The mapping
{
"goal": {
"mappings": {
"doc": {
"properties": {
"title": {
"type": "keyword"
},
// lot's of other fields here ...
"steps": {
"type": "nested",
"properties": {
"assignment_name": {
"type": "keyword"
},
"due_date": {
"type": "date"
}
// lots of other fields here
}
}
}
}
}
}
}
I want to:
Filter all document that have a specific assignment_name (e.g.user_a)
Sort the result by the next due_date, not taking other assignements into account.
This query gives me random result (no sortings):
{
"query":{
"bool":{
"filter":[
{
"nested":{
"path":"steps",
"query":{
"term":{
"steps.assignment_name":"user_a"
}
}
}
}
]
}
},
"sort":[
{
"steps.due_date":{
"order":"asc",
"nested":{
"path":"steps",
"filter":{
"term":{
"steps.assignment_name":"user_a"
}
}
}
}
}
],
"from":0,
"size":25
}
Firstly you need to ensure that datatype for steps field is nested. Then you have to use nested sorting to sort documents based on a nested document field.
The query would be:
{
"query": {
"bool": {
"filter": [
{
"nested": {
"path": "steps",
"query": {
"term": {
"steps.assignment_name": "user_a"
}
}
}
}
]
}
},
"sort": [
{
"steps.due_date": {
"order": "asc",
"nested": {
"path": "steps",
"filter": {
"term": {
"steps.assignment_name": "user_a"
}
}
}
}
}
]
}
The catch above is to use the same filter in sort as used in the main query to filter the documents. This ensures that the correct nested document's field value is considered to sort the documents.
can somebody help me to understand what Elastic means by nested. In documentations https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_nested_sorting_examples is an example which does not show how the document object looks like. It look like I should imagine the mapping from the search query. Query looks like:
POST /_search
{
"query": {
"nested": {
"path": "parent",
"query": {
"bool": {
"must": {"range": {"parent.age": {"gte": 21}}},
"filter": {
"nested": {
"path": "parent.child",
"query": {"match": {"parent.child.name": "matt"}}
}
}
}
}
}
},
"sort" : [
{
"parent.child.age" : {
"mode" : "min",
"order" : "asc",
"nested": {
"path": "parent",
"filter": {
"range": {"parent.age": {"gte": 21}}
},
"nested": {
"path": "parent.child",
"filter": {
"match": {"parent.child.name": "matt"}
}
}
}
}
}
]
}
Can somebody write a document structure on which this query will work?
Something like this.
{
"parent": {
"name": "Elasti Sorch",
"age": 23,
"child": [
{
"name": "Kibana Lion",
"age": 12
},
{
"name": "Matt",
"age": 15
}
]
}
}
In Elastic nested means it's an array of objects. To store an array of objects into a field in elastic search you have to map the field to a nested while creating the index.
PUT parent
{
"mappings": {
"doc":{
"properties": {
"name":{
"type": "text"
},
"age":{
"type": "integer"
},
"child":{
"type": "nested",
"properties": {
"name":{
"type":"text"
},
"age":{
"type":"integer"
}
}
}
}
}
}
}
and a sample nested document cab be inserted like this
POST parent/doc
{
"name":"abc",
"age":50,
"child":[
{
"name":"son1",
"age":25
},
{
"name":"adughter1",
"age":20
}
]
}
In short: I want to lookup for distinct values in some field of the document BUT only matching some filter. The problem is in array-fields.
Imagine there are following documents in ES 2.4:
[
{
"states": [
"Washington (US-WA)",
"California (US-CA)"
]
},
{
"states": [
"Washington (US-WA)"
]
}
]
I'd like my users to be able to lookup all possible states via typeahead, so I have the following query for the "wa" user request:
{
"query": {
"wildcard": {
"states.raw": "*wa*"
}
},
"aggregations": {
"typed": {
"terms": {
"field": "states.raw"
},
"aggregations": {
"typed_hits": {
"top_hits": {
"_source": { "includes": ["states"] }
}
}
}
}
}
}
states.raw is a sub-field with not_analyzed option
This query works pretty well unless I have an array of values like in the example - it returns both Washington and California. I do understand why it happens (query and aggregations are working on top of the document and the document contains both, even though only one option matched the filter), but I really want to only see Washington and don't want to add another layer of filtering on the application side for the ES results.
Is there a way to do so via single ES 2.4 request?
You could use the "Filtering Values" feature (see https://www.elastic.co/guide/en/elasticsearch/reference/2.4/search-aggregations-bucket-terms-aggregation.html#_filtering_values_2).
So, your request could look like:
POST /index/collection/_search?size=0
{
"aggregations": {
"typed": {
"terms": {
"field": "states.raw",
"include": ".*wa.*" // You need to carefully quote the "wa" string because it'll be used as part of RegExp
},
"aggregations": {
"typed_hits": {
"top_hits": {
"_source": { "includes": ["states"] }
}
}
}
}
}
}
I can't hold myself back, though, and not tell you that using wildcard with leading wildcard is not the best solution. Do, please please, consider using ngrams for this:
PUT states
{
"settings": {
"analysis": {
"filter": {
"ngrams": {
"type": "nGram",
"min_gram": "2",
"max_gram": "20"
}
},
"analyzer": {
"ngram_analyzer": {
"type": "custom",
"filter": [
"standard",
"lowercase",
"ngrams"
],
"tokenizer": "standard"
}
}
}
},
"mappings": {
"doc": {
"properties": {
"location": {
"properties": {
"states": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
},
"ngrams": {
"type": "string",
"analyzer": "ngram_analyzer"
}
}
}
}
}
}
}
}
}
POST states/doc/1
{
"text":"bla1",
"location": [
{
"states": [
"Washington (US-WA)",
"California (US-CA)"
]
},
{
"states": [
"Washington (US-WA)"
]
}
]
}
POST states/doc/2
{
"text":"bla2",
"location": [
{
"states": [
"Washington (US-WA)",
"California (US-CA)"
]
}
]
}
POST states/doc/3
{
"text":"bla3",
"location": [
{
"states": [
"California (US-CA)"
]
},
{
"states": [
"Illinois (US-IL)"
]
}
]
}
And the final query:
GET states/_search
{
"query": {
"term": {
"location.states.ngrams": {
"value": "sh"
}
}
},
"aggregations": {
"filtering_states": {
"terms": {
"field": "location.states.raw",
"include": ".*sh.*"
},
"aggs": {
"typed_hits": {
"top_hits": {
"_source": {
"includes": [
"location.states"
]
}
}
}
}
}
}
}
Here's the mapping:
PUT books-index
{
"mappings": {
"books": {
"properties": {
"tags": {
"type": "nested",
"fields": {
"name": {
"type": "string"
},
"weight": {
"type": "float"
}
}
}
}
}
}
}
Then doing a nested query using a field_value_factor fails with an error
GET books-index/books/_search
{
"query": {
"nested": {
"path": "tags",
"score_mode": "sum",
"query": {
"function_score": {
"query": {
"match": {
"tags.name": "world"
}
},
"field_value_factor": {
"field": "weight"
}
}
}
}
}
}
The error: "nested: ElasticsearchException[Unable to find a field mapper for field [weight]]"
Interestingly, if there's one book in the index with tags - there's no error and the query works well.
Why is this happening? how can I prevent the error when there are no books with tags in the index?
Any ideas?
Thank you!
P.S. There's also an issue on github for this: https://github.com/elastic/elasticsearch/issues/12871
it looks like your mapping is incorrect.
After PUTing the mapping you provided, try executing GET books-index/_mapping, It will show these results:
"books-index": {
"mappings": {
"books": {
"properties": {
"tags": {
"type": "nested"
}
}
}
}
}
It's missing name and weight! The problem with the mapping is that you used either you used fields instead of properties, or you forget to put in a second properties key.
I modified your mapping to reflect that you were looking for a nested name and type within tags, as it looks like that is what your query wants.
PUT books-index
{
"mappings": {
"books": {
"properties": {
"tags": {
"type": "nested",
"properties": { // <--- HERE!
"name": {
"type": "string"
},
"weight": {
"type": "float"
}
}
}
}
}
}
}