Filtered Query with Term on collection - elasticsearch

I'm taking an old project to maintain and I am stuck since a day on a query.
The elasticsearch version I use is 1.7 but I don't think this is relevant to my problem.
I have some teacher documents :
{
"id": 244,
"degree": [],
"teacherDiplomaRelation": [],
"user": {
"enabled": true
},
"teacherClassDisciplineRelation": [
SEE BELOW
}
The teacherClassDisciplineRelation is N times this format (for every couple levelTree/Discipline that I have)
{
"levelTree": {
"id": 34,
"label": "1st year of college",
"slugLastLevelDisplay": "college"
},
"discipline": {
"id": 1,
"label": "Maths",
"slug": "maths"
},
"cityLocation": "10.1010,10.1010"
}
Now i want to get all teacher enabled and having maths in their disciplines. my query is:
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"user.enabled": true
}
},
{
"term": {
"teacherClassDisciplineRelation.discipline.slug": "maths"
}
}
]
}
}
}
},
"size": {
"from": 0,
"size": 15
}
}
Mapping:
"teacherClassDisciplineRelation": {
"type": "nested",
"properties": {
"cityLocation": {
"type": "geo_point",
"store": true
},
"discipline": {
"properties": {
"id": {
"type": "string",
"store": true
},
"label": {
"type": "string",
"boost": 7.0,
"store": true,
"analyzer": "custom_analyzer"
},
"slug": {
"type": "string",
"boost": 7.0,
"index": "not_analyzed",
"store": true,
"norms": {
"enabled": true
}
}
}
}
Problem:
My query with "user.enabled": true give me some results,
My query with "teacherClassDisciplineRelation.discipline.slug": "maths" always gives me 0 result but I've checked in the index, I should have some results.
I'm new to elasticsearch but I can't find out why my result is always 0.
Any idea why?

Since teacherClassDisciplineRelation is a nested field. You have to use Nested Query.
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "teacherClassDisciplineRelation",
"query": {
"term": {
"teacherClassDisciplineRelation.discipline.slug": {
"value": "maths"
}
}
}
}
},
{
"term": {
"user.enabled": true
}
}
]
}
}
}
Hope this helps!!

Related

ElasticSearch won't search specific field

I have a problem searching a specific field inside my index.
little background:
On my project we need to search inside a terminology Server like FHIR but then our own.
So we have an object that contains a Code (123564/A), multiple translations as term/display (urine problem) and mapping to other codes that are equal to that code but in a different system (ICD-10, SNOMED-CT, ICPC-2,..) example what has been indexed:
{
"Code": "10008220/A1",
"EffectiveTime": "0001-01-01T00:00:00Z",
"Active": true,
"System": "ibui",
"Purpose": "",
"Descriptions": [
{
"DescriptionId": "2464cf5c-d4fc-4a61-b6bc-746d003cb4ef",
"Code": "10008220/A1",
"System": "ibui",
"Term": "gebroken arm",
"LanguageId": "3d50c237-0add-43e7-92a2-5edf1ac7c6ee",
"FSN": false,
"Preferred": true,
"EffectiveTime": "0001-01-01T00:00:00Z",
"Active": true,
"SendVersion": "2021-12-07T17:01:53.786755Z",
"Purpose": ""
},
{
"DescriptionId": "95501583-9f24-4964-bbc9-1a6e95eba30f",
"Code": "10008220/A1",
"System": "ibui",
"Term": "fracture du bras",
"LanguageId": "1238dde0-08df-4ae0-8676-59919f66737e",
"FSN": false,
"Preferred": true,
"EffectiveTime": "0001-01-01T00:00:00Z",
"Active": true,
"SendVersion": "2021-12-07T17:01:53.786755Z",
"Purpose": ""
}
],
"Mappings": [
{
"MappingId": "",
"FromSys": "ibui",
"From": "10008220/A1",
"ToSys": "icd-10",
"To": "T10",
"EffectiveTime": "0001-01-01T00:00:00Z",
"Active": true
},
{
"MappingId": "",
"FromSys": "ibui",
"From": "10008220/A1",
"ToSys": "icpc-2",
"To": "L76",
"EffectiveTime": "0001-01-01T00:00:00Z",
"Active": true
}
],
"SendVersion": "2021-12-07T17:01:53.786755Z"
}
The problem:
We can search on 2 different fields : Code & Term. and when searching we keep in mind that we have some filters for a specific language code (Dutch,..) or A system like ICD-10 or ICPC-2,..
I have a query that is working and returns the above object when searching in 1 field (Descriptions.Term) that is the following:
working query
{
"query": {
"bool": {
"must": {
"nested": {
"inner_hits": {
"highlight": {
"fields": {
"*": {}
}
}
},
"path": "Descriptions",
"query": {
"bool": {
"should": [
{
"multi_match": {
"fields": [
"Descriptions.Term",
"Descriptions.Term._2gram",
"Descriptions.Term._3gram"
],
"query": "gebroken*~ n",
"type": "bool_prefix"
}
}
],
"filter": [
{
"bool": {
"should": [
{
"term": {
"Descriptions.System": "ibui"
}
},{
"term": {
"Descriptions.System": "icd-10"
}
},{
"term": {
"Descriptions.System": "icpc-2"
}
}
],
"minimum_should_match": "1"
}
},
{
"term": {
"Descriptions.Active": "true"
}
},
{
"term": {
"Descriptions.LanguageId": "3d50c237-0add-43e7-92a2-5edf1ac7c6ee"
}
}
]
}
}
}
}
}
}
}
But when we somethings need to search in multiple fields.
When adding the Descriptions.Code field to the fields map the query is not working and I can't figure out why. I have it decleared inside my mapping so it should be searchable?
I'm searching for the Code of the object above in both fields (Descriptions.Term & Descriptions.Code) but it doesn't returns the hit.
not working query
{
"query": {
"bool": {
"must": {
"nested": {
"inner_hits": {
"highlight": {
"fields": {
"*": {}
}
}
},
"path": "Descriptions",
"query": {
"bool": {
"should": [
{
"multi_match": {
"fields": [
"Descriptions.Term",
"Descriptions.Term._2gram",
"Descriptions.Term._3gram",
"Descriptions.Code"
],
"query": "10008220*~ n",
"type": "bool_prefix"
}
}
],
"filter": [
{
"bool": {
"should": [
{
"term": {
"Descriptions.System": "ibui"
}
},{
"term": {
"Descriptions.System": "icd-10"
}
},{
"term": {
"Descriptions.System": "icpc-2"
}
}
],
"minimum_should_match": "1"
}
},
{
"term": {
"Descriptions.Active": "true"
}
},
{
"term": {
"Descriptions.LanguageId": "3d50c237-0add-43e7-92a2-5edf1ac7c6ee"
}
}
]
}
}
}
}
}
}
}
mapping:
{
"settings": {
"number_of_shards": 1,
"analysis": {
"analyzer": {
"autocomplete": {
"tokenizer": "custom_tokenizer"
}
},
"tokenizer": {
"custom_tokenizer": {
"type": "ngram",
"min_gram": 2,
"max_gram": 6,
"token_chars": [
"letter",
"digit",
"symbol",
"punctuation"
]
}
}
},
"max_ngram_diff" : "5"
},
"mappings": {
"properties": {
"Descriptions": {
"type": "nested",
"properties": {
"Term": {
"type": "search_as_you_type",
"analyzer": "autocomplete"
},
"Code": {
"type": "keyword",
"index": true
},
"System": {
"type": "keyword",
"index": true
},
"LanguageId": {
"type": "keyword",
"index": true
},
"Purpose": {
"type": "keyword",
"index": true
},
"Active": {
"type": "keyword",
"index": true
}
}
},
"Mappings": {
"properties": {
"To": {
"type": "keyword",
"index": true
},
"ToSys": {
"type": "keyword",
"index": true
}
}
}
}
}
}
Thank you for helping me out!

elasticsearch with range sub query in nested query

I am trying to get a nested query filter inside of a nested.
here is my es mapping: there is one "id" field(long) and a nested field called "my_field" with four sub fields in it.
{
"my_index": {
"mappings": {
"dynamic": "strict",
"properties": {
"id": {
"type": "long"
},
"my_field": {
"type": "nested",
"properties": {
"x": {
"type": "long"
},
"y": {
"type": "long"
},
"z": {
"type": "long"
},
"a": {
"type": "double"
},
"b": {
"type": "long"
}
}
}
}
}
}
}
My question is how to retrive the document with nested es query which contains sub range query in it.
For example, I'm trying to get two document id :11111 and id:22222 with nested query restriction "x > 15" or "a > 0.5" and also with inner hit size limitation, which is 20 here.
{
"_source": false,
"query": {
"bool": {
"must": {
"nested": {
"inner_hits": {
"size": 20
},
"path": "my_field",
"query": {
"bool": {
"should": [
{
"range": {
"x": {
"from": 15,
"include_lower": true,
"include_upper": true,
"to": null
}
}
},
{
"range": {
"a": {
"from": 0.5,
"include_lower": true,
"include_upper": true,
"to": null
}
}
}
]
}
}
}
},
"should": [
{
"term": {
"id": 11111
}
},
{
"term": {
"id": 22222
}
}
]
}
},
"timeout": "5000ms",
"track_total_hits": true
}
However, there are no hits return
Please use the dot notation in your query to include the complete path, e.g.,
"range": {
"my_field.x": { "from": ... }
}

Elastic search: Nested query with and condition on parent and child

I have 2 sample record like below:
{
"parents": [
{
"child": [
{
"is_deleted": false,
"child_id": -1,
"timestamp": 1483536052232
}
],
"parent_id": 810
},
{
"child": [
{
"is_deleted": true,
"child_id": 105,
"timestamp": 1483537567000
}
],
"parent_id": 42
}
]
},
{
"parents": [
{
"child": [
{
"is_deleted": false,
"child_id": 105,
"timestamp": 1483537567000
}
],
"parent_id": 42
}
]
}
and my mapping:
"properties": {
"parents": {
"type": "nested",
"properties": {
"parent_id": {
"type": "integer",
"doc_values": false
},
"child": {
"type": "nested",
"properties": {
"is_deleted": {
"type": "boolean",
"doc_values": false
},
"child_id": {
"type": "integer",
"doc_values": false
},
"timestamp": {
"type": "long",
"doc_values": false
}
}
}
}
}
I want to search by parent ID which has at least one child with is_deleted as false. For example if i will query with parent ID 42, i should get only 2nd document not first.
You should use nested query to query nested field.
Here is an example, but I don't know if this is the best solution at least it is a working one:
POST /test1/test/_search
{
"query": {
"nested": {
"path": "parents",
"query": {
"bool": {
"must": [
{
"match": {
"parents.parent_id": 42
}
},
{
"nested": {
"path": "parents.child",
"query": {
"term": {
"parents.child.is_deleted": "F"
}
}
}
}
]
}
}
}
}
}

ElasticSearch double nested sorting

I have documents which look like this (here is example):
{
"user": "xyz",
"state": "FINISHED",
"finishedTime": 1465566467161,
"jobCounters": {
"counterGroup": [
{
"counterGroupName": "org.apache.hadoop.mapreduce.FileSystemCounter",
"counter": [
{
"name": "FILE_BYTES_READ",
"mapCounterValue": 206509212380,
"totalCounterValue": 423273933523,
"reduceCounterValue": 216764721143
},
{
"name": "FILE_BYTES_WRITTEN",
"mapCounterValue": 442799895522,
"totalCounterValue": 659742824735,
"reduceCounterValue": 216942929213
},
{
"name": "HDFS_BYTES_READ",
"mapCounterValue": 207913352565,
"totalCounterValue": 207913352565,
"reduceCounterValue": 0
},
{
"name": "HDFS_BYTES_WRITTEN",
"mapCounterValue": 0,
"totalCounterValue": 89846725044,
"reduceCounterValue": 89846725044
}
]
},
{
"counterGroupName": "org.apache.hadoop.mapreduce.JobCounter",
"counter": [
{
"name": "TOTAL_LAUNCHED_MAPS",
"mapCounterValue": 0,
"totalCounterValue": 13394,
"reduceCounterValue": 0
},
{
"name": "TOTAL_LAUNCHED_REDUCES",
"mapCounterValue": 0,
"totalCounterValue": 720,
"reduceCounterValue": 0
}
]
}
]
}
}
Now I want to sort this data to get TOP 15 documents on the basis of totalCounterValue where counter.name is FILE_BYTES_READ. I have tried nested sorting on this but no matter which key name I write in counter.name, it is always sorting on the basis of HDFS_BYTES_READ. Can anyone please help me with my query.
{
"_source": true,
"size": 15,
"query": {
"bool": {
"must": [
{
"term": {
"state": {
"value": "FINISHED"
}
}
},
{
"range": {
"startedTime": {
"gte": "now - 4d",
"lte": "now"
}
}
}
]
}
},
"sort": [
{
"jobCounters.counterGroup.counter.totalCounterValue": {
"order": "desc",
"nested_path": "jobCounters.counterGroup",
"nested_filter": {
"nested": {
"path": "jobCounters.counterGroup.counter",
"filter": {
"term": {
"jobCounters.counterGroup.counter.name": "file_bytes_read"
}
}
}
}
}
}
]}
This is the mapping for jobCounters we have created:
"jobCounters": {
"type": "nested",
"include_in_parent": true,
"properties" : {
"counterGroup": {
"type": "nested",
"include_in_parent": true,
"properties": {
"counterGroupName": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"counter" : {
"type": "nested",
"include_in_parent": true,
"properties": {
"reduceCounterValue": {
"type": "long"
},
"name": {
"type": "string",
"analyzer": "english",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
},
"totalCounterValue": {
"type": "long"
},
"mapCounterValue": {
"type": "long"
}
}
}
}
}
}
}
I followed nested sorting documentation of ElasticSearch and came up with this query, but I don't know why it is always sorting the totalCounterValue of HDFS_BYTES_READ irrespective of jobCounters.counterGroup.counter.name's value.
you can try something like this,
curl -XGET 'http://localhost:9200/index/jobCounters/_search' -d '
{
"size": 15,
"query": {
"nested": {
"path": "jobCounters.counterGroup.counter",
"filter": {
"term": {
"jobCounters.counterGroup.counter.name": "file_bytes_read"
}
}
}
},
"sort": [
{
"jobCounters.counterGroup.counter.totalCounterValue": {
"order": "desc",
"nested_path": "jobCounters.counterGroup",
"nested_filter": {
"nested": {
"path": "jobCounters.counterGroup.counter",
"filter": {
"term": {
"jobCounters.counterGroup.counter.name": "file_bytes_read"
}
}
}
}
}
}
]
}
'
Read the end of this document. It explains that we have to repeat the same query in nested_filter too.

Showing Nested objects in Kibana

I have a nested object in document. In kibana 3, I am not able to add a panel for any of these nested object. E.g.
{
"location": [
0.023027
,
51.58011
],
"itemList": [
{
"make": "x",
"model": "y"
}
,
{
"make": "a",
"model": "b"
}
],
"somekey": "1234",
"dateTime": "01/10/2005 22:43"
}
}
...
...
And my mapping looks like:
{
"order": 0,
"template": "*",
"settings": {},
"mappings": {
"_default_": {
"dynamic": "true",
"dynamic_templates": [
{
"string_not_analyzed": {
"mapping": {
"index": "not_analyzed"
},
"match": "*",
"match_mapping_type": "string"
}
},
{
"timestamp_hours_minutes": {
"mapping": {
"type": "date",
"format": "dd/mm/yyyy HH:mm"
},
"match": "dateTime"
}
},
{
"geo_location_data":{
"mapping":{
"type": "geo_point"
},
"match": "location"
}
},
{
"my_nested_mapping":{
"mapping":{
"type": "nested",
"include_in_parent": true,
"store": "yes"
},
"match": "*List"
}
}
],
"_timestamp": {
"enabled": true,
"store": true,
"format": "yyyy-MM-dd HH:mm:ssZZ"
},
"date_detection": "false",
"_all": {
"enabled": true
}
}
}
}
On kibana 3 when adding a term chart, I can select in dropdown 'itemList.make', however in the pie chart they come all as empty. Can someone throw some light on this please. Is this even possible.
On inspection kibana runs following query that returns zero results when run independently from head plugin.
curl -XGET 'http://localhost:9200/data/_search?pretty' -d '{
"facets": {
"terms": {
"terms": {
"field": "itemList.make",
"size": 10,
"order": "count",
"exclude": []
},
"facet_filter": {
"fquery": {
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"query_string": {
"query": "*"
}
}
]
}
},
"filter": {
"bool": {
"must": [
{
"match_all": {}
}
]
}
}
}
}
}
}
}
},
"size": 0
}'
In Elasticsearch documentation nested queries have path defined which i don't see being added by kibana.
Is there a way to get nested objects displayed in a kibana 3 panel?

Resources