Elasticsearch Nested query not working as expected - elasticsearch

I am bit new to elastic search. I am trying a nested query to get the result soem thing like below sql in query DSL..means I wanna restrict the search to driver last name as well as the vehicle make as well..like below use case.
select driver.last_name,driver.vehicle.make,driver.vehicle.model from drivers
where driver.last_name='Hudson' and driver.vehicle.make"="Miller-Mete;
But this doesn't work in elastic search sql as well as Query DSL...
--> can we do the query like this in ES...like let me clarify..
if department has List[employees] in Elasticsearch denoarmalized data..
and i want to restrict the query to department_name and emp_position..
--> is this use case even possible in elastic search?
select department_name,emp_name,emp_salary,emp_position
where emp_position="Intern" and department.name="devlopment"
--> Below are mappings and search Query DSL...
PUT /drivers
{
"mappings": {
"properties": {
"driver": {
"type": "nested",
"properties": {
"last_name": {
"type": "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"vehicle": {
"type": "nested",
"properties": {
"make": {
"type": "text"
},
"model": {
"type": "text"
}
}
}
}
}
}
}
}
GET /drivers/_mapping
O/P:
{
"drivers" : {
"mappings" : {
"properties" : {
"driver" : {
"type" : "nested",
"properties" : {
"last_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"vehicle" : {
"type" : "nested",
"properties" : {
"make" : {
"type" : "text"
},
"model" : {
"type" : "text"
}
}
}
}
}
}
}
}
}
--> inserting documents..
PUT /drivers/_doc/1
{
"driver" : {
"last_name" : "McQueen",
"vehicle" : [
{
"make" : "Powell Motors",
"model" : "Canyonero"
},
{
"make" : "Miller-Meteor",
"model" : "Ecto-1"
}
]
}
PUT /drivers/_doc/2
{
"driver" : {
"last_name" : "Hudson",
"vehicle" : [
{
"make" : "Mifune",
"model" : "Mach Five"
},
{
"make" : "Miller-Meteor",
"model" : "Ecto-1"
}
]
}
}
--> Below is the search query dsl..this gives 0 results. Even i replace
"term": {
"driver.last_name.keyword": "McQueen"
}
with "match" or "filter" still gives 0 results...
GET /drivers/_search
{
"query": {
"nested": {
"path": "driver",
"query": {
"nested": {
"path": "driver.vehicle",
"query": {
"bool": {
"must": [
{ "match": { "driver.vehicle.make": "Powell Motors" } },
{ "match": { "driver.vehicle.model": "Canyonero" } },
{
"term": {
"driver.last_name.keyword": "McQueen"
}
}
]
}
}
}
}
}
}
}
==> below Query DSL gives 2 results...
GET /drivers/_search
{
"query": {
"nested": {
"path": "driver",
"query": {
"nested": {
"path": "driver.vehicle",
"query": {
"bool": {
"must": [
{ "match": { "driver.vehicle.make": "Miller-Meteor" } }
]
}
}
}
}
}
}
}
O/P:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 2,
"relation" : "eq"
},
"max_score" : 1.3097506,
"hits" : [
{
"_index" : "drivers",
"_type" : "_doc",
"_id" : "1",
"_score" : 1.3097506,
"_source" : {
"driver" : {
"last_name" : "McQueen",
"vehicle" : [
{
"make" : "Powell Motors",
"model" : "Canyonero"
},
{
"make" : "Miller-Meteor",
"model" : "Ecto-1"
}
]
}
}
},
{
"_index" : "drivers",
"_type" : "_doc",
"_id" : "2",
"_score" : 1.3097506,
"_source" : {
"driver" : {
"last_name" : "Hudson",
"vehicle" : [
{
"make" : "Mifune",
"model" : "Mach Five"
},
{
"make" : "Miller-Meteor",
"model" : "Ecto-1"
}
]
}
}
}
]
}
}
==> this gives "parsing_exception",
"reason" : "[bool] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
==> even replacing 1st query bool to "match" also gives this error...as below
GET /drivers/_search
{
"query": {
"nested": {
"path": "driver",
"query": {
"bool": {
"must": [
{"match": {
"driver.last_name.keyword": "Hudson"
}}
]
},
"nested": {
"path": "driver.vehicle",
"query": {
"bool": {
"must": [
{
"match": {
"driver.vehicle.make": "Miller-Meteor"
}
}
]
}
}
}
}
}
}

Related

Elasticsearch multiple fields query

I'm asking for your help.
elasticsearch create search query
first, search field is keyword type
data
"hits" : [
{
"_index" : "search_event",
"_type" : "_doc",
"_score" : 5.179434,
"_source" : {
"search_keyword" : [
{
"search" : "or",
"keyword" : "developer",
"type" : "18"
}
]
},
{
"_source" : {
"search_keyword" : [
{
"search" : "or",
"keyword" : "tail"
},
{
"search" : "or",
"keyword" : "cap"
},
{
"search" : "and",
"keyword" : "developer"
}
]
}
}
}
When searching,
Must be keyword=developer and search=or
"query": {
"bool": {
"filter": [
{
"term": {
"search_keyword.keyword": {
"value": "developer"
}
}
},
{
"term": {
"search_keyword.search": {
"value": "or"
}
}
}
]
}
}
}
However, 'keyword=developer and search=and' but also a search.
how do I write a query?
"hits" : [
{
"_index" : "search_event",
"_type" : "_doc",
"_score" : 5.179434,
"_source" : {
"search_keyword" : [
{
"search" : "or",
"keyword" : "developer",
"type" : "18"
},
{
"search" : "or",
"keyword" : "tail"
},
{
"search" : "or",
"keyword" : "cap"
},
{
"search" : "and",
"keyword" : "developer"
}
]
}
]
}
i wan't search 'keyword=developer and search=and' documents
only 'keyword=developer and search=or' documents
use below query
"query": {
"bool": {
"must": [ --> note instead of `filter`, it's `must` clause.
{
"term": {
"search_keyword.keyword": {
"value": "developer"
}
}
},
{
"term": {
"search_keyword.search": {
"value": "or"
}
}
}
]
}
}
}

Upsert document such that it would update the particular item in an array field

In Elasticsearch, say I have the document like this:
{
"inputs": [
{
"id": "1234",
"value": "ABCD"
},
{
"id": "5678",
"value": "EFGH"
}
]
}
Say, now, I wanted to update value of all items where id is "1234" to "XYZA". How can I do that using script in elasticsearch? I am not sure if I can do some for loop in script?
Mapping:
{
"inputs" : {
"mappings" : {
"properties" : {
"inputs" : {
"type" : "nested",
"properties" : {
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"value" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
}
Query:
You can use _update_by_query api. Query part will filter out documents and script will update the field
<1. When inputs is of nested type
POST inputs/_update_by_query
{
"script": {
"source": "for(a in ctx._source['inputs']){if(a.id=='1234') a.value=params.new_value; }",
"params": {
"new_value": "XYZA"
}
},
"query": {
"nested":{
"path":"inputs",
"query":{
"term":{
"inputs.id":1234
}
}
}
}
}
2. When inputs if of object type
POST inputs/_update_by_query
{
"script": {
"source": "for(a in ctx._source['inputs']){if(a.id=='1234') a.value=params.new_value; }",
"params": {
"new_value": "XYZA"
}
},
"query": {
"term": {
"inputs.id": 1234
}
}
}
Result:
"hits" : [
{
"_index" : "inputs",
"_type" : "_doc",
"_id" : "3uwrwHEBLcdvQ7OTrUmi",
"_score" : 1.0,
"_source" : {
"inputs" : [
{
"id" : "1234",
"value" : "XYZA"
},
{
"id" : "5678",
"value" : "EFGH"
}
]
}
}
]

Using named queries (matched_queries) for nested types in Elasticsearch?

Using named queries, I can get a list of the matched_queries for boolean expressions such as:
(query1) AND (query2 OR query3 OR true)
Here is an example of using named queries to match on top-level document fields:
DELETE test
PUT /test
PUT /test/_mapping/_doc
{
"properties": {
"name": {
"type": "text"
},
"type": {
"type": "text"
},
"TAGS": {
"type": "nested"
}
}
}
POST /test/_doc
{
"name" : "doc1",
"type": "msword",
"TAGS" : [
{
"ID" : "tag1",
"TYPE" : "BASIC"
},
{
"ID" : "tag2",
"TYPE" : "BASIC"
},
{
"ID" : "tag3",
"TYPE" : "BASIC"
}
]
}
# (query1) AND (query2 or query3 or true)
GET /test/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"name": {
"query": "doc1",
"_name": "query1"
}
}
}
],
"should": [
{
"match": {
"type": {
"query": "msword",
"_name": "query2"
}
}
},
{
"exists": {
"field": "type",
"_name": "query3"
}
}
]
}
}
}
The above query correctly returns all three matched_queries in the response:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.5753641,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "TKNJ9G4BbvPS27u-ZYux",
"_score" : 1.5753641,
"_source" : {
"name" : "doc1",
"type" : "msword",
"TAGS" : [
{
"ID" : "ds1",
"TYPE" : "BASIC"
},
{
"ID" : "wb1",
"TYPE" : "BASIC"
}
]
},
"matched_queries" : [
"query1",
"query2",
"query3"
]
}
]
}
}
However, I'm trying to run a similar search:
(query1) AND (query2 OR query3 OR true)
only this time on the nested TAGS object rather than top-level document fields.
I've tried the following query, but the problem is I need to supply the inner_hits object for nested objects in order to get the matched_queries in the response, and I can only add it to one of the three queries.
GET /test/_search
{
"query": {
"bool": {
"must": {
"nested": {
"path": "TAGS",
"query": {
"match": {
"TAGS.ID": {
"query": "tag1",
"_name": "tag1-query"
}
}
},
// "inner_hits" : {}
}
},
"should": [
{
"nested": {
"path": "TAGS",
"query": {
"match": {
"TAGS.ID": {
"query": "tag2",
"_name": "tag2-query"
}
}
},
// "inner_hits" : {}
}
},
{
"nested": {
"path": "TAGS",
"query": {
"match": {
"TAGS.ID": {
"query": "tag3",
"_name": "tag3-query"
}
}
},
// "inner_hits" : {}
}
}
]
}
}
}
Elasticsearch will complain if I add more than one 'inner_hits'. I've commented out the places above where I can add it, but each of these will only return the single matched query.
I want my response to this query to return:
"matched_queries" : [
"tag1-query",
"tag2-query",
"tag3-query"
]
Any help is much appreciated, thanks!
A colleague helpfully provided a solution to this; move the _named parameter to directly under each nested section:
GET /test/_search
{
"query": {
"bool": {
"must": {
"nested": {
"_name": "tag1-query",
"path": "TAGS",
"query": {
"match": {
"TAGS.ID": {
"query": "tag1"
}
}
}
}
},
"should": [
{
"nested": {
"_name": "tag2-query",
"path": "TAGS",
"query": {
"match": {
"TAGS.ID": {
"query": "tag2"
}
}
}
}
},
{
"nested": {
"_name": "tag3-query",
"path": "TAGS",
"query": {
"match": {
"TAGS.ID": {
"query": "tag3"
}
}
}
}
}
]
}
}
}
This correctly returns all three tags now in the matched_queries response:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 2.9424875,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "TaNy9G4BbvPS27u--oto",
"_score" : 2.9424875,
"_source" : {
"name" : "doc1",
"type" : "msword",
"TAGS" : [
{
"ID" : "ds1",
"TYPE" : "DATASOURCE"
},
{
"ID" : "wb1",
"TYPE" : "WORKBOOK"
},
{
"ID" : "wb2",
"TYPE" : "WORKBOOK"
}
]
},
"matched_queries" : [
"tag1-query",
"tag2-query",
"tag3-query"
]
}
]
}
}

Query string with AND operator in nested query not working. Any idea?

I want to get the document in which nested child contains both words Mifune AND Miller-Meteor.
For more detail of nested, I've gone through https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html
here are the mappings
{
"mappings" : {
"properties" : {
"driver" : {
"type" : "nested",
"properties" : {
"last_name" : {
"type" : "text"
},
"vehicle" : {
"type" : "nested",
"properties" : {
"make" : {
"type" : "text"
},
"model" : {
"type" : "text"
}
}
}
}
}
}
}
}
i've two documents in the index
{
"driver" : {
"last_name" : "McQueen",
"vehicle" : [
{
"make" : "Powell Motors",
"model" : "Canyonero"
},
{
"make" : "Miller-Meteor",
"model" : "Ecto-1"
}
]
}
},{
"driver" : {
"last_name" : "Hudson",
"vehicle" : [
{
"make" : "Mifune",
"model" : "Mach Five"
},
{
"make" : "Miller-Meteor",
"model" : "Ecto-1"
}
]
}
}
query as below
{
"query" : {
"nested" : {
"path" : "driver",
"query" : {
"nested" : {
"path" : "driver.vehicle",
"query" : {
"bool" : {
"must" : [
{ "match" : { "driver.vehicle.make" : "Mifune" } },
{ "match" : { "driver.vehicle.make" : "Miller-Meteor" } }
]
}
}
}
}
}
}
}
I tried the above query but it did not work
also tried with query_string AND operator but it also not worked
{
"query": {
"nested": {
"path": "driver",
"query": {
"nested": {
"path": "driver.vehicle",
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "Mifune AND Miller-Meteor",
"fields": ["driver.vehicle.make"]
}
}
]
}
}
}
}
}
}
}
This is how you should query multiple nested fields.
There are two Nested queries inside your must clause.
The bool->must operator should be outside of your internal nested fields.
GET my_index/_search
{
"query": {
"nested": {
"path": "driver",
"query": {
"bool": {
"must": [
{
"nested": {
"path": "driver.vehicle",
"query": {
"match": {
"driver.vehicle.make": "Mifune"
}
}
}
},
{
"nested": {
"path": "driver.vehicle",
"query": {
"match": {
"driver.vehicle.make": "Miller-Meteor"
}
}
}
}
]
}
}
}
}
}
Results:
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 2.769686,
"hits" : [
{
"_index" : "my_index",
"_type" : "_doc",
"_id" : "2",
"_score" : 2.769686,
"_source" : {
"driver" : {
"last_name" : "Hudson",
"vehicle" : [
{
"make" : "Mifune",
"model" : "Mach Five"
},
{
"make" : "Miller-Meteor",
"model" : "Ecto-1"
}
]
}
}
}
]
}
}
Nested DataType
Hope this helps

Elasticsearch - Conditional nested fetching

I have index mapping:
{
"dev.directory.3" : {
"mappings" : {
"profile" : {
"properties" : {
"email" : {
"type" : "string",
"index" : "not_analyzed"
},
"events" : {
"type" : "nested",
"properties" : {
"id" : {
"type" : "integer"
},
"name" : {
"type" : "string",
"index" : "not_analyzed"
},
}
}
}
}
}
}
}
with data:
"hits" : [ {
"_index" : "dev.directory.3",
"_type" : "profile",
"_id" : "1",
"_score" : 1.0,
"_source" : {
"email" : "test#dummy.com",
"events" : [
{
"id" : 111,
"name" : "ABC",
},
{
"id" : 222,
"name" : "DEF",
}
],
}
}]
I'd like to filter only matched nested elements instead of returning all events array - is this possible in ES?
Example query:
{
"nested" : {
"path" : "events",
"query" : {
"bool" : {
"filter" : [
{ "match" : { "events.id" : 222 } },
]
}
}
}
}
Eg. If I query for events.id=222 there should be only single element on the result list returned.
What strategy for would be the best to achieve this kind of requirement?
You can use inner_hits to only get the nested records which matched the query.
{
"query": {
"nested": {
"path": "events",
"query": {
"bool": {
"filter": [
{
"match": {
"events.id": 222
}
}
]
}
},
"inner_hits": {}
}
},
"_source": false
}
I am also excluding the source to get only nested hits

Resources