I need a quick help, i need to fetch the entire document only when my certain conditions are fulfilled in that same array.
Example
Conditions when in one array block all these three conditions fulfill. i.e.
"profile.bud.buddies.code": "1"
"profile.bud.buddies.moredata.key":"one"
"profile.bud.buddies.moredata.val": "0"
Unfortunately right now it is going through the entire document and trying to match the values in each of those arrays so it could be so that code=1 gets matched in one array, key=one in some other array and val=0 in the third array. What happens it in this case it returns me the entire document, whereas actually this was not fulfilled in one array alone so shouldn't have returned me the document.
I made the moredata as nested type but still cannot get through. Please help.
Query I am using
"query": {
"bool": {
"should": [
{
"match": {
"profile.bud.buddies.code": "1"
}
}
]
},
"nested": {
"path": "profile.bud.buddies.moredata",
"query": {
"bool": {
"must": [
{
"match": {
"profile.bud.buddies.moredata.key": "one"
}
},
{
"match": {
"profile.bud.buddies.moredata.val": "0"
}
}
]
}
}
}
}
Document Structure
"profile": {
"x":{},
"y":{},
"a":{},
"b":{},
"bud":{
"buddies": [
{
"code":"1",
"moredata": [
{
"key": "one",
"val": 0,
"setup": "2323",
"data": "myid"
},
{
"key": "two",
"val": 1,
"setup": "23",
"data": "id"
}]
},
{
"code":"2",
"moredata": [
{
"key": "two",
"val": 0,
"setup": "2323",
"data": "myid"
},
{
"key": "three",
"val": 1,
"setup": "23",
"data": "id"
}]
}]
}
This is how i have marked the mappings;
"profile": {
"bug": {
"properties": {
"buddies": {
"properties": {
"moredata": {
"type": "nested",
"properties": {
"key": {"type": "string"},
"val": {"type": "string"}
Your query structure is incorrect, it should be something like
"query": {
"bool": {
"must": [{
"match": {
"profile.bud.buddies.code": "1"
}
},
{
"nested": {
"path": "profile.bud.buddies.moredata",
"query": {
"bool": {
"must": [{
"match": {
"profile.bud.buddies.moredata.key": "one"
}
},
{
"match": {
"profile.bud.buddies.moredata.val": "0"
}
}
]
}
}
}
]
}
}
}
where the nested query is inside of the array of must clauses of the outer bool query. Note that profile.bud.buddies.moredata must be mapped as a nested data type.
Related
All,
I am working on building a NEST 6.x query that takes a serach term and looks in different fields in different indices.
This is the one I got so far but is not returning any results that I am expecting.
Please see the details below
Indices used
dev-sample-search
user-agents-search
The way the search should work is as follows.
The value in the query field(27921093) is searched against the
fields agentNumber, customerName, fileNumber, documentid(These are all
analyzed fileds).
The search should limit the documents to the agentNumbers the user
sampleuser#gmail.com has access to( sample data for
user-agents-search) is added below.
agentNumber, customerName, fileNumber, documentid and status are
part of the index dev-sample-search.
status field is defined as a keyword.
The fields in the user-agents-search index are all keywords
Sample user-agents-search index data:
{
"id": "sampleuser#gmail.com"",
"user": "sampleuser#gmail.com"",
"agentNumber": [
"123.456.789",
"1011.12.13.14"
]
}
Sample dev-sample-search index data:
{
"agentNumber": "123.456.789",
"customerName": "Bank of america",
"fileNumber":"test_file_1123",
"documentid":"1234456789"
}
GET dev-sample-search/_search
{
"from": 0,
"size": 10,
"query": {
"bool": {
"must": [
{
"multi_match": {
"type": "best_fields",
"query": "27921093",
"operator": "and",
"fields": [
"agentNumber",
"customerName",
"fileNumber",
"documentid^10"
]
}
}
],
"filter": [
{
"bool": {
"must": [
{
"terms": {
"agentNumber": {
"index": "user-agents-search",
"type": "_doc",
"user": "sampleuser#gmail.com",
"path": "agentNumber"
}
}
},
{
"bool": {
"must_not": [
{
"terms": {
"status": {
"value": "pending"
}
}
},
{
"term": {
"status": {
"value": "cancelled"
}
}
},
{
"term": {
"status": {
"value": "app cancelled"
}
}
}
],
"should": [
{
"term": {
"status": {
"value": "active"
}
}
},
{
"term": {
"status": {
"value": "terminated"
}
}
}
]
}
}
]
}
}
]
}
}
}
I see a couple of things that you may want to look at:
In the terms lookup query, "user": "sampleuser#gmail.com", should be "id": "sampleuser#gmail.com",.
If at least one should clause in the filter clause should match, set "minimum_should_match" : 1 on the bool query containing the should clause
I am learning to use Elasticsearch as a basic recommender engine.
My elasticsearch document contains records with nested entities as follows
PUT recs/user/1
{
"name" : "Brad Pitt",
"movies_liked": [
{
"name": "Forrest Gump",
"score": 1
},
{
"name": "Terminator",
"score": 4
},
{
"name": "Rambo",
"score": 4
},
{
"name": "Rocky",
"score": 4
},
{
"name": "Good Will Hunting",
"score": 2
}
]
}
PUT recs/user/2
{
"name" : "Tom Cruise",
"movies_liked": [
{
"name": "Forrest Gump",
"score": 2
},
{
"name": "Terminator",
"score": 1
},
{
"name": "Rocky IV",
"score": 1
},
{
"name": "Rocky",
"score": 1
},
{
"name": "Rocky II",
"score": 1
},
{
"name": "Predator",
"score": 4
}
]
}
I would like to search for users who specifically like "Forrest Gump","Terminator" and "Rambo".
I have used a nested query which currently looks like this
POST recs/user/_search
{
"query": {
"nested": {
"path": "movies_liked",
"query": {
"terms": {
"movies_liked.name": ["Forrest Gump","Terminator","Rambo"]
}
}
}
}
}
However when I execute this search, I expected to see only the first record which has all the required terms, but in the results I am getting both the records. In the second record the user clearly does not have "Rambo" in his liked list. I understand that this query is doing an "OR" operation with the given terms, How do I tweak this query to do an "AND" operation so that only the records having all the terms get matched?
How do I tweak this query to do an "AND" operation so that only the records having all the terms get matched?
By using a bool query:
POST recs/user/_search
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "movies_liked",
"query": {
"bool": {
"must": [
{
"terms": {
"movies_liked.name": [
"Forrest Gump"
]
}
}
]
}
}
}
},
{
"nested": {
"path": "movies_liked",
"query": {
"bool": {
"must": [
{
"terms": {
"movies_liked.name": [
"Terminator"
]
}
}
]
}
}
}
},
{
"nested": {
"path": "movies_liked",
"query": {
"bool": {
"must": [
{
"terms": {
"movies_liked.name": [
"Rambo"
]
}
}
]
}
}
}
}
]
}
}
}
Note that bool wraps around several nested queries, not the other way around. It is important because the scope of a nested query is the nested document, because it basically a hidden separate object.
Hope that helps!
I am trying to write an elastic search query where I have an array of sub-documents with exactly same structure and I have to check two conditions within the same sub document.
My data looks like this :
{
"_id": "5672ba7c0d71c93d159c408e",
"productId": "1723913526",
"title": "Digitab Tablet",
"instock": true,
"specifications": [{
"category": "In the box",
"name": "Test1",
"value": "No"
}, {
"category": "Warranty",
"name": "Test2",
"value": "Yes"
}, {
"category": "General",
"name": "Test3",
"value": "Abcd"
}],
}
}
What I am trying is : I should get products where specifications.name is "Test1" and specifications.value of the same sub document is "Yes"
Here is the Query that I have written for the this:
{
"query": {
"bool": {
"must": [
{
"wildcard": {
"specifications.name": "Test1*"
}
},
{
"term": {
"instock": "true"
}
},
{
"term": {
"specifications.value": "yes"
}
}
],
"must_not": [],
"should": []
}
}
}
The issue is - It should not return the product with id 1723913526 because "value": "Yes" is true for "name": "Test2" and not for "name": "Test1".
I hope the question is clear, please comment if more information is needed.
Thanks in advance.
You need to modify your mapping to indicate that specifications is a nested type.
Mapping like:
"mappings": {
"product": {
"properties": {
...
"specifications": {
"type": "nested"
}
}
}
}
And then querying like:
"bool": {
"must": [{
"nested": {
"path": "specifications",
"query": {
"bool": {
"must": [
{ "wildcard": { "specifications.first": "Test1*" }},
{ "term": { "specifications.value": "yes" }}
]
}
}
},
{
"term": { "instock": "true" }
}
}]
}
I've got a very small dataset of documents put in ES :
{"id":1, "name": "John", "team":{"code":"red", "position":"P"}}
{"id":2, "name": "Jack", "team":{"code":"red", "position":"S"}}
{"id":3, "name": "Emily", "team":{"code":"green", "position":"P"}}
{"id":4, "name": "Grace", "team":{"code":"green", "position":"P"}}
{"id":5, "name": "Steven", "team":[
{"code":"green", "position":"S"},
{"code":"red", "position":"S"}]}
{"id":6, "name": "Josephine", "team":{"code":"red", "position":"S"}}
{"id":7, "name": "Sydney", "team":[
{"code":"red", "position":"S"},
{"code":"green", "position":"P"}]}
I want to query ES for people who are in the red team, with position P.
With the request
curl -XPOST 'http://localhost:9200/teams/aff/_search' -d '{
"query": {
"bool": {
"must": [
{
"match": {
"team.code": "red"
}
},
{
"match": {
"team.position": "P"
}
}
]
}
}
}'
I've got a wrong result.
ES gives
"name": "John",
"team":
{ "code": "red", "position": "P" }
and
"name": "Sydney",
"team":
[
{ "code": "red", "position": "S"},
{ "code": "green", "position": "P"}
]
For the last entry, ES took the property code=red in the first record and took the property position=P in the second record.
How can I specify that the search must match the 2 two terms in the same record (within or not a list of nested records) ?
In fact, the good answer is only the document 1, with John.
Here is the gist that creates the dataset :
https://gist.github.com/flrt/4633ef59b9b9ec43d68f
Thanks in advance
When you index document like
{
"name": "Sydney",
"team": [
{"code": "red", "position": "S"},
{"code": "green","position": "P"}
]
}
ES implicitly create inner object for your field (team in particular example) and flattens it to structure like
{
'team.code': ['red', 'green'],
'team.position: ['S', 'P']
}
So you lose your order. To avoid this you need explicitly put nested mapping, index your document as always and query them with nested query
So, this
PUT so/nest/_mapping
{
"nest": {
"properties": {
"team": {
"type": "nested"
}
}
}
}
PUT so/nest/
{
"name": "Sydney",
"team": [
{
"code": "red",
"position": "S"
},
{
"code": "green",
"position": "P"
}
]
}
GET so/nest/_search
{
"query": {
"nested": {
"path": "team",
"query": {
"bool": {
"must": [
{
"match": {
"team.code": "red"
}
},
{
"match": {
"team.position": "P"
}
}
]
}
}
}
}
}
will result with empty hits.
Further reading on relation management: https://www.elastic.co/blog/managing-relations-inside-elasticsearch
You can use a Nested Query so that your searches happen individually on the subdocuments in the team array, rather than across the entire document.
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "team",
"query": {
"bool": {
"must": [
{ "match": { "team.code": "red" } },
{ "match": { "team.position": "P" } }
]
}
}
}
}
]
}
}
}
I have a following mapping
posts":{
"properties":{
"prop1": {
"type": "nested",
"properties": {
"item1": {
"type": "string",
"index": "not_analyzed"
},
"item2": {
"type": "string",
"index": "not_analyzed"
},
"item3": {
"type": "string",
"index": "not_analyzed"
}
}
},
"name": {
"type": "string",
"index": "not_analyzed"
}
}
}
Consider the objects indexed like following for these mapping
{
"name": "Name1",
"prop1": [
{
"item1": "val1",
"item2": "val2",
"item3": "val3"
},
{
"item1": "val1",
"item2": "val5",
"item3": "val6"
}
]
}
And another object
{
"name": "Name2",
"prop1": [
{
"item1": "val2",
"item2": "val7",
"item3": "val8"
},
{
"item1": "val12",
"item2": "val9",
"item3": "val10"
}
]
}
Now say i want to search documents which have prop1.item1 value to be either "val1" or "val2". I also want the result to be sorted in such a way that the document with both val1 and val2 would have more score than the one with only one of "val1" or "val2".
I have tried the following query but that doesnt seem to score based on number of matches
{
"query": {
"filtered": {
"query": {"match_all": {}},
"filter": {
"nested": {
"path": "prop1",
"filter": {
"or": [
{
"and": [
{"term": {"prop1.item1": "val1"}},
{"term": {"prop1.item2": "val2"}}
]
},
{
"and": [
{"term": {"prop1.item1": "val1"}},
{"term": {"prop1.item2": "val5"}}
]
},
{
"and": [
{"term": {"prop1.item1": "val12"}},
{"term": {"prop1.item2": "val9"}}
]
}
]
}
}
}
}
}
}
Now although it should give both documents, first document should have more score as it contains 2 of the things in the filter whereas second contains only one.
Can someone help with the right query to get results sorted based on most matches ?
The biggest problem you have with your query is that you are using a filter. Therefore no score is calculated. Than you use a match_all query which gives all documents a score of 1. Replace the filtered query with a query and use the bool query instead of the bool filter.
Hope that helps.
Scores aren't calculated on filters use a nested query instead:
{
"query": {
"nested": {
"score_mode": "sum",
"path": "prop1",
"query": {
"bool": {
"should": [{
"bool": {
"must": [{
"match": {
"prop1.item1": "val1"
}
},
{
"match": {
"prop1.item2": "val2"
}
}]
}
},
{
"bool": {
"must": [{
"match": {
"prop1.item1": "val1"
}
},
{
"match": {
"prop1.item2": "val5"
}
}]
}
},
{
"bool": {
"must": [{
"match": {
"prop1.item1": "val12"
}
},
{
"match": {
"prop1.item2": "val9"
}
}]
}
}]
}
}
}
}
}