I use ElasticSearch version 1.7.5 and I am trying to fetch all documents where missing some fields.
My mapping:
...
"participant": {
"properties": {
"id": {
"type": "string"
},
"firstName": {
"type": "string"
},
"lastName": {
"type": "string"
},
"name": {
"type": "string"
}
},
"coordinator": {
"properties": {
"id": {
"type": "string"
},
"firstName": {
"type": "string"
},
"lastName": {
"type": "string"
},
"name": {
"type": "string"
}
}
...
I want to query all documents that don't have assigned coordinator.id or participant.id yet.
My query looks like:
"query": {
"nested": {
"path": "coordinator, participant",
"query": {
"constant_score": {
"filter": {
"or": [
{
"missing": {
"field": "coordinator.id"
}
},
{
"missing": {
"field": "participant.id"
}
},
]
}
}
}
}
}
You do OR queries via the bool query:
https://www.elastic.co/guide/en/elasticsearch/reference/1.7/query-dsl-bool-filter.html
So this query would work:
{
"query": {
"bool": {
"should": [
{
"constant_score": {
"filter": {
"missing": {
"field": "participant.id"
}
}
}
},
{
"constant_score": {
"filter": {
"missing": {
"field": "coordinator.id"
}
}
}
}
]
}
}
}
I noticed that you were using a nested query though the mapping does not state that coordinator and participant are nested field types so that will not work:
https://www.elastic.co/guide/en/elasticsearch/reference/1.7/mapping-nested-type.html
Setting something as a nested type is only useful when you need to group search terms together so I don't think it is necessary for you.
Related
I have an ES index with this mapping:
{
"_doc": {
"dynamic": "false",
"properties": {
"original": {
"properties":{
"id": {
"type": "keyword"
},
"purchaseStatus": {
"type": "keyword"
},
"marketCode": {
"type": "keyword"
},
"salesProfiles": {
"type": "nested",
"properties": {
"marketCode": {
"type": "keyword"
},
"purchaseStatus": {
"type": "keyword"
}
}
}
}
},
"recommended": {
"properties":{
"id": {
"type": "keyword"
},
"purchaseStatus": {
"type": "keyword"
},
"marketCode": {
"type": "keyword"
},
"salesProfiles": {
"type": "nested",
"properties": {
"marketCode": {
"type": "keyword"
},
"purchaseStatus": {
"type": "keyword"
}
}
}
}
},
"distance": {
"type": "double"
},
"rank": {
"type": "double"
},
"source": {
"properties": {
"application": {
"type": "keyword"
},
"platform": {
"type": "keyword"
}
}
},
"timestamp": {
"properties": {
"createdAt": {
"type": "date"
},
"updatedAt": {
"type": "date"
}
}
}
}
},
"_default_": {
"dynamic": "false"
}
}
and I need to obtain the recommended docs with salesProfiles.marketCode equal to original.marketCode but my query doesn't return any buckets:
GET index/_search
{
"aggs": {
"similarities": {
"filter": {
"bool": {
"must": [
{
"term": {
"original.storefrontId": "12345"
}
},
{
"nested": {
"path": "recommended.salesProfiles",
"query": {
"bool": {
"must": [
{
"match": {
"recommended.salesProfiles.purchaseStatus": "PAID"
}
}
]
}
}
}
}
]
}
},
"aggs": {
"markets": {
"nested": {
"path": "recommended.salesProfiles"
},
"aggs": {
"recommendedMarket": {
"terms": {
"field": "recommended.salesProfiles.marketCode",
"size": 100
}
}
}
}
}
}
},
"explain": false
}
Any suggestion would be really appreciated. Thanks in advance!
Its hard to debug this without any example docs, but I think this might work
{
"size": 0,
"query": {
"bool": {
"must": [
{
"term": {
"original.storefrontId": "12345"
}
},
{
"nested": {
"path": "recommended.salesProfiles",
"query": {
"bool": {
"must": [
{
"match": {
"recommended.salesProfiles.purchaseStatus": "PAID"
}
}
]
}
}
}
}
]
}
},
"aggs": {
"Profiles": {
"nested": {
"path": "recommended.salesProfiles"
},
"aggs": {
"by_term": {
"terms": {
"field": "recommended.salesProfiles.marketCode",
"size": 100
}
}
}
}
}
}
I don't think you can use "nested" under the filter agg without being under a nested aggregation, so I believe that's why you didn't get any docs.
I basically moved all the filtering to the query and just aggregated the terms later
Is it possible using the Elastic Search _count API and having the following abbreviated ES template to find the count of sponsorships for all the campaigns by brandId?
sponsorshipSets and sponsorships are optional so it can be null.
{
"index_patterns": "campaigns*",
"order": 4,
"version": 4,
"aliases": {
"campaigns": {
}
},
"settings": {
"number_of_shards": 5
},
"mappings": {
"dynamic": "false",
"properties": {
"brandId": {
"type": "keyword"
},
"sponsorshipSets": {
"type": "nested",
"properties": {
"id": {
"type": "keyword"
},
"sponsorships": {
"type": "nested",
"properties": {
"id": {
"type": "keyword"
}
}
}
}
}
}
filter aggregation can be used to fetch docs with certain brand Id. Two Nested aggregations to point to sponsorship and value_count aggregation to get the count.
Query
{
"aggs": {
"selected_brand": {
"filter": {
"term": {
"brandId": "1"
}
}
},
"sponsorshipSets": {
"nested": {
"path": "sponsorshipSets"
},
"aggs": {
"sponsorships": {
"nested": {
"path": "sponsorshipSets.sponsorships"
},
"aggs": {
"count": {
"value_count": {
"field": "sponsorshipSets.sponsorships.id"
}
}
}
}
}
}
}
}
I found a solution without using Aggregations, it seems more accurate from the above and I can use the _count API.
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "sponsorshipSets.sponsorships",
"query": {
"bool": {
"filter": {
"exists": {
"field": "sponsorshipSets.sponsorships"
}
}
}
}
}
},
{
"term": {
"brandId": "b1d28821-3730-4266-8f55-eb69596004fb"
}
}
]
}
}
}
I am trying to fetch documents where is missing field "topic.description" and match term "fundedUnder.programme": "ABC".
Mapping:
...
"fundedUnder": {
"properties": {
"programme": {
"type": "string"
},
"subprogramme": {
"type": "string"
}
}
},
"topics": {
"type": "nested",
"include_in_parent": true,
"properties": {
"code": {
"type": "string",
"analyzer": "analyzer_keyword"
},
"description": {
"type": "string",
"analyzer": "analyzer_keyword"
},
"title": {
"type": "string",
"analyzer": "analyzer_keyword"
}
}
},
...
My Query looks like:
{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"missing": {
"field": "topics.description"
}
},
{
"term": {
"fundedUnder.programme" : "ABC"
}
}
]
}
}
}
}
}
This query found nothing and that is wrong, because I have in indexes a lot of documents with fundedUnder.programme == "ABC" and with missing field topics.description.
Thanks in advance.
ElasticSearch version 1.7.5
I believe this should work:
EDIT: updated to use version 1.7 Query DSL
{
"query": {
"filtered": {
"query": {
"match": { "fundedUnder.programme" : "ABC" }
},
"filter": {
"missing": { "field": "topics.description" }
}
}
}
}
I have the following mapping:
PUT /test
{
"mappings": {
"test": {
"properties": {
"parent": {
"type": "nested",
"properties": {
"#id": {
"type": "string",
"index": "not_analyzed"
},
"#type": {
"type": "string"
},
"child": {
"type": "nested",
"properties": {
"#id": {
"type": "string",
"index": "not_analyzed"
},
"subchild": {
"type": "nested",
"properties": {
"#id": {
"type": "string",
"index": "not_analyzed"
},
"hasA": {
"type": "nested",
"properties": {
"#value": {
"type": "string"
}
}
},
"hasB": {
"type": "nested",
"properties": {
"#id": {
"type": "string",
"index": "not_analyzed"
}
}
},
"hasC": {
"type": "nested",
"properties": {
"#id": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}
}
}
}
}
}
}
And the following document:
POST /test/test/1
{
"parent": {
"#id": "12345",
"#type": "test",
"child": [
{
"#id": "1",
"subchild": [
{
"#id": "1.1",
"hasA": {
"#value": "hasA value"
},
"hasB": {
"#id": "hasB_1"
},
"hasC": {
"#id": "hasC_1"
}
}
]
},
{
"#id": "2",
"subchild": [
{
"#id": "2.1",
"hasA": {
"#value": "hasA value"
},
"hasB": {
"#id": "hasB_2"
},
"hasC": {
"#id": "hasC_2"
}
}
]
}
]
}
}
And the following query:
POST test/test/_search
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"nested": {
"path": "parent.child.subchild.hasB",
"filter": {
"bool": {
"must": [
{
"term": {
"parent.child.subchild.hasB.#id": "hasB_2"
}
}
]
}
},
"_cache": false
}
}
}
}
}
I'm unable to set the path to just parent.child.subchild so that I can match on both hasB and hasC, it seems I can only select one nested item at a time. This is what I would like to be able to do:
POST test/test/_search
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"nested": {
"path": "parent.child.subchild",
"filter": {
"bool": {
"must": [
{
"term": {
"parent.child.subchild.hasB.#id": "hasB_2"
}
},
{
"term": {
"parent.child.subchild.hasC.#id": "hasC_2"
}
}
]
}
},
"_cache": false
}
}
}
}
}
Are you looking for something like this?
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"nested": {
"path": "parent.child.subchild",
"filter": {
"bool": {
"must": [
{
"nested": {
"path": "parent.child.subchild.hasB",
"query": {
"term": {
"parent.child.subchild.hasB.#id": "hasB_2"
}
}
}
},
{
"nested": {
"path": "parent.child.subchild.hasC",
"query": {
"term": {
"parent.child.subchild.hasC.#id": "hasC_2"
}
}
}
}
]
}
},
"_cache": false
}
}
}
}
}
Query for correct syntax for multi-level nested docs can be found here. Look at #martijnvg comments inside it.
ES Docs has not done a good job on explaining multi-level nested query.
Basically you need to nest subchild inside child inside parent and specify individual path. You will need three nested queries.
P.S - I have not tested this myself. Please let me know if it does not work.
I have a mapping like this
{
"experience": {
"type": "nested",
"properties": {
"end": {
"type": "string"
},
"organization": {
"type": "nested",
"properties": {
"details": {
"type": "string"
},
"name": {
"type": "string"
}
}
}
}
}
}
Now I want to make a query like this:
{
"nested": {
"path": "experience",
"query": {
"bool": {
"must": [{
"match": {
"experience.organization.name": {
"query": company_name,
"operator": "and"
}
}
}, {
"match": {
"experience.end": "Present"
}
}]
}
}
}
}
The above query is not returning any results, is this the correct way to index and query the above scenario?
I am confused about what should be the value of the path variable since organisation.name and end are not at the same level.
Here is a complete working sample with your code:
PUT index1/test1/_mapping
{
"test1": {
"properties": {
"experience": {
"type": "nested",
"properties": {
"end": {
"type": "string"
},
"organization": {
"type": "nested",
"properties": {
"details": {
"type": "string"
},
"name": {
"type": "string"
}
}
}
}
}
}
}
}
POST index1/test1
{
"experience": {
"end": "Present",
"organization": {
"name": "org1",
"details": "some details here"
}
}
}
GET index1/test1/_search
{
"query": {
"nested": {
"path": "experience",
"query": {
"bool": {
"must": [
{
"match": {
"end": "present"
}
},
{
"nested": {
"path": "experience.organization",
"query": {
"match": {
"name": "org1"
}
}
}
}
]
}
}
}
}
}
That being said, you have a double nested object here which you will probably find will work against you in the long run. I would consider flattening the data so that the nested is not necessary.