Here I need to fetch the key, value pair of particular index based on the some conditions.
I need only few fields in that index doc not all fields
I need only fields which i given in input.
GET /_search?&pretty=true&size=3
{
"query": {
"query_string": {
"query": "countryCode:SA AND serviceName:SMS",
"fields": ["level","cause","to"]
}
}
}
Output:
{
"took": 2854,
"timed_out": false,
"num_reduce_phases": 4,
"_shards": {
"total": 1891,
"successful": 1891,
"failed": 0
},
"hits": {
"total": 14032,
"max_score": 8.429943,
"hits": [
{
"_index": "postman-2019.01.21",
"_type": "syslog",
"_id": "AWhvN1KDl97BCeGFfgpe",
"_score": 8.429943,
"_source": {
"eId": "346589962",
"level": "info",
"prevStatus": "SUCCESS",
"cause": "SUCCESS",
"serviceName": "SMS",
"loggingAction": "SMS_CALLBACK_REPORTS",
"application": "POSTMAN",
"countryCode": "SA",
"client": "CRS",
"to": "+966572444531",
"externalServiceName": "gupshupInternationalChannelA",
"time": "Mon Jan 21 07:02:02 UTC 2019",
"category": "OTP",
"dId": "3762647059352507724-309778596902014991",
"uIdentifier": "2e262115-d09c-4bef-a04b-d0860d064930",
"epochTime": 1548054122491,
"status": "DELIVERED",
"#version": "1",
"#timestamp": "2019-01-21T07:02:03.124Z",
"path": "/mnt/logs/logstash",
"host": "ip-10-20-10-164",
"type": "syslog"
}
}
]
}
}
this is output i got, but i need only fields which i given in input
Here is what you can try :
POST cars/_search
{
"_source": {
"includes": [ "price_eur", "stk_year" ],
"excludes": [ "mileage" ]
},
"query" : {
"match" : { "maker" : "audi" }
}
}
POST cars/_search?&pretty=true&size=3
{
"_source": {
"includes": [ "price_eur", "stk_year" ],
"excludes": [ "mileage" ]
},
"query" : {
"match" : { "maker" : "audi" }
}
}
Related
I have an index, which stores a nested document. I wanna see this nested documents, for this purpose I used 'inner_hits' in request, but elastic returns nullPointerException. Do anyone meet with this problem?)
Request to elasticsearch using Postman:
GET http://localhost/my-index/_search
{
"query": {
"nested": {
"path": "address_object",
"query": {
"bool": {
"must": {
"term": {"address_object.city": "Paris"}
}
}
},
"inner_hits" : {}
}
}
}
Response with status code 200:
{
"took": 161,
"timed_out": false,
"_shards": {
"total": 2,
"successful": 1,
"skipped": 0,
"failed": 1,
"failures": [
{
"shard": 0,
"index": "my-index",
"node": "DWdD83KaTmUiodENQkGDww",
"reason": {
"type": "null_pointer_exception",
"reason": null
}
}
]
},
"hits": {
"total": 6500039,
"max_score": 2.1761138,
"hits": []
}
}
Elasticsearch version: 6.2.4
Lucene version: 7.2.1
Update:
Mapping:
{
"my-index": {
"mappings": {
"mytype": {
"dynamic": "false",
"_source": {
"enabled": false
},
"properties": {
"adverts_count": {
"type": "integer",
"store": true
},
...
"address_object": {
"type": "nested",
"properties": {
"adverts_count": {
"type": "integer",
"store": true
},
"city": {
"type": "keyword",
"store": true
}
}
},
...
Sample document:
{
"_index": "my-index",
"_type": "mytype",
"_id": "XDWrGncBdwNBWGEagAM2",
"_score": 2.1587489,
"fields": {
"is_target_page_shown": [
0
],
"updated_at": [
1612264276
],
"is_shown": [
0
],
"nb_queries": [
1
],
"search_query": [
"phone"
],
"target_category": [
15
],
"adverts_count": [
1
]
}
}
Extra information:
If I remove the "inner_hits": {} from search request, elastic returns nested documents(_index, _type, _id, _score), but ain't other fields(e.g city)
Also, as suggested in the comments, I tried setting to true ignore_unmapped, but it doesn't helped. The same nullPointerException.
I tried reproducing your issue, but as you have not provided the proper sample documents(one which you provided doesn't have the address_object properties), I used your mapping and below sample documents.
PUT index-name/_doc/1
{
"address_object" :{
"adverts_count" : 1,
"city": "paris"
}
}
PUT index-name/_doc/2
{
"address_object" :{
"adverts_count" : 1,
"city": "blr"
}
}
And when I use the same search provided by you.
POST 71907588/_search
{
"query": {
"nested": {
"path": "address_object",
"query": {
"bool": {
"must": {
"term": {
"address_object.city": "paris"
}
}
}
},
"inner_hits": {}
}
}
}
I get a proper response, matching paris as city as shown in the search response.
"hits": [
{
"_index": "71907588",
"_id": "1",
"_score": 0.6931471,
"_source": {
"address_object": {
"adverts_count": 1,
"city": "paris"
}
},
"inner_hits": {
"address_object": {
"hits": {
"total": {
"value": 1,
"relation": "eq"
},
"max_score": 0.6931471,
"hits": [
{
"_index": "71907588",
"_id": "1",
"_nested": {
"field": "address_object",
"offset": 0
},
"_score": 0.6931471,
"_source": {
"city": "paris",
"adverts_count": 1
}
}
]
}
}
}
}
]
Elasticsearch: 6.5.4
Issue: I'm executing a bool query (sample to follow) where I'm checking for the existence of a specific field. The issue is, I'm getting results back where the field does exist but has an empty array.
My question is, how do I properly execute a query and only get results where nlp is not added to the document at all.
Sample query:
{
"size": 100,
"sort": [{
"publishedAt": {
"order": "asc"
}
}],
"_source": {
"includes": ["nlp"]
},
"query": {
"bool": {
"must_not": {
"exists": {
"field": "nlp.categories.gcp"
}
}
}
}
}
Sample Mapping:
(This was automatically created by Elastic Search, with the exception of the null_value, I tried adding that).
{
"mapping": {
"article": {
"properties": {
"nlp": {
"properties": {
"categories": {
"properties": {
"gcp": {
"properties": {
"confidence": {
"type": "float"
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"null_value": "[]",
"ignore_above": 256
}
}
}
}
}
}
}
}
}
}
}
}
}
Sample Result:
{
"took": 68,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1126581,
"max_score": null,
"hits": [
{
"_index": "news",
"_type": "article",
"_id": "UTuVmmsBE1H01hY9Rn6i",
"_score": null,
"_source": {
"nlp": {
"categories": {
"gcp": []
}
}
},
"sort": [
1509940860000
]
},
{
"_index": "news",
"_type": "article",
"_id": "2w6PmmsBIpi-jAhhO13F",
"_score": null,
"_source": {
"nlp": {
"categories": {
"gcp": []
}
}
},
"sort": [
1510027260000
]
}
]
}
}
When the nlp.categories.gcp has values in it, a typical response would look like this.
{
"took": 26,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 475690,
"max_score": null,
"hits": [
{
"_index": "news",
"_type": "article",
"_id": "6Q6JmmsBIpi-jAhhAlcm",
"_score": null,
"_source": {
"nlp": {
"categories": {
"gcp": [
{
"confidence": 0.8999999761581421,
"name": "/Travel/Hotels & Accommodations"
}
]
}
}
},
"sort": [
1510215565000
]
},
{
"_index": "news",
"_type": "article",
"_id": "rzunmmsBE1H01hY9sLyE",
"_score": null,
"_source": {
"nlp": {
"categories": {
"gcp": [
{
"confidence": 0.9399999976158142,
"name": "/Travel/Hotels & Accommodations"
}
]
}
}
},
"sort": [
1510228881000
]
}
]
}
}
I have my mappings as below and I am doing a bool should query on name and other properties as shown below but what I need is that I want to filter CustomerPrices by CustomerId on response.
Each products have same CustomerIds so for eaxample;
product1 -CustomerPrice( CustomerId :1234 -Price:4)
CustomerPrice( CustomerId :567-Price:5)
.
.
Product2 - CustomerPrice(CustomerId :1234 -Price:8)
CustomerPrice(CustomerId :567-Price:10)
.
.
So according to that when I query Product1, response should have only customerPrice for customerId:1234
{
"Product": {
"properties": {
"CustomerPrices": {
"type": "nested",
"properties": {
"Price": {
"store": true,
"type": "float"
},
"CustomerId": {
"type": "integer"
}
}
},
"Name": {
"index": "not_analyzed",
"store": true,
"type": "string"
}
}
}
}
I tried following query but this is not filtering nested objects. I guess it filters product objects as it makes sense because all products have customerId:1234
"query":{
"bool":{
"should":[
{
"multi_match":{
"type":"best_fields",
"query":"product 1",
"fields":[
"Name^7"]
}
},
{
"multi_match":{
"type":"best_fields",
"query":"product 1",
"operator":"and",
"fields":[
"Code^10",
"ShortDescription^6"]
}
},
{
"nested":{
"query":{
"term":{
"CustomerPrices.CustomerId":{
"value":1234
}
}
},
"path":"CustomerPrices"
}
}]
}
},
I've spent some time on your question since it was interesting how this can be achieved and the only solution I found for now is relying on the inner_hits which gives the exact nested object the match was on. I've also deactivated the _source which isn't used anymore.
So given your mapping and having 2 products like:
PUT product/Product/product1
{
"CustomerPrices": [
{
"CustomerId": 1234,
"Price": 4
},
{
"CustomerId": 567,
"Price": 5
}
],
"Name": "John"
}
PUT product/Product/product2
{
"CustomerPrices": [
{
"CustomerId": 1234,
"Price": 8
},
{
"CustomerId": 567,
"Price": 10
}
],
"Name": "Bob"
}
When running the following query: (Used must just to see 1 result, works with should as well)
GET product/_search
{
"_source": false,
"query": {
"bool": {
"must": [
{ "match": { "Name": "Bob"}}
],
"filter": [
{
"nested" : {
"path" : "CustomerPrices",
"score_mode" : "avg",
"query" : {
"bool" : {
"should" : [
{ "match" : {"CustomerPrices.CustomerId" : 1234}}
]
}
},
"inner_hits": {}
}
}
]
}
}
}
I was able to get the result where only "Price" from customer with id 1234 was present:
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.2876821,
"hits": [
{
"_index": "product",
"_type": "Product",
"_id": "product2",
"_score": 0.2876821,
"inner_hits": {
"CustomerPrices": {
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "product",
"_type": "Product",
"_id": "product2",
"_nested": {
"field": "CustomerPrices",
"offset": 0
},
"_score": 1,
"_source": {
"CustomerId": 1234,
"Price": 8
}
}
]
}
}
}
}
]
}
}
Couldn't find an official way of returning partial results of the document by only having the matched nested object. Maybe something that we need to inform elasticsearch guys about to consider for some next releases. Hope it helps you.
I am trying to do a search within elasticsearch using the regexp filters. Following is my query:
{
"from": 0,
"size": 10,
"_source":["CODE"],
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"regexp" : {
"CODE" : {
"value" : "[0]?[0]?[0]?[0]?3410086456[0-9]?",
"flags_value" : 0,
"boost" : 20.0
}
}
},
{
"regexp" : {
"CODE" : {
"value" : "[0]?[0]?[0]?[0]?83560900204[0-9]?",
"flags_value" : 0,
"boost" : 20.0
}
}
}
]
}
},
{
"terms": {
"CODETYPE": [
"TYPE1", "TYPE2", "TYPE3"
]
}
}
]
}
}
}
Below is the result of the query:
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 20.091797,
"hits": [
{
"_index": "index1",
"_type": "type1",
"_id": "142242",
"_score": 20.091797,
"_source": {
"CODE": "003410086456"
}
},
{
"_index": "index1",
"_type": "type1",
"_id": "375897",
"_score": 20.091797,
"_source": {
"CODE": "083560900204"
}
}
]
}
}
What I need to get additionally in my output is the input term against which each result has matched. Something like this:
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 20.091797,
"hits": [
{
"_index": "index1",
"_type": "type1",
"_id": "142242",
"_score": 20.091797,
"_source": {
"CODE": "003410086456",
"INPUT": "3410086456"
}
},
{
"_index": "index1",
"_type": "type1",
"_id": "375897",
"_score": 20.091797,
"_source": {
"CODE": "083560900204",
"INPUT": "83560900204"
}
}
]
}
}
Notice the additional INPUT field above. That way I can map what pattern has mapped to which result. Is there any possibility in elasticsearch I can do this? I am currently unable to find any way of achieving this.
Appreciate your help on this. Let me know if I need to furnish any more information.
you could use highlighting, though it won't in _source, it would create a separate field highlight which gives the field value.
{
"from": 0,
"size": 10,
"_source": [
"CODE"
],
"query": {
"bool": {
"must": [
{
"bool": {
"should": [
{
"regexp": {
"CODE": {
"value": "[0]?[0]?[0]?[0]?3410086456[0-9]?",
"flags_value": 0,
"boost": 20
}
}
},
{
"regexp": {
"CODE": {
"value": "[0]?[0]?[0]?[0]?83560900204[0-9]?",
"flags_value": 0,
"boost": 20
}
}
}
]
}
},
{
"terms": {
"CODETYPE": [
"TYPE1",
"TYPE2",
"TYPE3"
]
}
}
]
}
},
"highlight": {
"fields": {
"CODE": {}
}
}
}
Refer: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-highlighting.html#search-request-highlighting
I've read some articles and documentation about queries/filters in nested objects but I can not make this sample work. Hopefully you can help me to check what is wrong. Bellow is the index and mapping settings I am using:
# Create Index
PUT agency
# Mapping
PUT agency/site/_mapping
{
"site": {
"properties": {
"name":{
"type":"string"
},
"phones": {
"type": "nested",
"properties":{
"is_confidential": { "type": "string" },
"number": { "type": "string" },
"description": {"type" : "string"}
}
}
}
}
}
# Indexing one document
PUT agency/site/1
{
"site":{
"name":"Burger Queen",
"phones":[
{
"is_confidential":"true",
"number":"10000000000",
"description":"Manager Phone"
},
{
"is_confidential":"false",
"number":"10000000001",
"description":"Public Line"
},
{
"is_confidential":"false",
"number":"10000000002",
"description":"Public Line 2"
},
{
"is_confidential":"false",
"number":"10000000003",
"description":"Complains Phone"
}
]
}
}
# Query the nested document (https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-query.html)
GET /agency/site/_search
{
"query": {
"bool": {
"must": [
{ "match": { "site.name": "Burger" }},
{
"nested": {
"path": "phones",
"query": {
"bool": {
"must": [
{ "match": { "phones.is_confidential": "false" }}
]
}}}}
]
}}}
# Results
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
}
}
Why I can not get any results ?
What I am trying to do is filter the parent documents by some term, for example: name = Burger and also filter nested documents to get only those phones where is_confidential = false.
Sample results without any filter applied in nested documents:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "agency",
"_type": "site",
"_id": "1",
"_score": 1,
"_source": {
"site": {
"name": "Burger Queen",
"phones": [
{
"is_confidential": "true",
"number": "10000000000",
"description": "Manager Phone"
},
{
"is_confidential": "false",
"number": "10000000001",
"description": "Public Line"
},
{
"is_confidential": "false",
"number": "10000000002",
"description": "Public Line 2"
},
{
"is_confidential": "false",
"number": "10000000003",
"description": "Complains Phone"
}
]
}
}
}
]
}
}
If the site contains several nested objects in the phones array, only those phones that are not confidential should be returned by elasticsearch.
Sample result when is_confidential = false filter is applied:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "agency",
"_type": "site",
"_id": "1",
"_score": 1,
"_source": {
"site": {
"name": "Burger Queen",
"phones": [
{
"is_confidential": "false",
"number": "10000000001",
"description": "Public Line"
},
{
"is_confidential": "false",
"number": "10000000002",
"description": "Public Line 2"
},
{
"is_confidential": "false",
"number": "10000000003",
"description": "Complains Phone"
}
]
}
}
}
]
}
}
Sample result when is_confidential = true:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "agency",
"_type": "site",
"_id": "1",
"_score": 1,
"_source": {
"site": {
"name": "Burger Queen",
"phones": [
{
"is_confidential": "true",
"number": "10000000000",
"description": "Manager Phone"
}
]
}
}
}
]
}
}
Is it possible to get those sample results shown above with elasticsearch nested filter (queries)? If is possible could you please show me sample?
You need to update your PUT clause
Indexing one document
PUT agency/site/1
{
"site":{ // <-- need to remove this as it will alter the mapping definition
"name":"Burger Queen",
"phones":[
{
"is_confidential":"true",
"number":"10000000000",
"description":"Manager Phone"
},
{
"is_confidential":"false",
"number":"10000000001",
"description":"Public Line"
},
{
"is_confidential":"false",
"number":"10000000002",
"description":"Public Line 2"
},
{
"is_confidential":"false",
"number":"10000000003",
"description":"Complains Phone"
}
]
}
}
GET agency/site/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"name": "burger"
}
},
{
"nested": {
"path": "phones",
"query": {
"term": {
"phones.is_confidential": {
"value": "true"
}
}
},
"inner_hits":{}
}
}
]
}
}
}
The matching nested-documents will be present in the inner hits response.
Sample Response :
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 2.2019112,
"hits": [
{
"_index": "agency",
"_type": "site",
"_id": "1",
"_score": 2.2019112,
"_source": {
"name": "Burger Queen",
"phones": [
{
"is_confidential": "true",
"number": "10000000000",
"description": "Manager Phone"
},
{
"is_confidential": "false",
"number": "10000000001",
"description": "Public Line"
},
{
"is_confidential": "false",
"number": "10000000002",
"description": "Public Line 2"
},
{
"is_confidential": "false",
"number": "10000000003",
"description": "Complains Phone"
}
]
},
"inner_hits": {
"phones": {
"hits": {
"total": 1,
"max_score": 1.9162908,
"hits": [
{
"_index": "agency",
"_type": "site",
"_id": "1",
"_nested": {
"field": "phones",
"offset": 0
},
"_score": 1.9162908,
"_source": {
"is_confidential": "true",
"number": "10000000000",
"description": "Manager Phone"
}
}
]
}
}
}
}
]
}
}