I need to filter the records closest to the given coordinates. From the below query removing script_fields will work (gives the results).
Need to get the distance for each matched results.
GET story/_search
{
"_source": [
"title.english", "location"
],
"query": {
"bool": {
"filter": [
{
"geo_distance": {
"distance": "1000km",
"location": {
"lat": 57.3079700,
"lon": 123.4977090
}
}
}
]
}
},
"script_fields": {
"distance": {
"script": "doc['location'].distanceInKm(57.3079700, 123.4977090)"
}
}
}
Below is the error
"failures" : [
{
"shard" : 1,
"index" : "story",
"node" : "asdf-asdf",
"reason" : {
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"doc['location'].distanceInKm(57.3079700, 123.4977090)",
" ^---- HERE"
],
"script" : "doc['location'].distanceInKm(57.3079700, 123.4977090)",
"lang" : "painless",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "dynamic method [org.elasticsearch.index.fielddata.ScriptDocValues.GeoPoints, distanceInKm/2] not found"
}
}
}
]
},
As #Sharath pointed out, distanceInKm is deprecated. These days you can use arcDistance and convert the value to km through dividing by 1000.
GET my-index-000001/_search
{
...
"script_fields": {
"distance": {
"script": "doc['location'].arcDistance(57.3079700, 123.4977090) / 1000"
}
}
}
Here's the list of currently supported geo methods and here's the arcDistance source.
Related
I'm trying to query ElasticSearch to match every document that in a list of list contains all the values requested, but I can't seem to find the perfect query.
Mapping:
"id" : {
"type" : "keyword"
},
"mainlist" : {
"properties" : {
"format" : {
"type" : "keyword"
},
"tags" : {
"type" : "keyword"
}
}
},
...
Documents:
doc1 {
"id" : "abc",
"mainlist" : [
{
"type" : "big",
"tags" : [
"tag1",
"tag2"
]
},
{
"type" : "small",
"tags" : [
"tag1"
]
}
]
},
doc2 {
"id" : "abc",
"mainlist" : [
{
"type" : "big",
"tags" : [
"tag1"
]
},
{
"type" : "small",
"tags" : [
"tag2"
]
}
]
},
doc3 {
"id" : "abc",
"mainlist" : [
{
"type" : "big",
"tags" : [
"tag1"
]
}
]
}
The query I've tried that got me closest to the result is:
GET /index/_doc/_search
{
"query": {
"bool": {
"must": [
{
"term": {
"mainlist.tags": "tag1"
}
},
{
"term": {
"mainlist.tags": "tag2"
}
}
]
}
}
}
although I get as result doc1 and doc2, while I'd only want doc1 as contains tag1 and tag2 in a single list element and not spread across both sublists.
How would I be able to achieve that?
Thanks for any help.
As mentioned by #caster, you need to use the nested data type and query as in normal way Elasticsearch treats them as object and relation between the elements are lost, as explained in offical doc.
You need to change both mapping and query to achieve the desired output as shown below.
Index mapping
{
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"mainlist" :{
"type" : "nested"
}
}
}
}
Sample Index doc according to your example, no change there
Query
{
"query": {
"nested": {
"path": "mainlist",
"query": {
"bool": {
"must": [
{
"term": {
"mainlist.tags": "tag1"
}
},
{
"match": {
"mainlist.tags": "tag2"
}
}
]
}
}
}
}
}
And result
hits": [
{
"_index": "71519931_new",
"_id": "1",
"_score": 0.9139043,
"_source": {
"id": "abc",
"mainlist": [
{
"type": "big",
"tags": [
"tag1",
"tag2"
]
},
{
"type": "small",
"tags": [
"tag1"
]
}
]
}
}
]
use nested field type，this is work for it
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/nested.html
Say I have a field, data.url. Some our logs contain this field, some do not. I want to return only results where data.url is more than, say, 50 characters long. Really I just need a list of URLs.
I'm trying:
GET _search
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "doc['data.url'].value.length() > 50",
"lang": "painless"
}
}
}
}
}
}
But get mixed errors:
{
"error" : {
"root_cause" : [
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
"doc['data.url'].value.length() > 50",
" ^---- HERE"
],
"script" : "doc['data.url'].value.length() > 50",
"lang" : "painless",
"position" : {
"offset" : 4,
"start" : 0,
"end" : 35
}
},
or
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.index.fielddata.ScriptDocValues$Strings.get(ScriptDocValues.java:496)",
"org.elasticsearch.index.fielddata.ScriptDocValues$Strings.getValue(ScriptDocValues.java:503)",
"doc['data.url'].value.length() > 50",
" ^---- HERE"
],
"script" : "doc['data.url'].value.length() > 50",
"lang" : "painless",
"position" : {
"offset" : 15,
"start" : 0,
"end" : 35
}
With
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "No field found for [data.url] in mapping with types []"
}
and sometimes
"caused_by" : {
"type" : "illegal_state_exception",
"reason" : "A document doesn't have a value for a field! Use doc[<field>].size()==0 to check if a document is missing a field!"
}
This field definitely exists; I can see it in the logs, search in the search field, and using term works:
GET _search
{
"query": {
"bool": {
"filter": {
"term": {
"data.url": "www.google.com"
}
}
}
}
}
What am I missing?
I'm using Elasticsearch 7.8.
Since you are using version 7.*, you need to use this below script query
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "doc['data.url.keyword'].length > 50",
"lang": "painless"
}
}
}
}
}
}
If data.url field is of keyword type, then ignore the ".keyword" at the end of the field
In elastic search index, how to get the documents where array field has length greater than 0?
I tried following multiple syntaxes but didn't get any breakthrough. I got same error in all of the syntaxes.
GET http://{{host}}:{{elasticSearchPort}}/student_details/_search
Syntax 1:
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "doc['enrolledCourses'].values.length > 0",
"lang": "painless"
}
}
}
}
}
}
Error:
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [enrolledCourses] in mapping with types []"
}
Syntax 2:
{
"query": {
"bool": {
"filter": {
"script": {
"script": {
"source": "doc['enrolledCourses'].values.size() > 0",
"lang": "painless"
}
}
}
}
}
}
Error:
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [enrolledCourses] in mapping with types []"
}
Syntax 3:
{
"query": {
"bool": {
"filter" : {
"script" : {
"script" : "doc['enrolledCourses'].values.size() > 0"
}
}
}
}
}
Error:
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [enrolledCourses] in mapping with types []"
}
Syntax 4:
{
"query": {
"bool": {
"filter" : {
"script" : {
"script" : "doc['enrolledCourses'].values.length > 0"
}
}
}
}
}
Error:
"caused_by": {
"type": "illegal_argument_exception",
"reason": "No field found for [enrolledCourses] in mapping with types []"
}
Please help me in solving this.
I don't know what version of elastic you run, then all my test I'd running on latest 7.9.0 version of Elasticsearch.
I will use painless script for scripting.
I put to documents to index test:
PUT test/_doc/1
{
"name": "Vasia",
"enrolledCourses" : ["test1", "test2"]
}
PUT test/_doc/2
{
"name": "Petya"
}
How you can see one document contains enrolledCourses field and second not.
In painless you don't need use values field and you can take length directly, this is according to painless documentation. Then I skip using values operator in my script:
GET test/_search
{
"query": {
"bool": {
"filter": [
{
"script": {
"script": {
"source": "doc['enrolledCourses'].length > 0",
"lang": "painless"
}
}
}
]
}
}
}
After running I'd received 2 different errors:
{
"type" : "script_exception",
"reason" : "runtime error",
"script_stack" : [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:757)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116)",
"org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:331)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:97)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:94)",
"java.base/java.security.AccessController.doPrivileged(AccessController.java:312)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:94)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)",
"doc['enrolledCourses'].length > 0",
" ^---- HERE"
]
}
and
{
"type" : "illegal_argument_exception",
"reason" : "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [enrolledCourses] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
Both of errors is pretty clear. First for document where field doesn't exists and second because Elasticsearch indexed string array field as default mapping type text.
Both of cases is very easy to fix by mapping enrolledCourses field as keyword.
In first case mapping will always provide empty field and in second keyword word be allow to run fielddata property.
PUT test
{
"settings": {
"number_of_replicas": 0
},
"mappings": {
"properties": {
"name": {
"type": "keyword"
},
"enrolledCourses": {
"type": "keyword"
}
}
}
}
Now I will receive right answer for query:
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 0.0,
"hits" : [
{
"_index" : "test",
"_type" : "_doc",
"_id" : "1",
"_score" : 0.0,
"_source" : {
"name" : "Vasia",
"enrolledCourses" : [
"test1",
"test2"
]
}
}
]
}
}
I have the following mapping:
"fullName" : {
"type" : "text",
"norms" : false,
"similarity" : "boolean",
"fields" : {
"raw" : {
"type" : "keyword"
},
"terms" : {
"type" : "token_count",
"analyzer" : "standard"
}
}
}
I want to display the value of terms field. When I do the following, I get the fullName but not the terms value
GET /_search
{"_source": ["fullName","fullName.terms"],
"query": {
"bool" : {
"must" : {
"script" : {
"script" : {
"source": "doc['fullName.terms'].value != 3,
"lang": "painless"
}
}
}
}
}
}
How can I get it?
You need to configure that your token count is stored - Here documentation
You should modify your mapping :
"terms" : {
"type" : "token_count",
"analyzer" : "standard",
"store": true
}
Then to retrive the value you need to explicitly ask for stored value in your query : ( here documentation )
GET /_search
{
"_source": [
"fullName"
],
"stored_fields": [
"fullName.terms"
],
"query": {
"bool": {
"must": {
"script": {
"script": {
"source": "doc['fullName.terms'].value != 3",
"lang": "painless"
}
}
}
}
}
}
I want to express this SQL in Elasticsearch:
select * from ticket where user_id = 1 and (class_a = 1000010 or class_b = 16);
I use a combining filter as below:
curl 'localhost:9200/ticket/_search?pretty' -d'
{
"query": {
"bool": {
"should": [
{"term": {"class_a": 1000010}},
{"term": {"class_b": 16}}
]
},
"filter": {
"term": {
"user_id": 1
}
}
}
}'
but got the error as below:
{
"error" : {
"root_cause" : [ {
"type" : "parse_exception",
"reason" : "failed to parse search source. expected field name but got [START_OBJECT]"
} ],
"type" : "search_phase_execution_exception",
"reason" : "all shards failed",
"phase" : "query_fetch",
"grouped" : true,
"failed_shards" : [ {
"shard" : 0,
"index" : "ticket",
"node" : "FO3-zhb1R1WCak381t88gQ",
"reason" : {
"type" : "parse_exception",
"reason" : "failed to parse search source. expected field name but got [START_OBJECT]"
}
} ]
},
"status" : 400
}
Anyone can help me? Thanks in advance!
You're almost there, you need to rewrite your query like this (i.e. move your filter inside the bool clause):
curl 'localhost:9200/ticket/_search?pretty' -d'{
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"term": {
"class_a": 1000010
}
},
{
"term": {
"class_b": 16
}
}
],
"filter": {
"term": {
"user_id": 1
}
}
}
}
}'