Using Elasticsearch 7.*, we have a field 'ElapsedTime' under the mapping and I am trying to write a query to generate output of that field as 'ElapsedTime' / 1000.
Tried below but no luck:
1)
GET /_search
{
"script_fields": {
"test1": {
"script": {
"lang": "painless",
"source": "params._source.ElapsedTime / 1000"
}
}
}
}
GET /_search
{
"script_fields": {
"test2": {
"script": {
"lang": "expression",
"source": "doc['ElapsedTime'] / 1000"
}
}
}
}
Errors:
null pointer exception
parse_exception: Field [ElapsedTime] does not exist in mappings
You need to run GET concrete-index/_search on a concrete-index and not on / which runs on all indexes of your cluster, where the chance of hitting an index which doesn't have ElapsedTime in its mapping is quite big.
Related
I am working on a Elasticsearch project. I want to get an additional column in response when an index is queried. Say for example, if I have an index with two columns num1 and num2, when this index is queried it should respond with two column (num1 and num2) but also with additional column add_result (which is actually a addition of two columns). If I query it normally like below it would respond with just two columns
{
query:{
match_all : {}
}
}
In my use case I have tried:
{
"runtime_mappings": {
"add_result": {
"type": "double",
"script": "emit(doc['file_count'].value + doc['follower_count'].value)"
}
},
"query": {
"match_all": {}
}
}
Yes, there are 2 ways:
1. Using runtime field
This feature is available since Elasticsearch 7.12. Simply make a GET request to the _search endpoint with the request body like this:
{
"runtime_mappings": {
"add_result": {
"type": "double",
"script": "emit(doc['num1'].value + doc['num2'].value)"
}
},
"fields": ["add_result", "num*"],
"query": {
"match_all": {}
}
}
You need to explicitly specify that you want to get your runtime fields back in the fields parameter.
2. Using script_field
The request looks like this:
{
"query": {
"match_all": {}
},
"fields": ["num*"],
"script_fields": {
"add_result": {
"script": {
"lang": "painless",
"source": "doc['num1'].value + doc['num2'].value"
}
}
}
}
Note that you still need to have the fields parameter, but you don't need to include your script_field (add_result in this case) in the fields parameter.
I'm using cosineSimilarity in elasticsearch for searching documents and the query looks like the following:
{
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "cosineSimilarity(params.queryVector, 'title_vector') + 1.0",
"params": {
"queryVector": list(feat)
}
}
}
}}
The issue here is that I'll be getting all the results despite the similarity score. I want to filter my results based on a threshold filter value.
I tried using bool with following script:
query = {
"query": {
"bool" : {
"must": {
"match_all": {}
},
"filter" : {
"script" : {
"source": "cosineSimilarity(params.queryVector, 'title_vector') + 1.0 > 1.4",
"params": {
"queryVector": list(feat)
}
}
}
}
}
}
But this throws an error:
RequestError(400, 'x_content_parse_exception', '[source] query malformed, no start_object after query name')
From Text similarity search with vector fields
Important limitations
The script_score query is designed to wrap a restrictive query, and modify the scores of the documents it returns. However, we’ve provided a match_all query, which means the script will be run over all documents in the index. This is a current limitation of vector similarity in Elasticsearch — vectors can be used for scoring documents, but not in the initial retrieval step. Support for retrieval based on vector similarity is an important area of ongoing work.
EDIT
Adding min_score to the request will filter out based on the calculated score after doing the match_all.
{
"min_score": 1.4,
"query": {
"script_score": {
"query": {
"match_all": {}
},
"script": {
"source": "cosineSimilarity(params.queryVector, 'title_vector') + 1.0",
"params": {
"queryVector": list(feat)
}
}
}
}
}
This behavior of running script with elastic search is quite puzzling to me. I am running a script to filter by time. I have got these methods from this elastic documentation.
This script with HoursOfDay works (date.hourOfDay)
{
"script": {
"script": {
"lang": "expression",
"source": "doc['#timestamp'].date.hourOfDay >= min && doc['#timestamp'].date.hourOfDay <= max",
"params": {
"min": 5,
"max": 8
}
}
}
}
Whereas, the script with Minutes doesnt return any results. (date.minuteOfDay)
{
"script": {
"script": {
"lang": "expression",
"source": "doc['#timestamp'].date.minuteOfDay >= min && doc['#timestamp'].date.minuteOfDay <= max",
"params": {
"min": 300,
"max": 480
}
}
}
}
Also i have tried few other way from this elastic document by using "date.getHourOfDay()" and "date.getMinutes()", and found similar behavior. getHourOfDay returns results and getMinutes() doesnt.
Am I missing something here or is there a specific configuration at the elastic level to enable this method. Thanks.
I think in source you can try params.min and params.max instead min and max respectively
I have a mapping like this:
{
printings: {
type: "nested",
properties: {
prop1: {type: "number"}
}
},
prop2: {type: "number"}
}
I then want to build a Painless query like this:
"script": {
"lang": "painless",
"inline": "doc['prop1'] > (3 * doc['printings.prop2'])"
}
However testing this in Sense doesn't work. If I replace the nested prop2 with a simple number then it does work. Is there a way to access both root and nested props in a single scripted query?
You can try below query.
{
"query": {
"script": {
"script": {
"lang": "painless",
"inline": "params['_source']['prop1'] > (2 * params['_source']['printings']['prop2'])"
}
}
}
}
But please keep this thing mind that _source is very slow. read more about here
Unfortunately, you cannot access nested context from root and you cannot access root context from nested because nested documents are separate documents, even though they are stored close to the parent. But you can solve it with a different mapping using copy_to field feature. Here is a mapping:
{
"mappings": {
"sample": {
"properties": {
"printings": {
"type": "nested",
"properties": {
"prop2": {
"type": "integer",
"copy_to": "child_prop2"
}
}
},
"prop1": {
"type": "integer"
},
"child_prop2": {
"type": "integer"
}
}
}
}
}
In this case the values from nested documents will be copied to the parent. You don't have to explicitly fill this new field, here is ax example of bulk indexing:
POST http://localhost:9200/_bulk HTTP/1.1
{"index":{"_index":"my_index","_type":"sample","_id":null}}
{"printings":[{"prop2":1},{"prop2":4}],"prop1":2}
{"index":{"_index":"my_index","_type":"sample","_id":null}}
{"printings":[{"prop2":0},{"prop2":1}],"prop1":2}
{"index":{"_index":"my_index","_type":"sample","_id":null}}
{"printings":[{"prop2":1},{"prop2":0}],"prop1":2}
After that you can use this query
{
"query": {
"script": {
"script": {
"inline": "doc['prop1'].value > (3 * doc['child_prop2'].value)",
"lang": "painless"
}
}
}
}
The first document won't match. The second one will match by the first subdocument. The third one will match by the second subdocument.
I have mapping of one filed is as follows in ES2.3
"move_in_ts": {
"type": "integer"
}
"move_out_ts": {
"type": "integer"
}
Sample document stores data as follows:
"move_in_ts": [
1475280000,
1475539200,
1475712000,
1475884800,
1477008000,
1477785600
]
I have a script in my DSL query (trying to find an integer in that array)
"script": "if(doc['move_in_ts'] && doc['move_in_ts'].values.contains('1475280000')){return 200;}"
and also tried this:
"script": "if(doc['move_in_ts'] && doc['move_in_ts'].contains('1475280000')){return 200;}"
and also tried this:
"script": "if(doc['move_in_ts'] && doc['move_in_ts'].contains(1475280000)){return 200;}"
and also tried this:
"script": "if(doc['move_in_ts'] && doc['move_in_ts'].values.contains(1475280000)){return 200;}"
but in all above cases, I get the following error:
"reason": {
"type": "null_pointer_exception",
"reason": null
}
It might be possible that this field doesn't exist at all in few documents (I cannot use filter in my use case, I need to have it in the script only)
What am I doing wrong or how to get it work?
I am not able to reproduce the issue(I am also using ES 2.3). You would also need toLong() to get the right results or it will give zero results. I created sample index like this.
PUT books
{
"mappings": {
"book":{
"properties": {
"move_in_ts":{
"type": "integer"
}
}
}
}
}
I indexed few docs
POST books/book
{
"move_in_ts" : null
}
POST books/book
{
"move_in_ts" : [4,null]
}
POST books/book
{
"move_in_ts" : []
}
POST books/book
{
"some_other_field" : "some value"
}
POST books/book
{
"move_in_ts" : [
1475280000,
1475539200,
1475712000,
1475884800,
1477008000,
1477785600
]
}
Then below query gives right result
GET books/book/_search
{
"query": {
"bool": {
"filter": {
"script": {
"script": "if(doc['move_in_ts'] && doc['move_in_ts'].values.contains(1475280000.toLong())){return 200;}"
}
}
}
}
}