Trying to update a nested geoip location field in elasticsearch - elasticsearch

Here is what I've tried:
POST orders/_update_by_query
{
"script" : "ctx._source.geoip += newElement",
"params": {
"newElement": {
"location" : "[40.730610, -73.935242]"
}
},
"query": {
"term": {
"CITY": {
"value": "nyc"
}
}
}
}
The above throws error Unknown key for a START_OBJECT in [params].
Second Attempt:
POST orders/_update_by_query
{
"script":{
"source":
"for (item in ctx._source.geoip){item.location = '[40.730610, -73.935242]'}",
"lang":"painless"
},
"query": {
"term": {
"CITY": {
"value": "nyc"
}
}
}
}
The above throws null pointer exception, and points to the period at source.geoip
I also tried changing the value of location to just test but receive the same errors.
Here is my mapping:
{
"orders" : {
"mappings" : {
"properties" : {
"geoip" : {
"dynamic" : "true",
"properties" : {
"location" : {
"type" : "geo_point"
}
}
}
}
}
I am using ES v7.2 and Kibana v7.2

A couple of issues in the 1st approach:
params need to be defined within the script object, not below it
newElement needs to be accessed using params.newElement
you cannot append += params.newElement to a nonexistent ctx._source.geoip
you cannot append an object to a single-value field -- you can just assign it
location is of the geo_point type, so either [40.730610, -73.935242] ([lon, lat]) or "-73.935242,40.730610" ("lat,lon"), but not a mixture of both
Working command:
POST orders/_update_by_query
{
"script": {
"inline": "ctx._source.geoip = params.newElement",
"params": {
"newElement": {
"location": [
40.73061,
-73.935242
]
}
}
},
"query": {
"term": {
"CITY": {
"value": "nyc"
}
}
}
}

Related

How does Elasticsearch store a float value into an keyword field

I have created this index:
{
"users" : {
"mappings" : {
"properties" : {
"user_id" : {
"type" : "keyword"
}
}
}
}
}
and I added this entry:
PUT users/_doc/1
{
"user_id": 4.0000
}
When I query like this, I always can get this entry
GET /_search
{
"query": {
"term": {
"user_id": {
"value": 4.0, // not exact same value as I put in
"boost": 1.0
}
}
}
}
I wonder why this happens.

ES query to match all elements in array

So I got this document with a
nested array that I want to filter with this query.
I want ES to return all documents where all items have changes = 0 and that only.
If document has even a single item in the list with a change = 1, that's discarded.
Is there any way I can achieve this starting from the query I have already wrote? Or should I use a script instead?
DOCUMENTS:
{
"id": "abc",
"_source" : {
"trips" : [
{
"type" : "home",
"changes" : 0
},
{
"type" : "home",
"changes" : 1
}
]
}
},
{
"id": "def",
"_source" : {
"trips" : [
{
"type" : "home",
"changes" : 0
},
{
"type" : "home",
"changes" : 0
}
]
}
}
QUERY:
GET trips_solutions/_search
{
"query": {
"bool": {
"must": [
{
"term": {
"id": {
"value": "abc"
}
}
},
{
"nested": {
"path": "trips",
"query": {
"range": {
"trips.changes": {
"gt": -1,
"lt": 1
}
}
}
}
}
]
}
}
}
EXPECTED RESULT:
{
"id": "def",
"_source" : {
"trips" : [
{
"type" : "home",
"changes" : 0
},
{
"type" : "home",
"changes" : 0
}
]
}
}
Elasticsearch version: 7.6.2
Already read this answers but they didn't help me:
https://discuss.elastic.co/t/how-to-match-all-item-in-nested-array/163873
ElasticSearch: How to query exact nested array
First off, if you filter by id: abc, you obviously won't be able to get id: def back.
Second, due to the nature of nested fields which are treated as separate subdocuments, you cannot query for all trips that have the changes equal to 0 -- the connection between the individual trips is lost and they "don't know about each other".
What you can do is return only the trips that matched your nested query using inner_hits:
GET trips_solutions/_search
{
"_source": "false",
"query": {
"bool": {
"must": [
{
"nested": {
"inner_hits": {},
"path": "trips",
"query": {
"term": {
"trips.changes": {
"value": 0
}
}
}
}
}
]
}
}
}
The easiest solution then is to dynamically save this nested info on a parent object like discussed here and using range/term query on the resulting array.
EDIT:
Here's how you do it using copy_to onto the doc's top level:
PUT trips_solutions
{
"mappings": {
"properties": {
"trips_changes": {
"type": "integer"
},
"trips": {
"type": "nested",
"properties": {
"changes": {
"type": "integer",
"copy_to": "trips_changes"
}
}
}
}
}
}
trips_changes will be an array of numbers -- I presume they're integers but more types are available.
Then syncing a few docs:
POST trips_solutions/_doc
{"trips":[{"type":"home","changes":0},{"type":"home","changes":1}]}
POST trips_solutions/_doc
{"trips":[{"type":"home","changes":0},{"type":"home","changes":0}]}
And finally querying:
GET trips_solutions/_search
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "trips",
"query": {
"term": {
"trips.changes": {
"value": 0
}
}
}
}
},
{
"script": {
"script": {
"source": "doc.trips_changes.stream().filter(val -> val != 0).count() == 0"
}
}
}
]
}
}
}
Note that we first filter normally using the nested term query to narrow down our search context (scripts are slow so this is useful). We then check if there are any non-zero changes in the accumulated top-level changes and reject those that apply.

Elasticsearch search template not working with terms query

I have an index with a long field and i am just trying to use search template to use terms query but it throws exception.
"pid": {
"type": "long"
}
Search Template:
PUT /_search/template/article_query_template
{
"template": {
"query": {
"terms": {
"pid": "{{articleId}}"
}
}
}
}
Search Query :
POST test2*/_search
{
"query": {
"template": {
"id": "article_query_template",
"params" : {
"articleId" : ["1"]
}
}
}
}
Exception : reason": "[terms] query does not support [pid]".
Its working without template. How to fix this issue.
Create your template like :
{
"source": {
"query": {
"term": {
"message": "{{query_string}}"
}
}
}
}
and pass params like :
{
"id": "<templateName>",
"params": {
"query_string": "search for these words"
}}
More details are available here.

Querying Nested JSON based on 1 term value

I have indexed JSON like below format
JSON:
{"work":[{"organization":"abc", end:"present"},{"organization":"edf", end:"old"}]}
{"work":[{"organization":"edf", end:"present"},{"organization":"abc", end:"old"}]}
I want to query records where organization is "abc" and end is "present"
but below query is not working
work.0.organization: "abc" AND work.0.end:"present"
No records are matched
if I give query like below
work.organization: "abc" AND work.end:"present"
Both the records are matched. Whereas only the first record is what I want
The matched record should be only the below
{"work":[{"organization":"abc", end:"present"},{"organization":"edf", end:"old"}]}
You have to use nested_types. First map work as nested type in elastic using following mappings
PUT index_name_3
{
"mappings": {
"document_type" : {
"properties": {
"work" : {
"type": "nested",
"properties": {
"organization" : {
"type" : "text"
},
"end" : {
"type" : "text"
}
}
}
}
}
}
}
Use the following query to do nested filter match and innerhits
{
"query": {
"nested": {
"path": "work",
"inner_hits": {},
"query": {
"bool": {
"must": [{
"term": {
"work.organization": {
"value": "abc"
}
}
},
{
"term": {
"work.end": {
"value": "present"
}
}
}
]
}
}
}
}
}

Is it possible to update nested field by query?

I am using update by query plugin (https://github.com/yakaz/elasticsearch-action-updatebyquery/) to update documents by query.
In my case, there is nested field in document, the mapping is something like this:
"mappings": {
"mytype": {
"properties": {
"Myfield1": {
"type": "nested",
"properties": {
"field1": {
"type": "string"
},
"field2": {
"type": "long"
}
}
},
"Title": {
"type": "string"
}
}
}
}
Then I want to update the nested field Myfield1 by query with following request:
But unfortunately, it does not work.
{
"query": {
"match": {
"Title": "elasticsearch"
}
},
"script": "ctx._source.Myfield1 = [{'nestfield1':'foo blabla...','nestfield2':100},{'nestfield1':'abc...','nestfield2':200}]"
}
Does update by query support nested object?
BTW: any other ways to update document by query?
Is the update by query plugin the only choice?
This example uses _update_by_query
POST indexname/type/_update_by_query
{
"query": {
"match": {
"Title": "elasticsearch"
}
},
"script": {
"source": "ctx._source.Myfield1= params.mifieldAsParam",
"params": {
"mifieldAsParam": [
{
"nestfield1": "foo blabla...",
"nestfield2": 100
},
{
"nestfield1": "abc...",
"nestfield2": 200
}
]
},
"lang": "painless"
}
}
Nested elements need to be iterated in painless script to update values
POST /index/_update_by_query
{
"script": {
"source": "for(int i=0;i<=ctx._source['Myfield1'].size()-1;i++){ctx._source.Myfield1[i].field1='foo blabla...';ctx._source.Myfield1[i].field2=100}",
"lang": "painless"
},
"query": {
"match": {
"Title": "elasticsearch"
}
}
}
Nested elements value update if index is known
POST /index/_update_by_query
{
"script": {
"source": "ctx._source.Myfield1[0].field1='foo blabla...';ctx._source.Myfield1[0].field2=100;ctx._source.Myfield1[1].field1='abc...';ctx._source.Myfield1[1].field2=200;",
"lang": "painless"
},
"query": {
"match": {
"Title": "elasticsearch"
}
}
}
You can try with params, something like this:
"query" : {
"match_all" : {}
},
"script" : "ctx._source.Myfield1 = Myfield1;",
"params": {
"Myfield1": {
"nestfield1": "foo blabla..."
}
}
In my case I'm moving the data from not nested fields in nested fields. I need to add fake information to initialize the nested field. It looks like that:
"query" : {
"match_all" : {}
},
"script" : "ctx._source.Myfield1 = Myfield1; ctx._source.Myfield1.nestfield1 = ctx._source.Myfield1Nestfield1; ctx._source.Myfield1.nestfield2 = ctx._source.Myfield1Nestfield2;",
"params": {
"Myfield1": {
"nestfield1": "init_data"
}
}

Resources