Check if id exist in array field or not using script_fields - elasticsearch

i like to get a script field if a specific value exist in a array .
Here is my mapping -
"mappings": {
"properties": {
"field1": {
"type": "text",
},
"field2": {
"type": "text",
},
}
}
Here is my example documents -
{
"field1": "hello 1",
"field2": ["id1","id3"]
}
{
"field1": "hello 2",
"field2": ["id2","id3"]
}
{
"field1": "hello 3",
}
{
"field1": "mobile 1",
"field2": ["id1","id4"]
}
Here is my search query -
{
"query": {
"match" : {
"title": "hello"
}
},
"script_fields":{
"testField2":{
"script": {
"source":"doc['field2'].value == params.id ? true : false",
"params": {
"id": "id1"
}
}
}
}
}
what i want to achieve is to get all those document which has "hello" in their title and add a custom field (testField2) in every doc . the value of that custom field would be True if "field2" contain a specific id ("id1") else False

If field2 is only mapped as text, you won't be able to use its field values (via doc['field2'].value). You can only use its _source:
POST your-index/_search
{
"query": {
"match": {
"field1": "hello"
}
},
"script_fields": {
"testField2": {
"script": {
"source": "params._source.containsKey('field2') ? params._source.field2.contains(params.id) : false",
"params": {
"id": "id1"
}
}
}
}
}
Note that accessing _source is not as efficient as using the doc values -- so in short, you were using the right strategy but with the wrong mapping.
In order to be able to access the doc values, either use the keyword data type or set the fielddata parameter on your existing text field to true. I'm going to go with the keyword here:
DELETE your-index
PUT your-index
{
"mappings": {
"properties": {
"field1": {
"type": "text"
},
"field2": {
"type": "keyword" <--
}
}
}
}
After reindexing (which is required after a "mapping breaking change") you can then call:
POST your-index/_search
{
"query": {
"match": {
"field1": "hello"
}
},
"script_fields": {
"testField2": {
"script": {
"source": "doc['field2'].size() != 0 ? doc['field2'].asList().contains(params.id) : false",
"params": {
"id": "id1"
}
}
}
}
}
Finally, notice the .asList() method. It's needed to convert the doc values which are an instance of the class ScriptDocValues.Strings to a consumable ArrayList on which you can then call the conventional .contains() method.

Related

Search for flattened field existence in ElasticSearch

I'm storing an arbitrary nested object as a flattened field "_meta" which contains various information related to a product.
Here is the mapping for that field:
"mappings": {
"dynamic": "strict",
"properties": {
"_meta": {
"type": "flattened"
},
...
So when trying to search for:
{
"query": {
"exists": {
"field": "_meta.user"
}
}
}
I'm expecting to retrieve all documents that have that field populated. I get zero hits, although if I search for a particular document, I can see that at least one document has that field populated:
"user": {
"origin_title": "some title",
"origin_title_en": "some other title",
"address": "some address",
"performed_orders_count": 0,
"phone": "some phone",
"name": "some name",
"tariff": null,
"proposal_image_background_color": null
},
So how exactly does searching through a flattened data field work?
Why I'm not getting any results?
Tldr;
It is because of the way flattened fields work.
In your case:
{
"_meta":{
"user": {
"name": "some name"
}
}
}
Elasticsearch available representation are:
{
"_meta": ["some name"],
"_meta.user.name": "some name"
}
To reproduce
For the set up:
PUT /74025685/
{
"mappings": {
"dynamic": "strict",
"properties": {
"_meta":{
"type": "flattened"
}
}
}
}
POST /_bulk
{"index":{"_index":"74025685"}}
{"_meta":{"user": "some user"}}
{"index":{"_index":"74025685"}}
{"_meta":{"user": null, "age": 10}}
{"index":{"_index":"74025685"}}
{"_meta":{"user": ""}}
{"index":{"_index":"74025685"}}
{"_meta":{"user": {"username": "some user"}}}
This query is going to find 2 records:
GET 74025685/_search
{
"query": {
"term": {
"_meta": {
"value": "some user"
}
}
}
}
This one, is only going to match the first documents:
GET 74025685/_search
{
"query": {
"term": {
"_meta.user": {
"value": "some user"
}
}
}
}
And so for the exist query:
This one will only return the last doc.
GET 74025685/_search
{
"query": {
"exists": {
"field": "_meta.user.username"
}
}
}
Whereas this one os going to return the 1st and 3rd:
GET 74025685/_search
{
"query": {
"exists": {
"field": "_meta.user"
}
}
}

ElasticSearch query nested path filter OR

I have following index:
PUT /ab11
{
"mappings": {
"properties": {
"product_id": {
"type": "keyword"
},
"data": {
"type": "nested",
"properties": {
"p_id": {
"type": "keyword"
}
}
}
}
}
}
PUT /ab11/_doc/1
{
"product_id": "123",
"data": [
{
"p_id": "a"
},
{
"p_id": "b"
},
{
"p_id": "c"
}
]
}
I want to do query like following sql does(NOTE: I want to do filter not query, because I don't care about score) :
select * from abc11 where data.pid = "a" or data.pid = "b"
You can do it like this because the terms query has OR semantics by default:
{
"query": {
"nested": {
"path": "data",
"query": {
"terms": {
"data.p_id": [
"a",
"b"
]
}
}
}
}
}
Basically, select all documents which have either "a" or "b" in their data.p_id nested docs.

Elasticsearch search length of array

Here is my object profile:
{
"mappings": {
"_doc": {
"properties": {
"name": {
"type": "text"
},
"posts": {
"properties": {
"id": {
"type": "text"
},
"create_date": {
"type": "long"
}
}
}
}
}
}
}
I want to make a search: return all profiles which
1. have name "bob"
2. and have more than 5 posts
Here is an example that I found, but it does not work
{
"query": {
"bool": {
"must": [
{
"term": {
"name": "bob"
}
}
],
"filter": [
{
"script": {
"script": "doc['posts'].values.size() > 5"
}
}
]
}
}
}
I get error "reason":"Variable [posts] is not defined."
update posts.id to keyword
{"id": {"type":"text"},"fields":{{"keyword":{"type":"keyword","ignore_above":256}}}}
Same error
"caused_by":{"type":"script_exception","reason":"compile error","script_stack":["doc[posts.id].values.size() > ..."," ^---- HERE"],"script":"doc[posts.id].values.size() > 5","lang":"painless","caused_by":{"type":"illegal_argument_exception","reason":"Variable [posts] is not defined."}}}}]},"status":400
According to https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html this is because one document is missing the field posts (I suppose).
You could add a filter: "if field posts exists" or use in the script the condition "if doc.containsKey('posts')...."

How to search on multiple fields in URI Search

I would like to perform an AND operation in ElasticSearch using the URI Search (q=). How do I do it?
If I have document like:
[{ "name":"Test 1", "pub":"2"}, { "name":"Test 2", "pub":"1"}, { "name":"A", "pub":"1"}]
And I would like to query for documents containing with a name containing "Test" AND where pub equals "1". How do I do that?
Thanks!
Assuming your document looks like this:
{
"my_field": [
{ "name":"Test 1", "pub":"2"},
{ "name":"Test 2", "pub":"1"},
{ "name":"A", "pub":"1"}
]
}
And the mapping of my_field is of type nested similar to this:
{
"mappings": {
"doc_type": {
"properties": {
"my_field": {
"type": "nested",
"properties": {
"name": { "type": "string" },
"pub": {"type": "integer" }
}
}
}
}
}
}
Then you can query your index and get the expected documents with the following nested query:
POST /_search
{
"query": {
"nested": {
"path": "my_field",
"query": {
"bool": {
"filter": [
{
"match": {
"name": "Test"
}
},
{
"match": {
"pub": 1
}
}
]
}
}
}
}
}
Actually you'd need nested fields. The following is a good resource.
https://www.elastic.co/guide/en/elasticsearch/guide/current/nested-objects.html

Is it possible to update nested field by query?

I am using update by query plugin (https://github.com/yakaz/elasticsearch-action-updatebyquery/) to update documents by query.
In my case, there is nested field in document, the mapping is something like this:
"mappings": {
"mytype": {
"properties": {
"Myfield1": {
"type": "nested",
"properties": {
"field1": {
"type": "string"
},
"field2": {
"type": "long"
}
}
},
"Title": {
"type": "string"
}
}
}
}
Then I want to update the nested field Myfield1 by query with following request:
But unfortunately, it does not work.
{
"query": {
"match": {
"Title": "elasticsearch"
}
},
"script": "ctx._source.Myfield1 = [{'nestfield1':'foo blabla...','nestfield2':100},{'nestfield1':'abc...','nestfield2':200}]"
}
Does update by query support nested object?
BTW: any other ways to update document by query?
Is the update by query plugin the only choice?
This example uses _update_by_query
POST indexname/type/_update_by_query
{
"query": {
"match": {
"Title": "elasticsearch"
}
},
"script": {
"source": "ctx._source.Myfield1= params.mifieldAsParam",
"params": {
"mifieldAsParam": [
{
"nestfield1": "foo blabla...",
"nestfield2": 100
},
{
"nestfield1": "abc...",
"nestfield2": 200
}
]
},
"lang": "painless"
}
}
Nested elements need to be iterated in painless script to update values
POST /index/_update_by_query
{
"script": {
"source": "for(int i=0;i<=ctx._source['Myfield1'].size()-1;i++){ctx._source.Myfield1[i].field1='foo blabla...';ctx._source.Myfield1[i].field2=100}",
"lang": "painless"
},
"query": {
"match": {
"Title": "elasticsearch"
}
}
}
Nested elements value update if index is known
POST /index/_update_by_query
{
"script": {
"source": "ctx._source.Myfield1[0].field1='foo blabla...';ctx._source.Myfield1[0].field2=100;ctx._source.Myfield1[1].field1='abc...';ctx._source.Myfield1[1].field2=200;",
"lang": "painless"
},
"query": {
"match": {
"Title": "elasticsearch"
}
}
}
You can try with params, something like this:
"query" : {
"match_all" : {}
},
"script" : "ctx._source.Myfield1 = Myfield1;",
"params": {
"Myfield1": {
"nestfield1": "foo blabla..."
}
}
In my case I'm moving the data from not nested fields in nested fields. I need to add fake information to initialize the nested field. It looks like that:
"query" : {
"match_all" : {}
},
"script" : "ctx._source.Myfield1 = Myfield1; ctx._source.Myfield1.nestfield1 = ctx._source.Myfield1Nestfield1; ctx._source.Myfield1.nestfield2 = ctx._source.Myfield1Nestfield2;",
"params": {
"Myfield1": {
"nestfield1": "init_data"
}
}

Resources