update a nested object in Elasticsearch - elasticsearch

i hope i can find some help
my problem is that i am triying to update a nasted object that i have.
here 's my index architecture
{
"grille_badge" : {
"mappings" : {
"properties" : {
"theme" : {
"type" : "nested",
"properties" : {
"categories" : {
"type" : "nested",
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"niveaux" : {
"type" : "nested",
"properties" : {
"grille_defis" : {
"type" : "nested",
"properties" : {
"defis" : {
"type" : "nested",
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
}
and here is exemple of my document that i wand to edit
{
"_index" : "grille_badge",
"_type" : "_doc",
"_id" : "1",
"_version" : 1,
"_seq_no" : 0,
"_primary_term" : 1,
"found" : true,
"_source" : {
"theme" : [
{
"name" : "foot",
"categories" : [
{
"name" : "categorie1",
"niveaux" : [
{
"name" : "bronze",
"grille_defis" : [
{
"name" : "grille1",
"defis" : [ ]
}
]
}
]
},
{
"name" : "categorie2",
"niveaux" : [
{
"name" : "argent",
"grille_defis" : [
{
"name" : "grille1",
"defis" : [ ]
}
]
}
]
},
{
"name" : "categorie3",
"niveaux" : [
{
"name" : "or",
"grille_defis" : [
{
"name" : "grille1",
"defis" : [ ]
}
]
}
]
}
]
}
]
}
}
And i am triying to add a Defis to my empty list, like the code below that i found in the documentation
POST index_name/_update/doc_id
{
"script": {
"source": "ctx._source.cats.add(params.cat)",
"params": {
"cat": {
"colors": 4,
"name": "Leon",
"breed": "Persian"
}
}
}
}
i tried to make my own script that can go deep into my defis list but still don't know how to do it.
here's what i tried to do
POST grille_badge/_update/1
{
"script": {
"source": "def theme_target = ctx._source.theme.findAll(our_theme theme.name == params.current_theme); def categorie_target = theme_target.findAll(our_categorie categories.name == params.current_categorie); def niveau_target = categorie_target.findAll(our_niveau niveaux.name == params.current_niveau); def grille_target = niveau_target.findAll(our_grille grille_defis.name == params.current_grille); grille_target.defis.add(params.defis_ajouter)",
"params": {
"current_theme": "foot",
"current_categorie": "categorie1",
"current_niveau": "niveau1",
"current_grille": "grille1",
"defis_ajouter": "i am a defis"
}
}
}
please could anyone help me !!
thanks for advance

Welcome to SO. Your question is quite confusing but here's how you update a doc by a script where your params are an instance of HashMap:
Index the original document w/ an id of 1 -- collapsed for brevity:
POST grille_badge/_doc/1
{"theme":[{"name":"foot","categories":[{"name":"categorie1","niveaux":[{"name":"bronze","grille_defis":[{"name":"grille1","defis":[]}]}]},{"name":"categorie2","niveaux":[{"name":"argent","grille_defis":[{"name":"grille1","defis":[]}]}]},{"name":"categorie3","niveaux":[{"name":"or","grille_defis":[{"name":"grille1","defis":[]}]}]}]}]}
Update doc#1 (note that you can also use a wide-ranging _update_by_query too). Also note that you need to verify whether cats exist before add-ing a hashmap in there.
POST grille_badge/_update/1
{
"script": {
"source": "if (ctx._source.cats instanceof List) { ctx._source.cats.add(params.cat) } else { ctx._source.cats = [params.cat] }",
"params": {
"cat": {
"colors": 4,
"name": "Leon",
"breed": "Persian"
}
}
}
}
Verify that cats are there:
GET grille_badge/_search
EDIT & ELABORATION
Here's how you do it:
POST grille_badge/_update/1
{
"script": {
"source": """
def theme_target = ctx._source.theme.find(theme -> theme.name == params.current_theme);
def categorie_target = theme_target.categories.find(categories -> categories.name == params.current_categorie);
def niveau_target = categorie_target.niveaux.find(niveaux -> niveaux.name == params.current_niveau);
def grille_target = niveau_target.grille_defis.find(grille_defis -> grille_defis.name == params.current_grille);
grille_target['defis'].add(["name" : params['defis_ajouter']]);
""",
"params": {
"current_theme": "foot",
"current_categorie": "categorie1",
"current_niveau": "bronze",
"current_grille": "grille1",
"defis_ajouter": "i am a defis"
}
}
}
Note the proper arrow notation in the findAll methods.
Also note that your own mapping defined the innermost defis as a nested array of key-value pairs -- this means you cannot push a plain string using the .add method -- you need to use a HashMap. You instatiate one with the one liner ["your_key" : "your_value"]. Good luck.

Related

match_only_text fields do not support sorting and aggregations elasticsearch

I would like to count and sort the number of occurred message on a field of type match_only_text. Using a DSL query the output needed to have to look like this:
{" Text message 1":615
" Text message 2":568
....}
So i tried this on kibana:
GET my_index_name/_search?size=0
{
"aggs": {
"type_promoted_count": {
"cardinality": {
"field": "message"
}
}
}
}
However i get this error:
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "match_only_text fields do not support sorting and aggregations"
}
I am interested in the field "message" this is its mapping:
"message" : {
"type" : "match_only_text"
}
This is a part of the index mapping:
"mappings" : {
"_meta" : {
"package" : {
"name" : "system"
},
"managed_by" : "ingest-manager",
"managed" : true
},
"_data_stream_timestamp" : {
"enabled" : true
},
"dynamic_templates" : [
{
"strings_as_keyword" : {
"match_mapping_type" : "string",
"mapping" : {
"ignore_above" : 1024,
"type" : "keyword"
}
}
}
],
"date_detection" : false,
"properties" : {
"#timestamp" : {
"type" : "date"
}
.
.
.
"message" : {
"type" : "match_only_text"
},
"process" : {
"properties" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
},
"pid" : {
"type" : "long"
}
}
},
"system" : {
"properties" : {
"syslog" : {
"type" : "object"
}
}
}
}
}
}
}
Please Help
Yes, by design, match_only_text is of the text field type family, hence you cannot aggregate on it.
You need to:
A. create a message.keyword sub-field in your mapping of type keyword:
PUT my_index_name/_mapping
{
"properties": {
"message" : {
"type" : "match_only_text",
"fields": {
"keyword": {
"type" : "keyword"
}
}
}
}
}
B. update the whole index (using _update_by_query) so the sub-field gets populated and
POST my_index_name/_update_by_query?wait_for_completion=false
Then, depending on the size of your index, call GET _tasks?actions=*byquery&detailed regularly to check the progress of the task.
C. run the aggregation on that sub-field.
POST my_index_name/_search
{
"size": 0,
"aggs": {
"type_promoted_count": {
"cardinality": {
"field": "message.keyword"
}
}
}
}

Elastic painless count unique occurrences

I'm using ELK stack version 7. What I need to do is to count the unique occurence of a value in my indexes.
My indexes are created by WSO2 Identity Server version 5.10 and they are so defined:
{
"login.wso2.node.ip-2021.03.11" : {
"aliases" : {
"alias_my_login" : { }
},
"mappings" : {
"dynamic" : "true",
"_meta" : { },
"_source" : {
"includes" : [ ],
"excludes" : [ ]
},
"dynamic_date_formats" : [
"strict_date_optional_time",
"yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
],
"dynamic_templates" : [ ],
"date_detection" : true,
"numeric_detection" : false,
"properties" : {
"#timestamp" : {
"type" : "date",
"format" : "strict_date_optional_time"
},
"#version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"host" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"instance_IP" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"instance_name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"java_class" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"level" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"log_message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"message" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"rr" : {
"type" : "text"
},
"tags" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tenant_id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"timestamp" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"settings" : {
"index" : {
"creation_date" : "1615481578543",
"number_of_shards" : "1",
"number_of_replicas" : "1",
"uuid" : "9o-UQnn-SKaj7LbhO8GYxQ",
"version" : {
"created" : "7070199"
},
"provided_name" : "login.wso2.node.ip-2021.03.11"
}
}
}
}
What I need to do is to check if in the message field I have a SAML2 Response XML and if so I need to access to one value of this XML and count the unique occurrences.
So far so good. The message field is multi mapping field. It is both text type and keyword type so I can use text type for full search and keyword type for aggregation, sorting and so on.
What I did is to write this painless script:
GET login.wso2.node.ip-2021.03.11/_search
{
"query": {
"bool": {
"filter": [
{
"script": {
"script": {
"source": "doc['message.keyword'].value.contains('SAML_MESSAGES_LOGFILE') && doc['message.keyword'].value.contains('TINIT-')"
}
}
}
]
}
},
"aggs": {
"distinct_cf_count": {
"scripted_metric": {
"params": {
"fieldName":"message"
},
"init_script": "state.list = []",
"map_script": """
//Controllo se c'è il campo message e se c'è fiscalnumber
//if(doc[params.fieldName] != null && doc[params.fieldName].size()==0 ){
// def matcher = /<saml2:Attribute FriendlyName="Codice Fiscale" Name="fiscalNumber"><saml2:AttributeValue xmlns:xs="http:\/\/www.w3.org\/2001\/XMLSchema" xmlns:xsi="http:\/\/www.w3.org\/2001\/XMLSchema-instance" xsi:type="xs:string">(.*)<\/saml2:AttributeValue><\/saml2:Attribute>/.matcher(doc[params.fieldName].value);
//if (matcher.find()) {
// state.list.add(matcher.group(1));
//}
if(doc[params.fieldName] != null && doc[params.fieldName].size()==0 && doc[params.fieldName].value.indexOf('TINIT-') > -1 ){
def valore = doc[params.fieldName].value;
def startIdx = valore.indexOf('TINIT-')+'TINIT-'.length();
state.list.add(valore.substring(startIdx, 16));
}
""",
"combine_script": "return state.list;",
"reduce_script": """
Map uniqueValueMap = new HashMap();
int count = 0;
for(shardList in states) {
if(shardList != null) {
for(key in shardList) {
if(!uniqueValueMap.containsKey(key)) {
count +=1;
uniqueValueMap.put(key, key);
}
}
}
}
return count;
"""
}
}
}
}
But I can't use regex because they are disabled and I should restart my ELK cluster in order to enable them. So I tried the contains and indexOf but I'm not able in counting the unique occurrences of this field.
Do you have any suggestion?
Thank you
Angelo
EDIT MORE INFO
gave a look. This check alwaus return 0 so it's like if message.keyword is always missing
"map_script": """
//Controllo se c'è il campo message e se c'è fiscalnumber
//if(doc[params.fieldName] != null && doc[params.fieldName].size()==0 ){
// def matcher = /<saml2:Attribute FriendlyName="Codice Fiscale" Name="fiscalNumber"><saml2:AttributeValue xmlns:xs="http:\/\/www.w3.org\/2001\/XMLSchema" xmlns:xsi="http:\/\/www.w3.org\/2001\/XMLSchema-instance" xsi:type="xs:string">(.*)<\/saml2:AttributeValue><\/saml2:Attribute>/.matcher(doc[params.fieldName].value);
//if (matcher.find()) {
// state.list.add(matcher.group(1));
//}
**if( doc[params.fieldName].size()==0 ){**
** state.list.add(UUID.randomUUID().toString());**
** }**
//else{
// def valore = doc[params.fieldName].value;
// def cf = valore.splitOnToken('TINIT-')[1].substring(16);
// state.list.add(cf);
//}
""",
Do you have any suggestion? I'm really blocked here... at 1 step to the solution
Thank you

Upsert document such that it would update the particular item in an array field

In Elasticsearch, say I have the document like this:
{
"inputs": [
{
"id": "1234",
"value": "ABCD"
},
{
"id": "5678",
"value": "EFGH"
}
]
}
Say, now, I wanted to update value of all items where id is "1234" to "XYZA". How can I do that using script in elasticsearch? I am not sure if I can do some for loop in script?
Mapping:
{
"inputs" : {
"mappings" : {
"properties" : {
"inputs" : {
"type" : "nested",
"properties" : {
"id" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"value" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
}
Query:
You can use _update_by_query api. Query part will filter out documents and script will update the field
<1. When inputs is of nested type
POST inputs/_update_by_query
{
"script": {
"source": "for(a in ctx._source['inputs']){if(a.id=='1234') a.value=params.new_value; }",
"params": {
"new_value": "XYZA"
}
},
"query": {
"nested":{
"path":"inputs",
"query":{
"term":{
"inputs.id":1234
}
}
}
}
}
2. When inputs if of object type
POST inputs/_update_by_query
{
"script": {
"source": "for(a in ctx._source['inputs']){if(a.id=='1234') a.value=params.new_value; }",
"params": {
"new_value": "XYZA"
}
},
"query": {
"term": {
"inputs.id": 1234
}
}
}
Result:
"hits" : [
{
"_index" : "inputs",
"_type" : "_doc",
"_id" : "3uwrwHEBLcdvQ7OTrUmi",
"_score" : 1.0,
"_source" : {
"inputs" : [
{
"id" : "1234",
"value" : "XYZA"
},
{
"id" : "5678",
"value" : "EFGH"
}
]
}
}
]

query to find all docs that match with exact terms with all the fields in the query

I have a simple doc structure as follows.
{
"did" : "1",
"uid" : "user1",
"mid" : "pc-linux1",
"path" : "/tmp/path1"
}
I need to query elastic ,that matches all fields exactly
GET index2/_search
{
"query": {
"bool":{
"must": [
{
"term" : { "uid" : "user1"}
},
{
"term" : { "mid" : "pc-linux1"}
},
{
"term" : { "did" : "1"}
},
{
"term" : { "path" : "/tmp/path1"}
}
]
}
}
}
The matching should happen without any kind of elastic 'analysis' on keywords, so that "/tmp/path1" is matched as a full term.
I tried to use a custom mapping: with
"index" : false
which does not work.
PUT /index2?include_type_name=true
{
"mappings" : {
"_doc": {
"properties" : {
"did" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
},
"mid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
},
"path" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
},
"uid" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"index" : false,
"ignore_above" : 256
}
}
}
}
}
}
}
I am using elastic7.0 and few posts suggesting a custom mapping with
"index" : "not_analysed"
does not get accepted as a valid mapping in elastic 7.0
Any suggestions?
If you want to match exact terms, try this query:
GET index2/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"uid": "user1"
}
},
{
"match": {
"mid": "pc-linux1"
}
},
{
"match": {
"did": "1"
}
},
{
"match": {
"path": "/tmp/path1"
}
}
]
}
}
}

Is it possible to define default mapping for an inner object in ElasticSearch?

Say I have a document like this:
{
"events" : [
{
"event_id" : 123,
"props" : {
"version": "33"
},
{
"event_id" : 124,
"props" : {
"version": "44a"
}
]
}
Is it possible to specify that the events.props.version be mapped to some type?
I've tried:
{
"template" : "logstash-*",
...
"mappings" : {
"_default_" : {
"properties" : {
"events.props.version" : { "type" : "string" }
}
}
}
}
But that doesn't seem to work.
Please have a look at mapping API in elasticsearch Mapping API.
To set any analyzer in the inner element we need to consider each and every inner field as a separate properties set. try the following
{
"mappings": {
"properties": {
"events": {
"properties": {
"event_id": {
"type": "string",
"analyzer": "keyword"
},
"props": {
"properties": {
"version": {
"type": "string"
}
}
}
}
}
}
}
}
if this not works please provide me you mapping.
Sure, but you need to use the "object" type:
From the doc ( https://www.elastic.co/guide/en/elasticsearch/reference/1.5/mapping-object-type.html ) if you want to map
{
"tweet" : {
"person" : {
"name" : {
"first_name" : "Shay",
"last_name" : "Banon"
},
"sid" : "12345"
},
"message" : "This is a tweet!"
}
}
you can write:
{
"tweet" : {
"properties" : {
"person" : {
"type" : "object",
"properties" : {
"name" : {
"type" : "object",
"properties" : {
"first_name" : {"type" : "string"},
"last_name" : {"type" : "string"}
}
},
"sid" : {"type" : "string", "index" : "not_analyzed"}
}
},
"message" : {"type" : "string"}
}
}
}

Resources