Logstash : Is there a way to change some of the properties in document while migrating - elasticsearch

Question

I have been migrating some of the indexes from self-hosted Elasticsearch to AmazonElasticSearch using Logstash. While migrating the documents, We need to change the field names in the index based on some logic.
Our Logstash Config file
input {
elasticsearch {
hosts => ["https://staing-example.com:443"]
user => "userName"
password => "password"
index => "testingindex"
size => 100
scroll => "1m"
}
}
filter {
}
output {
amazon_es {
hosts => ["https://example.us-east-1.es.amazonaws.com:443"]
region => "us-east-1"
aws_access_key_id => "access_key_id"
aws_secret_access_key => "access_key_id"
index => "testingindex"
}
stdout{
codec => rubydebug
}
}
Here it is one of the documents for the testingIndex from our self-hosted elastic search
{
"uniqueIdentifier" => "e32d331b-ce5f-45c8-beca-b729707fca48",
"createdDate" => 1527592562743,
"interactionInfo" => [
{
"value" => "Hello this is testing",
"title" => "msg",
"interactionInfoId" => "8c091cb9-e51b-42f2-acad-79ad1fe685d8"
},
{
**"value"** => """"{"edited":false,"imgSrc":"asdfadf/soruce","cont":"Collaborated in <b class=\"mention\" gid=\"4UIZjuFzMXiu2Ege6cF3R4q8dwaKb9pE\">#2222222</b> ","chatMessageObjStr":"Btester has quoted your feed","userLogin":"test.comal#google.co","userId":"tester123"}"""",
"title" => "msgMeta",
"interactionInfoId" => "f6c7203b-2bde-4cc9-a85e-08567f082af3"
}
],
"componentId" => "compId",
"status" => [
"delivered"
]
},
"accountId" => "test123",
"applicationId" => "appId"
}
This is what we are expecting when documents get migrated to our AmazonElasticSearch
{
"uniqueIdentifier" => "e32d331b-ce5f-45c8-beca-b729707fca48",
"createdDate" => 1527592562743,
"interactionInfo" => [
{
"value" => "Hello this is testing",
"title" => "msg",
"interactionInfoId" => "8c091cb9-e51b-42f2-acad-79ad1fe685d8"
},
{
**"value-keyword"** => """"{"edited":false,"imgSrc":"asdfadf/soruce","cont":"Collaborated in <b class=\"mention\" gid=\"4UIZjuFzMXiu2Ege6cF3R4q8dwaKb9pE\">#2222222</b> ","chatMessageObjStr":"Btester has quoted your feed","userLogin":"test.comal#google.co","userId":"tester123"}"""",
"title" => "msgMeta",
"interactionInfoId" => "f6c7203b-2bde-4cc9-a85e-08567f082af3"
}
],
"componentId" => "compId",
"status" => [
"delivered"
]
},
"accountId" => "test123",
"applicationId" => "appId"
}
What we need is to change the "value" field to "value-keyword" wherever we find some JSON format. Is there any other filter in Logstash to achieve this

Answer 1

As documented in the Logstash website:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename
You can use the mutate filter, applying the rename function.
For example:
filter {
mutate {
replace => { "old-field" => "new-field" }
}
}
For nested fields, you could just pass the path of the field:
filter {
mutate {
replace => { "[interactionInfo][value]" => "[interactionInfo][value-keyword]" }
}
}

Answer 2

Try adding this to your filter:
filter {
ruby {
code => "event.get('interactionInfo').each { |item| if item['value'].match(/{.+}/) then item['value-keyword'] = item.delete('value') end }"
}
}