I am using below output block to upsert the document and incrementing the counter (partial updates) for an existing document with matching ID.
Currently, post first doc entry into elasticsearch , "script" has not impact through the subsequent update calls. It is not incrementing the counter value
Below is the output block of logstash using upsert and script:
`output {
stdout { }
elasticsearch {
hosts => "localhost"
index => "test_dest"
script => "ctx._source.views+=1"
script_lang => "painless"
script_type => "inline"
# scripted_upsert => true
doc_as_upsert => true
document_id => "%{[userId]}"
}
stdout {
codec => "json"
}
}`
Try to set the action to "update" and scripted_upsert to "true" as below:
output {
elasticsearch {
action => "update"
hosts => "localhost"
index => "test_dest"
script => "ctx._source.views+=1"
script_lang => "painless"
script_type => "inline"
scripted_upsert => true
document_id => "%{[userId]}"
}
}
Related
We have an alias in elasticsearch and this alias is mapped to one index at a time. This index will get change daily.
Daily we have to write the data to two indexes:
employee-%{+YYYY.MM.dd} and
one more is to the index that is mapped with alias.
To write the data to the first index no issues but how to write the data to the index that is mapped with a particular alias.
We are using kafka, Logstash pipeline to push the data and below is the pipeline:
input {
kafka {
bootstrap_servers => "SomeServer"
client_dns_lookup => "use_all_dns_ips"
topics => ["TOPIC_NAME"]
codec => json
group_id => "kalavakuri"
decorate_events => true
consumer_threads => 2
security_protocol => "SSL"
ssl_keystore_location => "${SSL_KEYSTORE_LOCATION}"
ssl_keystore_password => "${SSL_KEYSTORE_PASSWORD}"
ssl_key_password => "${SSL_KEYSTORE_PASSWORD}"
ssl_truststore_location => "${SSL_TRUSTSTORE_LOCATION}"
}
}
filter {
json {
source => "message"
}
}
output {
if [type] == "EMP" {
elasticsearch {
document_id => "%{id}"
index => "employee-%{+YYYY.MM.dd}"
hosts => ["SomeHost"]
user => "${DEFAULT_LOGSTASH_USER}"
password => "${DEFAULT_LOGSTASH_USER_PW}"
cacert => "/etc/logstash/certs/tls.crt"
action => "update"
doc_as_upsert => true
}
} else if [type] == "STD" {
elasticsearch {
document_id => "%{id}"
index => "employee-%{+YYYY.MM.dd}"
hosts => ["SomeHost"]
user => "${DEFAULT_LOGSTASH_USER}"
password => "${DEFAULT_LOGSTASH_USER_PW}"
cacert => "/etc/logstash/certs/tls.crt"
scripted_upsert => true
action => "update"
upsert => {}
script => "
if (ctx._source.associatedparties == null) {
ctx._source.associatedparties = [];
}
ctx._source.associatedparties.add(params.event.get('associatedparty'));
"
}
}
}
In above pipeline configuration currently we are pushing the data to the first index. I want to know, how to push the data to the index that is mapped with a particular alias in elasticsearch.
To get the index details that is mapped with alias we are using the command in elasticsearch GET _cat/aliases/ra_employee
Is there any way to query the elasticsearch and get the index details based on the alias.
i use logstash 7.9.3 and with this version i have problems to create right index name like logstash-2021.01.01. I need first 9 days of month with 0.
with this config logstash-%{+yyyy.MM.dd} result is => logstash-2021.01.01-000001
with this config logstash-%{+yyyy.MM.d} result is => logstash-2021.01.1
input {
redis {
host => "someip_of_redis"
data_type => "list"
key => "logstash"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["http://someip_of_elastic:9200"]
index => "logstash-%{+yyyy.MM.dd}"
}
}
Thank you in advance
to disable it, i add to config following ilm_enabled => false
input {
redis {
host => "someip_of_redis"
data_type => "list"
key => "logstash"
codec => "json"
}
}
output {
elasticsearch {
hosts => ["http://someip_of_elastic:9200"]
ilm_enabled => false
index => "logstash-%{+yyyy.MM.dd}"
}
}
Hello I am getting this error when I try to add value to existing array field in elasticseach, and my logstash output configuration is:
elasticsearch {
document_id => 1
action => "update"
hosts => ["X.X.X.X:9200"]
index => "test"
script_lang => "painless"
script_type => "inline"
script => 'ctx._source.arrat.add(event("[file][fuid]"))'
}
The error i was getting is
error"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to execute script", "caused_by"=>{"type"=>"script_exception", "reason"=>"compile error", "script_stack"=>["ctx._source.arrat.add(event(\"[file][fuid]\"))", " ^---- HERE"], "script"=>"ctx._source.arrat.add(event(\"[file][fuid]\"))", "lang"=>"painless", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Unknown call [event] with [1] arguments."}}}}}}.
Below is the logstash configuration
input {
beats {
port => "12109"
}
}
filter {
mutate {
id => "brolog-files-rename-raw-fields"
rename => { "[ts]" => "[file][ts]"
"[fuid]" => "[file][fuid]"
"[tx_hosts]" => "[file][tx_hosts]"
"[rx_hosts]" => "[file][rx_hosts]"
"[conn_uids]" => "[file][conn_uids]"
"[source]" => "[file][source]"
"[depth]" => "[file][depth]"
"[analyzers]" => "[file][analyzers]"
"[mime_type]" => "[file][mime_type]"
"[duration]" => "[file][duration]"
"[is_orig]" => "[file][is_orig]"
"[seen_bytes]" => "[file][seen_bytes]"
"[missing_bytes]" => "[file][missing_bytes]"
"[overflow_bytes]" => "[file][overflow_bytes]"
"[timedout]" => "[file][timedout]"
"[md5]" => "[file][md5]"
"[sha1]" => "[file][sha1]"
}
}
}
output{
stdout { codec => rubydebug}
elasticsearch {
document_id => 1
action => "update"
doc_as_upsert => "true"
hosts => ["X.X.X.X:9200"]
index => "test"
script_lang => "painless"
script_type => "inline"
script => 'ctx._source.arrat.add(event.[file][fuid])'
}
}
i am getting data in json format.
Is there any way I can use logstash configuration file to scale output accordingly with different types/indexes ?
For eg.,
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "index_resources"
if(%{some_field_id}==kb){
document_type => "document_type"
document_id => "%{some_id}"
}
else {
document_type => "other_document_type"
document_id => "%{some_other_id}"
}
}
Yes you could route your documents to multiple indexes within your logstash itself. Output could look something like this:
output {
stdout {codec => rubydebug}
if %{some_field_id} == "kb" { <---- insert your condition here
elasticsearch {
host => "localhost"
protocol => "http"
index => "index1"
document_type => "document_type"
document_id => "%{some_id}"
}
} else {
elasticsearch {
host => "localhost"
protocol => "http"
index => "index2"
document_type => "other_document_type"
document_id => "%{some_other_id}"
}
}
}
This thread might help you as well.
i have duplicate data in Logstash
how could i remove this duplication?
my input is:
input
input {
file {
path => "/var/log/flask/access*"
type => "flask_access"
max_open_files => 409599
}
stdin{}
}
filter
filter of files is :
filter {
mutate { replace => { "type" => "flask_access" } }
grok {
match => { "message" => "%{FLASKACCESS}" }
}
mutate {
add_field => {
"temp" => "%{uniqueid} %{method}"
}
}
if "Entering" in [api_status] {
aggregate {
task_id => "%{temp}"
code => "map['blockedprocess'] = 2"
map_action => "create"
}
}
if "Entering" in [api_status] or "Leaving" in [api_status]{
aggregate {
task_id => "%{temp}"
code => "map['blockedprocess'] -= 1"
map_action => "update"
}
}
if "End Task" in [api_status] {
aggregate {
task_id => "%{temp}"
code => "event['blockedprocess'] = map['blockedprocess']"
map_action => "update"
end_of_task => true
timeout => 120
}
}
}
Take a look at the image, the same data log, at the same time, and I just sent one log request.
i solve it
i create a unique id by ('document_id') in output section
document_id point to my temp and temp is my unique id in my project
my output changed to:
output {
elasticsearch {
hosts => ["localhost:9200"]
document_id => "%{temp}"
# sniffing => true
# manage_template => false
# index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
# document_type => "%{[#metadata][type]}"
}
stdout { codec => rubydebug }
}
Executing tests in my local lab, I've just found out that logstash is sensitive to the number of its config files that are kept in /etc/logstash/conf.d directory.
If config files are more than 1, then we can see duplicates for the same record.
So, try to remove all backup configs from /etc/logstash/conf.d directory and perform logstash restart.