how to java application to logstash? - elasticsearch

Question

I want to send data from java application to logstash.
Sending it to " curl " is good, but it does not send it to "java restTemplate"
"curl" example : OK
$ curl -XPOST -H "Content-Type: application/x-ndjson" "http://10.97.8.151:18080" --data-binary #data.txt
data.txt
{"index":{"_index": "myIndex","_type":"myType"}}
{"data1":"value1","data2":"value2","data3":"value3"}
This works well. However, " Java restTemplate " did not work as follows.
Everything else is identical and the data form is different.
{"index":{"_index": "myIndex","_type":"myType"}}\n{"data1":"value1","data2":"value2","data3":"value3"}
I have tried to describe the type of data in the form of "application/x-ndjson"and I don't know where it was wrong.
How do you transfer data from " java application " to " logstash " in " java application "?
The " logstash config " file is as follows :
input {
http {
host => "0.0.0.0"
port => "12345"
codec => es_bulk {
}
}
}
output {
elasticsearch {
hosts => "x.x.x.x:yyyy"
index => "{[#metadata][_index]}"
document_type => "{[#metadata][_type]}"
template_name => "api"
}
stdout { codec => rubydebug { metadata => true } }
}
Ask for help.

Answer 1

The best way to do this is not to use the es_bulk codec in your http input, but simply let your events be plain text messages.
Then, instead of using an elasticsearch output, you can use the http output like this:
input {
http {
host => "0.0.0.0"
port => "12345"
codec => "plain"
}
}
output {
http {
http_method => "post"
url => "http://x.x.x.x:yyyy/_bulk"
format => "message"
message => "%{message}"
}
}
The big plus of this method is that your Logstash pipeline doesn't have to parse the "bulk" input just to recreate the same "bulk" output. In this case, you're simply using Logstash as a "pass-through". I don't see the value of this, though, you could have your Java application send the bulk payload directly to ES.