Error while parsing csv to kafka in logstash - elasticsearch

Question

I am trying to send csv data to kafka using LogStash implementing my own configuration script named test.conf.
I got this error while parsing.
Using JAVA_HOME defined java: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-05-24 19:12:08.565 [main] runner - Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.262-b10 on 1.8.0_262-b10 +indy +jit [linux-x86_64]"}
[FATAL] 2021-05-24 19:12:08.616 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:530:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:290:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:201:in `block in validate_all'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:200:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:317:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:273:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:88:in `<main>'"]}
[ERROR] 2021-05-24 19:12:08.623 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
This is the command used to run logstash.
/usr/share/logstash/bin/logstash -f test.conf
Here is the config file.
input {
file {
path => "/home/data/*.csv"
start_position =>"beginning"
sincedb_path => "/dev/null"
}
}
filter {
mutate {
add_field => {
"timestamp" => "%{Date} %{Time}"
}
}
date { match => ["timestamp", "dd-MM-YYYY HH:mm:ss"]}
csv {
remove_field => ["Date", "Time"]
}
grok {
match => { "message" => [
"^%{DATE:timestamp},%{NUMBER:ab},%{NUMBER:cd},%{NUMBER:ef},%{NUMBER:gh},%{NUMBER:ij},%{NUMBER:kl},%{NUMBER:mn},%{NUMBER:op},%{NUMBER:qr},%{NUMBER:st},%{NUMBER:uv},%{NUMBER:wx},%{NUMBER:yz}$"
]
}
}
}
output {
stdout { codec => rubydebug }
if "_grokparsefailure" not in [tags] {
kafka {
codec => "json"
topic_id => "abcd1234"
bootstrap_servers => "192.16.12.119:9092"
}
}
}
Please help me with this.

Answer 1

First of all make sure about the the availability of your server ip with given port(192.16.12.119) with "telnet 192.16.12.119 9092 .
after that you forgot one field in in Kafka output section, Add the group_id field in your output Kafka section such as
Kafka{group_id => "35834"
topics => ["Your topic name"]
bootstrap_server => "192.16.12.199:9092"
codec => json}
If it doesn't worked again then change you bootstrap as "advertise type" look like below
bootstrap.servers => "advertised.listeners=PLAINTEXT://192.16.12.199:9092"
at the end do not metion you server or system ip in internet, It's not safe ;)

Answer 2

Your config is not even being loaded, you have a FATAL error when starting logstash.
[FATAL] 2021-05-24 19:12:08.616 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>,
The user that you are using to run logstash does not have permissions to write in this directory, it needs permission to write to the path.data directory or logstash won't start.
Your logstash.yml file also is not being loaded.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
You need first to give permissions to the user running logstash to write into the path.data, you can change the path.data in the logstash.yml file, then you can pass the path to that file in the command line.
Considering that you installed logstash using a package manager like yum or apt, your logstash.yml file will be in the directory /etc/logstash/.
So you need to run logstash this way:
/usr/share/logstash/bin/logstash -f /path/to/your/config.conf --path.settings /etc/logstash/.
In the logstash.yml you need to set path.data to a directory where the user has permissions to write.
path.data: /path/to/writable/directory