I am trying to start logstash 5.4 on my linux rhel 6 server but i'm getting the following message:
WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Xms1G .Xmx64G
Error: Could not find or load main class .Xmx64G
Following is my logstash.conf in which I'm try to ingest data from sqlserver
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-5.1.42-bin.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://9.37.92.72:1433;databaseName=KaiserPermanente;"
jdbc_user => "sa"
jdbc_password => "passw0rd!"
statement => "select * from IEVDIncident ;"
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "kaiserpermanente"
}
stdout { codec => json_lines }
}
Please tell me how can I resolve this one. Thanks
It seems you have an environment variable JAVA_OPTS with value -Xms1G .Xmx64G so it overrides logstash options. You need to change your variable to -Xms1G -Xmx64G. Replace . with -.
Related
Hi please have a look at below issue. I am clueless how to fix this issue.
I've downloaded ElasticSrearch -6.2.4 and Logstash - 6.2.4 on the window machine.
I'm trying to import data from MySQL to ElasticSearch using LogStash. but I'm getting the below error :
C:\logstash-6.2.4\bin>logstash -f logstash.conf
Error: Could not find or load main class Files\Apache
here are the steps I'm following:
first I started the ElasticSearch which is running perfectly on the port 9200.
then I've added the below Scripts in logstash.yml which has all the migration instructions.
# ------------ MySQL to ElasticSearch -------------
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/MySQL_ElasticSearch_Demo"
# The user we wish to execute our statement as
jdbc_user => "root"
jdbc_password => "root"
# The path to our downloaded jdbc driver
jdbc_driver_library => "C:\mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
# our query
statement => "SELECT * FROM user"
}
}
output {
stdout { codec => json_lines }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "users"
"document_type" => "usersData"
}
}
I'm trying to run the logstash via command prompt using below command:
C:\logstash-6.2.4\bin>logstash -f logstash.conf
Error: Could not find or load main class Files\Apache
===> any help will be much appreciated. thanks in advance!
Logstash.conf
input { tcp { port => 7199 } } output { elasticsearch { hosts => ["localhost"] } }
Cassandra running on 7199 port and jhipster application running on localhost:8080.
we are unable to add into logstash by my_application
No log4j2 file found.
I think you can use the JDBC plugin:
https://github.com/logstash-plugins/logstash-input-jdbc
input {
jdbc {
jdbc_connection_string => "jdbc:cassandra://hostname:XXXX" # Your port
jdbc_user => "user" # The user value
jdbc_password => "password" # The password
jdbc_driver_library => "$PATH/cassandra_driver.jar" # Jar path
jdbc_driver_class => "org.apache.cassandra.cql.jdbc.CassandraDriver" # Driver
statement => "SELECT * FROM keyspace.my_table" # Your query
}
}
I had the same issue. The issue was solved by downloading a Cassandra JDBC from DatabaseSchema.
also when You want to add the jar files, add it in the
logstashFolder/logstash-core/lib/jar
there seems to be a bug with logstash which only looks this path for external jar files.
also if there were some jar files that were duplicated use the latest ones.
I am getting below error while using Mongodb Java Driver to ready data from MongoDB and push it to ElasticSearch-
Error: mongodb.jdbc.MongoDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
Plateform Info:
OS- RHEL 6.6
Logstash- 5.5.0
Elasticsearch- 5.5.0
Mongodb- 3.2.13
Jars- mongodb-driver-core-3.4.2.jar, mongo-java-driver-3.4.2.jar and bson-3.4.2.jar
Logstash config
input{
jdbc{
jdbc_driver_library => "/home/pdwiwe/logstash-5.5.0/bin/mongo-java-driver-3.4.2.jar"
jdbc_driver_class => "mongodb.jdbc.MongoDriver"
jdbc_connection_string => "jdbc:mongo://hostname:27017?authSource=admin"
jdbc_user => "user"
jdbc_password => "pwd"
statement => "select * from system.users"
}
}
output {
if "_grokparsefailure" not in [tags]{
elasticsearch {
hosts => [ "localhost:9200" ]
index => "mongodb-data"
}
}
}
Logstash Service Start:
/home/pdwiwe/logstash-5.5.0/bin$ sh logstash -f mongo.conf
mongodb.jdbc.MongoDriver is not a Driver class in the mongo-java-driver.
AFAIK - this driver does not support JDBC
Various JDBC drivers have wrapped the mongo-java-driver such as Unity, Simba, DbSchema
After watching this tutorial ;
https://www.youtube.com/watch?v=ZnI_rlrei1s
I'm trying to fetch my localhost mysql (using laravel valet mysql) using logstash with jdbc to sent to elasticsearch server .
This is my config :
# file: db.conf
input {
jdbc {
# MySQL jdbc connection string to our database, mydb
jdb_connection_string => "jdbc:mysql://localhost:3306/dragon"
# The user we wish to execute our statement as
jdbc_user => "root"
# The user password
jdbc_password => ""
# The path to our downloaded jdbc driver
jdbc_driver_library => "/Users/muaz/downloads/logstash/mysql-connector-java-5.1.39-bin.jar"
# The name of the deliver clas for MySQL
jdbc_driver_class => "com.mysql.jdbc.Driver"
# Our query
statement = "SELECT * from Receipt"
}
}
output {
# stdout { codec => json_lines }
elasticsearch {
# protocol = https
index => "power_receipt"
document_type => "Receipt"
document_id => "%{id}"
host => "https://search-power-yidhfhkidiiddcccyhyrijaagamu.ap-southeast-1.es.amazonaws.com"
}
}
And i run it using command (in logstash folder) :
./bin/logstash agent -f db.conf
It produce :
fetched an invalid config {:config=>"# file: db.conf\ninput {\n\tjdbc {\n\t\t# MySQL jdbc connection string to our database, mydb\n\t\tjdb_connection_string => \"jdbc:mysql://localhost:3306/dragon\"\n\t\t# The user we wish to execute our statement as \n\t\tjdbc_user => \"root\"\n\t\t# The user password\n\t\tjdbc_password => \"\"\n\t\t# The path to our downloaded jdbc driver\n\t\tjdbc_driver_library => \"/Users/muaz/downloads/logstash/mysql-connector-java-5.1.39-bin.jar\"\n\t\t# The name of the deliver clas for MySQL\n\t\tjdbc_driver_class => \"com.mysql.jdbc.Driver\"\n\t\t# Our query\n\t\tstatement = \"SELECT * from Receipt\"\n\t}\n}\noutput {\n\t# stdout { codec => json_lines }\n\telasticsearch {\n\t\t# protocol = https\n\t\tindex => \"slurp_receipt\"\n\t\tdocument_type => \"Receipt\"\n\t\tdocument_id => \"%{id}\"\n\t\thost => \"https://search-power-yidhfhkidiiddcccyhyrijaagamu.ap-southeast-1.es.amazonaws.com\"\n\t}\n}\n\n\n", :reason=>"Expected one of #, => at line 15, column 13 (byte 521) after # file: db.conf\ninput {\n\tjdbc {\n\t\t# MySQL jdbc connection string to our database, mydb\n\t\tjdb_connection_string => \"jdbc:mysql://localhost:3306/dragon\"\n\t\t# The user we wish to execute our statement as \n\t\tjdbc_user => \"root\"\n\t\t# The user password\n\t\tjdbc_password => \"\"\n\t\t# The path to our downloaded jdbc driver\n\t\tjdbc_driver_library => \"/Users/muaz/downloads/logstash/mysql-connector-java-5.1.39-bin.jar\"\n\t\t# The name of the deliver clas for MySQL\n\t\tjdbc_driver_class => \"com.mysql.jdbc.Driver\"\n\t\t# Our query\n\t\tstatement ", :level=>:error}
How to solve it?
Thank you
You have a typo on the last line of your jdbc input
statement = "SELECT * from Receipt"
should read
statement => "SELECT * from Receipt"
Also in your elasticsearch output you need to change
host => "https://search-power-yidhfhkidiiddcccyhyrijaagamu.ap-southeast-1.es.amazonaws.com"
to
hosts => ["https://search-power-yidhfhkidiiddcccyhyrijaagamu.ap-southeast-1.es.amazonaws.com"]
below is my configuration -
input {
jdbc {
jdbc_driver_class => "org.apache.derby.jdbc.EmbeddedDriver"
jdbc_connection_string => "jdbc:derby://localhost:3306/test"
jdbc_user => "root"
jdbc_password => "password"
statement => "select * from user"
}
}
filter {
#csv {
#columns => ["cdatetime", "address", "district", "beat", "grid", "crimedescr", "ucr_ncic_code",
# "latitude","longitude"]
#separator => ","
}
#}
output {
elasticsearch { host => localhost protocol => http index => test user => pavan password => password }
stdout { codec => rubydebug }
}
When I run the config file, I get the following error -
The error reported is:
org.apache.derby.jdbc.EmbeddedDriver not loaded
Can anyone please help me resolve it? My current Java version is -
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
Any reference links about JDBC input configuration would be appreciated.