Im using the Below JDBC code in Logstash for updating the already existing index in Elasticsearch, without duplicating rows or adding the updated row as another new row.
Versions: Elasticsearch, Logstash and Kibana are v7.1.0.
input {
jdbc {
jdbc_connection_string => "jdbc:sqlserver://DB01:1433;databasename=testdb;integratedSecurity=true"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_driver_library => "C:\Program Files\sqljdbc_6.2\enu\mssql-jdbc-6.2.2.jre8.jar"
jdbc_user => nil
statement => "SELECT * from data WHERE updated_on > :sql_last_value ORDER BY updated_on"
use_column_value =>true
tracking_column =>updated_on
tracking_column_type => "timestamp"
}
}
output {
elasticsearch { hosts => ["localhost:9200"]
index => "datau"
action=>update
document_id => "%{id}"
doc_as_upsert =>true}
stdout { codec => rubydebug }
}
when i run the above in logstash (logstash -f myfile.conf)
the below error appars.
[2019-08-21T10:46:33,864][ERROR][logstash.outputs.elasticsearch] Failed to insta ll template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://localhost:9200/_template/logstash'", :class=>"LogStash::Outputs::Elasti cSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["D:/ELK 6.4.0/log stash-6.4.0/logstash-6.4.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasti csearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adap ter.rb:80:in `perform_request'", "D:/ELK 6.4.0/logstash-6.4.0/logstash-6.4.0/ven dor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstas h/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'"...
Where am i gong wrong?
Remove } from this line:
doc_as_upsert =>true}
The issue was with the version compatibility.
I had used 6.4 logstash on 7.1 Elasticsearch.
Once my logstash was upgraded, the issue was resolved.
Thanks!
Related
There is my envirmoment.
ElasticSearch 7.8.0 ,Logstash 7.14.0
I tried transfer MySQL data to ElasticSearch.There is my pipline config.
input {
stdin {
}
jdbc {
jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/test?characterEncoding=UTF-8&useSSL=false&autoReconnect=true"
jdbc_user => "username"
jdbc_password => "password"
jdbc_driver_library => "/etc/logstash/pipeline/mysql-connector-java-8.0.26.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
codec => plain { charset => "UTF-8"}
tracking_column => created_at
record_last_run => true
last_run_metadata_path => "/opt/logstash/lastrun/.logstash_jdbc_last_run"
statement_filepath => "/etc/logstash/pipeline/jdbc.sql"
clean_run => false
schedule => "* * * * *"
type => "std"
}
}
output {
elasticsearch {
hosts => "127.0.0.1:9200"
index => "test"
document_type => "testdb"
document_id => "%{id}"
}
stdout {
codec => json_lines
}
}
When I start the logstash, It looks some error,I cannt sync data..
[2021-08-11T01:11:50,378][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"http://127.0.0.1:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '400' contacting Elasticsearch at URL 'http://127.0.0.1:9200/_license'"}
[2021-08-11T01:11:50,378][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"http://127.0.0.1:9200/"}
[2021-08-11T01:11:51,645][ERROR][logstash.outputs.elasticsearch][main][d38f4c43e6fdc69444ff9aadc94fa4db5da791aa7892cf45e98fa87844fb0d43] Elasticsearch setup did not complete normally, please review previously logged errors {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
[2021-08-11T01:11:55,385][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}
I start logstash with docker,in the docker environment,it connect elasticsearch in shell(curl 127.0.0.1:9200) correctly.
Why does Logstash visit http://127.0.0.1:9200/_license?
Logstash Version Should be Same as ElasticSearch Version.Thanks for #leandrojmp and #Val.
The offical docker container has default setting,it contain xpack config,so I should editlogstash.yaml,remove xpack config or change it as note.
Some other advise.
With the license change from elastic, versions after 7.10 are not open source anymore and will only work with Elastic distribution of Elasticsearch using the Elastic License (the free one or one of the paid tiers).
I am trying to get data from mongodb to elastic using logstash
but i get below errors:
Exception when executing JDBC query {:exception=>#<Sequel::DatabaseError: Java::OrgLogstash::Missing
ConverterException:
below is my config file:
input{
jdbc{
jdbc_driver_library => "D:/mongojdbc1.2.jar"
jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
jdbc_connection_string => "jdbc:mongodb://localhost:27017/users"
jdbc_user => ""
jdbc_validate_connection => true
statement => "db.user_details.find({})"
}
}
output {
elasticsearch {
hosts => 'http://localhost:9200'
index => 'person_data'
document_type => "person_data"
}
stdout { codec => rubydebug }
}
This possibly happens because certain data type in Mongo might not be convertible to data type in Elasticsearch. May be you should try to select few columns and then see which one is failing.
I didn't find any proper documentation in output plugins of logsatsh ,for loading data into AWS ES,i do find
output plugin only speaks the HTTP protocol. without specifying port 9200 can we load data in AWS ES
input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost/elasticsearch"
jdbc_user => "root"
jdbc_password => "empower"
#jdbc_validate_connection => true
jdbc_driver_library => "/home/wtc082/Documents/com.mysql.jdbc_5.1.5.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "SELECT * FROM index_part_content_local LIMIT 10;"
schedule => "1 * * * *"
#codec => "json"
}
}
output {
elasticsearch {
index => "mysqltest"
document_type => "mysqltest_type"
document_id => "%{partnum}"
hosts => "AWSURI"
}
}
Can we do like this ?
Actually, I was using the log stash 2.4 to load data from Mysql to ES 5.X version when I used the log stash 5.X version it solved my issue.I didn't get any error while running the conf file
Thanks Val
I am trying to index data from mysql db to elasticsearch using logstash. Logstash is running without errors but the problem is, it indexing only one row from my SELECT query.
Below are the versions of softwares I am using:
elastic search : 2.4.1
logstash: 5.1.1
mysql: 5.7.17
jdbc_driver_library: mysql-connector-java-5.1.40-bin.jar
I am not sure if this is because logstash and elasticsearch versions are different.
Below is my pipeline configuration:
input {
jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.40-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"
jdbc_user => "user"
jdbc_password => "password"
schedule => "* * * * *"
statement => "SELECT * FROM employee"
use_column_value => true
tracking_column => "id"
}
}
output {
elasticsearch {
index => "logstash"
document_type => "sometype"
document_id => "%{uid}"
hosts => ["localhost:9200"]
}
}
It seems like the tracking_column (id) which you're using in the jdbc plugin and the document_id (uid) in the output is different. What if you have both of them same since it'll be easy to get all the records by id and push them into ES using the same id as well which could look more understandable:
document_id => "%{id}" <-- make sure you've got the exact spellings
And also please try adding this following line to your jdbc input after tracking_column:
tracking_column_type => "numeric"
Additionally to make sure that you don't have the .logstash_jdbc_last_run file existing when you're running the logstash file include the following line as well:
clean_run => true
So this is how your jdbc input should look like:
jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.40-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"
jdbc_user => "user"
jdbc_password => "password"
schedule => "* * * * *"
statement => "SELECT * FROM employee"
use_column_value => true
tracking_column => "id"
tracking_column_type => "numeric"
clean_run => true
}
Other than that the conf seems to be fine, unless you're willing to have :sql_last_value where if you only wanted to update the newly added records in your database table. Hope it helps!
I am trying to create a data pipeline where Logstash jdbc plugin get some data with SQL query every 5 minutes and ElasticSearch output plugin puts data from the input plugin into ElasticSearch server. I want this output plugin to partial-updates existing document in ElasticSearch server. my Logstash configuration file looks like:
input {
jdbc {
jdbc_driver_library => "/Users/hello/logstash-2.3.2/lib/mysql-connector-java-5.1.34.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:13306/mysqlDB”
jdbc_user => “root”
jdbc_password => “1234”
last_run_metadata_path => "/Users/hello/.logstash_last_run_display"
statement => "SELECT * FROM checkout WHERE checkout_no between :sql_last_value + 1 and :sql_last_value + 5 ORDER BY checkout_no ASC"
schedule => “*/5 * * * *"
use_column_value => true
tracking_column => “checkout_no”
}
}
output {
stdout { codec => json_lines }
elasticsearch {
action => "update"
index => "ecs"
document_type => “checkout”
document_id => “%{checkout_no}"
hosts => ["localhost:9200"]
}
}
the problem is that ElasticSearch output plugin appears not to call partial update API such as /{index}/{type}/{id}/_update. the manual just lists actions such as index, delete, create, update, But it doesn’t mention each action calls which REST API URL, i.e) Whether update action calls /{index}/{type}/{id}/_update or /{index}/{type}/{id} api (upsert). I would like to call partial update api from elastic search output plugin? Is it possible?
set both doc_as_upsert => true and action => "update" works in my production script.
output {
elasticsearch {
hosts => ["es_host"]
document_id => "%{id}" # !!! the id here MUST be the same
index => "logstash-my-index"
timeout => 30
workers => 1
doc_as_upsert => true
action => "update"
}
}
It is possible. The Elasticsearch output plugin has a series of upsert options that correspond to the ones in Elasticsearch update API:
upsert itself: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-upsert
scripted_upsert: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-scripted_upsert
doc_as_upsert: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-doc_as_upsert