I'm having issues connecting Logstash using Oracle wallet. I am getting an invalid login/password even though I am trying to connect through the wallet, without using said login/password.
[ERROR][logstash.inputs.jdbc ] Unable to connect to database. Tried 2 times {:error_message=>"Java::JavaSql::SQLException: ORA-01017 Invalid Username/Password"}
Here's my Logstash jdbc input file :
input {
jdbc {
jdbc_driver_library => "${ORACLE_HOME}/dmu/jlib/ojdbc10.jar,${ORACLE_HOME}/ucp/lib/ucp.jar,${ORACLE_HOME}/jlib/oraclepki.jar,${ORACLE_HOME}/jlib/osdt_cert.jar,${ORACLE_HOME}/jlib/osdt_core.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:/#name_high?TNS_ADMIN=${ORACLE_HOME}/network/admin/wallet"
jdbc_user => ""
schedule => "* * * * * *"
statement => "SELECT * from table"
connection_retry_attempts => 2
connection_retry_attempts_wait_time => 5
jdbc_pool_timeout => 20
jdbc_validation_timeout => 600
}
}
So I use the following drivers :
ojdbc10.jar
ucp.jar
oraclepki.jar
osdt_cert.jar
osdt_core.jar
I made sure every environement variable paths are correct, every access rights correctly set.
I believe the connection string syntaxe is correct since it does find the tnsnames.ora and attempt a connexion to the right host, port, sid.
I omitted the field jdbc_password in the jdbc input as it is not required and I don't want jdbc to think I want to use it. I left the jdbc_user field but empty as it is apparently required even though I'm trying not to use it.
I'm able to connect to the database using sqlplus as follows:
sqlplus /#name_high
So what I was missing was the java property wallet_location wich append to be the same path as the TNS_ADMIN in my case.
It needs to be loaded by the script that launch the JRE when starting Logstash as such :
-Doracle.net.wallet_location=${WALLET_PATH}
You can refer to the blog for steps with different versions of the driver. Also, you can refer to the blog to know different ways to set connection properties.
Related
I am trying to use logstash to connect my MSSQL server with elastic for real time sync. I have installed elastic 8.3.3 and kibana 8.3.3, I am able to log in to kibana and create some sample index users_test and one sample text variable "name". Elastic is running on https://localhost:9200/, i verified by logging into and got the success message.
I have created a sample config file for logstash
input {
jdbc {
jdbc_driver_library => "D:\sqljdbc_12.2.0.0_enu\sqljdbc_12.2\enu\mssql-jdbc-12.2.0.jre8.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://<dbpath>/<dbname>"
jdbc_user => <db_userid>
jdbc_password => <db_passwd>
statement => "SELECT * FROM Users"
}
}
output {
elasticsearch {
hosts => ["https://localhost:9200/"]
index => "users_test"
}
}
i run it with logstash.bat -f sql.conf and get the error message. logstash is also
Attempted to resurrect connection to dead ES instance, but got an error {:url=>"https://localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [https://localhost:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
The error message says
unable to find valid certification path to requested target
The reason is explained here
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target and javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
This SunCertPathBuilderException indicates that a certificate was returned during the handshake that is not trusted. This message is seen on the client side of the connection. The SSLException is seen on the server side of the connection. The CA certificate that signed the returned certificate was not found in the keystore or truststore and needs to be added to trust this certificate.
You need to add the cacert parameter to your elasticsearch output to validate the certificate returned from the server and you also probably need to add a username and password as well:
elasticsearch {
hosts => ["https://localhost:9200/"]
index => "users_test"
cacert => "/path/to/the/ca/cert"
user => "username"
password => "password"
}
I have installed Oracle databese 18c XE, and Oracle APEX 21.2.0,
I was getting the below error when trying to (Create REST Data Source) for APEX Application
"ORA-24247: network access denied by access control list(ACL)"
after doing some research I connected sys as sysdba using SQL Plus, and then executed the below:
ALTER SESSION SET CONTAINER = XEDBA1;
after that I executed the below commands:
BEGIN
DBMS_NETWORK_ACL_ADMIN.APPEND_HOST_ACE(
host => '*',
ace => xs$ace_type(privilege_list => xs$name_list('connect', 'resolve'),
principal_name => 'APEX_210200',
principal_type => xs_acl.ptype_db));
END;
/
BEGIN
DBMS_NETWORK_ACL_ADMIN.APPEND_HOST_ACE(
host => '*',
ace => xs$ace_type(privilege_list => xs$name_list('connect', 'resolve'),
principal_name => 'MDS_SCHEMA',
principal_type => xs_acl.ptype_db));
END;
/
ACL Commands
where:
APEX_210200 is the INTERNAL workspace schema
MDS_SCHEMA is the schema for the workspace which the application was build
Now I'm getting the error "ORA-28860: Fatal SSL error" when trying to create REST Data Source
Create REST Data Source
ORA-28860
And I can't send emails knowing that SMTP server is set correctly
Please help so I can Create REST Data Source and send Emails through my APEX application
I'm trying to get the data from my Oracle Database to a Databricks Cluster. But I think I'm doing it wrong:
On the cluster library I just installed the ojdbc8.jar and then after that I opened a notebook and did this to connect:
CREATE TABLE oracle_table
USING org.apache.spark.sql.jdbc
OPTIONS (
dbtable 'table_name',
driver 'oracle.jdbc.driver.OracleDriver',
user 'username',
password 'pasword',
url 'jdbc:oracle:thin://#<hostname>:1521/<db>')
And it says:
java.sql.SQLException: Invalid Oracle URL specified
Can someone help? I've been reading documentations but there's no clear instruction on how I should actually install this jar step by step. I might be using the wrong jar? Thanks!
I have managed to set this up in Python/PySpark as follows:
jdbcUrl = "jdbc:oracle:thin:#//hostName:port/databaseName"
connectionProperties = {
"user" : username,
"password" : password,
"driver" : "oracle.jdbc.driver.OracleDriver"
}
query = "(select * from mySchema.myTable )"
df = spark.read.jdbc(url=jdbcUrl, table=query1, properties=connectionProperties)
I am using the Oracle JDBC Thin Driver instantclient-basic-linux.x64-21.5.0.0.0, as available on the Oracle webpages. The current version is 21.7 I think, but it should work the same way.
Check this link to understand the two different notations for jdbc URLs
Has anyone faced this issue while connecting to an Oracle DB from Logstash JDBC Input Plugin :
[2021-07-22T11:22:42,912][ERROR][logstash.inputs.jdbc ][main][9615a4202c23f17db8abee168682c63c25349f105e2579d02086d38fe8145d97] Unable to connect to database. Trying again {:error_message=>"Java::JavaSql::SQLRecoverableException: IO Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond, Authentication lapse 0 ms."}
[2021-07-22T11:22:42,922][ERROR][logstash.inputs.jdbc ][main][b9b24f18d4550bc4a001bc364e9e4d369504fce0c99e762583d6da267f2f7e5e] Unable to connect to database. Tried 1 times {:error_message=>"Java::JavaSql::SQLRecoverableException: IO Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond, Authentication lapse 0 ms."}
[2021-07-22T11:22:42,924][ERROR][logstash.inputs.jdbc ][main][5d2187e4deef8e15edbd453b49ce621da2d805def7d3a2a1a45abb4b261c5a7f] Unable to connect to database. Tried 1 times {:error_message=>"Java::JavaSql::SQLRecoverableException: IO Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond, Authentication lapse 0 ms."}
[2021-07-22T11:22:42,943][ERROR][logstash.javapipeline ][main][5d2187e4deef8e15edbd453b49ce621da2d805def7d3a2a1a45abb4b261c5a7f] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_user=>"<User>", jdbc_paging_enabled=>true, jdbc_password=><password>, statement=>"select<Fields> from <Table>", jdbc_driver_library=>"<Path>/lib/ojdbc8-21.1.0.0.jar", jdbc_connection_string=>"jdbc:oracle:thin:#<HOST>:<PORT>:<SID>", id=>"5d2187e4deef8e15edbd453b49ce621da2d805def7d3a2a1a45abb4b261c5a7f", jdbc_driver_class=>"Java::oracle.jdbc.driver.OracleDriver", type=>"cis-api-gateway-request", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_00b6436f-afd3-4f1d-8450-d627939f640e", enable_metric=>true, charset=>"UTF-8">, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, plugin_timezone=>"utc", last_run_metadata_path=>"<Path>/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true, use_prepared_statements=>false>
Error: Java::JavaSql::SQLRecoverableException: IO Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond, Authentication lapse 0 ms.
Exception: Sequel::DatabaseConnectionError
Stack: oracle.jdbc.driver.T4CConnection.handleLogonIOException(oracle/jdbc/driver/T4CConnection.java:913)
I am able to connect with the same credentials and connection details through SQL developer.
Here is my Logstash JDBC Input:
jdbc {
jdbc_connection_string => "jdbc:oracle:thin:#<ORACLE_HOST>:<ORACLE_PORT>:<ORACLE_SID>"
jdbc_user => "<USER>"
jdbc_password => "<PASSWORD>"
jdbc_driver_library => "<PATH>/lib/ojdbc8-21.1.0.0.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_paging_enabled => true
connection_retry_attempts => 5
connection_retry_attempts_wait_time => 10
statement=> "select<Fields> from <Table>"
type => "<Temp Val>"
}
My Oracle version is 19.3 and Logstash-7.9.1 (windows)
I see that you are using ojdbc8.jar from oracle jdbc driver 21.1 release. Try a simple standalone java test to connect your db and see if you get same error or not. If the issue reproduce then paste the full error stack trace. SQLDeveloper might be using a different release ojdbc jar.
I'm trying to install the jdbc plugin for Logstash on a AWS EC2 server so I can query a MySQL database.
Logstash is working but I can't get the JDBC plugin to work. When I run Logstash I get this error:
Error: com.mysql.jdbc.Driver not loaded. Are you sure you've included
the correct jdbc driver in :jdbc_driver_library?
I tried installing the JDBC plugin based on this:
https://discuss.elastic.co/t/getting-started-with-jdbc-input-plugin/77162
The install looked like it worked:
[ec2-user# settings]$ sudo /usr/share/logstash/bin/logstash-plugin install logstash-input-jdbc
Validating logstash-input-jdbc
Installing logstash-input-jdbc
Installation successful
But when I try to find the .jar file for the mysql connector it does not seem to exist on the system:
[ec2-user# logstash]$ sudo find / -name "*connector-java*"
This is what my Logstash config file looks like:
input {
jdbc {
jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://***:3306/website"
jdbc_user => "***"
jdbc_password => "***"
statement => "SELECT * from runtime_ProdFull WHERE `MfPN` LIKE 'KO8%'"
}
}
How do I identify the correct name for jdbc_driver_library and how do I find it's path so I can add it to my config?
Thanks,
Phil
SOLUTION:
I downloaded the latest platform independent version from https://dev.mysql.com/downloads/connector/j/ to my PC, unzipped the mysql-connector-java-8.0.15.jar file and uploaded to the EC2 server via WinSCP to /var/lib/logstash/.
Then changed the jdbc driver to:
jdbc_driver_library => "/var/lib/logstash/mysql-connector-java-8.0.15.jar"
You have to download JDBC driver by yourself. Go to this website and download appropriate driver for your database and then put it somewhere in your EC2. Then you just have to specify path to the downloaded driver in this field jdbc_driver_library and you're good to go.