i want to create an origin source from oracle. so I choos as origin oracle cdc. then I configured each parameter:
Schema Name
Table
Username
Password
JDBC Connection String
but when I run the process, i find into my log:
2017-08-22 11:07:22,447 test/testb156f588-dbd7-4e4c-8896-caf658d14d77 ERROR Error while connecting to DB
com.streamsets.pipeline.api.StageException: JDBC_06 - Failed to initialize connection pool: java.lang.RuntimeException: Unable to get driver instance for jdbcUrl=jdbc:oracle:thin:#(DESCRIPTION =
(ENABLE=BROKEN)
(ADDRESS = (PROTOCOL = TCP)(HOST = myhost)(PORT = myport))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl.WORLD)))
at com.streamsets.pipeline.lib.jdbc.JdbcUtil.createDataSourceForRead(JdbcUtil.java:638)
at com.streamsets.pipeline.stage.origin.jdbc.cdc.oracle.OracleCDCSource.init(OracleCDCSource.java:643)
at com.streamsets.pipeline.api.base.BaseStage.init(BaseStage.java:52)
at com.streamsets.pipeline.configurablestage.DStage.init(DStage.java:40)
at com.streamsets.datacollector.runner.StageRuntime.init(StageRuntime.java:156)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:105)
at com.streamsets.datacollector.runner.StagePipe.init(StagePipe.java:53)
at com.streamsets.datacollector.runner.Pipeline.initPipe(Pipeline.java:299)
at com.streamsets.datacollector.runner.Pipeline.init(Pipeline.java:214)
at com.streamsets.datacollector.execution.runner.common.ProductionPipeline.run(ProductionPipeline.java:96)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunnable.run(ProductionPipelineRunnable.java:79)
at com.streamsets.datacollector.execution.runner.standalone.StandaloneRunner.start(StandaloneRunner.java:646)
at com.streamsets.datacollector.execution.runner.common.AsyncRunner.lambda$start$3(AsyncRunner.java:143)
at com.streamsets.pipeline.lib.executor.SafeScheduledExecutorService$SafeCallable.call(SafeScheduledExecutorService.java:233)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Unable to get driver instance for jdbcUrl=jdbc:oracle:thin:#(DESCRIPTION =
(ENABLE=BROKEN)
(ADDRESS = (PROTOCOL = TCP)(HOST = myhost)(PORT = myport))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl.WORLD)))
at com.zaxxer.hikari.util.DriverDataSource.<init>(DriverDataSource.java:88)
at com.zaxxer.hikari.pool.PoolElf.initializeDataSource(PoolElf.java:157)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:113)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:73)
at com.streamsets.pipeline.lib.jdbc.JdbcUtil.createDataSourceForRead(JdbcUtil.java:630)
... 19 more
Caused by: java.sql.SQLException: No suitable driver
at java.sql.DriverManager.getDriver(DriverManager.java:315)
at com.zaxxer.hikari.util.DriverDataSource.<init>(DriverDataSource.java:81)
... 23 more
Do you have any idea?
I had to add the ojdbc jar under the repository
/opt/streamsets-extra/streamsets-datacollector-jdbc-lib/lib
Seems that it's missing to set environment variable STREAMSETS_LIBRARIES_EXTRA_DIR which's a part of installing external library installation process.
There are three types of installation depending on initialization style of Data Collector :
Manually start
Using SysV Init ( supported by CentOS 6, Oracle Linux 6, Red Hat
Enterprise Linux 6 or Ubuntu 14.04 LTS )
Using Systemd Init ( supported by CentOS 7, Oracle Linux 7, Red Hat
Enterprise Linux 7 or Ubuntu 16.04 LTS )
If you're installing SDC
being started manually, then variable STREAMSETS_LIBRARIES_EXTRA_DIR is expected to be set from the command line by
export STREAMSETS_LIBRARIES_EXTRA_DIR="/opt/streamsets-data-collector/streamsets-data-collector-3.15.0/streamsets-libs-extras/"
starting as a service, then this parameter already would already exist
within the $SDC_DIST/libexec/_sdc file as
STREAMSETS_LIBRARIES_EXTRA_DIR="${STREAMSETS_LIBRARIES_EXTRA_DIR:=${SDC_DIST}/streamsets-libs-extras}"
where $SDC_DIST variable is the extraction directory for SDC installation file(tarball or RPM).
and the same path should be added to the file $SDC_CONF/sdc-security.policy such as
grant codebase "file:///opt/streamsets-data-collector/streamsets-data-collector-3.15.0/streamsets-libs-extras/-" {
permission java.security.AllPermission;
};
where $SDC_CONF variable is typically defined by the path /etc/sdc
Now, we're ready to log in Data Collector Console in order to add JDBC External library following these steps :
In Data Collector, in the top right toolbar, click the Package
Manager icon:
In the navigation panel, click External Libraries :
Data Collector lists any currently installed external libraries.
Immediately under the top right toolbar, click the Install
External Libraries icon:
In the Install External Libraries dialog box, select the stage
library JDBC from the file system (assuming you were already registered and downloaded streamsets-datacollector-jdbc-lib);
and the choose the .jar file such as ojdbc8.jar which can be downloaded from JDBC and UCP Downloads page ( in my case I've chosen the link named Oracle Database 12c Release 2 (12.2.0.1) drivers due to my remote DB version ).
As the last step, Don't forget to click Cancel in the Install External Libraries window, and then run the following command :
service sdc restart ( for SysV Init)
or
systemctl restart sdc ( for Systemd Init )
( You can click Restart Data Collector in the Install External Libraries window provided that you started the Data Collector manually from the command line. )
Under the Legacy configuration tab try specifying the JDBC Driver Class name as oracle.jdbc.driver.OracleDriver
Related
I am developing a Spark Streaming application which would listen to a folder (partitioned as yyyyMMdd) and aggregate the number of records written per minutes then persist the results to an Oracle table.
I have developed a JDBCSink (ForeachWriter) and in the open method I'm trying to open a connection to Oracle but I am getting "oracle.net.ns.NetException: could not resolve the connect identifier" exception while creating the Oracle connection. I am using Oracle wallet (SSO) and I'm able to connect over sqlplus using this wallet by setting TNS_ADMIN environment variable.
I am pushing the tnsnames.ora, sqlnet.ora, cwallet.sso and ewallet.p12 with the spark-submit --files option, and I have verified the files are pushed to the executors with the SparkFiles.get method in the sink class. I have also added third party Oracle dependencies for Oracle wallet with spark-submit --jars option (namely ojdbc7.jar,oraclepki.jar,osdt_cert.jar,osdt_core.jar)
The code piece for opening the connection is as follows:
Class.forName("oracle.jdbc.driver.OracleDriver")
System.setProperty("oracle.net.tns_admin", new Path(SparkFiles.get("tnsnames.ora")).getParent.getName)
val ds = new OracleDataSource()
val props = new Properties()
props.setProperty(OracleConnection.CONNECTION_PROPERTY_WALLET_LOCATION,
new Path(SparkFiles.get("cwallet.sso")).getParent.getName)
ds.setConnectionProperties(props)
ds.setURL("jdbc:oracle:thin:#xe")
I have tried to isolate the problem
The Oracle version is 12.1.0.2 (I am using a Docker image)
spark-submit2 ^
--master local ^
--files "%CWD%\wlt\tnsnames.ora,%CWD%\wlt\sqlnet.ora,%CWD%\wlt\cwallet.sso,%CWD%\wlt\ewallet.p12" ^
--jars "%CWD%\lib\ojdbc7.jar,%CWD%\lib\oraclepki.jar,%CWD%\lib\osdt_cert.jar,%CWD%\lib\osdt_core.jar" ^
--class OJDBCSinkMain ^
.\target\spark-streaming-ojdbc-sink-1.0-SNAPSHOT-jar-with-dependencies.jar
My sqlnet.ora file is as follows:
NAMES.DIRECTORY_PATH=(TNSNAMES, EZCONNECT)
SQLNET.WALLET_OVERRIDE=TRUE
SSL_CLIENT_AUTHENTICATION=FALSE
SSL_VERSION=0
and my tnsnames.ora file is:
xe =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = xe)
)
)
I also verified the credentials for Oracle service xe exists in my wallet:
comment mkstore -wrl . -listCredential
List credential (index: connect_string username)
1: xe system
Do you have any comments? Thanks in advance.
My mistake, I needed a rubber dock to spot!!! SparkFiles.get("tnsnames.ora")).getParent.getName returns relative path, not the absolute. My problem is solved now.
I'm installing Oracle Database 12c Release 2 Installer, and it jumps error message
[INS 20802] Oracle Net Configuration Assistant Failed.
and the log flie error message is
INFO: Parameter "maskpasswords" = false
WARNING: Skipping line: Parameter "maskpasswords" = false
INFO: Read: Done parsing command line arguments.
INFO: Done parsing command line arguments.
WARNING: Skipping line: Done parsing command line arguments.
INFO: Read: Oracle Net Services Configuration:
INFO: Oracle Net Services Configuration:
WARNING: Skipping line: Oracle Net Services Configuration:
INFO: Read: ProfileException: Could not create Profile: TNS-04414: File error
INFO: ProfileException: Could not create Profile: TNS-04414: File error
WARNING: Skipping line: ProfileException: Could not create Profile: TNS-04414: File error
INFO: Read: Error: null
INFO: Error: null
WARNING: Skipping line: Error: null
INFO: Read: Check the trace file for details: E:\app\test2\cfgtoollogs\netca\trace_OraDB12Home1-2006184ŕň2754.log
INFO: Check the trace file for details: E:\app\test2\cfgtoollogs\netca\trace_OraDB12Home1-2006184ŕň2754.log
WARNING: Skipping line: Check the trace file for details: E:\app\test2\cfgtoollogs\netca\trace_OraDB12Home1-2006184ŕň2754.log
INFO: Read: Oracle Net Services configuration failed. The exit code is 1
INFO: Oracle Net Services configuration failed. The exit code is 1
WARNING: Skipping line: Oracle Net Services configuration failed. The exit code is 1
INFO: Completed Plugin named: Oracle Net Configuration Assistant
INFO: Oracle Net Configuration Assistant failed.
INFO: Oracle Net Configuration Assistant failed.
INFO: ConfigClient.executeSelectedToolsInAggregate action performed
INFO: Exiting ConfigClient.executeSelectedToolsInAggregate method
INFO:
The Runconfig command constructed is E:\app\test2\product\12.2.0\dbhome_1\oui\bin\runConfig.bat ORACLE_HOME=E:\app\test2\product\12.2.0\dbhome_1 MODE=perform ACTION=configure RERUN=false %*
INFO: Since the option is to overwrite the existing E:\app\test2\product\12.2.0\dbhome_1\cfgtoollogs\configToolFailedCommands file, backing it up
INFO: The backed up file name is E:\app\test2\product\12.2.0\dbhome_1\cfgtoollogs\configToolFailedCommands.bak.5
INFO: Adding ExitStatus SUCCESS_MINUS_RECTOOL to the exit status set
INFO: ConfigClient.saveSession method called
INFO: Calling event ConfigSessionEnding
INFO: ConfigClient.endSession method called
INFO: Completed Configuration
How to solve this error and let install going forward?
Screenshot:
EDIT
I then try to use DBCA to create, but unfortunately first time it hang stuck at 44%, 2/3/4 time it stuck at 59% ...
This is a related article but not work ... #Oracle Corporation help me with your product please...
I believe you apply for this :
TNS-04414 While Configuring A Listener Using NetCA Or Net Configuration Assistant (Doc ID 467391.1)
Is it a listener.ora file is already present in the $ORACLE_HOME/network/admin directory ?. If so, try to review the syntax of it. You are trying to add a new listener using Oracle Net Configuration Assistant (NETCA).
Check listener.ora file for missing parenthesis:
LISTENER =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1) <--missing closing parenthesis
(ADDRESS = (PROTOCOL = TCP)(HOST = myhost)(PORT = 1521))
)
Correct the syntax error in the listener.ora. The corrected listener.ora entry would appear as -
LISTENER =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
(ADDRESS = (PROTOCOL = TCP)(HOST = myhost)(PORT = 1521))
)
or
Rename the present listener.ora file and create a new file using NETCA.
Note - Renaming the present lisetner.ora file and creating a new one with NETCA will remove all the old listener configurations.
Hope it helps
Can you provide the content of this file ?
E:\app\test2\cfgtoollogs\netca\trace_OraDB12Home1-2006184ŕň2754.log
How to configure Oracle to be able to query a MS Access database (.accdb) that is lcoated in the same server (localhost)?
Which steps to follow?
Version of Oracle: 11g -> 11.2.0.1.0
Version of Windows: Windows Server 2008
Steps to complete:
Create System DSN with Microsoft Access Driver
Modify listener.ora file
Create the file initaccess.ora
Modify tnsnames.ora file
Restart listener
Check connection with access database
Create the DB Link in Oracle
STEP 1: Create system DSN
First go to ODBC Data Source Administrator click on System DSN tab. The create a DSN with the following parameter (you can choose the name link that you prefer but then you need to keep the same name all over the process).
In our case:
Driver: Microsoft Access Driver (*.mdb,*.accdb)
Name: access
Description: (doesn't matter)
Database -> Select -> (Select the .accdb or .mdb database)
Save the configuration
Make sure you see this listed once you save it:
Name - Driver
access - Microsoft Access Driver (*.mdb,*.accdb)
STEP 2: Modify listener.ora file
Let's locate the file. In our case:
E:\app\Administrador\product\11.2.0\dbhome_1\NETWORK\ADMIN
We need to add the following text at the end. By the way, look at I posted as 'oracle_home' the directory in C: drive. That's because I had more than one directory related to Oracle, and I still don't understand it but worked in my case. Maybe in yours there's only one.
Parameters you need to modify:
In first block:
SID_NAME: name chosen in the step above. The name of the dsn created ('access' in our case)
ORACLE_HOME: Home directory of Oracle
PROGRAM = dg4odbc (I think that from Oracle 11g and beyond, 'dg4odbc' is mandatory.
In second block:
HOST: The name of the hostname (not sure if localhost' or an IP address are supported, but I guess they are)
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC=
(SID_NAME = access)
(ORACLE_HOME = C:\app\Administrador\product\11.2.0\dbhome_1)
(PROGRAM = dg4odbc)
)
)
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = host_name)(PORT = 1521))
)
)
Make sure there is no space before 'SID_LIST_LISTENER' or 'LISTENER' in the file you modify:
STEP 3: Create the initaccess.ora file
Go to:
E:\app\Administrador\product\11.2.0\dbhome_1\hs\admin
There, make a copy of 'initdg4odbc.ora' file (I deleted it after I made a copy). Add the following text to the file:
HS_FDS_CONNECT_INFO = access
HS_FDS_TRACE_LEVEL = 0
HS_FDS_CONNECT_INFO: Here we write the name of the link created in the ODBC source tool in the first step, so it's 'access'
HS_FDS_TRACE_LEVEL: Leave it with a 0
STEP 4: Modify tnsnames.ora file:
Let's locate the file. In our case:
E:\app\Administrador\product\11.2.0\dbhome_1\NETWORK\ADMIN
Parameters to modify:
HOST: hostname of the server
SID: this is the name of the 'initaccess.ora' file BUT taking away the 'init' part and '.ora' file extension: then 'access'.
(HS=OK) = don't forget this.
access =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = host_name) (PORT = 1521))
(CONNECT_DATA =
(SID = access)
)
(HS=OK)
)
Be sure that you don't leave spaces before the 'access =' parameter.
STEP 5: Restart the listener
Open a command line terminal
Go to: C:\app\Administrador\product\11.2.0\dbhome_1\BIN
Stop the listener
C:\app\Administrador\product\11.2.0\dbhome_1\BIN>LSNRCTL.EXE stop
You should see a message saying it was successfull
Restart the listener
C:\app\Administrador\product\11.2.0\dbhome_1\BIN>LSNRCTL.EXE start
You should see a message saying it was successfull
You should also see, (in the output, when restarting the server) a message talking about the 'access' service we just added
El servicio "access" tiene 1 instancia(s).
La instancia "access", con estado UNKNOWN, tiene 1 manejador(es) para este ser
vicio...
El comando ha terminado correctamente
STEP 6: Test connection with the access database
Open a command line terminal
Go to: C:\app\Administrador\product\11.2.0\dbhome_1\BIN
Run tnsping.exe (as argument, pass the name of the link: 'access'):
C:\app\Administrador\product\11.2.0\dbhome_1\BIN>tnsping.exe access
You should see a message saying it was correctly run and displaying the delay in msecs
TNS Ping Utility for 64-bit Windows: Version 11.2.0.1.0 - Production on 06-JUN-2
014 11:04:35
Copyright (c) 1997, 2010, Oracle. All rights reserved.
Archivos de parßmetros utilizados:
E:\app\Administrador\product\11.2.0\dbhome_1\network\admin\sqlnet.ora
Adaptador TNSNAMES utilizado para resolver el alias
Intentando contactar con (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = CUPCA
KE) (PORT = 1521)) (CONNECT_DATA = (SID = access)) (HS=OK))
Realizado correctamente (20 mseg)
STEP 7: Create the DB Link in Oracle and query the Ms Access database:
Create the DB Link (always as **sys):**
CREATE public DATABASE LINK accessdblink USING 'access';
* 'access': Same name as in tnsnames.ora -> 'access = '
Query the database:
SELECT * FROM table_name#accessdblink;
I have this message while installing ORACLE 11g Release1 on Windows 7 64bit
before Installing the Oracle DB I created the "Microsoft LoopBack Adapter" & set the adapter to be the first order
Checking Network Configuration requirements ...
Check complete. The overall result of this check is: Failed <<<<
Problem: The install has detected that the primary IP address of the
system is DHCP-assigned.
Recommendation: Oracle supports installations on systems with
DHCP-assigned IP addresses; However, before you can do this, you must
configure the Microsoft LoopBack Adapter to be the primary network
adapter on the system. See the Installation Guide for more details on
installing the software on systems configured with DHCP.
And After the installation done I tried to test the connection using SQL Plus I faced this error
-- This is the tnsnames.ora contants
# tnsnames.ora Network Configuration File: E:\ORACLE\Oracle_Base\product\11.1.0\db_1\network\admin\tnsnames.ora
# Generated by Oracle configuration tools.
MTAHAPC =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = MTahaPC.lan)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = MTAHAPC)
)
)
-- this is the listener.ora
# listener.ora Network Configuration File: E:\ORACLE\Oracle_Base\product\11.1.0\db_1\network\admin\listener.ora
# Generated by Oracle configuration tools.
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = MTahaPC.lan)(PORT = 1521))
)
)
Sorry, this is the first trail to install Oracle DB, and I get tired from it
That's a warning indicating that your system has DHCP allocation of IPs. This isn't strictly supported because when the service lookup happens, the TNSNAMES.ora file will have an entry for an IP which may or may not correspond to your system since DHCP would mean the IP may be assigned to other computers when the lease is up.
You can choose to ignore this by clicking on the checkbox next to it to indicate that you want to skip the test.
I'm trying to implement JDBCRealm in tomcat (as described in http://tomcat.apache.org/tomcat-4.1-doc/realm-howto.html#Configuring%20a%20Realm) to check credential agains Oracle Database. The thing is that I want database to communicate over SSL. So I configured listener to use TCPS. Like that:
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<hostname>)(PORT=1521)))<br>
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=<hostname>)(PORT=1512)))
Generated certificates, etc. I have no problem connecting to database using tcps from sqlplus or from WLS (I can use connection property oracle.net.ssl_cipher_suites=(SSL_DH_anon_WITH_3DES_EDE_CBC_SHA,SSL_DH_anon_WITH_RC4_128_MD5,SSL_DH_anon_WITH_DES_CBC_SHA) there without any problem).
However, I cant get JDBCRealm to work with the database over TCPS. If I configure realm like this:
<Realm className="org.apache.catalina.realm.JDBCRealm" debug="99" driverName="oracle.jdbc.driver.OracleDriver" connectionURL="jdbc:oracle:thin:#(DESCRIPTION = (ADDRESS = (PROTOCOL = TCPS)(HOST = <hostname>)(PORT = 1512)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = <service name>)))" connectionName="<login>" connectionPassword="<password>" userTable="users" userNameCol="user_name" userCredCol="user_pass" userRoleTable="user_roles" roleNameCol="role_name" />
I'm getting following error:
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
I imported certificates in JKS store which I configured in Tomcat like this:
With no success.
I'm not a professional in Tomcat (more in databases). I'll appreciate any help or pointing me in right direction. Thanks in advance!
Error seems to indicate that SSL certificate is not trusted.
I suggest to verify that you indeed have certificate in your trust store (there is a command line tool in JDK to list trust store content, you can Google it) and then add following parameter in Tomcat startup script:
javax.net.ssl.trustStore=<path to trust store>