we had one SSIS package with Oracle 11 Client, we would run our daily query with 30min to 1 hour run time.
we had to upgrade our oracle clients as one of our other oracle source got upgraded.
post upgrade to Oracle 12c, our daily job run time increased.
oracle DBA said, its not running in parallel, as its occupying only one processor.
when we run the same query from SQL Developer or toad, its running in parallel. but if we run from SSIS OLEDB Source component its not running in parallel.
I'm clue less with this behavior. any solution will be helpful.
ask me more clarifications if required.
Trying to figure out the issue
I tried to search on this topic, i didn't find a lot of informations but i think it is based on the OLEDB Connection string provided in the OLEDB Connection Manager.
Check the following Oracle documentation it may give you some insights:
Features of OraOLEDB
In the link above, in the Distributed Transactions part, they mentioned that:
The DistribTX attribute specifies whether sessions are enabled to enlist in distributed transactions. Valid values are 0 (disabled) and 1 (enabled). The default is 1 which indicates that sessions are enabled for distributed transaction enlistments.
Sessions enabled for distributed transaction enlistments cannot run statements that use the direct path load and parallel DML capabilities of the Oracle database. Such statements are executed as conventional path serial statements.
I am not sure if this could help, but it is not bad to give a try.
Oracle Attunity Connectors
Instead of using OLEDB Source to read from oracle, it is better to use Oracle Attunity Connectors for SSIS which guarantee higher performance than OLEDB Source:
Microsoft Connectors By Attunity
Attunity's high speed connectors for Oracle and Teradata have been selected by Microsoft to be included with SQL Server Integration Services (SSIS).
Related
We have a legacy process that runs on SSIS 2016 on Windows Server 2016, executes custom queries against databases on remote servers, pulls the results (thousands or sometimes millions of records) and stores them in a local SQL Server database. These other databases are on DB2 and Oracle 19c.
This process has always connected using an OLE DB driver and a data flow with OLE DB source and destination components. It also has always been slow.
Because of some article we read recently talking about how OLE DB transfers only 1 record at a time, but with ADO.NET this network transfer could be done in batches (is this even true?), we decided to try to use an ADO.NET driver to connect to DB2 and replace the OLE DB source and destination components by ADO.NET components.
The transfer we were using as test case, which involved 46 million records, basically flew and we could see it bring down around 10K records at a time. Something that used to run in 13 hours ran in 24 minutes with no other changes. Some small tweaks in the query allowed us to bring that time even lower to 11 minutes.
This is obviously major and we want to be able to replicate it with our Oracle data sources. Network bandwidth seems to have been the main issue, so we want to be able to transfer data from Oracle 19c to our SQL Server 2016 databases using SSIS in batches, but want to ask the experts what the best/fastest way to do this is.
Is Microsoft Connector for Oracle the way to go as far as driver? Since we're not on SQL Server 2019, this article says we also need to install the Oracle Client and Microsoft Connector Version 4.0 for Oracle by Attunity. What exactly is the Oracle Client? Is it one of these? If so, which one, based on our setup?
Also, should we use ADO.NET components in the data flow just like we did with DB2? In other words, is the single record vs. record batches difference driven by the driver used to connect, the type of components in the data flow or both need to go hand in hand for this to work?
Thanks in advance for your responses!
OLEDB connections are not slow by themselves - it's a matter or what features the driver has available to it. It sounds like the ADO.NET driver for DB2 allows bulk insert and the OLEDB one does not.
Regarding Oracle, the attunity driver is the way to go. You'll need to install the oracle driver as well. The links that you have look correct to me but I don't have access to test.
Also, please note that dataflows will batch data by default in increments of the buffer size. 10k rows for example.
I need to run a performance comparison before activating Oracle TDE on our database (12c).
My idea was to monitor executed queries for a specific user (Java EE App Server) and then replay these queries on a cloned Oracle TDE instance to compare response times.
We found the Oracle AWR feature, however, in this report over 90% of the queries are "system" queries which were not triggered by the App Server's JDBC driver(, but by internal oracle jobs/optimizations?).
Is there some functionality to restrict AWR monitoring to only use queries of a specific user/driver or is there another functionality I could use for conducting my performance tests?
Thanks!
Oracle's Automated Workload Repository (AWR) captures all database activity. You have some flexibility in the depth of statistics it captures, but you can't filter the way you are asking for. When you product an AWR report, you will get everything the database performed, and specifically for the SQL data, you'll get the top or highest load SQL's. There are some additional thresholds you can set to get more SQL in the report, so instead of the top-10, you can get the top-50. This can be set as a default or per AWR report.
More can be found here
Oracle workload repository package
We have a requirement to implement replication (of transactions) from a production database to a test database (across a db link). We're using 11g enterprise edition.
This is because it's not just the production data that's needed, the intention is that code releases in the test database would be tested against real-world transactions from the prod system, without the need for those transactions to be done manually. If a transaction fails then in the test system that worked in the prod system, something is wrong with the release.
It doesn't necessarily have to be in real-time however, a delay is acceptable.
There must be zero risk though on the production transactions failing, due an issue with the replication.
What are the options here? I believe Streams is deprecated in 12c, should that be of concern? GoldenGate is additional ££, which I'm afraid rules it out in this case. Some kind of custom trigger-based solution... too much risk on the prod system as far as I can see.
Any advice greatly appreciated!
As of Oracle release 12c (12.1.x), Oracle Advanced Replication and Oracle Streams are being discontinued. Oracle Golden Gate is going to replace all features of Oracle Advanced Replication and Oracle Streams.
Please review this list of possible alternatives (to name a few):
SharePlex
DBVisit
DBConvert
the designing.making/deploying/management of a distributed database?
or do i need to buy the versions for the above?
if it 10g express has this support, will there be a central point to which my web application will connect, or do i have to write connection strings to connect to each server making "the distributed database "
please also refer to this post
Oracle distributed databases and MSVC
Oracle Express Edition is limited to a single CPU and 4GB of data on a single server. That's why it's free. If you want to use it in a distributed configuration you will have to write the glue yourself.
The following shows the replication capabilities as described for ms sql server on oracle
website.
"the text inside the " " are my questions/confusion AGAINST what oracle website says"
Please clear these points, what do you think? Is this article out of date?
Replication functionality in Microsoft
SQL Server or Sybase Adaptive Server
has the following characteristics:
Unidirectional "merge replication supports!!"
Table-based, not transaction-based "in transaction replication new
changes/transactions are replicated in
real time!!!"
No automatic conflict resolution (must be manual) "Really? what about
merge replication"??
Heterogeneous replication through Open Database Connectivity (ODBC)
In addition to the preceding
characteristics, Microsoft SQL Server
7.0 replication provides heterogeneous replication through ODBC.
Oracle replication has richer
replication functionality, which
includes the following:
Bi-directional
Any database object can be replicated
Automatic resynchronization
Automatic conflict resolution
Heterogeneous replication provided through gateways
Is marketing material about direct competitor accurate, up to date and unbiased? Use your brain, user287745...
SQL Server 7.0 is rather old. Since then 4 (four) more major editions have shipped (2000, 2005, 2008 and R2). If you want to read about SQL Server Replication, you better read on the MSDN site. SQL Server Replication.
Outdated. Really. or worse, whoever wrote that at oracle... left information out on purpose.