When I use the OLE DB Destination Editor on SSIS to connect to an oracle database, it takes a long time (15-30 minutes) to retrieve the list of tables in the database.
The database is not so big (400 tables and 50 views) and is based on Oracle 12.
I have done some SSIS packages connected to other Oracle databases before and it has never been so long. What can I look at to determine what is the issue and how to solve it ?
Thanks,
Related
We have a legacy process that runs on SSIS 2016 on Windows Server 2016, executes custom queries against databases on remote servers, pulls the results (thousands or sometimes millions of records) and stores them in a local SQL Server database. These other databases are on DB2 and Oracle 19c.
This process has always connected using an OLE DB driver and a data flow with OLE DB source and destination components. It also has always been slow.
Because of some article we read recently talking about how OLE DB transfers only 1 record at a time, but with ADO.NET this network transfer could be done in batches (is this even true?), we decided to try to use an ADO.NET driver to connect to DB2 and replace the OLE DB source and destination components by ADO.NET components.
The transfer we were using as test case, which involved 46 million records, basically flew and we could see it bring down around 10K records at a time. Something that used to run in 13 hours ran in 24 minutes with no other changes. Some small tweaks in the query allowed us to bring that time even lower to 11 minutes.
This is obviously major and we want to be able to replicate it with our Oracle data sources. Network bandwidth seems to have been the main issue, so we want to be able to transfer data from Oracle 19c to our SQL Server 2016 databases using SSIS in batches, but want to ask the experts what the best/fastest way to do this is.
Is Microsoft Connector for Oracle the way to go as far as driver? Since we're not on SQL Server 2019, this article says we also need to install the Oracle Client and Microsoft Connector Version 4.0 for Oracle by Attunity. What exactly is the Oracle Client? Is it one of these? If so, which one, based on our setup?
Also, should we use ADO.NET components in the data flow just like we did with DB2? In other words, is the single record vs. record batches difference driven by the driver used to connect, the type of components in the data flow or both need to go hand in hand for this to work?
Thanks in advance for your responses!
OLEDB connections are not slow by themselves - it's a matter or what features the driver has available to it. It sounds like the ADO.NET driver for DB2 allows bulk insert and the OLEDB one does not.
Regarding Oracle, the attunity driver is the way to go. You'll need to install the oracle driver as well. The links that you have look correct to me but I don't have access to test.
Also, please note that dataflows will batch data by default in increments of the buffer size. 10k rows for example.
I have a Oracle 10g database currently in a server.
I will create a web which may have massive requests to get data from the Oracle 10g. So, to prevent degrade the Oracle 10g database and server performance, I would like to setup an Oracle Express 11g to another server, periodically (say 15 minutes) get the data from Oracle 10g to Oracle Express 11g.
Finally the web will just get the data from the new server. And the only loading to the Oracle 10g is the data synchronization. There are just 2 tables in Oracle 10g need to sync.
My question is how can I set up the data synchronization?
For example, can I set up linked server, and set materialized view with refresh period between 2 oracles.
I am new to Oracle, welcome any useful reference materials and idea.
We have two divisions in our company, one uses E1 on Oracle 11g the other uses SAP on Oracle 11g.
We also have a SQL Server system we use to data warehouse information once a night from both system to run our report server against.
The question I have is for pooled tables in SAP, such as A016, how would I get that information out of SAP?
Currently we have SSIS's setup with a linked server to the two Oracle servers which pull the data we need I just don't have the knowledge of SAP to find the Pooled tables.
if I can't pull the pooled tables because they don't physically exist is there a tool I can use in SAP to find out what tables the pooled table is getting it's information from? This way I can rebuild that table in SQL using a open query and some fun Joins.
Thanks
You have to access those tables using the application server. They can't be accessed directly from the database.
You'll probably want to write an ABAP program to extract the data you need go from there.
I have a Oracle server which is very highly loaded. I want to Select big table with several millions rows of data and download the data using SQL Developer. How I can do this with little performance impact of the production database?
use oracle's export tool for most efficient dump of data from a table.
example
exp user/password tables=mytable query="optional where clause here"
Note you have to have oracle client installed to use it.
I'm trying to load a table from SQL Server using the Microsoft OLE DB Provider into an Oracle table (using the Oracle Provider for OLE DB). The package is a straight forward OLE DB Source (SQL Server) -> OLE DB Destination (Oracle).
I'm using SQL Server 2008 R2 and Oracle 11g.
Every time I run the package, I get a different number of rows in the destination table, and BIDS reports fewer rows read than there are in the source table. The number of rows returned is different each time I run it. I get no errors or kickouts, but the boxes for the source and destination remain yellow even after BIDS says "Package completed successfully".
Dumping the source table into a flat file instead of the Oracle destination works fine, and I get all the rows that I expect. I can use this flat file to pull the information into the Oracle destination table without problems as well.
Even though I have a work-around, I want to understand why this is happening, and what I can do to resolve this problem without having to use flat files.
Edit: Looks like even using the flat file to Oracle doesn't bring over all the rows. The first time was just luck?
Edit/Update: Running the package out of Integration Services (not BIDS) seems to have eliminated the problem (tested three times). Still don't understand why this is happening though.