I have two database connections in sql developer active lets say DB1 and DB2. I am working on ETL validation. So I want to check if data from Table1 of DB1 is populated correctly in Table2 of DB2.
To access tables from this two connections how can I write a query?
Any help on this will be helpful
There are common ways to validate if the ETL is correct:
You can run 2 queries to calculate the line counts separately against to Table1#DB1 and Table2#DB2, and compare the line counts between them.
Or perform some aggregate functions, such as sum(), avg()...etc on tables in DB1/DB2.
Related
I have two tables with the same table structure. One in oracle and the other in Postgres. I would like to compare the data between the two tables. I cannot use DB_Link, because of some connectivity issues.
I have copied both the contents to an excel sheet. But still having issues comparing the data.
Please suggest a suitable option to compare the data between the two tables.
I have the following scenario and need to solve it in ORACLE:
Table A is on a DB-server
Table B is on a different server
Table A will be populated with data.
Whenever something is inserted to Table A, i want to copy it to Table B.
Table B nearly has similar columns, but sometimes I just want to get
the content from 2 columns from tableA and concatenate it and save it to
Table B.
I am not very familiar with ORACLE, but after researching on GOOGLE
some say that you can do it with TRIGGERS or VIEWS, how would you do it?
So in general, there is a table which will be populated and its content
should be copien to a different table.
This is the solution I came up so far
create public database link
other_db
connect to
user
identified by
pw
using 'tns-entry';
CREATE TRIGGER modify_remote_my_table
AFTER INSERT ON my_table
BEGIN INSERT INTO ....?
END;
/
How can I select the latest row that was inserted?
If the databases of these two tables are in two different servers, then you will need a database link (db-link) to be created in Table A schema so that it can access(read/write) the Table B data using db-link.
Step 1: Create a database link in Table A server db pointing to Table B server DB
Step 2: Create a trigger for Table A, which helps in inserting data to the table B using database link. You can customize ( concatenate the values) inside the trigger before inserting it into table B.
This link should help you
http://searchoracle.techtarget.com/tip/How-to-create-a-database-link-in-Oracle
Yes you can do this with triggers. But there may be a few disadvantages.
What if database B is not available? -> Exception handling in you trigger.
What if database B was not available for 2h? You inserted data into database A which is now missing in database B. -> Do crazy things with temporarily inserting it into a cache table in database A.
Performance. Well, the performance for inserting a lot of data will be ugly. Each time you insert data, Oracle will start the PL/SQL engine to insert the data into the remote database.
Maybe you could think about using MViews (Materialized Views) to replicate the data via database link. Later you can build your queries so that they access tables from database B and add the required data from database A by joining the MViews.
You can also use fast refresh to replicate the data (almost) realtime.
From perspective of an Oracle Database Admin this would make a lot more sense than the trigger approach.
try this code
database links are considered rather insecure and oracle own options are having licences associated these days, some of the other options are deprecated as well.
https://gist.github.com/anonymous/e3051239ba401e416565cdd912e0de8c
uses ora_rowscn to sync tables across two different oracle databases.
I am developing a java app which can connect to oracle database and selecting column names from any tables, after selecting columns i have to query the data from those tables which the user select in my java app, now my question is how can i join all tables in the database so that query returns data successfully, i want to connect to any oracle schema to a specific, i will make the logic in java, but i am unable to find the query which can extract the data from all tables, i tried natural join among all tables but it has dependency of having same name of connecting columns. so i want to know any generic way which can work in all conditions.
As others have mentioned.. it seems that there are other tools out there that you probably should leverage prior to trying to roll your own complex solution.
With that said if you wish to roll your own solution you could look into using some of oracle's dictionary tables. Such as:
Select * from all_tables;
Select * from all_tab_cols;
Hey EXPERIENCED SSIS DEVELOPERS, I need your help.
High-Level Requirements
Query SQL Server table (on a different server than my SSIS server) resulting in about 200-300k records results set.
Use three output colums for each row to lookup date in Oracle database.
Insert or Update SQL Server table with results.
Use SSIS.
SQL Server 2008
Sounds easy, right?
Here is what I have done:
Created on Control Flow Execute SQL Task that gets a recordset from SQL Server. Very fast, easy query, like select field1, field2, field 3 from table where condition > 0. That's it. Takes less than a second.
Created a variable (evaluated as expression) for the Oracle query that uses the results set from the above in the WHERE clause.
Created a ForEachLoop Container that takes the results (from #1 above) for each row in the recordset and runs it through a Data Flow that uses the Oracle query (from #2 above) with Data access mode: SQL command from variable against an Oracle data source. Fast, simple query with only about 6 columns returned.
Data Conversion - obvious reasons - changing 3 columns from Oracle data types to SQL Server data types.
OLE DB Destination to insert to SQL Server using Fast Load to staging table.
It works perfectly! Hooray! Bad news - it is very, very slow. When I say slow, I mean it process 3000 records per hour. Holy moly - so freaking slow.
Question: am I missing a way to speed it up? It seems like the ForEachLoop Container is the bottleneck. Growl.
Important Points:
- I have NO write access in Oracle environment, so don't even suggest a potential solution that requires it. Not a possibility. At all.
Oracle sources do not allow for direct parameter definition. So no SELECT FIELD FROM TABLE WHERE ?. Don't suggest it - doesn't work.
Ideas
- Should I find a way to break down the results of the Execute SQL task and send them through several ForEachLoop Containers for faster processing?
Is there another design that is more appropriate?
Is there a script I can use that is faster?
Would it be faster to create a temporary table in memory and populate it - then use the results to bulk insert to SQL Server? Does this work when using an Oracle data source?
ANY OTHER IDEAS?
I have over million records in these tables in both the databases.
I am trying to figure out data in both the tables acros databases.
SELECT COUNT(*) FROM DB1.MYTABLE WHERE SEQ_NO NOT IN(SELECT SEQ_NO FROM DB2.MYTABLE) AND FILENAME NOT LIKE '%{%'
and PT_TYPE NOT IN(15,24,268,284,285,286,12,17,9,290,214,73) AND STTS=1
The query is taking ages. Is there any way I can make it fast?
Appreciate your help in advance
Do you actually mean different databases? Or do you mean different schemas? You talk about different databases but the syntax appears to be using tables in two different schemas, not two different databases. I don't see any references to a database link which would be needed if there were two different databases but perhaps DB2.MYTABLE is supposed to be a synonym for MYTABLE#DB2.
It would be helpful if you could post the query plan that is generated. It would also be useful to indicate what indexes exist and how selective each of these predicates is. My guess is that modifying the query to be
SELECT count(*)
FROM schema1.mytable a
WHERE NOT EXISTS (
SELECT 1
FROM schema2.mytable b
WHERE a.seq_no = b.seq_no )
AND a.filename NOT LIKE '%{%'
AND a.pt_type NOT IN (15,24,268,284,285,286,12,17,9,290,214,73)
AND a.stts = 1
might be more efficient if most of the rows in SCHEMA1.MYTABLE are eliminated because the SEQ_NO exists in SCHEMA2.MYTABLE.