I would like to backup part of a database with Insert and DDL statements with output similar to what you can get with TOAD or SQL Developer, but from a script.
Details:
This is to be able to see changes and differences with source control.
We are using SQL Developer, LINUX tools and Python (With Oracle).
try this? Change settings as per your need.
Exporting Schema:
1) Run following in command prompt (not mandatory though)
SET NLS_LANG AMERICAN_AMERICA.UTF.8
2) Once above is set run the below, run the below. Change username/password/schemaname, path to export
exp userid=<schemaname>/<pwd>#<dbname or SID> file=<localpath\1.dmp>
log=<localpath\2.log> buffer=1000000000 feedback=25000
direct=y recordlength=64000 owner=<schemaname>
There must be tools for what you are trying to do (other than TOAD and PL/SQL).
This is a great project you have if you intend to write the code for this. You must work as follows:
For DDLs
. Write a script that list all the objects you need to extract; this will be tables, schemas, procedures, functions:
for x in (
select * from dba_objects where object_type in ('PACKAGES', 'TABLES', '>others>') loop
--- things to do with x.>columns<
end loop;
for u in (
select * from dba_users where username in ( <list of needed schemas> ) loop
)
--- things to do with users
end loop;
. Then from above list, you have Oracle function dbms_metadata.get_ddl that generates the DDL. e.g. with users:
begin
for u in (select * from dba_users where <list of needed schemas>
order by username)
loop
dbms_metadata.get_ddl('USER', u.username);
end loop;
end;
For the inserts
(this could take time to generate on databases with large tables (thousands of rows is already a lot for my approach).
I advise you create a generic function that generates the insert statements. This is rather simple as long as it does not include BLOB and CLOB. You take each column of each table.
Remember to generate the inserts in an orderly way, otherwise the comparison will not be possible.
The generic function should take a table as input. From the table name, it gets the columns from dba_tables, then selects the table.
--- plsql code below lists the columns of the tables
begin
for t in (select * from dba_tables order by table_name) loop
for c in (select * from dba_tab_columns where table_name=t.table_name and owner=t.owner order by owner, table_name, column_name) loop
dbms_output.put_line(c.table_name||'.'||c.column_name);
end loop;
end loop;
end;
Here is how you could collect the values in an array, then loop on this array to display the insert statements as text (with dbms_output.put_line)
declare
type t_v is table of varchar2(32000);
t_vchars t_v;
v_ins varchar2(2000):=' ';
v_sel varchar2(2000):=' ';
v_res varchar2(4030):=' ';
begin
for t in (select * from dba_tables order by table_name) loop
v_ins :='insert into '||t.table_name||'(';
v_sel :=' select ';
for c in (select * from dba_tab_columns where table_name=t.table_name and owner=t.owner) loop
v_ins:= v_ins||c.column_name||',';
v_sel:= v_sel||'''''''||'||c.column_name||'||'''''',';
end loop;
-- remove last comma
v_ins:= substr(v_ins,1,length(v_ins)-1);
v_sel:= substr(v_sel,1,length(v_sel)-1);
v_ins:= v_ins||')';
v_sel:= v_sel||' from dual';
v_res:= 'select '''||v_ins||v_sel||''' from '||t.table_name;
-- above, you still need to insert an order by, but I let you choose which solution
execute immediate v_res bulk collect into t_vchars;
FOR i in 1..t_vchars.count LOOP
dbms_output.put_line(t_vchars(i)||';');
END LOOP;
end loop;
end;
After you're done and want to test the whole script.
One tricky thing when you want to test and run the inserts is if you have foreign key. You must deactivate them before inserting, otherwise you need to do the inserts in a ordered way, which is not easy to do.
for c in (select * from dba_constraints)
loop
--- disable constraint
end loop;
Then after inserts are done
for c in (select * from dba_constraints)
loop
--- enable constraint
end loop;
Related
I am creating a temp table
CREATE TABLE TMP_PRTimeSum AS
SELECT DISTINCT p.employee,
SUM(p.wage_amount) AS wage_sum,
SUM(p.hours) AS hour_sum
I then want to do a select from that temp table and show the results. I want to do it all in one query. It works if I run them as two separate queries. Is there any way to do this in one query?
CREATE GLOBAL TEMPORARY TABLE a
ON COMMIT PRESERVE ROWS
AS
select * from b;
(add where 1=0 too if you didn't want to initially populate it for the current session with all the data from b).
Is there any way to do this in one query?
No, you need to use one DDL statement (CREATE TABLE) to create the table and one DML statement (SELECT) to query the newly created table.
If you want to be really pedantic then you could do it in a single PL/SQL statement:
DECLARE
cur SYS_REFCURSOR;
v_employee VARCHAR2(20);
v_wage NUMBER;
v_hours NUMBER;
BEGIN
EXECUTE IMMEDIATE
'CREATE TABLE TMP_PRTimeSum (employee, wage_sum, hour_sum) AS
SELECT P.EMPLOYEE,
SUM(P.WAGE_AMOUNT),
SUM(P.HOURS)
FROM table_name p
GROUP BY p.Employee';
OPEN cur FOR 'SELECT * FROM TMP_PRTimeSum';
LOOP
FETCH cur INTO v_employee, v_wage, v_hours;
EXIT WHEN cur%NOTFOUND;
DBMS_OUTPUT.PUT_LINE(v_employee || ', ' || v_wage || ', ' || v_hours);
END LOOP;
END;
/
db<>fiddle here
However, you are then wrapping the two SQL statements in a PL/SQL anonymous block so, depending on how you want to count it then it is either one composite statement, two SQL statements or three (2 SQL and 1 PL/SQL) statements.
I want to create procedure, that will use cursor, which is the same for arbitrary tables. My current one looks like this:
create or replace
PROCEDURE
some_name(
p_talbe_name IN VARCHAR2,
p_chunk_size IN NUMBER,
p_row_limit IN NUMBER
) AS
CURSOR v_cur IS
SELECT common_column,
ora_hash(substr(common_column, 1, 15), p_chunk_size) as chunk_number
-- Here it can find the table!
FROM p_table_name;
TYPE t_sample IS TALBE OF v_cur%rowtype;
v_sample t_sample;
BEGIN
OPEN v_cur;
LOOP FETCH v_cur BULK COLLECT INTO v_sample LIMIT p_row_limit;
FORALL i IN v_sample.first .. v_sample.last
INSERT INTO chunks VALUES v_sample(i);
COMMIT;
EXIT WHEN v_cur%notfound;
END LOOP;
CLOSE v_cur;
END;
The problem is that it cannot find the table named p_table_name which I want to parametrize. The thing is that I need to create chunks based on hashes for common_column which exists in all intended tables. How to deal with that problem? Maybe there is the equivalent oracle code that will do the same thing? Then I need the same efficiency for the query. Thanks!
I would do this as a single insert-as-select statement, complicated only by the fact you're passing in the table_name, so we need to use dynamic sql.
I would do it something like:
CREATE OR REPLACE PROCEDURE some_name(p_table_name IN VARCHAR2,
p_chunk_size IN NUMBER,
p_row_limit IN NUMBER) AS
v_table_name VARCHAR2(32); -- 30 characters for the tablename, 2 for doublequotes in case of case sensitive names, e.g. "table_name"
v_insert_sql CLOB;
BEGIN
-- Sanitise the passed in table_name, to ensure it meets the rules for being an identifier name. This is to avoid SQL injection in the dynamic SQL
-- statement we'll be using later.
v_table_name := DBMS_ASSERT.ENQUOTE_LITERAL(p_table_name);
v_insert_sql := 'insert into chunks (common_column_name, chunk_number)'||CHR(10)|| -- replace the column names with the actual names of your chunks table columns.
'select common_column,'||CHR(10)||
' ora_hash(substr(common_column, 1, 15), :p_chunk_size) AS chunk_number'||CHR(10)||
'from '||v_table_name||CHR(10)||
'where rownum <= :p_row_limit';
-- Used for debug purposes, so you can see the definition of the statement that's going to be run.
-- Remove before putting the code in production / convert to proper logging code:
dbms_output.put_line(v_insert_sql);
-- Now run the statement:
EXECUTE IMMEDIATE v_insert_sql USING p_chunk_size, p_row_limit;
-- I've included the p_row_limit in the above statement, since I'm not sure if your original code loops through all the rows once it processes the
-- first p_row_limit rows. If you need to insert all rows from the p_table_name into the chunks table, remove the predicate from the insert sql and the extra bind variable passed into the execute immediate.
END some_name;
/
By using a single insert-as-select statement, you are using the most efficient way of doing the work. Doing the bulk collect (which you were using) would use up memory (storing the data in the array) and cause extra context switches between the PL/SQL and SQL engines that the insert-as-select statement avoids.
The database-schema (Source and target) are very large (each has over 350 tables). I have got the task to somehow merge these two tables into one. The data itself (whats in the tables) has to be migrated. I have to be careful that there are no double entries for primary keys before or while merging the schemata. Has anybody ever done that already and would be able to provide me his solution or could anyone help me get a approach to the task? My approaches all failed and my advisor just tells me to get help online :/
To my approach:
I have tried using the "all_constraints" table to get all pks from my db.
SELECT cols.table_name, cols.column_name, cols.position, cons.status, cons.owner
FROM all_constraints cons, all_cons_columns cols
WHERE cols.owner = 'DB'
AND cons.constraint_type = 'P'
AND cons.constraint_name = cols.constraint_name
AND cons.owner = cols.owner
ORDER BY cols.table_name, cols.position;
I also "know" that there has to be a sequence for the primary keys to add values to it:
CREATE SEQUENCE seq_pk_addition
MINVALUE 1
MAXVALUE 99999999999999999999
START WITH 1
INCREMENT BY 1
CACHE 20;
Because I am a noob if it comes to pl/sql (or sql in general)
So how/what I should do next? :/
Here is a link for an ERD of the database: https://ufile.io/9tdoj
virus scan: https://www.virustotal.com/#/file/dbe5f418115e50313a2268fb33a924cc8cb57a43bc85b3bbf5f6a571b184627e/detection
As promised to help in my comment, i had prepared a dynamic code which you can try to get the data merged with the source and target tables. The logic is as below:
Step1: Get all the table names from the SOURCE schema. In the query below you can you need to replace the schema(owner) name respectively. For testing purpose i had taken only 1 table so when you run it,remove the table name filtering clause.
Step2: Get the constrained columns names for the table. This is used to prepared the ON clause which would be later used for MERGE statement.
Step3: Get the non-constrainted column names for the table. This would be used in UPDATE clause while using MERGE.
Step4: Prepare the insert list when the data doesnot match ON conditon of MERGE statement.
Read my inline comments to understand each step.
CREATE OR REPLACE PROCEDURE COPY_TABLE
AS
Type OBJ_NME is table of varchar2(100) index by pls_integer;
--To hold Table name
v_obj_nm OBJ_NME ;
--To hold Columns of table
v_col_nm OBJ_NME;
v_othr_col_nm OBJ_NME;
on_clause VARCHAR2(2000);
upd_clause VARCHAR2(4000);
cntr number:=0;
v_sql VARCHAR2(4000);
col_list1 VARCHAR2(4000);
col_list2 VARCHAR2(4000);
col_list3 VARCHAR2(4000);
col_list4 varchar2(4000);
col_list5 VARCHAR2(4000);
col_list6 VARCHAR2(4000);
col_list7 VARCHAR2(4000);
col_list8 varchar2(4000);
BEGIN
--Get Source table names
SELECT OBJECT_NAME
BULK COLLECT INTO v_obj_nm
FROM all_objects
WHERE owner LIKE 'RU%' -- Replace `RU%` with your Source schema name here
AND object_type = 'TABLE'
and object_name ='TEST'; --remove this condition if you want this to run for all tables
FOR I IN 1..v_obj_nm.count
loop
--Columns with Constraints
SELECT column_name
bulk collect into v_col_nm
FROM user_cons_columns
WHERE table_name = v_obj_nm(i);
--Columns without Constraints remain columns of table
SELECT *
BULK COLLECT INTO v_othr_col_nm
from (
SELECT column_name
FROM user_tab_cols
WHERE table_name = v_obj_nm(i)
MINUS
SELECT column_name
FROM user_cons_columns
WHERE table_name = v_obj_nm(i));
--Prepare Update Clause
FOR l IN 1..v_othr_col_nm.count
loop
cntr:=cntr+1;
upd_clause := 't1.'||v_othr_col_nm(l)||' = t2.' ||v_othr_col_nm(l);
upd_clause:=upd_clause ||' and ' ;
col_list1:= 't1.'||v_othr_col_nm(l) ||',';
col_list2:= col_list2||col_list1;
col_list5:= 't2.'||v_othr_col_nm(l) ||',';
col_list6:= col_list6||col_list5;
IF (cntr = v_othr_col_nm.count)
THEN
--dbms_output.put_line('YES');
upd_clause:=rtrim(upd_clause,' and');
col_list2:=rtrim( col_list2,',');
col_list6:=rtrim( col_list6,',');
END IF;
dbms_output.put_line(col_list2||col_list6);
--dbms_output.put_line(upd_clause);
End loop;
--Update caluse ends
cntr:=0; --Counter reset
--Prepare ON clause
FOR k IN 1..v_col_nm.count
loop
cntr:=cntr+1;
--dbms_output.put_line(v_col_nm.count || cntr);
on_clause := 't1.'||v_col_nm(k)||' = t2.' ||v_col_nm(k);
on_clause:=on_clause ||' and ' ;
col_list3:= 't1.'||v_col_nm(k) ||',';
col_list4:= col_list4||col_list3;
col_list7:= 't2.'||v_col_nm(k) ||',';
col_list8:= col_list8||col_list7;
IF (cntr = v_col_nm.count)
THEN
--dbms_output.put_line('YES');
on_clause:=rtrim(on_clause,' and');
col_list4:=rtrim( col_list4,',');
col_list8:=rtrim( col_list8,',');
end if;
dbms_output.put_line(col_list4||col_list8);
-- ON clause ends
--Prepare merge Statement
v_sql:= 'MERGE INTO '|| v_obj_nm(i)||' t1--put target schema name before v_obj_nm
USING (SELECT * FROM '|| v_obj_nm(i)||') t2-- put source schema name befire v_obj_nm here
ON ('||on_clause||')
WHEN MATCHED THEN
UPDATE
SET '||upd_clause ||
' WHEN NOT MATCHED THEN
INSERT
('||col_list2||','
||col_list4||
')
VALUES
('||col_list6||','
||col_list8||
')';
dbms_output.put_line(v_sql);
execute immediate v_sql;
end loop;
End loop;
END;
/
Execution:
exec COPY_TABLE
Output:
anonymous block completed
PS: i have tested this with a table with 2 columns out of which i was having unique key constraint .The DDL of table is as below:
At the end i wish you could understand my code(you being a noob) and implement something similar if the above fails for your requirement.
CREATE TABLE TEST
( COL2 NUMBER,
COLUMN1 VARCHAR2(20 BYTE),
CONSTRAINT TEST_UK1 UNIQUE (COLUMN1)
) ;
Oh dear! Normally, such a question would be quickly closed as "too broad", but we need to support victims of evil advisors!
As for the effort, I would need a week full time for an experienced expert plus two days quality checking for an experierenced QA engineer.
First of all, there is no way that such a complex data merge will work on the first try. That means that you'll need test copies of both schemas that can be easily rebuild. And you'll need a place to try it out. Normally this is done with an export of both schemas and an empty dev database.
Next, you need both schemas close enough to be able to compare the data. I'd do it with an import of the export files mentione above. If the schema names are identical than rename one during import.
Next, I'd doublecheck if the structure is really identical, with queries like
SELECT a.owner, a.table_name, b.owner, b.table_name
FROM all_tables a
FULL JOIN all_tables b
ON a.table_name = b.table_name
AND a.owner = 'SCHEMAA'
AND b.owner = 'SCHEMAB'
WHERE a.owner IS NULL or b.owner IS NULL;
Next, I'd check if the primary and unique keys have overlaps:
SELECT id FROM schemaa.table1
INTERSECT
SELECT id FROM schemab.table1;
As there are 300+ tables, I'd generate those queries:
DECLARE
stmt VARCHAR2(30000);
n NUMBER;
schema_a CONSTANT VARCHAR2(128 BYTE) := 'SCHEMAA';
schema_b CONSTANT VARCHAR2(128 BYTE) := 'SCHEMAB';
BEGIN
FOR c IN (SELECT owner, constraint_name, table_name,
(SELECT LISTAGG(column_name,',') WITHIN GROUP (ORDER BY position)
FROM all_cons_columns c
WHERE s.owner = c.owner
AND s.constraint_name = c.constraint_name) AS cols
FROM all_constraints s
WHERE s.constraint_type IN ('P')
AND s.owner = schema_a)
LOOP
dbms_output.put_line('Checking pk '||c.constraint_name||' on table '||c.table_name);
stmt := 'SELECT count(*) FROM '||schema_a||'.'||c.table_name
||' JOIN '||schema_b||'.'||c.table_name
|| ' USING ('||c.cols||')';
--dbms_output.put_line('Query '||stmt);
EXECUTE IMMEDIATE stmt INTO n;
dbms_output.put_line('Found '||n||' overlapping primary keys in table '||c.table_name);
END LOOP;
END;
/
1st of all, for 350 tables, most probably, would need an dynamic SQL.
Declare a CURSOR or a COLLECTION - table of VARCHAR2 with all table names.
Declare a string variable to build the dynamic SQL.
loop through the entire list of the tables name and, for each table generates a string which will be executed as SQL with EXECUTE IMMEDIATE command.
The dynamic SQL which will be built, should insert the values from source table into the target table. In case the PK already exists, in target table, should be checked the field which represents the last updated date and if in source table it is bigger than in target table, then perform an update in target table, else, do nothing.
I have got stuck in below and getting syntax error - Please help.
Basically I am using a collection to store few department ids and then would like to use these department ids as a filter condition while inserting data into emp table in FORALL statement.
Below is sample code:
while compiling this code i am getting error, my requirement is to use INSERT INTO table select * from table and cannot avoid it so please suggest.
create or replace Procedure abc(dblink VARCHAR2)
CURSOR dept_id is select dept_ids from dept;
TYPE nt_dept_detail IS TABLE OF VARCHAR2(25);
l_dept_array nt_dept_detail;
Begin
OPEN dept_id;
FETCH dept_id BULK COLLECT INTO l_dept_array;
IF l_dept_array.COUNT() > 0 THEN
FORALL i IN 1..l_dept_array.COUNT SAVE EXCEPTIONS
EXECUTE IMMEDIATE 'INSERT INTO stg_emp SELECT
Dept,''DEPT_10'' FROM dept_emp'||dblink||' WHERE
dept_id = '||l_dept_array(i)||'';
COMMIT;
END IF;
CLOSE dept_id;
end abc;
Why are you bothering to use cursors, arrays etc in the first place? Why can't you just do a simple insert as select?
Problems with your procedure as listed above:
You don't declare procedures like Procedure abc () - for a standalone procedure, you would do create or replace procedure abc as, or in a package: procedure abc is
You reference a variable called "dblink" that isn't declared anywhere.
You didn't put end abc; at the end of your procedure (I hope that was just a mis-c&p?)
You're effectively doing a simple insert as select, but you're way over-complicating it, plus you're making your code less performant.
You've not listed the column names that you're trying to insert into; if stg_emp has more than two columns or ends up having columns added, your code is going to fail.
Assuming your dblink name isn't known until runtime, then here's something that would do what you're after:
create Procedure abc (dblink in varchar2)
is
begin
execute immediate 'insert into stg_emp select dept, ''DEPT_10'' from dept_emp#'||dblink||
' where dept_id in (select dept_ids from dept)';
commit;
end abc;
/
If, however, you do know the dblink name, then you'd just get rid of the execute immediate and do:
create Procedure abc (dblink in varchar2)
is
begin
insert into stg_emp -- best to list the column names you're inserting into here
select dept, 'DEPT_10'
from dept_emp#dblink
where dept_id in (select dept_ids from dept);
commit;
end abc;
/
There appears te be a lot wrong with this code.
1) why the execute immediate? Is there any explicit requirement for that? No, than don't use it
2) where is the dblink variable declared?
3) as Boneist already stated, why not a simple subselect in the insert statement?
INSERT INTO stg_emp SELECT
Dept,'DEPT_10' FROM dept_emp#dblink WHERE
dept_id in (select dept_ids from dept );
For one, it would make the code actually readable ;)
Is there a way to update all statistics for all tables regardless of the owner?
I found this sniplet, but I'm not sure if this will grab all tables....
BEGIN
FOR A IN ( SELECT owner FROM SYS.all_tables ) LOOP
execute immediate
EXEC dbms_stats.gather_schema_stats( 'A.owner', cascade='TRUE');
END LOOP;
END;
Use DBMS_STATS.GATHER_DATABASE_STATS:
begin
dbms_stats.gather_database_stats;
end;
/
No the DBMS_STATS package can do at most one schema at a time.
You can use the script below to gather stats for all objects types in all schemas. The one you listed has a couple of issues (needless execute immediate, `A.owner' is a string but it should be an object, etc).
You can add additional schemas to skip in the IN list as you probably don't want to do this for the built in schemas (they're mostly static anyway so it'd be waste). Also, you'll need to have the appropriate privileges for each schema you are gathering stats on (or be logged in as a DBA).
Gather stats on all objects (probably what you really want):
BEGIN
FOR rec IN (SELECT *
FROM all_users
WHERE username NOT IN ('SYS','SYSDBA'))
LOOP
dbms_stats.gather_schema_stats(rec.username);
END LOOP;
END;
Gather stats on just tables:
BEGIN
FOR rec IN (SELECT *
FROM all_tables
WHERE owner NOT IN ('SYS','SYSDBA'))
LOOP
dbms_stats.gather_table_stats(rec.owner, rec.table_name);
END LOOP;
END;
I did a modification to #sehrope procedure to skip locked stats and IOT tables as they will through exceptions.
BEGIN
FOR rec IN (SELECT a.owner, a.table_name
FROM all_tables a, dba_tab_statistics s
WHERE a.owner NOT IN ('SYS','SYSDBA')
AND
(a.iot_type IS NULL
OR
a.iot_type != 'IOT_OVERFLOW')
and a.owner = s.owner and a.table_name = s.table_name and s.STATTYPE_LOCKED is null)
LOOP
dbms_stats.gather_table_stats(rec.owner, rec.table_name);
END LOOP;
END;