After all, I achieved the best performance so far moving one billion rows from my source to target table and my PO is happy with this but, I am thinking if there is a better way of doing it?
Here is my code snippet, Thanks to marmiteBomber for correcting my script here.
DECLARE
v_parallel_degree NUMBER := 8;
PROCEDURE MOVE_DATA(p_target_tabname VARCHAR2,
p_source_tabname VARCHAR2,
p_part_name VARCHAR2)
AS
BEGIN
EXECUTE IMMEDIATE 'INSERT /*+PARALLEL(DEFAULT)*/ INTO '|| p_target_tabname || ' NOLOGGING
SELECT /*+PARALLEL(dmf,DEFAULT)*/*
FROM ' || p_source_tabname ||' PARTITION('|| p_part_name ||');
END;
BEGIN
EXECUTE IMMEDIATE 'ALTER SESSION ENABLE PARALLEL DML';
EXECUTE IMMEDIATE 'ALTER SESSION FORCE PARALLEL QUERY PARALLEL ' || v_parallel_degree;
EXECUTE IMMEDIATE 'ALTER SESSION FORCE PARALLEL DDL PARALLEL ' || v_parallel_degree;
EXECUTE IMMEDIATE 'DROP INDEX idx_pk';
FOR i IN (SELECT partition_name
FROM DBA_PARTITIONS
WHERE table_name = 'MYSOURCETABLE')
LOOP
MOVE_DATA('MYTARGETTABLE','MYSOURCETABLE',i.partition_name);
END LOOP;
EXECUTE IMMEDIATE 'CREATE UNIQUE INDEX idx_pk ON MYTARGETTABLE
(COL1,COL2,COL3)
LOCAL
NOLOGGING PARALLEL ' || v_parallel_degree;
END;
Now with this code, creating the index after the insert is taking around 1 hour. I have done some research and found instead of drop and create the index on the partition table, we can set the index to UNUSABLE and rebuild the same. But in my case after modifying the index to UNUSABLE I am not able to insert the data and facing an exception ORA-01502: index … or partition of such index is in unusable state
If I get this strategy to work, I have a plan to submit a job which will start a new session to rebuild the index and continue to my next partition in the current session.
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
PL/SQL Release 12.1.0.2.0 - Production
CORE 12.1.0.2.0 Production
will be upgraded to 19c.
Any suggestions are appreciated in advance.
Thanks
VB
Related
I am working on an Oracle SQL query. The query is getting flagged by Fortify SCA for Privilege Management: Default Function or Procedure Rights. Can someone help me with the correct way of using the query?
The query I want to use:
CREATE OR REPLACE PROCEDURE "reset_sequence"
IS
l_value NUMBER;
BEGIN
EXECUTE IMMEDIATE 'SELECT "ordering_seq".nextval FROM dual' INTO l_value;
EXECUTE IMMEDIATE 'ALTER SEQUENCE "ordering_seq" INCREMENT BY -' || l_value || ' MINVALUE 0';
EXECUTE IMMEDIATE 'SELECT "ordering_seq".nextval FROM dual' INTO l_value;
EXECUTE IMMEDIATE 'ALTER SEQUENCE "ordering_seq" INCREMENT BY 1 MINVALUE 0';
END;
I can spot a couple of issues here:
There's no authid clause, so it defaults to definer
execute immediate with string concatenation
This means anyone with execute privileges on the procedure is running with the full rights of the procedure owner. And with string concatenation, there's the risk of SQL injection. Yes, even with numbers.
Also you can get the next value of a sequence by assigning it. No need for execute immediate.
To be safe, I'd make the following changes:
Add authid current_user
Explicitly to_char the increment, avoiding attacks on this
Giving:
create sequence ordering_seq
start with 100;
select ordering_seq.nextval from dual;
NEXTVAL
100
CREATE OR REPLACE PROCEDURE reset_sequence
authid current_user
IS
l_value NUMBER;
BEGIN
l_value := ordering_seq.nextval;
EXECUTE IMMEDIATE 'ALTER SEQUENCE ordering_seq INCREMENT BY -' ||
to_char ( l_value, 'TM', 'NLS_Numeric_Characters = ''.,''' ) ||
' MINVALUE 0';
l_value := ordering_seq.nextval;
EXECUTE IMMEDIATE 'ALTER SEQUENCE ordering_seq INCREMENT BY 1 MINVALUE 0';
END;
/
exec reset_sequence;
select ordering_seq.nextval from dual;
NEXTVAL
1
Of course, using invoker's rights means you have to give out alter sequence rights to whoever calls. Which brings its own issues. To overcome this you could use Code-Based Access Control.
From the Fortify docs:
Privilege Management: Default Package Rights
PLSQL/TSQL
Abstract
Packages without an AUTHID clause default to AUTHID DEFINER.
Explanation
PL/SQL packages can be either AUTHID DEFINER or AUTHID CURRENT_USER. Functions and procedures in a package with definer's rights execute under the privileges of the user that defines the package. This can allow updates and access to specific pieces of data without granting access to entire tables or schemas. In a package with invoker's rights, or AUTHID CURRENT_USER, functions and procedures execute under the privileges of the user who invokes them. This does not allow a user to gain access to data it didn't already have access to. If no AUTHID clause is provided, the package defaults to definer's rights.
Packages are usually defined by SYS or another highly privileged user, making any exploits of the code potentially more dangerous.
So it seems you just need to add an AUTHID clause, even if that is just explicitly stating the default value again (though you should establish the correct value, of course).
Not relevant, but neither select needs to be dynamic - you may have chosen to do that so it looks more consistent with the alter statements, but it isn't necessary; and they don't even need to be selects any more - you could do:
CREATE OR REPLACE PROCEDURE "reset_sequence"
AUTHID CURRENT_USER
IS
l_value NUMBER;
BEGIN
l_value := "ordering_seq".nextval;
EXECUTE IMMEDIATE 'ALTER SEQUENCE "ordering_seq" INCREMENT BY -' || l_value || ' MINVALUE 0';
l_value := "ordering_seq".nextval;
EXECUTE IMMEDIATE 'ALTER SEQUENCE "ordering_seq" INCREMENT BY 1 MINVALUE 0';
END;
My Materialized view underlying table is having 6M of records and its going for COMPLETE refresh each time. The MVIEW underlying query is having 18 table join(all table in the same schema). Its blocking the PGA memory each time when refresh happening and I am getting warning. So my question is below.
1) Shall I change the Materialized view to a stored procedure? Stored procedure will have the cursor with having same query as Mview and it will load the data to a new table by using BULK COLLECT and LIMIT 1000 rows at a time. We will write an Autosys job(daily once at non business hour) to call the procedure.
Below is my changed PROC:
CREATE OR REPLACE PROCEDURE proc_reporting IS
CURSOR cur_rows IS
(Old MVIEW query)
TYPE reporting_type IS TABLE OF reporting_test%ROWTYPE INDEX BY PLS_INTEGER;
l_reporting_type reporting_type;
BEGIN
EXECUTE IMMEDIATE 'TRUNCATE TABLE reporting_test';
OPEN cur_rows;
LOOP
FETCH cur_rows BULK COLLECT INTO l_reporting_type LIMIT 1000;
EXIT WHEN l_reporting_type.count = 0;
BEGIN
FORALL i IN l_reporting_type.FIRST .. l_reporting_type.LAST SAVE EXCEPTIONS
INSERT INTO reporting_test VALUES (l_reporting_type(i).column_1 .. l_reporting_type(i).column_n);
EXCEPTION
WHEN ex_dml_errors THEN
l_error_count := SQL%BULK_EXCEPTIONS.count;
DBMS_OUTPUT.put_line('Number of failures: ' || l_error_count);
FOR i IN 1 .. l_error_count
LOOP
DBMS_OUTPUT.put_line('Error: ' || i ||
' Array Index: ' || SQL%BULK_EXCEPTIONS(i).error_index ||
' Message: ' || SQLERRM(-SQL%BULK_EXCEPTIONS(i).ERROR_CODE));
END LOOP;
END;
END LOOP;
CLOSE cur_rows;
END;
I agree that your first approach should be to examine the MView SQL for tuning. If you have license for AWR, you can run it through the SQL Tuning Advisor. Also please specify the PGA error, it may be that your DBA needs to increase pga_aggregate_target and/or pga_aggregate_limit. In many cases, increasing the PGA and/or SGA provides a significant performance improvement, but I always try SQL tuning first.
I have changed the underlying query of MVIEW and broken the MATVIEW query into two parts i.e two MVIEW. I moved all the one to one relationship table to the new MVIEW and made it fast refresh. We are refrencing the new MVIEW to the main MVIEW. By this approach, less data is getting loaded into PGA memory. We have made the new MVIEW to FAST load also to enhance the refresh mode. Now I am not getting warning of blocking the PGA memory.
I want to rebuild/unusable a partitioned index in pl-sql block. I have written in below code in pl-sql block,
declare m_ErrorMsg varchar2(100);
begin
execute immediate
'Alter Index TCMS.TTC_PERF_IDX01 Rebuild Partition
TMP_TRN_CUSTOMER_0;';
EXCEPTION WHEN OTHERS THEN
m_ErrorMsg := SUBSTR(SQLERRM(SQLCODE), 1, 200) ;
dbms_output.put_line(m_ErrorMsg);
end;
Above code is giving exception: 'ORA-14010: this physical attribute may not be specified for an index partition'.
If I execute below command on Command Promt then it's working,
Alter Index TCMS.TTC_PERF_IDX01 Rebuild Partition TMP_TRN_CUSTOMER_0;
Please suggest me correct way to write it in plsql bloc.
Clearly you cannot use ; with execute immediate clause.
your statment should be:
execute immediate 'Alter Index TCMS.TTC_PERF_IDX01 Rebuild Partition TMP_TRN_CUSTOMER_0';
We have a stored procedure that inserts data into a table from some temporary tables in oracle database. There is an update statement after the insert that updates a flag in the same table based on certain checks. At the end of the stored procedure commit happens.
The problem is that the update works on 95% cases but in some cases it fails to update. When we try to run it again without changing anything, it works. Even trying to execute the same stored procedure on the same data at some other time, works perfectly. I haven't found any issues in the logic in the stored procedure. I feel there is some database level issue which we are not able to find (maybe related to concurrency). Any ideas on this would be very helpful.
Without seeing the source code we will just be guessing. The most obvious suggestion that I can think of is that it hits an exception somewhere along the way in some cases and never gets as far as the commit. Another possibility is that there is a lock on the table during execution when it fails.
Probably the best thing to investigate further would be to add an exception handler that writes the exceptions to some table or file and see what error is raised e.g.
-- create a logging table
create table tmp_error_log (timestamp timestamp(0), Error_test varchar2(1000));
-- add a variable to your procedure declaration
v_sql varchar2(1000);
-- add an exception handler just before the final end; statement on your procedure
exception
when others then
begin
v_sql := 'insert into tmp_error_log values(''' ||
to_char(sysdate, 'DD-MON-YYYY HH24:MI:SS') || ''', ''' || SQLERRM || ''')';
dbms_output.put_line(v_sql);
execute immediate v_sql;
commit;
end;
-- see what you get in the table
select * from tmp_error_log;
CREATE OR REPLACE PROCEDURE test
AS
sql_stmt VARCHAR2(200);
BEGIN
sql_stmt := 'ALTER TABLE daily_table PARTITIONS p1 , p2 into PARTITION p2';
EXECUTE IMMEDIATE sql_stmt;
END ;
/
The above procedure is giving me the following error -
ORA-01031: insufficient privileges
ORA-06512: at "test", line 8
ORA-06512: at line 6
But if I run the ALTER Command directly on the sql prompt, I am not receiving any error..
I am wondering what permission I need to provide the user to perform the merge from the procedure.
I fixed the issue by using AUTHID CURRENT_USER
CREATE OR REPLACE PROCEDURE test AUTHID CURRENT_USER
AS
sql_stmt VARCHAR2(200);
BEGIN
sql_stmt := 'ALTER TABLE daily_table PARTITIONS p1 , p2 into PARTITION p2';
EXECUTE IMMEDIATE sql_stmt;
END ;
/
Note: you should never use DDL in stored procedures. Below is the sample of very, very badly designed code which should be avoided.
CREATE OR REPLACE PROCEDURE test
AS
sql_stmt VARCHAR2(200);
BEGIN
sql_stmt := 'GRANT ALTER ON daily_table TO your_user';
EXECUTE IMMEDIATE sql_stmt;
sql_stmt := 'ALTER TABLE daily_table PARTITIONS p1 , p2 into PARTITION p2';
EXECUTE IMMEDIATE sql_stmt;
END ;
/
I'm sure that if someone creates a stored proc SPECIFICALLY for DDL, then they realize that it is commited automatically.
I do use DDL in stored procedures: most commonly to truncate summary tables that the procedure will then re-populate; now and then for DDLish tasks such as renaming columns of an imported table to conform to Oracle's standard rules for an identifier, or for creating primary keys and sequences for named tables. Generally I use
dbms_utility.exec_ddl_statement(blah);
rather than
EXECUTE IMMEDIATE blah;
a prejudice I won't attempt to justify. I will say, that having packaged procedure supplied and documented by Oracle suggests that it is not to be prohibited across the board.