I have to write a dynamic sql cursor where there are several possibilities in which the select query will be generated. Hence I am chosing dynamic and I am Using DBMS_SQL package to dynamically create a cursor and dynamically fetch the data.
However , Result set is going to be huge . around 11GB (there are 2.4 million records and the select statement will be approx 80 cols long assumning about 50Byte varchar per column)
Hence I cannot open the cursor at once . I want to know if there is a feature wherein i can fetch the data from the curosr keeping the curosr open for Blocks of say 1000 records at time(I will have to do this dynamically)
Please find the code attached which only fetches and prints the value of the columns (one sample case ) I want to use bul collect here \
Thanks
---------------code sample--------------------------------------
--create or replace type TY_DIMDEAL AS TABLE OF VARCHAR2(50) ;
create or replace procedure TEST_PROC (po_recordset out sys_refcursor)
as
v_col_cnt INTEGER;
v_ind NUMBER;
rec_tab DBMS_SQL.desc_tab;
v_cursor NUMBER;
lvar_output number:=0;
lvar_output1 varchar2(100);
lvar_output3 varchar2(100);
lvar_output2 varchar2(100);
LVAR_TY_DIMDEAL TY_DIMDEAL;
lvarcol varchar2(100);
begin
--
LVAR_TY_DIMDEAL := TY_DIMDEAL();
lvar_output1 := '';
v_cursor := dbms_sql.open_cursor;
dbms_sql.parse(v_cursor, 'select to_char(Field1) , to_char(fiel2) , to_char(field3) from table,table2 ', dbms_sql.native);
dbms_sql.describe_columns(v_cursor, v_col_cnt, rec_tab);
FOR v_pos in 1..rec_tab.LAST LOOP
LVAR_TY_DIMDEAL.EXTEND();
DBMS_SQL.define_column( v_cursor, v_pos ,LVAR_TY_DIMDEAL(v_pos),20);
END LOOP;
-- DBMS_SQL.define_column( v_cursor, 1 ,lvar_output1,20);
--DBMS_SQL.define_column( v_cursor, 2 ,lvar_output2,20);
--DBMS_SQL.define_column( v_cursor, 3 ,lvar_output3,20);
v_ind := dbms_sql.execute( v_cursor );
LOOP
v_ind := DBMS_SQL.FETCH_ROWS( v_cursor );
EXIT WHEN v_ind = 0;
lvar_output := lvar_output+1;
dbms_output.put_line ('row number '||lvar_output) ;
FOR v_col_seq IN 1 .. rec_tab.COUNT LOOP
LVAR_TY_DIMDEAL(v_col_seq):= '';
DBMS_SQL.COLUMN_VALUE( v_cursor, v_col_seq,LVAR_TY_DIMDEAL(v_col_seq));
dbms_output.put_line (LVAR_TY_DIMDEAL(v_col_seq));
END LOOP;
END LOOP;
end TEST_PROC;
Fetching data from a cursor in blocks of reasonable size, while keeping the cursor open, is one of PL/SQL Best Practices.
The above document (see Code 38 item) sketches an approach for when the select list is not known until runtime. Basically:
Define an appropriate type to fetch results into. Let's assume that all the returned columns will by of of type VARCHAR2:
-- inside DECLARE
Ty_FetchResults IS TABLE OF DBMS_SQL.VARCHAR2_TABLE;
lvar_results Ty_FetchResults;
Before each call to DBMS_SQL.FETCH_ROWS, call DBMS_SQL.DEFINE_ARRAY to enable batch fetching.
Call DBMS_SQL.FETCH_ROWS to fetch 1000 rows from the cursor.
Call DBMS_SQL.COLUMN_VALUE to copy the fetched data into your result array.
Process the results, record by record, in a FOR loop. Don't worry about the number of fetched records: if there are records to process, the FOR loop will run correctly; if the result array is empty, the FOR loop will not run.
Exit from the loop when the number of fetched records is less than the expected size.
Remember to DBMS_SQL.CLOSE the cursor.
Your loop body should look like this:
LOOP
FOR j IN 1..v_col_cnt LOOP
DBMS_SQL.DEFINE_ARRAY(v_cursor, j, lvar_results(j), 1000, 1);
END LOOP;
v_ind := DBMS_SQL.FETCH_ROWS(v_cursor);
FOR j IN 1..v_col_cnt LOOP
lvar_results(j).DELETE;
DBMS_SQL.COLUMN_VALUE(v_cursor, j, lvar_results(j));
END LOOP;
-- process the results, record by record
FOR i IN 1..lvar_results(1).COUNT LOOP
-- process a single record...
-- your logic goes here
END LOOP;
EXIT WHEN lvar_results(1).COUNT < 1000;
END LOOP;
-- don't forget: DBMS_CLOSE(v_cursor);
See also Doing SQL from PL/SQL: Best and Worst Practices.
LIMIT CLAUSE CAN COME TO RESCUE!
PL/SQL collections are essentially arrays in memory, so massive
collections can have a detrimental effect on system performance due to
the amount of memory they require. In some situations, it may be
necessary to split the data being processed into chunks to make the
code more memory-friendly. This “chunking” can be achieved using the
LIMIT clause of the BULK COLLECT syntax.
YOU CAN USE LIMIT CLAUSE AFTER BULK COLLECT INTO CLAUSE TO LIMIT YOUR RS.
AFTER YOU EXCEED TO LIMIT YOU CAN FETCH REMAINING ROWS.
SEE THIS ARTICLE
http://www.dba-oracle.com/plsql/t_plsql_limit_clause.htm
Related
I have question while processing records in two different looping method i.e WHILE & FOR LOOP please refer to following codes
DECLARE
TYPE rc_emp_type IS RECORD ( empno NUMBER,ename VARCHAR(30),sal NUMBER, comm NUMBER);
TYPE rc_emp_tab IS TABLE OF rc_emp_type;
l_emp_rec rc_emp_tab := rc_emp_tab();
TYPE rc_emp_calc_type IS RECORD ( empno NUMBER,
ename VARCHAR(30),
sal NUMBER,
comm NUMBER,
new_sal NUMBER);
TYPE rc_emp_calc_tab IS TABLE OF rc_emp_calc_type;
l_emp_calc_rec rc_emp_calc_tab := rc_emp_calc_tab();
l_emp_fcalc_rec rc_emp_calc_tab := rc_emp_calc_tab();
l_idx NUMBER;
l_start_time TIMESTAMP;
l_end_time TIMESTAMP;
l_exe_time TIMESTAMP;
BEGIN
SELECT empno,ename,sal,comm
BULK COLLECT INTO l_emp_rec
FROM emp;
l_idx := l_emp_rec.FIRST;
WHILE l_idx IS NOT NULL
LOOP
l_emp_calc_rec.EXTEND;
l_emp_calc_rec(l_emp_calc_rec.LAST).empno := l_emp_rec(l_idx).empno;
l_emp_calc_rec(l_emp_calc_rec.LAST).ename := l_emp_rec(l_idx).ename;
l_emp_calc_rec(l_emp_calc_rec.LAST).sal := l_emp_rec(l_idx).sal;
l_emp_calc_rec(l_emp_calc_rec.LAST).comm := l_emp_rec(l_idx).comm;
l_emp_calc_rec(l_emp_calc_rec.LAST).new_sal := NVL(l_emp_rec(l_idx).sal,0) + NVL(l_emp_rec(l_idx).comm,0);
l_idx := l_emp_rec.NEXT(l_idx);
END LOOP;
FOR l_idx IN l_emp_rec.FIRST .. l_emp_rec.LAST
LOOP
l_emp_fcalc_rec.EXTEND;
l_emp_fcalc_rec(l_emp_fcalc_rec.LAST).empno := l_emp_rec(l_idx).empno;
l_emp_fcalc_rec(l_emp_fcalc_rec.LAST).ename := l_emp_rec(l_idx).ename;
l_emp_fcalc_rec(l_emp_fcalc_rec.LAST).sal := l_emp_rec(l_idx).sal;
l_emp_fcalc_rec(l_emp_fcalc_rec.LAST).comm := l_emp_rec(l_idx).comm;
l_emp_fcalc_rec(l_emp_fcalc_rec.LAST).new_sal := NVL(l_emp_rec(l_idx).sal,0) + NVL(l_emp_rec(l_idx).comm,0);
END LOOP;
END;
Out of these two above procedure which is the efficient way of looping
If you know your collection will be densely-filled, as is the case with a collection filled by BULK COLLECT, I suggest you use a numeric FOR loop. That assumes densely-filled and therefore is most appropriate in that context.
Whenever you are not 100% certain that your collection is densely-filled, you should use a WHILE loop and the FIRST-NEXT or LAST-PRIOR methods to iterate through the collection.
You might argue that you might as well just use the WHILE loop all the time. Performance will be fine, memory consumption is not different....BUT: you might "hide" an error this way. If the collection is supposed to be dense, but it is not not, you will never know.
Finally, there is one way in which the WHILE loop could be a better performer than a FOR loop: if your collection is very sparse (eg, elements populated only in index values -1M, 0, 1M, 2M, 3M, etc.), the FOR loop will raise lots of NO_DATA_FOUND exceptions. Handling and continuing for all those exceptions will make loop execution very slow.
Out of these two above procedure which is the efficient way of looping
While dealing with collection,For Loop sometimes throws error when the collection is Sparse. In that case its beneficial to use WHILE LOOP. Both looping mechanism is equal in performance.
Sparse:- A collection is sparse if there is at least one index value between the lowest and highest defined index values that is not defined. For example, a sparse collection has an element assigned to index value 1 and another to index value 10 but nothing in between. The opposite of a sparse collection is a dense one.
Use a numeric FOR loop when
Your collection is densely filled (every index value between the lowest and the highest is defined)
You want to scan the entire collection, not terminating your scan if some condition is met
Conversely, use a WHILE loop when
Your collection may be sparse
You might terminate the loop before you have iterated through all the elements in the collection
As #XING points the difference is not in how efficient they are, but in what happens with sparse collections. Your example does not face this issue as both are built with bulk collect so there are no gaps in the index values. But this is NOT always the case. The following demo shows the difference between them.
declare
cursor c_numbers is
select level+23 num -- 23 has no particulat significence
from dual
connect by level <= 5; -- nor does 5
type base_set is table of c_numbers%rowtype;
while_set base_set;
for_set base_set;
while_index integer; -- need to define while loop index
begin
-- populate both while and for arrays.
open c_numbers;
fetch c_numbers bulk collect into while_set;
close c_numbers;
open c_numbers;
fetch c_numbers bulk collect into for_set;
close c_numbers;
-- Make sparse
while_set.delete(3);
for_set.delete(3);
-- loop through with while
while_index := while_set.first;
while while_index is not null
loop
begin
dbms_output.put_line('While_Set(' ||
while_index ||
') = ' ||
while_set(while_index).num
);
while_index := while_set.next(while_index);
exception
when others then
dbms_output.put_line('Error in While_Set(' ||
while_index ||
') Message=' ||
sqlerrm
);
end;
end loop;
-- loop through with for
for for_index in for_set.first .. for_set.last
loop
begin
dbms_output.put_line('For_Set(' ||
for_index ||
') = ' ||
for_set(for_index).num
);
exception
when others then
dbms_output.put_line('Error in For_Set(' ||
for_index ||
') Message=' ||
sqlerrm
);
end;
end loop;
end;
Also try a for loop with a collection defines as:
type state_populations_t is table of number index by varchar2(20);
state_populations state_populations_t;
And yes, that line is in production code and has run for years,
I tinkered together the following PL/SQL BULK-COLLECT which works astonishingly fast for updates on huge tables (>50.000.000). The only problem is, that it does not perform the updates of the remaining < 5000 rows per table. 5000 is the given limit for the FETCH instruction:
DECLARE
-- source table cursor (only columns to be updated)
CURSOR base_table_cur IS
select a.rowid, TARGET_COLUMN from TARGET_TABLE a
where TARGET_COLUMN is null;
TYPE base_type IS
TABLE OF base_table_cur%rowtype INDEX BY PLS_INTEGER;
base_tab base_type;
-- new data
CURSOR new_data_cur IS
select a.rowid,
coalesce(b.SOURCE_COLUMN, 'FILL_VALUE'||a.JOIN_COLUMN) TARGET_COLUMN from TARGET_TABLE a
left outer join SOURCE_TABLE b
on a.JOIN_COLUMN=b.JOIN_COLUMN
where a.TARGET_COLUMN is null;
TYPE new_data_type IS TABLE OF new_data_cur%rowtype INDEX BY PLS_INTEGER;
new_data_tab new_data_type;
TYPE row_id_type IS TABLE OF ROWID INDEX BY PLS_INTEGER;
row_id_tab row_id_type;
TYPE rt_update_cols IS RECORD (
TARGET_COLUMN TARGET_TABLE.TARGET_COLUMN%TYPE
);
TYPE update_cols_type IS
TABLE OF rt_update_cols INDEX BY PLS_INTEGER;
update_cols_tab update_cols_type;
dml_errors EXCEPTION;
PRAGMA exception_init ( dml_errors,-24381 );
BEGIN
OPEN base_table_cur;
OPEN new_data_cur;
LOOP
FETCH base_table_cur BULK COLLECT INTO base_tab LIMIT 5000;
IF base_table_cur%notfound THEN
DBMS_OUTPUT.PUT_LINE('Nothing to update. Exiting.');
EXIT;
END IF;
FETCH new_data_cur BULK COLLECT INTO new_data_tab LIMIT 5000;
FOR i IN base_tab.first..base_tab.last LOOP
row_id_tab(i) := new_data_tab(i).rowid;
update_cols_tab(i).TARGET_COLUMN := new_data_tab(i).TARGET_COLUMN;
END LOOP;
FORALL i IN base_tab.first..base_tab.last SAVE EXCEPTIONS
UPDATE (SELECT TARGET_COLUMN FROM TARGET_TABLE)
SET row = update_cols_tab(i)
WHERE ROWID = row_id_tab(i);
COMMIT;
EXIT WHEN base_tab.count < 5000; -- changing to 1 didn't help!
END LOOP;
COMMIT;
CLOSE base_table_cur;
CLOSE new_data_cur;
EXCEPTION
WHEN dml_errors THEN
FOR i IN 1..SQL%bulk_exceptions.count LOOP
dbms_output.put_line('Some error occured');
END LOOP;
END;
Where is my mistake? It looks correct to me though.
The problem is this line:
IF base_table_cur%notfound THEN
The cursor meets %NOTFOUND when the number of records found is less than the LIMIT value. So if the last fetch is not exactly 5000 those records won't be processed.
It's a common gotcha for people using BULK COLLECT ... LIMIT for the first time. The solution is to change the exit condition to
EXIT when base_tab.count() = 0;
"I need to ensure, that the base_table_cur is not empty and exit if it is. I'l get an error if it is empty"
The new_data_cur cursor includes the table which is selected in base_table_cur cursor. So I don't think you need the two loops. You need a simple test to see whether the first cursor returns something, then just loop round the second cursor.
I'm not entirely clear on your logic, so I have changed as little as possible to demonstrate the sort of structure I think you need. However, the UPDATE statement looks a little odd, so you may still run into issues.
OPEN base_table_cur;
FETCH base_table_cur BULK COLLECT INTO base_tab LIMIT 1;
if base_table_tab.count = 0 then
DBMS_OUTPUT.PUT_LINE('Nothing to update. Exiting.');
else
OPEN new_data_cur;
LOOP
FETCH new_data_cur BULK COLLECT INTO new_data_tab LIMIT 5000;
exit when new_data_tab.count() = 0;
FOR i IN base_tab.first..base_tab.last LOOP
row_id_tab(i) := new_data_tab(i).rowid;
update_cols_tab(i).TARGET_COLUMN := new_data_tab(i).TARGET_COLUMN;
END LOOP;
FORALL i IN base_tab.first..base_tab.last SAVE EXCEPTIONS
UPDATE (SELECT TARGET_COLUMN FROM TARGET_TABLE)
SET row = update_cols_tab(i)
WHERE ROWID = row_id_tab(i);
END LOOP;
CLOSE new_data_cur;
end if;
COMMIT;
CLOSE base_table_cur;
I have this simple oracle plsql procedure:
declare
cursor A is
select column_A
from A_TAB; -- no order by
begin
for rec_ in A loop
procedure_A(rec_.column_A);
end loop;
end;
And this is running now for ages.
When I look into sys.v_$sql_bind_capture, value_string column, I can see the current value of bound column_A, and thankfully, that value keeps changing every few minutes.
As the cursor was not sorted by anything, is there a way to see how many more records to go (until this is finished)?
In other words I would need to see the currently fetched values of the query from that cursor. Where to look for it?
This is Oracle 12 database.
You can use dbms_application_info.set_session_longops to do this. The results are visible in V$SESSION_LONGOPS.
In your example, that could do something like:
DECLARE
rindex BINARY_INTEGER;
slno BINARY_INTEGER;
totalwork number;
sofar number;
obj BINARY_INTEGER;
cursor A is
select column_A,
COUNT(*) OVER () cnt
from A_TAB; -- no order by
begin
rindex := dbms_application_info.set_session_longops_nohint;
sofar := 0;
for rec_ in A loop
totalwork := rec_.cnt;
sofar := sofar + 1;
dbms_application_info.set_session_longops(rindex,
slno,
'Process a_tab',
'A_TAB',
0,
sofar,
totalwork,
'table',
'rows');
procedure_A(rec_.column_A);
end loop;
end;
Note that in order to get the totalwork value, I've used the analytic COUNT() function to get the total number of rows within the resultset. You could run a separate query to get the count before looping through your original cursor, if that is faster. You'd have to test both methods to work out which would be fastest for your data etc.
Of course, depending on what procedure_a does, you might be able to avoid the need to monitor the progress if you can refactor things so that all the work is being done in a single SQL statement. My answer above assumes that it's not possible to do that. If it is, I highly recommend you refactor your code instead!
Please help me to solve this ORACLE PL/SQL problem.
I have the following code:
BEGIN
DECLARE
P_COMMIT_STEP NUMBER := 10000; -- Commit every 10000 record copied
V_QUERY VARCHAR2 (4000) := NULL;
MY_CURSOR SYS_REFCURSOR;
TYPE FETCH_ARRAY IS TABLE OF MY_TABLE_BACKUP%ROWTYPE;
S_ARRAY FETCH_ARRAY;
BEGIN
V_QUERY := 'SELECT * FROM MY_TABLE_BACKUP';
OPEN MY_CURSOR FOR V_QUERY;
LOOP
FETCH MY_CURSOR
BULK COLLECT INTO S_ARRAY
LIMIT P_COMMIT_STEP;
FORALL I IN 1 .. S_ARRAY.COUNT
INSERT INTO MY_TABLE_BIS /*+ APPEND */
VALUES S_ARRAY (I);
COMMIT;
EXIT WHEN MY_CURSOR%NOTFOUND;
END LOOP;
CLOSE MY_CURSOR;
COMMIT;
END;
END;
Since the commit step is 10000, the copy works for a multiple of 10000 record.
So, if the original table has 1000010 records, only 1000000 records will be copied.
Where is the error?
The code seems correct, in my opinion.
Thank you very much for considering my request.
As noted in this article, you shouldn't rely on %NOTFOUND with bulk collect and forall. Check how many rows were fetched:
LOOP
FETCH MY_CURSOR
BULK COLLECT INTO S_ARRAY
LIMIT P_COMMIT_STEP;
FORALL I IN 1 .. S_ARRAY.COUNT
INSERT INTO MY_TABLE_BIS /*+ APPEND */
VALUES S_ARRAY (I);
COMMIT;
EXIT WHEN S_ARRAY.COUNT < P_COMMIT_STEP;
END LOOP;
Your first thousand iterations will get 10000 rows, so the count will equal your limit for all of those, and it will continue. The next one will get only 10 rows, so it will exit after the forall.
It should still work with the %NOTFOUND check at the end of the loop - the article is really talking about it being an issue if you use it to exit before processing the partial batch - and the documentation shows that pattern; but in some circumstances it seems not to. Having said that, I can't reproduce the issue from your code in 11.2.0.3.
Incidentally, committing inside a loop is generally a bad idea unless you've made the block restartable.
HI small modification to your Forall loop
FORALL I IN S_ARRAY.first .. S_ARRAY.last
INSERT INTO MY_TABLE_BIS /*+ APPEND */
VALUES S_ARRAY (I);
I think it will work for you.
Is there a quick way to flush REF CURSOR into a temporary table in oracle
CURSOR size will vary between 5K to 100K rows
currently I am using a for loop, but its very slow
There is no way to change the OUT parameter for SOME_FUNCTION_CALL function.
DECLARE
v_return SYS_REFCURSOR;
VAR_A varchar2(100);
VAR_B varchar2(100);
BEGIN
v_return := SOME_FUNCTION_CALL();
LOOP
FETCH v_return into VAR_A,VAR_B
EXIT WHEN v_return%NOTFOUND
INSERT INTO temp_table(a,b) values (VAR_A,VAR_B);
END LOOP;
CLOSE v_return;
END LOOP;
Probably not.
You could potentially make the code more efficient by doing a BULK COLLECT into local collections rather than doing row-by-row fetches. Something like
DECLARE
TYPE vc100_tbl IS TABLE OF VARCHAR2(100);
l_return SYS_REFCURSOR;
l_var_a_tbl vc100_tbl;
l_var_b_tbl vc100_tbl;
BEGIN
l_return := some_function_call();
LOOP
FETCH l_return
BULK COLLECT INTO l_var_a_tbl, l_var_b_tbl
LIMIT 100;
EXIT WHEN l_var_a_tbl.count = 0;
FORALL i IN 1 .. l_var_a_tbl.count
INSERT INTO temp_table( a, b )
VALUES( l_var_a_tbl(i), l_var_b_tbl(i) );
END LOOP;
close l_return;
END;
That will reduce the number of context shifts taking place between the SQL and PL/SQL engines. But depending on your definition of "very slow", it seems unlikely that context shifts are your most pressing problem. Were I to guess, I'd wager that if your code is "very slow" for most definitions of that term, the problem is likely that the query that is being used to open the cursor in some_function is slow and that most of your time is being spent executing that query in order to generate the rows that you want to fetch. If that's the case, you'd want to optimize the query.
Architecturally, this approach also seems rather problematic. Even if you can't change the definition of the function, can't you do something to factor out the query the function is executing so that you can use the same code without calling the function? For example, could you not factor out the query the function calls into a view and then call that view from your code?