Is there a quick way to flush REF CURSOR into a temporary table in oracle
CURSOR size will vary between 5K to 100K rows
currently I am using a for loop, but its very slow
There is no way to change the OUT parameter for SOME_FUNCTION_CALL function.
DECLARE
v_return SYS_REFCURSOR;
VAR_A varchar2(100);
VAR_B varchar2(100);
BEGIN
v_return := SOME_FUNCTION_CALL();
LOOP
FETCH v_return into VAR_A,VAR_B
EXIT WHEN v_return%NOTFOUND
INSERT INTO temp_table(a,b) values (VAR_A,VAR_B);
END LOOP;
CLOSE v_return;
END LOOP;
Probably not.
You could potentially make the code more efficient by doing a BULK COLLECT into local collections rather than doing row-by-row fetches. Something like
DECLARE
TYPE vc100_tbl IS TABLE OF VARCHAR2(100);
l_return SYS_REFCURSOR;
l_var_a_tbl vc100_tbl;
l_var_b_tbl vc100_tbl;
BEGIN
l_return := some_function_call();
LOOP
FETCH l_return
BULK COLLECT INTO l_var_a_tbl, l_var_b_tbl
LIMIT 100;
EXIT WHEN l_var_a_tbl.count = 0;
FORALL i IN 1 .. l_var_a_tbl.count
INSERT INTO temp_table( a, b )
VALUES( l_var_a_tbl(i), l_var_b_tbl(i) );
END LOOP;
close l_return;
END;
That will reduce the number of context shifts taking place between the SQL and PL/SQL engines. But depending on your definition of "very slow", it seems unlikely that context shifts are your most pressing problem. Were I to guess, I'd wager that if your code is "very slow" for most definitions of that term, the problem is likely that the query that is being used to open the cursor in some_function is slow and that most of your time is being spent executing that query in order to generate the rows that you want to fetch. If that's the case, you'd want to optimize the query.
Architecturally, this approach also seems rather problematic. Even if you can't change the definition of the function, can't you do something to factor out the query the function is executing so that you can use the same code without calling the function? For example, could you not factor out the query the function calls into a view and then call that view from your code?
Related
I have a Main cursor that is working fine.
declare
v_firm_id number;
amount number;
v_total_sum TABLE_TEMP.TOTAL_SUM%TYPE;
CURSOR MT_CURSOR IS
SELECT firm_id FROM t_firm;
BEGIN
OPEN MT_CURSOR;
LOOP
FETCH MT_CURSOR INTO v_firm_id;
EXIT WHEN MT_CURSOR%NOTFOUND;
DBMS_OUTPUT.PUT_LINE(to_char(sysdate, 'mi:ss') ||'--- '|| v_firm_id)
INSERT INTO TABLE_TEMP(TOTAL_SUM) VALUES(v_firm_id)
COMMIT;
END LOOP;
DBMS_LOCK.SLEEP(20);
BEGIN
FOR loop_emp IN
(SELECT TOTAL_SUM INTO v_total_sum FROM TABLE_TEMP)
LOOP
dbms_output.put_line(to_char(sysdate, 'mi:ss') ||'--- '|| v_total_sum || '-TEST--');
END LOOP loop_emp;
END;
end;
Everything Works fine except dbms_output.put_line(v_total_sum || '---');
I do not get any data there. I get the correct number of rows. which it inserted.
The problem is the cursor FOR loop has a redundant into clause which it appears the compiler silently ignores, and so v_total_sum is never used.
Try this:
begin
for r in (
select firm_id from t_firm
)
loop
insert into table_temp (total_sum) values (r.firm_id);
end loop;
dbms_lock.sleep(20);
for r in (
select total_sum from table_temp
)
loop
dbms_output.put_line(r.total_sum || '---');
end loop;
commit;
end;
If this had been a stored procedure rather than an anonymous block and you had PL/SQL compiler warnings enabled with alter session set plsql_warnings = 'ENABLE:ALL'; (or the equivalent preference setting in your IDE) then you would have seen:
PLW-05016: INTO clause should not be specified here
I also moved the commit to the end so you only commit once.
To summarise the comments below, the Cursor FOR loop construction declares, opens, fetches and closes the cursor for you, and is potentially faster because it fetches in batches of 100 (or similar - I haven't tested in recent versions). Simpler code has less chance of bugs and is easier to maintain in the future, for example if you need to add a column to the cursor.
Note the original version had:
for loop_emp in (...)
loop
...
end loop loop_emp;
This is misleading because loop_emp is the name of the record, not the cursor or the loop. The compiler is ignoring the text after end loop although really it should at least warn you. If you wanted to name the loop, you would use a label like <<LOOP_EMP>> above it. (I always name my loop records r, similar to the i you often see used in numeric loops.)
I have this simple oracle plsql procedure:
declare
cursor A is
select column_A
from A_TAB; -- no order by
begin
for rec_ in A loop
procedure_A(rec_.column_A);
end loop;
end;
And this is running now for ages.
When I look into sys.v_$sql_bind_capture, value_string column, I can see the current value of bound column_A, and thankfully, that value keeps changing every few minutes.
As the cursor was not sorted by anything, is there a way to see how many more records to go (until this is finished)?
In other words I would need to see the currently fetched values of the query from that cursor. Where to look for it?
This is Oracle 12 database.
You can use dbms_application_info.set_session_longops to do this. The results are visible in V$SESSION_LONGOPS.
In your example, that could do something like:
DECLARE
rindex BINARY_INTEGER;
slno BINARY_INTEGER;
totalwork number;
sofar number;
obj BINARY_INTEGER;
cursor A is
select column_A,
COUNT(*) OVER () cnt
from A_TAB; -- no order by
begin
rindex := dbms_application_info.set_session_longops_nohint;
sofar := 0;
for rec_ in A loop
totalwork := rec_.cnt;
sofar := sofar + 1;
dbms_application_info.set_session_longops(rindex,
slno,
'Process a_tab',
'A_TAB',
0,
sofar,
totalwork,
'table',
'rows');
procedure_A(rec_.column_A);
end loop;
end;
Note that in order to get the totalwork value, I've used the analytic COUNT() function to get the total number of rows within the resultset. You could run a separate query to get the count before looping through your original cursor, if that is faster. You'd have to test both methods to work out which would be fastest for your data etc.
Of course, depending on what procedure_a does, you might be able to avoid the need to monitor the progress if you can refactor things so that all the work is being done in a single SQL statement. My answer above assumes that it's not possible to do that. If it is, I highly recommend you refactor your code instead!
I have a routine written in T-SQL for SQL Server. We are migrating to Oracle so I am trying to port it to PL/SQL. Here is the T-SQL routine (simplified); note the use of the table-valued variable which, in Oracle, will become a "nested table" type PL/SQL variable. The main thrust of my question is on the best ways of working with such "collection" objects within PL/SQL. Several operations in the ported code (second code sample, below) are quite awkward, where they seemed a lot easier in the SQL Server original:
DECLARE #MyValueCollection TABLE( value VARCHAR(4000) );
DECLARE #valueForThisRow VARCHAR(4000);
DECLARE #dataItem1Val INT, #dataItem2Val INT, #dataItem3Val INT, #dataItem4Val INT;
DECLARE theCursor CURSOR FAST_FORWARD FOR
SELECT DataItem1, DataItem2, DataItem3, DataItem4 FROM DataTable;
OPEN theCursor;
FETCH NEXT FROM theCursor INTO #dataItem1Val, #dataItem2Val, #dataItem3Val, #dataItem4Val;
WHILE ##FETCH_STATUS = 0
BEGIN
-- About 50 lines of logic that evaluates #dataItem1Val, #dataItem2Val, #dataItem3Val, #dataItem4Val and constructs #valueForThisRow
SET #valueForThisRow = 'whatever';
-- !!! This is the row that seems to have no natural Oracle equivalent
INSERT INTO #MyValueCollection VALUES(#valueForThisRow);
FETCH NEXT FROM theCursor INTO #dataItem1Val, #dataItem2Val, #dataItem3Val, #dataItem4Val;
END;
CLOSE theCursor;
DEALLOCATE theCursor;
-- !!! output all the results; this also seems harder than it needs to be in Oracle
SELECT * FROM #MyValueCollection;
I have been able to port pretty much everything, but in two places (see comments in the code), the logic is a lot more complex than the old SQL Server way, and I wonder if there might be, in Oracle, some more graceful way that is eluding me:
set serveroutput on; -- needed for DBMS_OUTPUT; see below
DECLARE
TYPE StringList IS TABLE OF VARCHAR2(4000);
myValueCollection StringList;
dummyTempCollection StringList; -- needed for my kludge; see below
valueForThisRow VARCHAR2(4000);
BEGIN
-- build all the sql statements
FOR c IN (
SELECT DataItem1, DataItem2, DataItem3, DataItem4 FROM DataTable;
)
LOOP
-- About 50 lines of logic that evaluates c.DataItem1, c.DataItem2, c.DataItem3, c.DataItem4 and constructs valueForThisRow
valueForThisRow := 'whatever';
-- This seems way harder than it should be; I would rather not need an extra dummy collection
SELECT valueForThisRow BULK COLLECT INTO dummyTempCollection FROM dual; -- overwrites content of dummy temp
myValueCollection := myValueCollection MULTISET UNION dummyTempCollection; -- merges into main collection
END LOOP;
-- output all the results... again, there's no shorter/easier/more-compact/single-line equivalent?
IF myValueCollection.COUNT > 0
THEN
FOR indx IN myValueCollection.FIRST .. myValueCollection.LAST
LOOP
DBMS_OUTPUT.PUT_LINE(myValueCollection(indx));
END LOOP;
END IF;
END;
/
Thanks in advance for any help!
Personally, I'd take the "50 lines of logic", move it into a function that you call in your SQL statement, and then do a simple BULK COLLECT to load the data into your local collection.
Assuming that you really want to load data element-by-element into the collection, you can simplify the code that loads the collection
DECLARE
TYPE StringList IS TABLE OF VARCHAR2(4000);
myValueCollection StringList := StringList();
valueForThisRow VARCHAR2(4000);
BEGIN
-- build all the sql statements
FOR c IN (
SELECT DataItem1, DataItem2, DataItem3, DataItem4 FROM DataTable;
)
LOOP
-- About 50 lines of logic that evaluates c.DataItem1, c.DataItem2, c.DataItem3, c.DataItem4 and constructs valueForThisRow
valueForThisRow := 'whatever';
myValueCollection.extend();
myValueCollection( myValueCollection.count ) := valueForThisRow;
END LOOP;
-- output all the results... again, there's no shorter/easier/more-compact/single-line equivalent?
IF myValueCollection.COUNT > 0
THEN
FOR indx IN myValueCollection.FIRST .. myValueCollection.LAST
LOOP
DBMS_OUTPUT.PUT_LINE(myValueCollection(indx));
END LOOP;
END IF;
END;
/
If you declare the collection as an associative array, you could avoid calling extend to increase the size of the collection. If you know the number of elements that you are going to load into the collection, you could pass that to a single extend call outside the loop. Potentially, you can also eliminate the valueForThisRow local variable and just operate on elements in the collection.
As for the code that processes the collection, what is it that you are really trying to do? It would be highly unusual for production code to write to dbms_output and expect that anyone will see the output during normal processing. That will influence the way that you would write that code. Assuming that your intention is really to just call dbms_output, knowing that will generally send the data into the ether
FOR indx IN 1 .. myValueCollection.count
LOOP
dbms_output.put_line( myValueCollection(indx) );
END LOOP;
This works when you have a dense collection (all indexes between 1 and the count of the collection exist and have values). If you might have a sparse collection, you would want to use FIRST, NEXT, and LAST in a loop but that's a bit more code.
I have to write a dynamic sql cursor where there are several possibilities in which the select query will be generated. Hence I am chosing dynamic and I am Using DBMS_SQL package to dynamically create a cursor and dynamically fetch the data.
However , Result set is going to be huge . around 11GB (there are 2.4 million records and the select statement will be approx 80 cols long assumning about 50Byte varchar per column)
Hence I cannot open the cursor at once . I want to know if there is a feature wherein i can fetch the data from the curosr keeping the curosr open for Blocks of say 1000 records at time(I will have to do this dynamically)
Please find the code attached which only fetches and prints the value of the columns (one sample case ) I want to use bul collect here \
Thanks
---------------code sample--------------------------------------
--create or replace type TY_DIMDEAL AS TABLE OF VARCHAR2(50) ;
create or replace procedure TEST_PROC (po_recordset out sys_refcursor)
as
v_col_cnt INTEGER;
v_ind NUMBER;
rec_tab DBMS_SQL.desc_tab;
v_cursor NUMBER;
lvar_output number:=0;
lvar_output1 varchar2(100);
lvar_output3 varchar2(100);
lvar_output2 varchar2(100);
LVAR_TY_DIMDEAL TY_DIMDEAL;
lvarcol varchar2(100);
begin
--
LVAR_TY_DIMDEAL := TY_DIMDEAL();
lvar_output1 := '';
v_cursor := dbms_sql.open_cursor;
dbms_sql.parse(v_cursor, 'select to_char(Field1) , to_char(fiel2) , to_char(field3) from table,table2 ', dbms_sql.native);
dbms_sql.describe_columns(v_cursor, v_col_cnt, rec_tab);
FOR v_pos in 1..rec_tab.LAST LOOP
LVAR_TY_DIMDEAL.EXTEND();
DBMS_SQL.define_column( v_cursor, v_pos ,LVAR_TY_DIMDEAL(v_pos),20);
END LOOP;
-- DBMS_SQL.define_column( v_cursor, 1 ,lvar_output1,20);
--DBMS_SQL.define_column( v_cursor, 2 ,lvar_output2,20);
--DBMS_SQL.define_column( v_cursor, 3 ,lvar_output3,20);
v_ind := dbms_sql.execute( v_cursor );
LOOP
v_ind := DBMS_SQL.FETCH_ROWS( v_cursor );
EXIT WHEN v_ind = 0;
lvar_output := lvar_output+1;
dbms_output.put_line ('row number '||lvar_output) ;
FOR v_col_seq IN 1 .. rec_tab.COUNT LOOP
LVAR_TY_DIMDEAL(v_col_seq):= '';
DBMS_SQL.COLUMN_VALUE( v_cursor, v_col_seq,LVAR_TY_DIMDEAL(v_col_seq));
dbms_output.put_line (LVAR_TY_DIMDEAL(v_col_seq));
END LOOP;
END LOOP;
end TEST_PROC;
Fetching data from a cursor in blocks of reasonable size, while keeping the cursor open, is one of PL/SQL Best Practices.
The above document (see Code 38 item) sketches an approach for when the select list is not known until runtime. Basically:
Define an appropriate type to fetch results into. Let's assume that all the returned columns will by of of type VARCHAR2:
-- inside DECLARE
Ty_FetchResults IS TABLE OF DBMS_SQL.VARCHAR2_TABLE;
lvar_results Ty_FetchResults;
Before each call to DBMS_SQL.FETCH_ROWS, call DBMS_SQL.DEFINE_ARRAY to enable batch fetching.
Call DBMS_SQL.FETCH_ROWS to fetch 1000 rows from the cursor.
Call DBMS_SQL.COLUMN_VALUE to copy the fetched data into your result array.
Process the results, record by record, in a FOR loop. Don't worry about the number of fetched records: if there are records to process, the FOR loop will run correctly; if the result array is empty, the FOR loop will not run.
Exit from the loop when the number of fetched records is less than the expected size.
Remember to DBMS_SQL.CLOSE the cursor.
Your loop body should look like this:
LOOP
FOR j IN 1..v_col_cnt LOOP
DBMS_SQL.DEFINE_ARRAY(v_cursor, j, lvar_results(j), 1000, 1);
END LOOP;
v_ind := DBMS_SQL.FETCH_ROWS(v_cursor);
FOR j IN 1..v_col_cnt LOOP
lvar_results(j).DELETE;
DBMS_SQL.COLUMN_VALUE(v_cursor, j, lvar_results(j));
END LOOP;
-- process the results, record by record
FOR i IN 1..lvar_results(1).COUNT LOOP
-- process a single record...
-- your logic goes here
END LOOP;
EXIT WHEN lvar_results(1).COUNT < 1000;
END LOOP;
-- don't forget: DBMS_CLOSE(v_cursor);
See also Doing SQL from PL/SQL: Best and Worst Practices.
LIMIT CLAUSE CAN COME TO RESCUE!
PL/SQL collections are essentially arrays in memory, so massive
collections can have a detrimental effect on system performance due to
the amount of memory they require. In some situations, it may be
necessary to split the data being processed into chunks to make the
code more memory-friendly. This “chunking” can be achieved using the
LIMIT clause of the BULK COLLECT syntax.
YOU CAN USE LIMIT CLAUSE AFTER BULK COLLECT INTO CLAUSE TO LIMIT YOUR RS.
AFTER YOU EXCEED TO LIMIT YOU CAN FETCH REMAINING ROWS.
SEE THIS ARTICLE
http://www.dba-oracle.com/plsql/t_plsql_limit_clause.htm
W.r.t code below I can not declare the type of fetch-into-variable as the underlying table's %ROWTYPE because the SYS_REFCURSOR is on a select that joins two tables and also selects a few functions called on the attributes of the underlying two tables; i.e I can't declare as L_RECORD T%ROWTYPE
---
DECLARE
P_RS SYS_REFCURSOR;
L_RECORD P_RS%ROWTYPE;
BEGIN
CAPITALEXTRACT(
P_RS => P_RS
);
OPEN P_RS;
LOOP
BEGIN
FETCH P_RS INTO L_RECORD;
EXIT WHEN P_RS%NOTFOUND;
...
EXCEPTION
WHEN OTHERS THEN
...
END;
END LOOP;
CLOSE P_RS;
END;
--------
CREATE or REPLACE PROCEDURE CAPITALEXTRACT
(
p_rs OUT SYS_REFCURSOR
) AS
BEGIN
OPEN p_rs for
select t.*,tminusone.*, f(t.cash), g(t.cash) FROM T t, TMINUSONE tminusone
where t.ticket=tminusone.ticket;
END CAPITALEXTRACT;
Of course I don't want to define a static table R with columns as returned in the SYS_REFCURSOR and then declare as L_RECORD R%ROWTYPE.
And hence the question:
how to declare %ROWTYPE of a variable that is a weakly typed SYS_REFCURSOR ?
The short answer is, you can't. You'd need to define a variable for each column that wil be returned.
DECLARE
P_RS SYS_REFCURSOR;
L_T_COL1 T.COL1%TYPE;
L_T_COL1 T.COL2%TYPE;
...
And then fetch into the list of columns:
FETCH P_RS INTO L_T_COL1, L_T_COL2, ... ;
This is painful but manageable as long as you know what you're expecting in the ref cursor. Using T.* in your procedure makes this fragile though, as adding a column to the table would break the code that thinks it knows what columns there are and what order they're in. (You can also break it between environments if the tables aren't built consistently - I've seen places where column ordering is different in different environments). You'll probably want to make sure you're only selecting the columns you really care about anyway, to avoid having to define variables for things you'll never read.
From 11g you can use the DBMS_SQL package to convert your sys_refcursor into a DBMS_SQL cursor, and you can interrogate that to determine the columns. Just as an example of what you can do, this will print out the value of every column in every row, with the column name:
DECLARE
P_RS SYS_REFCURSOR;
L_COLS NUMBER;
L_DESC DBMS_SQL.DESC_TAB;
L_CURS INTEGER;
L_VARCHAR VARCHAR2(4000);
BEGIN
CAPITALEXTRACT(P_RS => P_RS);
L_CURS := DBMS_SQL.TO_CURSOR_NUMBER(P_RS);
DBMS_SQL.DESCRIBE_COLUMNS(C => L_CURS, COL_CNT => L_COLS,
DESC_T => L_DESC);
FOR i IN 1..L_COLS LOOP
DBMS_SQL.DEFINE_COLUMN(L_CURS, i, L_VARCHAR, 4000);
END LOOP;
WHILE DBMS_SQL.FETCH_ROWS(L_CURS) > 0 LOOP
FOR i IN 1..L_COLS LOOP
DBMS_SQL.COLUMN_VALUE(L_CURS, i, L_VARCHAR);
DBMS_OUTPUT.PUT_LINE('Row ' || DBMS_SQL.LAST_ROW_COUNT
|| ': ' || l_desc(i).col_name
|| ' = ' || L_VARCHAR);
END LOOP;
END LOOP;
DBMS_SQL.CLOSE_CURSOR(L_CURS);
END;
/
That's not of much practical use, and for brevity I'm treating every value as a string since I just want to print it anyway. Look at the docs and search for examples for more practical applications.
If you only want a few columns from your ref cursor you could, I suppose, loop around l_desc and record the position where column_name is whatever you're interested in, as a numeric variable; you could then refer to the column by that variable later where you would normally use the name in a cursor loop. Depends what you're doing with the data.
But unless you're expecting to not know the column order you're getting back, which is unlikely since you seem to control the procedure - and assuming you get rid of the .*s - you're probably much better off reducing the returned columns to the minimum you need and just declaring them all individually.