I have a table with employees and for each employees have few bills.
I declare 2 cursors, one for all employees(distinct) and second cursor for all bills for one employees. Now, open 1st cursor with all employees, fetch one, open second cursor (based on employee from 1st cursor) with all bills for employee. To reuse a second cursor for all employees, I open and close second cursor for each employee. This thing spend a lot of time. How to reuse a second cursor instead reopen or any good idea?
Part of code in Pro*C:
struct sforc1 {
long nis_rad[ROWS_FETCHED_C1];
long sec_nis[ROWS_FETCHED_C1];
/*char f_fact[9];
long sec_rec;*/
}forc1;
struct sforc2 {
long nis_rad[ROWS_FETCHED_C2];
long sec_nis[ROWS_FETCHED_C2];
char f_fact[ROWS_FETCHED_C2][9];
long sec_rec[ROWS_FETCHED_C2];
char f_p_camb_est[ROWS_FETCHED_C2][9];
char op_cambest[ROWS_FETCHED_C2][9];
}forc2;
void main (void)
{
exec sql declare c1 cursor for
select distinct nis_rad, sec_nis
from recibos
where ((imp_tot_rec - imp_cta)>0) and f_p_camb_est = '29991231';
exec sql declare c2 cursor for
select nis_rad, sec_nis, f_fact, sec_rec, f_p_camb_est, op_cambest
from recibos
where ((imp_tot_rec - imp_cta)>0) and f_p_camb_est = '29991231' and nis_rad = :forc1.nis_rad[i] and sec_nis=:forc1.sec_nis[i];
exec sql open c1;
while(1){
exec sql fetch c1 into :forc1;
rows_this_time1 = sqlca.sqlerrd[2]-rows_before1;
rows_before1 = sqlca.sqlerrd[2];
if (rows_this_time1==0){
break;
}
for(i=0;i<rows_this_time1;++i){
exec sql open c2;
rows_before2 = 0;
while(1){
exec sql fetch c2 into :forc2;
rows_this_time2 = sqlca.sqlerrd[2]-rows_before2;
rows_before2=sqlca.sqlerrd[2];
if(rows_this_time2==0){
break;
}
for(j=0;j<rows_this_time2;++j){
strcpy(forc2.f_p_camb_est[j], "20161212");
strcpy(forc2.op_cambest[j], "SIMD0943");
}
EXEC SQL
update recibos
set f_p_camb_est = :forc2.f_p_camb_est,
op_cambest = :forc2.op_cambest
where nis_rad = :forc2.nis_rad
and sec_nis = :forc2.sec_nis
and f_fact = :forc2.f_fact
and sec_rec = :forc2.sec_rec;
}
exec sql close c2;
}
exec sql close c1;
exec sql commit;
exec sql open c1;
rows_before1 = 0;
}
exec sql close c1;
}
nis_rad and sec_nis is a employee_id(primary key). Each nis_rad have few bills f_fact(bills)
For processing 10000 nis_rad's spend 30 min, and 28-29 min is for re-open second cursor(c2)
UP. Deleted previously example
I would say inplace of opening explicit cursor, let oracle use implicit cursor and use simple loops to achieve your requirements:
declare
begin
for rec in ( select employee_id
from employees )
Loop
for rcrd in (Select bill from employee_bill
where employee_id = rec.employee_id)
loop
/***Process your bills***/
end loop;
end loop;
end;
I've edited an answer.
You can't reuse date from cursor without opening it again unless you cache all results in memory. Cursor is like pointer to data when you read record it already points to next so you can't get back.
Problem in your code is using SQL imperative way. You shouldn't pull all records to your application and then apply logic. Think about how to write a query that will return only records you need to process. It may be even faster to execute 2-3 queries that will return records for each section of your code than pull all data and then check logic in application.
The problem with your code is that it's doing an awful lot of context switching, so it's no wonder that it's slow.
From what I can tell, your code is doing something like:
In Pro*C, go to the database
In the database, open the first cursor
Back in Pro*C, ask the database for a row from the cursor
The database sends back a row
In Pro*C, go to the database
In the database, open the second cursor, passing in the values from the first cursor
In Pro*C, ask the database for a row from the second cursor
The database sends back a row
In Pro*C, ask the database to update the row(s) based on the results from the second cursor
The database updates the rows.
Repeat steps 7-10 until there are no more rows to fetch from cursor 2
Repeat steps 3.11 until there are no more rows to fetch from cursor 1
I think you can see that there's an awful lot of unnecessary to-ing and fro-ing going on there... and in fact, it's likely that that's where the majority of your time is being spent.
Were it not for your mentor's requirements (which I assume are for teaching purposes only), you could do this in a single update statement, like so:
update recibos
set f_p_camb_est = '20161212',
op_cambest = 'SIMD0943'
where (imp_tot_rec - imp_cta) > 0
and f_p_camb_est = '29991231';
(N.B. if f_p_camb_est is of DATE datatype, then the string literals need to be converted into DATEs using DATE or to_date(), e.g. DATE '29991231', to_date('20161212', 'yyyymmdd'), unless you're going to pass in a parameter that's already of DATE datatype.)
Doing it this way means that you can plug the update statement into your Pro*C app and it will do:
In Pro*C, ask the database to execute the update statement
The database executes the update statement
Control is returned back to Pro*C.
That's 2 context switches in total - a massive reduction than from your original method!
However, your mentor has asked for (IMHO) the daft way of updating the rows by doing it one nis_rad at a time.
In that case, you are still far better off doing the bulk of the work in the database - there is *no* point in sending rows back to your app from the database only for your app to do nothing with them except pass them back into the database. The way you would do this on the database side is in PL/SQL, like so:
begin
for rec in (select distinct nis_rad, sec_nis
from recibos
where (imp_tot_rec - imp_cta) > 0
and f_p_camb_est = '29991231')
loop
update recibos
set f_p_camb_est = '20161212',
op_cambest = 'SIMD0943'
where nis_rad = rec.nis_rad
and (imp_tot_rec - imp_cta) > 0
and f_p_camb_est = '29991231';
end loop;
end;
/
You could either call this directly from Pro*C as an anonymous block or you could create it as a procedure in the database (potentially with parameters), and then call the procedure from Pro*C.
Either way, the overall flow looks like:
In Pro*C, ask the database to execute the PL/SQL
The database executes the PL/SQL
Control is returned back to Pro*C.
I know this looks like 2 context switches, but you also need to take account of the context switching within the database, when it switches between the PL/SQL and SQL engines, which looks something like:
The PL/SQL engine requests a cursor
The SQL engine opens the cursor
The PL/SQL engine requests a row from the cursor
The SQL engine passes a row back
The PL/SQL engine stored the information in the rec record
The PL/SQL engine asks the SQL engine to execute the update statement, based on the nis_rad value from the rec record
The SQL engine runs the update statement
Repeat steps 1 - 7 until no more rows are returned
which, as I'm sure you can see is a lot more than the 2 context switches we had overall if you had just run the single update statement instead.
Hopefully that clears things up a bit for you?
Related
Is there a hint to generate execution plan ignoring the existing one from the shared pool?
There is not a hint to create an execution plan that ignores plans in the shared pool. A more common way of phrasing this question is: how do I get Oracle to always perform a hard parse?
There are a few weird situations where this behavior is required. It would be helpful to fully explain your reason for needing this, as the solution varies depending why you need it.
Strange performance problem. Oracle performs some dynamic re-optimization of SQL statements after the first run, like adaptive cursor sharing and cardinality feedback. In the rare case when those features backfire you might want to disable them.
Dynamic query. You have a dynamic query that used Oracle data cartridge to fetch data in the parse step, but Oracle won't execute the parse step because the query looks static to Oracle.
Misunderstanding. Something has gone wrong and this is an XY problem.
Solutions
The simplest way to solve this problem are by using Thorsten Kettner's solution of changing the query each time.
If that's not an option, the second simplest solution is to flush the query from the shared pool, like this:
--This only works one node at a time.
begin
for statements in
(
select distinct address, hash_value
from gv$sql
where sql_id = '33t9pk44udr4x'
order by 1,2
) loop
sys.dbms_shared_pool.purge(statements.address||','||statements.hash_value, 'C');
end loop;
end;
/
If you have no control over the SQL, and need to fix the problem using a side-effect style solution, Jonathan Lewis and Randolf Geist have a solution using Virtual Private Database, that adds a unique predicate to each SQL statement on a specific table. You asked for something weird, here's a weird solution. Buckle up.
-- Create a random predicate for each query on a specific table.
create table hard_parse_test_rand as
select * from all_objects
where rownum <= 1000;
begin
dbms_stats.gather_table_stats(null, 'hard_parse_test_rand');
end;
/
create or replace package pkg_rls_force_hard_parse_rand is
function force_hard_parse (in_schema varchar2, in_object varchar2) return varchar2;
end pkg_rls_force_hard_parse_rand;
/
create or replace package body pkg_rls_force_hard_parse_rand is
function force_hard_parse (in_schema varchar2, in_object varchar2) return varchar2
is
s_predicate varchar2(100);
n_random pls_integer;
begin
n_random := round(dbms_random.value(1, 1000000));
-- s_predicate := '1 = 1';
s_predicate := to_char(n_random, 'TM') || ' = ' || to_char(n_random, 'TM');
-- s_predicate := 'object_type = ''TABLE''';
return s_predicate;
end force_hard_parse;
end pkg_rls_force_hard_parse_rand;
/
begin
DBMS_RLS.ADD_POLICY (USER, 'hard_parse_test_rand', 'hard_parse_policy', USER, 'pkg_rls_force_hard_parse_rand.force_hard_parse', 'select');
end;
/
alter system flush shared_pool;
You can see the hard-parsing in action by running the same query multiple times:
select * from hard_parse_test_rand;
select * from hard_parse_test_rand;
select * from hard_parse_test_rand;
select * from hard_parse_test_rand;
Now there are three entries in GV$SQL for each execution. There's some odd behavior in Virtual Private Database that parses the query multiple times, even though the final text looks the same.
select *
from gv$sql
where sql_text like '%hard_parse_test_rand%'
and sql_text not like '%quine%'
order by 1;
I think there is no hint indicating that Oracle shall find a new execution plan everytime it runs the query.
This is something we'd want for select * from mytable where is_active = :active, with is_active being 1 for very few rows and 0 for maybe billions of other rows. We'd want an index access for :active = 1 and a full table scan for :active = 0 then. Two different plans.
As far as I know, Oracle uses bind variable peeking in later versions, so with a look at the statistics it really comes up with different execution plans for different bind varibale content. But in older versions it did not, and thus we'd want some hint saying "make a new plan" there.
Oracle only re-used an execution plan for exactly the same query. It sufficed to add a mere blank to get a new plan. Hence a solution might be to generate the query everytime you want to run it with a random number included in a comment:
select /* 1234567 */ * from mytable where is_active = :active;
Or just don't use bind variables, if this is the problem you want to address:
select * from mytable where is_active = 0;
select * from mytable where is_active = 1;
wondering if there is way to validate a query before executing
Is there way to check/validate Query without executing it?
One way that we validate SQL is to add a condition to the SQL that could never be true.
Example:
long ll_rc
long ll_result
string ls_sql, ls_test
string ls_message
//Arbitrary SQL
ls_sql = "SELECT * FROM DUAL"
//This SQL when executed will always return 0 if successful.
ls_test = "select count(*) from ( " + ls_sql + " WHERE 1 = 2 )"
DECLARE l_cursor DYNAMIC CURSOR FOR SQLSA ;
PREPARE SQLSA FROM :ls_test;
OPEN DYNAMIC l_cursor;
ll_rc = SQLCA.SQLCODE
choose case ll_rc
case 0
//Success
ls_message = "SQL is properly formed"
case 100
//Fetched row not found. This should not be the case since we only opened the cursor
ls_message = SQLCA.SQLERRTEXT
case -1
//Error; the statement failed. Use SQLErrText or SQLDBCode to obtain the detail.
ls_message = SQLCA.SQLERRTEXT
end choose
CLOSE l_cursor ; //This will fail if open cursor failed.
messagebox( "Result", ls_message )
Note: If your SQL is VERY complicated, which I suspect it isn't, the database optimizer may take several seconds to prepare your SQL. It will be significantly less time than if you run the entire query.
Since the database is the final arbitrator for what is "valid" (table and column names and such) the general answer is no. Now you could come up with a class in PB which checks statement syntax, object names, etc. so you wouldn't have to touch the db but it would be obsolete as soon as any changes were made to the db.
Put the select statement in any script and compile it. Part of the work will be to check the SQL syntax against the database you are connected to.
Watch out: you need at least one bound variable in the column list of your SQL statement. This is not the case for other DML statements.
Example:
in my case:
select noms into :ls_ttt from contacts;
results in a message Unknown columns 'noms' in 'field list'.
However,
select nom into :ls_ttt from contacts;
does not show any error.
Hope this helps.
I have several Oracle functions that are similar to the one below. I don't know much about Oracle and although I have made in roads on a major query re-write. I'd like to ask for some help on how to convert this function to SQL Server 2008.
I have tried using the online conversion tool at www.sqlines.com and benefited from many pages there... but not successful in converting this function....
Thanks in advance, John
Oracle source:
function OfficeIDMainPhoneID(p_ID t_OfficeID)
return t_OfficePhoneID
is
wPhID t_OfficePhoneID;
wPhID1 t_OfficePhoneID;
cursor cr_phone
is
select Office_PHONE_ID,IS_PHONE_PRIMARY
from Office_PHONE
where Office_ID = p_ID
order by SEQ_NUMBER;
begin
wPhID :=NULL;
wPhID1:=NULL;
for wp in cr_phone
loop
if wPhID is NULL
then wPhID1:=wp.Office_PHONE_ID;
end if;
if wp.IS_PHONE_PRIMARY = 'Y'
then
wPhID:=wp.Office_PHONE_ID;
Exit;
end if;
end loop;
if wPhID is NULL
then wPhID:=wPhID1;
end if;
return(wPhID);
end OfficeIDMainPhoneID;
SQL Server attempt:
create function OfficeIDMainPhoneID(#p_ID t_OfficeID)
returns t_OfficePhoneID
as
begin
declare #wPhID t_OfficePhoneID;
declare #wPhID1 t_OfficePhoneID;
declare cr_phone cursor local
for
select Office_PHONE_ID,IS_PHONE_PRIMARY
from Office_PHONE
where Office_ID = #p_ID
order by SEQ_NUMBER;
set #wPhID =NULL;
set #wPhID1=NULL;
declare wp cursor for cr_phone
open wp;
fetch wp into;
while ##fetch_status=0
begin
if #wPhID is NULL
begin set #wPhID1=wp.Office_PHONE_ID;
end
if wp.IS_PHONE_PRIMARY = 'Y'
begin
set #wPhID=wp.Office_PHONE_ID;
Exit;
end
fetch wp into;
end;
close wp;
deallocate wp;
if #wPhID is NULL
begin set #wPhID=#wPhID1;
end
return(#wPhID);
end ;
To answer the question about the functions as written
If you just want to fix the cursor so it works, one problem is the two "fetch wp into;" statements. You are saying "fetch the data and put it into" and then not giving it anything to put it into. Declare a couple of variables, put the data into them, then later use the variables, not the code. You need one variable per item returned in your cursor definition, so one each for Office_PHONE_ID and IS_PHONE_PRIMARY.
Also, you are trying to declare variables (and the function) as t_OfficePhoneID, I suspect that should be something like INT OR BIGINT instead (whatever the table definition for the column is).
Declare #OP_ID INT, #ISPRIMARY CHAR(1) --Or whatever the column is
later (in two locations),
fetch wp into (#OP_ID, #ISPRIMARY);
then use #OP_ID instead of wp.Office_PHONE_ID, and so on.
HOWEVER, I would throw away all the code in the function after declaring #wPhID, and do something else. Cursors suck if you can get what you want with a simple set based request. If you work your way through the oracle code, it is doing the following:
Get the id of the first phone number marked primary (in sequence order). If it didn't find one of those, just get the id of the first non-primary phone number in sequence order. You can do that with the following
set #wPhID = select TOP 1 Office_PHONE_ID
from Office_PHONE
where Office_ID = #p_ID
order by CASE WHEN IS_PHONE_PRIMARY = 'Y' THEN 0 ELSE 1 END, SEQ_NUMBER;
Return #wPhID and you're done.
I used "CASE WHEN IS_PHONE_PRIMARY = 'Y' THEN 0 ELSE 1 END" in the order by because I don't know what other values are possible, so this will always work. If you know the only possible values are 'Y' and 'N', you could use something like the following instead
order by IS_PHONE_PRIMARY DESC, SEQ_NUMBER;
I would like to have your advise how to implement the plsql. Below is the situation that i want to do..
select * from table A
loop - get each records from #1 step, and execute the store procedure, processMe(a.field1,a.field2,a.field3 || "test",a.field4);
i dont have any idea how to implement something like this. Below is sample parameter for processMe
processMe(
number_name IN VARCHAR,
location IN VARCHAR,
name_test IN VARCHAR,
gender IN VARCHAR )
Begin
select objId into obj_Id from tableUser where name = number_name ;
select locId into loc_Id from tableLoc where loc = location;
insert into tableOther(obj_id,loc_id,name_test,gender)
values (obj_Id ,loc_Id, name_test, gender)
End;
FOR rec IN (SELECT *
FROM table a)
LOOP
processMe( rec.field1,
rec.field2,
rec.field3 || 'test',
rec.field4 );
END LOOP;
does what you ask. You probably want to explicitly list the columns you actually want in the SELECT list rather than doing a SELECT * (particularly if there is an index on the four columns you actually want that could be used rather than doing a table scan or if there are columns you don't need that contain a large amount of data). Depending on the data volume, it would probably be more efficient if a version of processMe were defined that could accept collections rather than processing data on a row-by-row bases as well.
i just add some process. but this is just a sample. By the way, why
you said that this is not a good idea using loop? i interested to know
Performance wise, If you can avoid looping through a result set executing some other DMLs inside a loop, do it.
There is PL/SQL engine and there is SQL engine. Every time PL/SQL engine stumbles upon a SQL statement, whether it's a select, insert, or any other DML statement, it has to send it to the SQL engine for the execution. It calls context switching. Placing DML statement inside a loop will cause the switch(for each DML statement if there are more than one of them) as many times as many times the body of a loop has to be executed. It can be a cause of a serious performance degradation. if you have to loop, say, through a collection, use foreach loop, it minimizes context switching by executing DML statements in batches.
Luckily, your code can be rewritten as a single SQL statement, avoiding for loop entirely:
insert into tableOther(obj_id,loc_id,name_test,gender)
select tu.objId
, tl.locid
, concat(a.field3, 'test')
, a.field4
from table a
join tableUser tu
on (a.field1 = tu.name)
join tableLoc tl
on (tu.field2 = tl.loc)
You can put that insert statement into a procedure, if you want. PL/SQL will have to sent this SQL statement to the SQL engine anyway, but it will only be one call.
You can use a variable declared using a cursor rowtype. Something like this:
declare
cursor my_cursor is
select * from table;
reg my_cursor%rowtype;
begin
for reg in my_cursor loop
--
processMe(reg.field1, reg.field2, reg.field3 || "test", reg.field4);
--
end loop;
end;
I have a procedure which returns ref cursor as output parameter. I need to find a way to get the count of no.of records in the cursor. Currently I have count fetched by repeating the same select query which is hindering the performance.
ex:
create or replace package temp
TYPE metacur IS REF CURSOR;
PROCEDURE prcSumm (
pStartDate IN DATE,
pEndDate IN DATE,
pKey IN NUMBER,
pCursor OUT metacur
) ;
package body temp is
procedure prcSumm(
pStartDate IN DATE,
pEndDate IN DATE,
pKey IN NUMBER,
pCursor OUT metacur
)
IS
vCount NUMBER;
BEGIN
vCount := 0;
select count(*) into vCount
from customer c, program p, custprog cp
where c.custno = cp.custno
and cp.programid = p.programid
and p.programid = pKey
and c.lastupdate >= pStartDate
and c.lastupdate < pEndDate;
OPEN pCursor for SELECT
c.custno, p.programid, c.fname, c.lname, c.address1, c.address2, cp.plan
from customer c, program p, custprog cp
where c.custno = cp.custno
and cp.programid = p.programid
and p.programid = pKey
and c.lastupdate >= pStartDate
and c.lastupdate < pEndDate;
end prcSumm;
Is there a way to get the no.of rows in the out cursor into vCount.
Thanks!
Oracle does not, in general, know how many rows will be fetched from a cursor until the last fetch finds no more rows to return. Since Oracle doesn't know how many rows will be returned, you can't either without fetching all the rows (as you're doing here when you re-run the query).
Unless you are using a single-user system or you are using a non-default transaction isolation level (which would introduce additional complications), there is no guarantee that the number of rows that your cursor will return and the count(*) the second query returns would match. It is entirely possible that another session committed a change between the time that you opened the cursor and the time that you ran the count(*).
If you are really determined to produce an accurate count, you could add a cnt column defined as count(*) over () to the query you're using to open the cursor. Every row in the cursor would then have a column cnt which would tell you the total number of rows that will be returned. Oracle has to do more work to generate the cnt but it's less work than running the same query twice.
Architecturally, though, it doesn't make sense to return a result and a count from the same piece of code. Determining the count is something that the caller should be responsible for since the caller has to be able to iterate through the results. Every caller should be able to handle the obvious boundary cases (i.e. the query returns 0 rows) without needing a separate count. And every caller should be able to iterate through the results without needing to know how many results there will be. Every single time I've seen someone try to follow the pattern of returning a cursor and a count, the correct answer has been to redesign the procedure and fix whatever error on the caller prompted the design.