I have a question regarding how queries executed through
'execute immediate' is treated in the library cache (We use Oracle 11).
Let's say I have a function like this:
FUNCTION get_meta_map_value (
getfield IN VARCHAR2,
searchfield IN VARCHAR2,
searchvalue IN VARCHAR2
) RETURN VARCHAR2 IS
v_outvalue VARCHAR2(32767);
sql_stmt VARCHAR2(2000) := 'SELECT '||getfield||' FROM field_mapping, metadata '||
'WHERE field_mapping.metadataid = metadata.metadataid AND rownum = 1 AND '||searchfield||' = :1';
BEGIN
EXECUTE IMMEDIATE sql_stmt INTO v_outvalue USING searchvalue;
...
The getfield and searchfield are in one installation always the same (but has other values in another installation, so that is why we use dynamic sql)
So this leaves us with an sql that only differs in the searchvalue (which is a parameter).
This function is called in a loop that executes x times, from inside another stored procedure.
The stored procedure is executed y times during the connection life time, through ODBC connection.
And there are z connections, but each of them uses the same database login.
Now let us also assume that the searchvalue changes b times during one loop.
Question 1:
When calculating how many copies of the sql will be kept in the library cache,
can we disregard the different values the searchvalue can have (b), as the value is sent as a parameter to execute immediate?
Question 2:
Will the loop cause a hard parse of the query x times (query will be created in library cache x times), or can Oracle reuse the query?
(We assume that the searchvalue is the same for all calls in this question here, for simplicity)
Question 3:
Does the y (number of times the stored procedure is called from odbc during the lifetime of one connection)
also multiply the amount of copies of the query that are kept in library cache?
Question 4:
Does the z (number of simultaneous connections with same db login)
multiply the amount of copies of the query that are kept in library cache?
Main question:
What behaviour should I expect here?
Is the behaviour configurable?
The cause for this question is that we have had this code is production for 4 years, and now one of our customer gets back to us and says "This query fills our whole SGA, and Oracle says it's your fault".
The number of different combinations of getfield and searchfield should determine how many "copies" there will be. I use teh word "copies" cautiously because Oracke will treat each variation as distinct. Since you are using a bind variable for searchvalue so however many values you have for this will not add to the query count.
In short, it looks like your code is OK.
Number of connections should not increase the hard parses.
Ask for a AWR report to see exactly how many of these queries are in the SGA, and how many hard parses are being triggered.
I will disagree that the number of connections will not increase the hard parse count for the posted code because the last I knew dynamic SQL cannot be shared between sessions. Since the generated SQL uses a bind variable it should generate a reusable statement by the session, but it will not be sharable between user sessions. As a general rule dynamic SQL should be used only for infrequently executed statements. You may want to refer to the following:
- -
Designing applications for performance and scalability An Oracle White Paper July 2005
https://www.oracle.com/technetwork/database/performance/designing-applications-for-performa-131870.pdf
- -
enter code here
Related
I've seen lots of posts regarding the use of cursors in PL/SQL to return data to a calling application, but none of them touch on the issue I believe I'm having with this technique. I am fairly new to Oracle, but have extensive experience with MSSQL Server. In SQL Server, when building queries to be called by an application for returning data, I usually put the SELECT statement inside a stored proc with/without parameters, and let the stored proc execute the statement(s) and return the data automatically. I've learned that with PL/SQL, you must store the resulting dataset in a cursor and then consume the cursor.
We have a query that doesn't necessarily return huge amounts of rows (~5K - 10K rows), however the dataset is very wide as it's composed of 1400+ columns. Running the SQL query itself in SQL Developer returns results instantaneously. However, calling a procedure that opens a cursor for the same query takes 5+ minutes to finish.
CREATE OR REPLACE PROCEDURE PROCNAME(RESULTS OUT SYS_REFCURSOR)
AS
BEGIN
OPEN RESULTS FOR
<SELECT_query_with_1400+_columns>
...
END;
After doing some debugging to try to get to the root cause of the slowness, I'm leaning towards the cursor returning one row at a time very slowly. I can actually see this real-time by converting the proc code into a PL/SQL block and using DBMS_SQL.return_result(RESULTS) after the SELECT query. When running this, I can see each row show up in the Script output window in SQL Developer one at a time. If this is exactly how the cursor returns the data to the calling application, then I can definitely see how this is the bottleneck as it could take 5-10 minutes to finish returning all 5K-10K rows. If I remove columns from the SELECT query, the cursor displays all the rows much faster, so it does seem like the large amount of columns is an issue using a cursor.
Knowing that running the SQL query by itself returns instant results, how could I get this same performance out of a cursor? It doesn't seem like it's possible. Is the answer putting the embedded SQL in the application code and not using a procedure/cursor to return data in this scenario? We are using Oracle 12c in our environment.
Edit: Just want to address how I am testing performance using the regular SELECT query vs the PL/SQL block with cursor method:
SELECT (takes ~27 seconds to return ~6K rows):
SELECT <1400+_columns>
FROM <table_name>;
PL/SQL with cursor (takes ~5-10 minutes to return ~6K rows):
DECLARE RESULTS SYS_REFCURSOR;
BEGIN
OPEN RESULTS FOR
SELECT <1400+_columns>
FROM <table_name>;
DBMS_SQL.return_result(RESULTS);
END;
Some of the comments are referencing what happens in the console application once all the data is returned, but I am only speaking regarding the performance of the two methods described above within Oracle\SQL Developer. Hope this helps clarify the point I'm trying to convey.
You can run a SQL Monitor report for the two executions of the SQL; that will show you exactly where the time is being spent. I would also consider running the two approaches in separate snapshot intervals and checking into the output from an AWR Differences report and ADDM Compare Report; you'd probably be surprised at the amazing detail these comparison reports provide.
Also, even though > 255 columns in a table is a "no-no" according to Oracle as it will fragment your record across > 1 database blocks, thus increasing the IO time needed to retrieve the results, I suspect the differences in the two approaches that you are seeing is not an IO problem since in straight SQL you report fast result fetching all. Therefore, I suspect more of a memory problem. As you probably know, PL/SQL code will use the Program Global Area (PGA), so I would check the parameter pga_aggregate_target and bump it up to say 5 GB (just guessing). An ADDM report run for the interval when the code ran will tell you if the advisor recommends a change to that parameter.
I want to write pl/sql code which utilizes a Cursor and Bulk Collect to retrieve my data. My database has rows in the order of millions, and sometimes I have to query it to fetch nearly all records on client's request. I do the querying and subsequent processing in batches, so as to not congest the server and show incremental progress to the client. I have seen that digging down for later batches takes considerably more time, which is why I am trying to do it by way of cursor.
Here is what should be simple pl/sql around my main sql query:
declare
cursor device_row_cur
is
select /my_query_details/;
type l_device_rows is table of device_row_cur%rowtype;
out_entries l_device_rows := l_device_rows();
begin
open device_row_cur;
fetch device_row_cur
bulk collect into out_entries
limit 100;
close device_row_cur;
end;
I am doing batches of 100, and fetching them into out_entries. The problem is that this block compiles and executes just fine, but doesn't return the data rows it fetched. I would like it to return those rows just the way a select would. How can this be achieved? Any ideas?
An anonymous block can't return anything. You can assign values to a bind variable, including a collection type or ref cursor, inside the block. But the collection would have to be defined, as well as declared, outside the block. That is, it would have to be a type you can use in plain SQL, not something defined in PL/SQL. At the moment you're using a PL/SQL type that is defined within the block, and a variable that is declared within the block too - so it's out of scope to the client, and wouldn't be a valid type outside it either. (It also doesn't need to be initialised, but that's a minor issue).
Dpending on how it will really be consumed, one option is to use a ref cursor, and you can declare and display that through SQL*Plus or SQL Developer with the variable and print commands. For example:
variable rc sys_refcursor
begin
open :rc for ( select ... /* your cursor statement */ );
end;
/
print rc
You can do something similar from a client application, e.g. have a function returning a ref cursor or a procedure with an out parameter that is a ref cursor, and bind that from the application. Then iterate over the ref cursor as a result set. But the details depend on the language your application is using.
Another option is to have a pipelined function that returns a table type - again defined at SQL level (with create type) not in PL/SQL - which might consume fewer resources than a collection that's returned in one go.
But I'd have to question why you're doing this. You said "digging down for later batches takes considerably more time", which sounds like you're using a paging mechanism in your query, generating a row number and then picking out a range of 100 within that. If your client/application wants to get all the rows then it would be simpler to have a single query execution but fetch the result set in batches.
Unfortunately without any information about the application this is just speculation...
I studied this excellent paper on optimizing pagination:
http://www.inf.unideb.hu/~gabora/pagination/article/Gabor_Andras_pagination_article.pdf
I used technique 6 mainly. It describes how to limit query to fetch page x and onward. For added improvement, you can limit it further to fetch page x alone. If used right, it can bring a performance improvement by a factor of 1000.
Instead of returning custom table rows (which is very hard, if not impossible to interface with Java), I eneded up opening a sys_refcursor in my pl/sql which can be interfaced such as:
OracleCallableStatement stmt = (OracleCallableStatement) connection.prepareCall(sql);
stmt.registerOutParameter(someIndex, OracleTypes.CURSOR);
stmt.execute();
resultSet = stmt.getCursor(idx);
I'm writting some stored functions in Oracle. One of these is a really basic function who take a string as parameter and return an another string. Here is my function:
CREATE OR REPLACE
FUNCTION get_mail_custcode (
custcodeParam IN customer_table.custcode%TYPE)
RETURN VARCHAR2
IS
mail_rc contact_table.email%TYPE;
BEGIN
SELECT cc.email
INTO mail_rc
FROM contact_table cc, customer_table cu
WHERE cu.customer_id = cc.customer_id
AND cu.custcode like custcodeParam ;
RETURN mail_rc ;
END;
So it's not working.. The function seems to work well but is executed without any end.. The function is working time and time, I manually cancel the operation after 2 or 3 minutes (this query give normally instant result).
After writing the query again and again I finally (and randomly) change the cu.custcode like custcodeParam into a cu.custcode = custcodeParam and it is working!!
So my question is why? Why I can't use a like comparator in a stored function? Why this makes no error but the function run indefinitly.
Thanks.
Cursors are all treated identically in Oracle. A query in a function will be treated exactly the same as a query you enter manually through SQL*Plus.
However, what may differ in your example is how Oracle works with variables. The following two queries are fundamentally different to the optimizer:
SELECT * FROM tab WHERE code LIKE 'FOO%';
and
variable v_code VARCHAR2(4)
EXEC :v_code := 'FOO%';
SELECT * FROM tab WHERE code LIKE :v_code;
In the first case the optimizer looks at the constant FOO% and can instantly tell that an index on code is perfectly suited to retrieve the rows rapidly via an index RANGE SCAN.
In the second case, the optimizer has to consider that :V_CODE is not constant. The purpose of the optimizer is to determine a plan for a query that will be shared by successive executions of the same query (because computing a plan is expensive).
The behaviour of the optimizer will depend upon your version of Oracle:
In old Oracle versions (9i and before), the value of the variable was ignored to build the plan. In effect Oracle had to build a plan that would be efficiently, whatever value was passed to it. In your case this likely would result in a full scan because Oracle had to take the least risky option and consider that FOO% was as likely a value as %FOO (the latter can't be accessed efficiently via an index range scan).
In 10g Oracle introduced bind peeking: now the optimizer can access the value of the variable and produce a suitable plan. The main problem is that in most cases a query can only have one plan, which means that the value of the first variable ever passed to the function will force the execution plan for all further executions. If the first value to be passed is %FOO, a FULL SCAN will likely be chosen.
In 11g, Oracle has "intelligent cursor sharing": a single query can share more than one plan, in the example above the value FOO% would use a RANGE SCAN while %FOO would probably use a FULL SCAN.
What version of Oracle are you using?
update
In 10g and before if this function is used often without wildcards you should rewrite it to acknowledge the optimizer behaviour:
BEGIN
IF instr(custcodeParam, '%') > 0 OR instr(custcodeParam, '_') > 0 THEN
SELECT cc.email
INTO mail_rc
FROM contact_table cc, customer_table cu
WHERE cu.customer_id = cc.customer_id
AND cu.custcode LIKE custcodeParam;
ELSE
SELECT cc.email
INTO mail_rc
FROM contact_table cc, customer_table cu
WHERE cu.customer_id = cc.customer_id
AND cu.custcode = custcodeParam;
END IF;
RETURN mail_rc;
END;
I am having some trouble with a procedure; when run for “big” sets (800+ parents, 1300+ children), it is very slow (30 - 60 secs).
Basic idea is to fetch all parent records (and their respective children) fitting a certain search criteria, along with 3 additional pieces of information that will have to be computed.
My approach to the problem was
to create a custom record type with additional fields for the computed values.
A reference to this record type can then be passed around to each function, controlled by a main processing function.
As a value is computed for each parent record, tack it onto the record.
Each procedure GET_PARENT_RECORDS and GET_CHILD_RECORDS are called once per search, and each computing functions are run N times (where N is the number of parent and/or child records).
Question 1: Is this the right approach? (weakly typed cursors, pipelined functions) If not, then how should I have approached the problem, assuming I can have a re-do?
Question 2: Barring a complete rewrite, is there anything obvious that can be improved in the code provided?
Question 3: Or could something else be wrong, as I notice that the same slow query returned in 20 secs when I've run the procedures a few times?
Package definition
create or replace
PACKAGE THIS_PKG AS
Type parentCursor IS REF CURSOR;
Type childCursor IS REF CURSOR;
Type ParentRecordType IS RECORD (
other_columns,
Extra_column_A,
Extra_column_B,
Extra_column_C,
Row_num);
--associative array
TYPE ParentArray IS TABLE OF ParentRecordType;
FUNCTION processParents(
p IN THIS_PKG. parentCursor
) RETURN ParentArray
PIPELINED
;
FUNCTION countSomething(some params…)
RETURN INT;
FUNCTION checkCondX (SomeParent IN ParentRecordType)
RETURN VARCHAR2;
FUNCTION checkCondY (SomeParent IN ParentRecordType)
RETURN VARCHAR2;
PROCEDURE GET_PARENT_RECORDS( other_parameters, Parents OUT THIS_PKG.parentCursor);
PROCEDURE GET_CHILD_RECORDS( other_parameters, Children OUT THIS_PKG.childCursor);
END THIS_PKG;
Package Body
-- omitted
FUNCTION processParents(
p IN THIS_PKG.parentCursor
) RETURN ParentArray
PIPELINED
IS
out_rec ParentArray;
someParent ParentRecordType;
BEGIN
LOOP
FETCH p BULK COLLECT INTO out_rec LIMIT 100;
FOR i IN 1 .. out_rec.COUNT
LOOP
out_rec(i).extra_column_A := countSomething (out_rec(i).field1, out_rec(i).field2);
out_rec(i).extra_column_B := checkCondX(out_rec(i));
out_rec(i).extra_column_C := checkCondY(out_rec(i));
pipe row(out_rec(i));
END LOOP;
EXIT WHEN p%NOTFOUND;
END LOOP;
RETURN;
END processParents;
PROCEDURE GET_PARENT_RECORDS(
some_columns,
Parents OUT THIS_PKG. parentCursor) IS
BEGIN
OPEN Parents FOR
SELECT *
FROM TABLE(processParents (CURSOR(
SELECT *
FROM (
--some select statement with quite a few where clause
--to simulate dynamic search (from pre-canned search options)
)
))) abc
WHERE abc.extra_column_C like '%xyz%' --(xyz is a user given value)
;
END GET_PARENT_RECORDS;
Update
Did some exploring yesterday and came across the Quest Batch SQL Optimizer (from Toad). I plugged in the package and here's what I got.
Batch Optimizer results
Complex query
Problematic query
What is happening in the row processing section? A lot of time may be spent in these countSomething, checkCondX/Y functions. Are they also making SQL calls? I'd check the performance of the table function without the additional predicates first. It may be better to simply create a query which does this all in SQL rather than functions - if you can do this it will be very much quicker than calling out to a function for every row.
out_rec(i).extra_column_A := countSomething (out_rec(i).field1, out_rec(i).field2);
out_rec(i).extra_column_B := checkCondX(out_rec(i));
out_rec(i).extra_column_C := checkCondY(out_rec(i));
Also the explain plan you provided is interesting as the optimiser believes there is only 1 row being returned from all tables (cardinality 1). If this isn't the case then the query plan will not be optimal. It may be necessary to gather statistics, use dynamic sampling or cardinality hints on the table function.
Finally, look into DBMS_SQLTUNE.REPORT_SQL_MONITOR which provides a detailed report on your sql. Unless the query is dynamically identified as needing monitoring your need to add the /*+ MONITOR */ hint. This provides more details like number of rows returned, execution count and other interesting tidbits not available in an explain plan.
SELECT /*+ MONITOR */
FROM slow_query;
-- then run sqltune to get a report
SELECT *
FROM TABLE(DBMS_SQLTUNE.REPORT_SQL_MONITOR());
Quest Batch SQL Optimizer (from Toad) or any other tool will be unable to help you taking into consideration that they don't understand what you do inside the functions. The problem is in "FETCH p BULK COLLECT INTO out_rec LIMIT 100;". The quality of the query that is passed into p actually defines the final execution plan and run times. Pipelining is not the reason for slowness. When you run your procedure several times, Oracle uses cached data. My best advice is: use Java instead of PL/SQL for this particular purpose, it will be more simpler for understanding.
I have a statement such as below but its padded out to do 1000 calls at a time. Anything over that throws a PLS-123 error Program Too Large Diana Nodes
begin
sp_myprocedure(....)
sp_myprocedure(....)
sp_myprocedure(....)
sp_myprocedure(....)
end
We are moving to 11g and I was wondering if this limitation could be increased to 2000 for example.
Thanks
"I have a statement such as below but its padded out to do 1000 calls
at a time"
This is a very bad programming strategy. Writing the same thing multiple times is a code smell. Anytime we find ourselves programming with cut'n'paste and then a bit of editing is a time when we should stop and ask ourselves, 'hmmm is there a better way to do this?'
"The parameters are different for each stored procedure call"
Yes, but the parameters have to come from somewhere. Presumably at the moment you are hard-coding them one thousand times. Yuck.
A better solution would be to store them in a table. Then you could write a simple loop. Like this:
for prec in ( select p1, p2 from my_parameters
order by id -- if ordering is important
)
loop
sp_myprocedure(prec.p1, prec.p2);
end loop;
Because you are storing the parameters in a table you can have as many calls to that proc as you like, and you are not bound by the Diana node limit.
True you will have to move your parameter values to a table, but it is not harder to maintain data in a table than it is to maintain hardcoded values in source code.
If you're just moving from 10g then I don't believe the limit has changed. So, if you're having problems now then you'll have them again in 11g. Take a look at this Ask Tom article. A general suggestion is to put your procedure in a package. Or, break it down into smaller blocks. If you're only getting the error when running the block which calls the procedure 1000 times and in the procedure on its own then I suggest you try as APC says and loop through it instead as this should reduce the number of nodes.