I have a performance problem.
First PL/SQL (most time never ends and OS database process is always over 90%):
DECLARE
myId nvarchar2(10) := '0;WF21izb0';
BEGIN
insert into MY_TABLE (select * from MY_VIEW where ID = myId);
END;
Second PL/SQL (ends with successfull result in 50s):
BEGIN
insert into MY_TABLE (select * from MY_VIEW where ID = '0;WF21izb0');
END;
select count(*) from MY_VIEW
is also a not ending call, there are a lot of table joins behind this view.
select count(*) from MY_VIEW where ID = '0;WF21izb0'
ends in 50s with count=60000.
Can somebody explain me the reason why my first PL/SQL is not finishing after 50s? What is the difference between using static string and declared parameter?
It boils down to what the DB engine knows about your data and your query, when preparing the query execution plan.
When a literal is placed in your query, it is a part of your query, so is known to the engine responsible for preparing the plan. It can take that literal value into account and decide on an execution plan, that is suitable, e.g. based on the DB data statistics (e.g. that this value is rare).
When you are using a PL/SQL variable, the actual query, for which the plan is determined, is different. It's something like:
insert into MY_TABLE (select * from MY_VIEW where ID = :param)
As you can see, the DB engine has now no information on the value, which will be used, when the query gets executed. So the best plan for such a scenario, is to prepare something, which is averagely good for most of the probable values (i.e. see what values in the DB will match this place most often, i.e. values that are prevalent).
If your data is unbalanced, and the '0;WF21izb0' value is rare (or even non-existent) in your data, a selective index may be used to narrow down, what needs to be processed, relatively soon in the critical parts of the execution plan. This plan will however backfire, when you'll use a value, which is all over the place - use of the index will be counter-productive. A better plan for such case may be a full table scan. Possibly the same one, which is used when executing select count(*) from MY_VIEW.
If you are faced with a scenario, where you do not know the filtering value upfront, you'll have to analyze the view code, and try to adjust it so it can be effectively used also for less "selective" values. You could try applying some optimizer hints to the query. You could also resign from using a view, and try your luck with a tabular function, where you can push your filtering predicates to the exact spots of the query, where they can be used most effectively.
Edit:
All in all, follow the advises from the question comments, and examine your execution plans and execution profile data. You should be able to find the culprit. From there it may not be obvious, what the solution is, but still, you know your data and relations much better than us.
I was checking some traces, but after reading the comment of APC and the answer of Hilarion i end up in this solution:
declare
sql_stmt VARCHAR2(200);
id VARCHAR2(10) := '0;WF21izb0';
BEGIN
sql_stmt := 'insert into MY_TABLE (select * from MY_VIEW where ID = :1)';
EXECUTE IMMEDIATE sql_stmt using id;
END;
This is done in 50s, and id can be now a function/procedure parameter.
Thanks for the comments.
Related
I have a table that is filled with many data validation queries. For example a row:
SELECT end_time - start_time
FROM mt_process_status
WHERE process_id = <PROCESS_ID> AND ref_date = <REF_DATE>
I have to execute all these SQL statements filling the values within '<>' with run time values and check if the performance of the process has not changed.
Can this be done with a stored procedure? I want to understand what the solution would look like. Any links to documentation of this sort of thing, anything to guide me in the right direction.
Somewhere there's a bunch of analysts in your organization telling each other, "We've done the difficult stuff, we've defined the queries. All the database has to do is execute them, how hard can that be?" Answer: very hard.
Let's take the query you posted:
select end_time - start_time from mt_process_status where process_id = <PROCESS_ID> AND ref_date = <REF_DATE>
It's easy enough to use replace(the_str, '<PROCESS_ID>', 1234) to substitute a value. But for ref_date that's presumably a date, so it needs to be replace(the_str, '<REF_DATE>', 'date ''2017-01-01'''). Starting to get icky, and that's just handling literals. It will be even ickier when the substitution values are passed as parameters.
Of course I've made an assumption that PROCESS_ID is numeric. Maybe it isn't. Who can tell? Is there a data dictionary where these details are defined?
It would be easier if the query was defined with dynamic SQL placeholders:
select end_time - start_time from mt_process_status where process_id = :PROCESS_ID AND ref_date = :REF_DATE
Then you could forget about the replace and simply run
execute immediate the_str
using 1234, date '2017-01-01'
into whatever;
But you still need to know how many placeholders there are, in what order they occur and what datatype they are. It may feel like this is soft-coded and configurable but there is still a really hard dependency between the query and the program which calls it.
Plus you have lost the ability to do impact analysis. What queries will be affected when you change mt_process_status? Who can tell?
CREATE OR REPLACE PROCEDURE MY_PROC(g_id)
AS
l_query varchar2(1000);
l_duration number;
BEGIN
-- Get the query from the table
select query into l_query from my_table where id=g_id;
-- l_query contains "select end_time - start_time from mt_process_status where process_id = <PROCESS_ID> and ref_date = <REF_DATE>"
-- Put value to replace the tags (take care of code injection...)
l_query := REPLACE(l_query,'<PROCESS_ID>', something);
l_query := REPLACE(l_query,'<REF_DATE>', something_else);
EXECUTE IMMEDIATE lquery RETURN INTO l_duration;
-- Do what you have to do...
-- If you do a function, then you can:
-- RETURN l_duration;
END;
/
Your question is not clear. This is the answer to your question. Use variables in a procedure.
regards.
I would put the reference data in another table and do it as a join
First, I want to make it clear that the question is not about the materialized views feature.
Suppose, I have a table function that returns a pre-defined set of columns.
When a function call is submitted as
SELECT col1, col2, col3
FROM TABLE(my_tfn(:p1))
WHERE col4 = 'X';
I can evaluate the parameter and choose what queries to execute.
I can either open one of the pre-defined cursors, or I can assemble my query dynamically.
What if instead of evaluating the parameter I want to evaluate the text of the requesting query?
For example, if my function returns 20 columns but the query is only requesting 4,
I can assign NULLs to remaining 16 clumns of the return type, and execute fewer joins.
Or I can push the filter down to my dynamic query.
Is there a way to make this happen?
More generally, is there a way to look at the requesting query before exuting the function?
There is no robust way to identify the SQL that called a PL/SQL object.
Below is a not-so-robust way to identify the calling SQL. I've used code like this before, but only in special circumstances where I knew that the PL/SQL would never run concurrently.
This seems like it should be so simple. The data dictionary tracks all sessions and running SQL. You can find the current session with sys_context('userenv', 'sid'), match that to GV$SESSION, and then get either SQL_ID and PREV_SQL_ID. But neither of those contain the calling SQL. There's even a CURRENT_SQL in SYS_CONTEXT, but it's only for fine-grained auditing.
Instead, the calling SQL must be found by a string search. Using a unique name for the PL/SQL object will help filter out unrelated statements. To prevent re-running for old statements, the SQL must be individually purged from the shared pool as soon as it is found. This could lead to race conditions so this approach will only work if it's never called concurrently.
--Create simple test type for function.
create or replace type clob_table is table of clob;
--Table function that returns the SQL that called it.
--This requires elevated privileges to run.
--To simplify the code, run this as SYS:
-- "grant execute on sys.dbms_shared_pool to your_user;"
--(If you don't want to do that, convert this to invoker's rights and use dynamic SQL.)
create or replace function my_tfn return clob_table is
v_my_type clob_table;
type string_table is table of varchar2(4000);
v_addresses string_table;
v_hash_values string_table;
begin
--Get calling SQL based on the SQL text.
select sql_fulltext, address, hash_value
bulk collect into v_my_type, v_addresses, v_hash_values
from gv$sql
--Make sure there is something unique in the query.
where sql_fulltext like '%my_tfn%'
--But don't include this query!
--(Normally creating a quine is a challenge, but in V$SQL it's more of
-- a challenge to avoid quines.)
and sql_fulltext not like '%quine%';
--Flush the SQL statements immediately, so they won't show up in next run.
for i in 1 .. v_addresses.count loop
sys.dbms_shared_pool.purge(v_addresses(i)||', '||v_hash_values(i), 'C');
end loop;
--Return the SQL statement(s).
return v_my_type;
end;
/
Now queries like these will return themselves, demonstrating that the PL/SQL code was reading the SQL that called it:
SELECT * FROM TABLE(my_tfn) where 1=1;
SELECT * FROM TABLE(my_tfn) where 2=2;
But even if you go through all this trouble - what are you going to do with the results? Parsing SQL is insanely difficult unless you can ensure that everyone always follows strict syntax rules.
I have a query, something like
select * from table1 where :iparam is null or iparam = field1;
On field1 there is a non-unique index, but oracle (11g) don't want to use it. As i understand, it optimize query not in run-time, but at compiling. I'm using such query in stored procedures. I wonder, if there is a way, to tell oracle, to use an indexes?
I know about "hints" but i would like to use something on all project, like some optimizer argument, to optimize queries in run-time.
where :iparam is null or :iparam = field1;
Oracle has no way of knowing in advance if you will pass NULL value for :iparam
If you do, full scan is the best way to access data. If you don't, index might be better. You can split this statement in two parts using IF, then there will be no ambiguity.
If you have a lot of fields to compare, dynamic sql migh be a better way.
IF :param1 IS NOT NULL THEN
v_sql := v_sql||' and field1 = :param1';
ELSE
v_sql := v_sql||' and nvl(:param1,1) = 1';
END IF;
ELSE part is for easyer usage of USING.
It is not true that the execution plan is determined at the time of package compilation. It will be determined just before the query is actually executed.
How the optimizer decides to run the query depends on many things. Foremost the availability of statistics. These give the optimizer something to go on. How many records are in the table. How many different values are in the index.
Here is an article that goes into more detail:
http://joco.name/2014/01/05/why-wouldnt-oracle-use-a-perfectly-valid-index/
First off, my background is in SQL Server. Using CTEs (Common Table Expressions) is a breeze and converting it to a stored procedure with variables doesn't require any changes to the structure of the SQL other than replacing entered values with variable names.
In Oracle PL/SQL however, it is a completely different matter. My CTEs work fine as straight SQL, but once I try to wrap them as PL/SQL I run into a host of issues. From my understanding, a SELECT now needs an INTO which will only hold the results of a single record. However, I am wanting the entire recordset of multiple values.
My apologies if I am missing the obvious here. I'm thinking that 99% of my problem is the paradigm shift I need to make.
Given the following example:
NOTE: I am greatly over simplifying the SQL here. I do know the below example can be done in a single SQL statement. The actual SQL is much more complex. It's the fundamentals I am looking for here.
WITH A as (SELECT * FROM EMPLOYEES WHERE DEPARTMENT = 200),
B as (SELECT * FROM A WHERE EMPLOYEE_START_DATE > date '2014-02-01'),
C as (SELECT * FROM B WHERE EMPLOYEE_TYPE = 'SALARY')
SELECT 'COUNTS' as Total,
(SELECT COUNT(*) FROM A) as 'DEPT_TOTAL',
(SELECT COUNT(*) FROM B) as 'NEW_EMPLOYEES',
(SELECT COUNT(*) FROM C) as 'NEW_SALARIED'
FROM A
WHERE rowcount = 1;
Now if I want to make this into PL/SQL with variables that are passed in or predefined at the top, it's not a simple matter of declaring the variables, popping values into them, and changing my hard-coded values into variables and running it. NOTE: I do know that I can simply change the hard-coded values to variables like :Department, :StartDate, and :Type, but again, I am oversimplifying the example.
There are three issues I am facing here that I am trying to wrap my head around:
1) What would be the best way to rewrite this using PL/SQL with declared variables? The CTEs now have to go INTO something. But then I am dealing with one row at a time as opposed to the entire table. So CTE 'A' is a single row at a time, and CTE B will only see the single row as opposed to all of the data results of A, etc. I do know that I will most likely have to use CURSORS to traverse the records, which somehow seems to over complicate this.
2) The output now has to use DBMS_OUTPUT. For multiple records, I will have to use a CURSOR with FETCH (or a FOR...LOOP). Yes?
3) Is there going to a big performance issue with this vs. straight SQL in regards to speed and resources used?
Thanks in advance and again, my apologies if I am missing something really obvious here!
First, this has nothing to do with CTEs. This behavior would be the same with a simple select * from table query. The difference is that with T-SQL, the query goes into an implicit cursor which is returned to the caller. When executing the SP from Management Studio this is convenient. The result set appears in the data window as if we had executed the query directly. But this is actually non-standard behavior. Oracle has the more standard behavior which might be stated as "the result set of any query that isn't directed into a cursor must be directed to variables." When directed into variables, then the query must return only one row.
To duplicate the behavior of T-SQL, you just have to explicitly declare and return the cursor. Then the calling code fetches from the cursor the entire result set but one row at a time. You don't get the convenience of Sql Developer or PL/SQL Developer diverting the result set to the data display window, but you can't have everything.
However, as we don't generally write SPs just to be called from the IDE, it is easier to work with Oracle's explicit cursors than SQL Server's implicit ones. Just google "oracle return ref cursor to caller" to get a whole lot of good material.
Simplest way is to wrap it into an implicit for loop
begin
for i in (select object_id, object_name
from user_objects
where rownum = 1) loop
-- Do something with the resultset
dbms_output.put_line (i.object_id || ' ' || i.object_name);
end loop;
end;
Single row query without the need to predefine the variables.
I'm writting some stored functions in Oracle. One of these is a really basic function who take a string as parameter and return an another string. Here is my function:
CREATE OR REPLACE
FUNCTION get_mail_custcode (
custcodeParam IN customer_table.custcode%TYPE)
RETURN VARCHAR2
IS
mail_rc contact_table.email%TYPE;
BEGIN
SELECT cc.email
INTO mail_rc
FROM contact_table cc, customer_table cu
WHERE cu.customer_id = cc.customer_id
AND cu.custcode like custcodeParam ;
RETURN mail_rc ;
END;
So it's not working.. The function seems to work well but is executed without any end.. The function is working time and time, I manually cancel the operation after 2 or 3 minutes (this query give normally instant result).
After writing the query again and again I finally (and randomly) change the cu.custcode like custcodeParam into a cu.custcode = custcodeParam and it is working!!
So my question is why? Why I can't use a like comparator in a stored function? Why this makes no error but the function run indefinitly.
Thanks.
Cursors are all treated identically in Oracle. A query in a function will be treated exactly the same as a query you enter manually through SQL*Plus.
However, what may differ in your example is how Oracle works with variables. The following two queries are fundamentally different to the optimizer:
SELECT * FROM tab WHERE code LIKE 'FOO%';
and
variable v_code VARCHAR2(4)
EXEC :v_code := 'FOO%';
SELECT * FROM tab WHERE code LIKE :v_code;
In the first case the optimizer looks at the constant FOO% and can instantly tell that an index on code is perfectly suited to retrieve the rows rapidly via an index RANGE SCAN.
In the second case, the optimizer has to consider that :V_CODE is not constant. The purpose of the optimizer is to determine a plan for a query that will be shared by successive executions of the same query (because computing a plan is expensive).
The behaviour of the optimizer will depend upon your version of Oracle:
In old Oracle versions (9i and before), the value of the variable was ignored to build the plan. In effect Oracle had to build a plan that would be efficiently, whatever value was passed to it. In your case this likely would result in a full scan because Oracle had to take the least risky option and consider that FOO% was as likely a value as %FOO (the latter can't be accessed efficiently via an index range scan).
In 10g Oracle introduced bind peeking: now the optimizer can access the value of the variable and produce a suitable plan. The main problem is that in most cases a query can only have one plan, which means that the value of the first variable ever passed to the function will force the execution plan for all further executions. If the first value to be passed is %FOO, a FULL SCAN will likely be chosen.
In 11g, Oracle has "intelligent cursor sharing": a single query can share more than one plan, in the example above the value FOO% would use a RANGE SCAN while %FOO would probably use a FULL SCAN.
What version of Oracle are you using?
update
In 10g and before if this function is used often without wildcards you should rewrite it to acknowledge the optimizer behaviour:
BEGIN
IF instr(custcodeParam, '%') > 0 OR instr(custcodeParam, '_') > 0 THEN
SELECT cc.email
INTO mail_rc
FROM contact_table cc, customer_table cu
WHERE cu.customer_id = cc.customer_id
AND cu.custcode LIKE custcodeParam;
ELSE
SELECT cc.email
INTO mail_rc
FROM contact_table cc, customer_table cu
WHERE cu.customer_id = cc.customer_id
AND cu.custcode = custcodeParam;
END IF;
RETURN mail_rc;
END;