I have a query, something like
select * from table1 where :iparam is null or iparam = field1;
On field1 there is a non-unique index, but oracle (11g) don't want to use it. As i understand, it optimize query not in run-time, but at compiling. I'm using such query in stored procedures. I wonder, if there is a way, to tell oracle, to use an indexes?
I know about "hints" but i would like to use something on all project, like some optimizer argument, to optimize queries in run-time.
where :iparam is null or :iparam = field1;
Oracle has no way of knowing in advance if you will pass NULL value for :iparam
If you do, full scan is the best way to access data. If you don't, index might be better. You can split this statement in two parts using IF, then there will be no ambiguity.
If you have a lot of fields to compare, dynamic sql migh be a better way.
IF :param1 IS NOT NULL THEN
v_sql := v_sql||' and field1 = :param1';
ELSE
v_sql := v_sql||' and nvl(:param1,1) = 1';
END IF;
ELSE part is for easyer usage of USING.
It is not true that the execution plan is determined at the time of package compilation. It will be determined just before the query is actually executed.
How the optimizer decides to run the query depends on many things. Foremost the availability of statistics. These give the optimizer something to go on. How many records are in the table. How many different values are in the index.
Here is an article that goes into more detail:
http://joco.name/2014/01/05/why-wouldnt-oracle-use-a-perfectly-valid-index/
Related
I have a performance problem.
First PL/SQL (most time never ends and OS database process is always over 90%):
DECLARE
myId nvarchar2(10) := '0;WF21izb0';
BEGIN
insert into MY_TABLE (select * from MY_VIEW where ID = myId);
END;
Second PL/SQL (ends with successfull result in 50s):
BEGIN
insert into MY_TABLE (select * from MY_VIEW where ID = '0;WF21izb0');
END;
select count(*) from MY_VIEW
is also a not ending call, there are a lot of table joins behind this view.
select count(*) from MY_VIEW where ID = '0;WF21izb0'
ends in 50s with count=60000.
Can somebody explain me the reason why my first PL/SQL is not finishing after 50s? What is the difference between using static string and declared parameter?
It boils down to what the DB engine knows about your data and your query, when preparing the query execution plan.
When a literal is placed in your query, it is a part of your query, so is known to the engine responsible for preparing the plan. It can take that literal value into account and decide on an execution plan, that is suitable, e.g. based on the DB data statistics (e.g. that this value is rare).
When you are using a PL/SQL variable, the actual query, for which the plan is determined, is different. It's something like:
insert into MY_TABLE (select * from MY_VIEW where ID = :param)
As you can see, the DB engine has now no information on the value, which will be used, when the query gets executed. So the best plan for such a scenario, is to prepare something, which is averagely good for most of the probable values (i.e. see what values in the DB will match this place most often, i.e. values that are prevalent).
If your data is unbalanced, and the '0;WF21izb0' value is rare (or even non-existent) in your data, a selective index may be used to narrow down, what needs to be processed, relatively soon in the critical parts of the execution plan. This plan will however backfire, when you'll use a value, which is all over the place - use of the index will be counter-productive. A better plan for such case may be a full table scan. Possibly the same one, which is used when executing select count(*) from MY_VIEW.
If you are faced with a scenario, where you do not know the filtering value upfront, you'll have to analyze the view code, and try to adjust it so it can be effectively used also for less "selective" values. You could try applying some optimizer hints to the query. You could also resign from using a view, and try your luck with a tabular function, where you can push your filtering predicates to the exact spots of the query, where they can be used most effectively.
Edit:
All in all, follow the advises from the question comments, and examine your execution plans and execution profile data. You should be able to find the culprit. From there it may not be obvious, what the solution is, but still, you know your data and relations much better than us.
I was checking some traces, but after reading the comment of APC and the answer of Hilarion i end up in this solution:
declare
sql_stmt VARCHAR2(200);
id VARCHAR2(10) := '0;WF21izb0';
BEGIN
sql_stmt := 'insert into MY_TABLE (select * from MY_VIEW where ID = :1)';
EXECUTE IMMEDIATE sql_stmt using id;
END;
This is done in 50s, and id can be now a function/procedure parameter.
Thanks for the comments.
Is there any possibility of avoiding full scans, when dealing with incoming NULL query parameters in a stored procedure? Suppose I have 4 parameters, that the user sends from a form and tries to look for an exact match in the table, like this:
SELECT *
FROM table1 t1
WHERE ((:qParam1 is null) OR (t1.col1 = :qParam1)) AND
((:qParam2 is null) OR (t1.col2 = :qParam2)) AND
((:qParam3 is null) OR (t1.col3 = :qParam3)) AND
((:qParam4 is null) OR (t1.col4 = :qParam4));
So when this part of the procedure executes, because of NULL check, it will do a FTS, since the procedure has already been compiled and the execution plan determined. It would need 2^4 different queries to be written inside the procedure in order to always use the most efficient plan considering the incoming query parameters (and considerably more if the input parameter number increases). My question is - is there any way, except for dynamic SQL, to avoid the FTS in these type of queries?
Maybe not. Oracle does not store nulls in the index so it cannot ever use an index when there's a possibility of null in the predicate. If your columns are nullable then no. Having said that, there's a good chance it's the best plan anyway - your query is so vague (ok - flexible) that it would be hard pressed to build a single, useful plan anyway. Oracle is quite smart with flexible plans but there's not really much to go on here.
If you do have nullable columns and an index you might be able to bodge it with
t1.col1 = :qParam2 and t1.col1 is not null
in case it's not smart enough to work that out for itself.
First off, my background is in SQL Server. Using CTEs (Common Table Expressions) is a breeze and converting it to a stored procedure with variables doesn't require any changes to the structure of the SQL other than replacing entered values with variable names.
In Oracle PL/SQL however, it is a completely different matter. My CTEs work fine as straight SQL, but once I try to wrap them as PL/SQL I run into a host of issues. From my understanding, a SELECT now needs an INTO which will only hold the results of a single record. However, I am wanting the entire recordset of multiple values.
My apologies if I am missing the obvious here. I'm thinking that 99% of my problem is the paradigm shift I need to make.
Given the following example:
NOTE: I am greatly over simplifying the SQL here. I do know the below example can be done in a single SQL statement. The actual SQL is much more complex. It's the fundamentals I am looking for here.
WITH A as (SELECT * FROM EMPLOYEES WHERE DEPARTMENT = 200),
B as (SELECT * FROM A WHERE EMPLOYEE_START_DATE > date '2014-02-01'),
C as (SELECT * FROM B WHERE EMPLOYEE_TYPE = 'SALARY')
SELECT 'COUNTS' as Total,
(SELECT COUNT(*) FROM A) as 'DEPT_TOTAL',
(SELECT COUNT(*) FROM B) as 'NEW_EMPLOYEES',
(SELECT COUNT(*) FROM C) as 'NEW_SALARIED'
FROM A
WHERE rowcount = 1;
Now if I want to make this into PL/SQL with variables that are passed in or predefined at the top, it's not a simple matter of declaring the variables, popping values into them, and changing my hard-coded values into variables and running it. NOTE: I do know that I can simply change the hard-coded values to variables like :Department, :StartDate, and :Type, but again, I am oversimplifying the example.
There are three issues I am facing here that I am trying to wrap my head around:
1) What would be the best way to rewrite this using PL/SQL with declared variables? The CTEs now have to go INTO something. But then I am dealing with one row at a time as opposed to the entire table. So CTE 'A' is a single row at a time, and CTE B will only see the single row as opposed to all of the data results of A, etc. I do know that I will most likely have to use CURSORS to traverse the records, which somehow seems to over complicate this.
2) The output now has to use DBMS_OUTPUT. For multiple records, I will have to use a CURSOR with FETCH (or a FOR...LOOP). Yes?
3) Is there going to a big performance issue with this vs. straight SQL in regards to speed and resources used?
Thanks in advance and again, my apologies if I am missing something really obvious here!
First, this has nothing to do with CTEs. This behavior would be the same with a simple select * from table query. The difference is that with T-SQL, the query goes into an implicit cursor which is returned to the caller. When executing the SP from Management Studio this is convenient. The result set appears in the data window as if we had executed the query directly. But this is actually non-standard behavior. Oracle has the more standard behavior which might be stated as "the result set of any query that isn't directed into a cursor must be directed to variables." When directed into variables, then the query must return only one row.
To duplicate the behavior of T-SQL, you just have to explicitly declare and return the cursor. Then the calling code fetches from the cursor the entire result set but one row at a time. You don't get the convenience of Sql Developer or PL/SQL Developer diverting the result set to the data display window, but you can't have everything.
However, as we don't generally write SPs just to be called from the IDE, it is easier to work with Oracle's explicit cursors than SQL Server's implicit ones. Just google "oracle return ref cursor to caller" to get a whole lot of good material.
Simplest way is to wrap it into an implicit for loop
begin
for i in (select object_id, object_name
from user_objects
where rownum = 1) loop
-- Do something with the resultset
dbms_output.put_line (i.object_id || ' ' || i.object_name);
end loop;
end;
Single row query without the need to predefine the variables.
I'm trying to understand how no_index actually speeds up a query and haven't been able to find documentation online to explain it.
For example I have this query that ran extremely slow
select *
from <tablename>
where field1_ like '%someGenericString%' and
field1_ <> 'someSpecificString' and
Action_='_someAction_' and
Timestamp_ >= trunc(sysdate - 2)
And one of our DBAs was able to speed it up significantly by doing this
select /*+ NO_INDEX(TAB_000000000019) */ *
from <tablename>
where field1_ like '%someGenericString%' and
field1_ <> 'someSpecificString' and
Action_='_someAction_' and
Timestamp_ >= trunc(sysdate - 2)
And I can't figure out why? I would like to figure out why this works so I can see if I can apply it to another query (this one a join) to speed it up because it's taking even longer to run.
Thanks!
** Update **
Here's what I know about the table in the example.
It's a 'partitioned table'
TAB_000000000019 is the table not a column in it
field1 is indexed
Oracle's optimizer makes judgements on how best to run a query, and to do this it uses a large number of statistics gathered about the tables and indexes. Based on these stats, it decides whether or not to use an index, or to just do a table scan, for example.
Critically, these stats are not automatically up-to-date, because they can be very expensive to gather. In cases where the stats are not up to date, the optimizer can make the "wrong" decision, and perhaps use an index when it would actually be faster to do a table scan.
If this is known by the DBA/developer, they can give hints (which is what NO_INDEX is) to the optimizer, telling it not to use a given index because it's known to slow things down, often due to out-of-date stats.
In your example, TAB_000000000019 will refer to an index or a table (I'm guessing an index, since it looks like an auto-generated name).
It's a bit of a black art, to be honest, but that's the gist of it, as I understand things.
Disclaimer: I'm not a DBA, but I've dabbled in that area.
Per your update: If field1 is the only indexed field, then the original query was likely doing a fast full scan on that index (i.e. reading through every entry in the index and checking against the filter conditions on field1), then using those results to find the rows in the table and filter on the other conditions. The conditions on field1 are such that an index unique scan or range scan (i.e. looking up specific values or ranges of values in the index) would not be possible.
Likely the optimizer chose this path because there are two filter predicates on field1. The optimizer would calculate estimated selectivity for each of these and then multiply them to determine their combined selectivity. But in many cases this will significantly underestimate the number of rows that will match the condition.
The NO_INDEX hint eliminates this option from the optimizer's consideration, so it essentially goes with the plan it thinks is next best -- possibly in this case using partition elimination based on one of the other filter conditions in the query.
Using an index degrades query performance if it results in more disk IO compared to querying the table with an index.
This can be demonstrated with a simple table:
create table tq84_ix_test (
a number(15) primary key,
b varchar2(20),
c number(1)
);
The following block fills 1 Million records into this table. Every 250th record is filled with a rare value in column b while all the others are filled with frequent value:
declare
rows_inserted number := 0;
begin
while rows_inserted < 1000000 loop
if mod(rows_inserted, 250) = 0 then
insert into tq84_ix_test values (
-1 * rows_inserted,
'rare value',
1);
rows_inserted := rows_inserted + 1;
else
begin
insert into tq84_ix_test values (
trunc(dbms_random.value(1, 1e15)),
'frequent value',
trunc(dbms_random.value(0,2))
);
rows_inserted := rows_inserted + 1;
exception when dup_val_on_index then
null;
end;
end if;
end loop;
end;
/
An index is put on the column
create index tq84_index on tq84_ix_test (b);
The same query, but once with index and once without index, differ in performance. Check it out for yourself:
set timing on
select /*+ no_index(tq84_ix_test) */
sum(c)
from
tq84_ix_test
where
b = 'frequent value';
select /*+ index(tq84_ix_test tq84_index) */
sum(c)
from
tq84_ix_test
where
b = 'frequent value';
Why is it? In the case without the index, all database blocks are read, in sequential order. Usually, this is costly and therefore considered bad. In normal situation, with an index, such a "full table scan" can be reduced to reading say 2 to 5 index database blocks plus reading the one database block that contains the record that the index points to. With the example here, it is different altogether: the entire index is read and for (almost) each entry in the index, a database block is read, too. So, not only is the entire table read, but also the index. Note, that this behaviour would differ if c were also in the index because in that case Oracle could choose to get the value of c from the index instead of going the detour to the table.
So, to generalize the issue: if the index does not pick few records then it might be beneficial to not use it.
Something to note about indexes is that they are precomputed values based on the row order and the data in the field. In this specific case you say that field1 is indexed and you are using it in the query as follows:
where field1_ like '%someGenericString%' and
field1_ <> 'someSpecificString'
In the query snippet above the filter is on both a variable piece of data since the percent (%) character cradles the string and then on another specific string. This means that the default Oracle optimization that doesn't use an optimizer hint will try to find the string inside the indexed field first and also find if the data it is a sub-string of the data in the field, then it will check that the data doesn't match another specific string. After the index is checked the other columns are then checked. This is a very slow process if repeated.
The NO_INDEX hint proposed by the DBA removes the optimizer's preference to use an index and will likely allow the optimizer to choose the faster comparisons first and not necessarily force index comparison first and then compare other columns.
The following is slow because it compares the string and its sub-strings:
field1_ like '%someGenericString%'
While the following is faster because it is specific:
field1_ like 'someSpecificString'
So the reason to use the NO_INDEX hint is if you have comparisons on the index that slow things down. If the index field is compared against more specific data then the index comparison is usually faster.
I say usually because when the indexed field contains more redundant data like in the example #Atish mentions above, it will have to go through a long list of comparison negatives before a positive comparison is returned. Hints produce varying results because both the database design and the data in the tables affect how fast a query performs. So in order to apply hints you need to know if the individual comparisons you hint to the optimizer will be faster on your data set. There are no shortcuts in this process. Applying hints should happen after proper SQL queries have been written because hints should be based on the real data.
Check out this hints reference: http://docs.oracle.com/cd/B19306_01/server.102/b14211/hintsref.htm
To add to what Rene' and Dave have said, this is what I have actually observed in a production situation:
If the condition(s) on the indexed field returns too many matches, Oracle is better off doing a Full Table Scan.
We had a report program querying a very large indexed table - the index was on a region code and the query specified the exact region code, so Oracle CBO uses the index.
Unfortunately, one specific region code accounted for 90% of the tables entries.
As long as the report was run for one of the other (minor) region codes, it completed in less than 30 minutes, but for the major region code it took many hours.
Adding a hint to the SQL to force a full table scan solved the problem.
Hope this helps.
I had read somewhere that using a % in front of query like '%someGenericString%' will lead to Oracle ignoring the INDEX on that field. Maybe that explains why the query is running slow.
To optimize SELECT queries, I run them both with and without an index and measure the difference. I run a bunch of different similar queries and try to select different data to make sure that caching doesn't throw off the results. However, on very large tables, indexes take a really long time to create, and I have several different ideas about what indexes would be appropriate.
Is it possible in Oracle (or any other database for that matter) to perform a query but tell the database to not use a certain index when performing the query? Or just turn off the index entirely, but be able to easily switch it back on without having to re-index the entire table? This would make it much easier to test, since I can create all the indexes I'm thinking about all at once, then try my queries using different ones.
Alternatively, is there any better way to go about optimizing queries on large tables and know which indexes would be best to create?
You can set index visibility in 11g -
ALTER INDEX idx1 [ INVISIBLE | VISIBLE ]
this makes it unusable by the optimizer, but oracle still updates the index when data is added or removed. This makes it easy to test performance with the index disabled without having to remove & rebuild the whole index.
See here for the oracle docs on index visibility
You can use the NO_INDEX hint in the queries to ignore the indexes - see docs for further details. The SQL Access Advisor is an Oracle utility that will recommend indexing strategies.
Well you can write the query in such a way that it wont use index(using expression instead of a value)
For example
Select * from foobar where column1 = 'result' --uses index on column1
To avoid using index for a number and varchar
Select * from foobar where column1 + 0 = 5 -- simple expression to disable the index
Select * from foobar where column1 || '' = 'result' --simple expression to disable the index
Or you can just use NVL to disable the index in the query without worrying about the column's data type
Select * from foobar where nvl(column1,column1) = 'result' --i love this way :D
Similarly you can use index hints
like /* Index(E employee_id) */ to use indexes.
P.S. This is all the paraphrased from Dan Tow's Book SQL Tuning. I started reading it a few days back :)