I have a table with column status. It is a string nullable column. I also have an index on this field only. Why is the following query not using the index?
select /*+ index(m IDX_STATUS) */ * from messages m where m.status = :1
Try running the query without named (bind) parameters. Sometimes that makes a big difference.
select * from messages m where m.status = 'P'
It may turn out that you don't even need a hint to trigger index usage.
A possible explanation is that the column contains many equal values, for example 90% of rows has status = 'D' (low-cardinality column). Now we can understand Oracle, why it did not use the index :) It simply doesn't make sense on value 'D', but is reasonable for other values. I would prefer Oracle to consider my hint (I know better), but that seems undoable.
In general there is an extremely useful guide Oracle SQL Tuning Guide: The index is being ignored. Still it does not mention the situation, where omitting bind parameter makes the problem go away. That's why I insisted on asking the question on SO.
Related
Is there any possibility of avoiding full scans, when dealing with incoming NULL query parameters in a stored procedure? Suppose I have 4 parameters, that the user sends from a form and tries to look for an exact match in the table, like this:
SELECT *
FROM table1 t1
WHERE ((:qParam1 is null) OR (t1.col1 = :qParam1)) AND
((:qParam2 is null) OR (t1.col2 = :qParam2)) AND
((:qParam3 is null) OR (t1.col3 = :qParam3)) AND
((:qParam4 is null) OR (t1.col4 = :qParam4));
So when this part of the procedure executes, because of NULL check, it will do a FTS, since the procedure has already been compiled and the execution plan determined. It would need 2^4 different queries to be written inside the procedure in order to always use the most efficient plan considering the incoming query parameters (and considerably more if the input parameter number increases). My question is - is there any way, except for dynamic SQL, to avoid the FTS in these type of queries?
Maybe not. Oracle does not store nulls in the index so it cannot ever use an index when there's a possibility of null in the predicate. If your columns are nullable then no. Having said that, there's a good chance it's the best plan anyway - your query is so vague (ok - flexible) that it would be hard pressed to build a single, useful plan anyway. Oracle is quite smart with flexible plans but there's not really much to go on here.
If you do have nullable columns and an index you might be able to bodge it with
t1.col1 = :qParam2 and t1.col1 is not null
in case it's not smart enough to work that out for itself.
We did refactoring and replaced 2 similar requests with parameterized request
a.isGood = :1
after that request that used this parameter with parameter 'Y' was executed longer that usually (become almost the same with parameter 'N'). We used alter system flush shared_pool command and request for parameter 'Y' has completed fast (as before refactoring) while request with parameter 'N' hangs for a long time.
As you could understand number of lines in data base with parameter 'N' much more then with 'Y'
Oracle 10g
Why it happened?
I assume that you have an index on that column, otherwise the performance would be the same regardless of the Y/N combination. I have seen this happening quite bit on 10g+ due to Oracle's optimizer Bind Peeking combined to histograms on columns with skewed data distribution. The histograms get created automatically when one gathers tables statistics using the parameter method_opt with 'FOR ALL COLUMNS SIZE AUTO' (among other values). Oracle optimizes the query for the value in the bind variables provided in the very first execution of that query. If you run the query with Y the first time, Oracle might want to use an index instead of a full table scan, since Y will return a small quantity of rows. The next time you run the query with N, then Oracle will repeat the first execution plan, which happens to be a poor choice for N, since it will return the vast majority of rows.
The execution plans are cached in the SGA. Once you flush it, you get a brand new execution plan the very first time the query runs again.
My suggestion is:
Obtain the explain plan of both original queries (one with a hardcoded Y and one with a hardcode N). Investigate if the two plans use different indexes or one has a much higher Cost than the other. I have the feeling that one uses a full table scan and the other uses an index. The first one should be faster for N and the second should be faster for Y.
Try to remove the statistics on the table and see if it makes a difference on the query that has the bind variable. Later you need to gather statistics again for the table or other queries on that table might suffer.
You can also gather statistics for that one table using method_opt => FOR ALL COLUMNS SIZE 1. That will keep the statistics without the histograms on any columns of that table.
A bitmap index on this column might fix the issue as well. Indexes on a column that have only two possible values (Y and N) are not exactly very efficient.
If column isGood has 99,000 'N' values and 1,000 'Y' values and you run with the condition isGood = 'Y', then it may be appropriate to use an index to find the results: you are returning 1% of the rows. If you run the query with the condition isGood = 'N', a full table scan would be more appropriate since you are returning most of the table anyway. If you were to use an index for the N condition, you would be doing an extra index lookup for every data item lookup.
Although the general rule is that bind parameters are good, it can be problematic in this kind of instance if really two different plans are required for the query. With the bind parameter scenario:
SELECT * FROM x WHERE isGood = :1
The statement will be parsed and a plan computed and saved in the sql cache. The same plan will be used for both query scenarios which is not desirable. But:
SELECT * FROM x WHERE isGood = 'Y'
SELECT * FROM x WHERE isGood = 'N'
will result in two plans being stored in the sql cache, hopefully each with the appropriate plan for the query. Version 11g avoids this problem with adaptive cursor sharing, which can use different plans for different bind variable values.
You need to look at your plans (EXPLAIN PLAN) to see what is happening in your case. Flush the cache, try one method, examine the plan; try the other, examine the plan. It might give you an idea what is happening in your case. There are a bunch of other topics you might follow up on that may help, for example:
using a hint to force the use of an index
cursor_sharing parameter
histograms on statistics
I'm currently working with a database that has two indexes for a specific table. The index I want has two columns "Name" (varchar2) and "Time" (number). When I write the query
SELECT SOMETHING
FROM MYTABLE
WHERE NAME = 'SOME-NAME'
AND TIME BETWEEN STARTVALUE AND ENDVALUE
(where STARTVALUE and ENDVALUE are numbers) it does not use the index. However if I use the following query instead
SELECT SOMETHING
FROM MYTABLE
WHERE NAME = 'SOME-NAME'
AND TIME BETWEEN MY_FUNC('STARTQUAL') AND MY_FUNC('ENDQUAL')
it does.
The only difference I can think of is that MY_FUNC explicitly returns a value of type NUMBER - is it possible that the query optimizer is confused about the data type for STARTVALUE and ENDVALUE specified explicitly and is refusing to use the index (I saw some similar threads that mentioned a type conflict was the cause).
Note:
The value being returned by MY_FUNC is EXACTLY the same value that I am specifying in the first query.
The index in question is UNDOUBTEDLY (absolutely no question) the correct index to be using and execution times are orders of magnitude faster when it does.
I have even specified a query hint with the first query and it refuses to use the index.
I know there must be something silly / simple that I'm overlooking but I just can't see it.
Thanks in advance for your assistance.
Alternatively, Oracle could be optimizing the queries differently based on whether the query involves literal values or bound values.
SELECT SOMETHING
FROM MYTABLE
WHERE NAME = 'SOME-NAME'
AND TIME BETWEEN 7 AND 41;
I'll bet Oracle knows something about the distribution of data in the TIME column, and is making a guess - perhaps using outdated statistics - as to what percentage of rows and blocks (i.e. the selectivity) of that column is. Check to see if there's a histogram on that column.
However, a query like this:
SELECT SOMETHING
FROM MYTABLE
WHERE NAME = 'SOME-NAME'
AND TIME BETWEEN MY_FUNC('7') AND MY_FUNC('41');
is likely to be optimized as semantically equivalent to:
SELECT SOMETHING
FROM MYTABLE
WHERE NAME = 'SOME-NAME'
AND TIME BETWEEN :some_bind AND :some_other_bind;
Because Oracle doesn't know what MY_FUNC('7') does - or even that MY_FUNC('7') will always return the same value of 7 - unless you've told Oracle the function's deterministic. So my experience is that Oracle takes a stab in the dark, for the most part, and tends to prefer an index with a high clustering factor. It seems to guess that even if the index isn't the best choice, at least it minimizes the downside risk by visiting as few data blocks as possible.
My recommendation is to find out for yourself why it's behaving differently - take a 10053 trace of each query:
alter session set events = '10053 trace name context forever;
run sql
alter session set events = '10053 trace name context off;
SELECT SOMETHING
FROM MYTABLE
WHERE NAME = 'SOME-NAME'
AND TIME BETWEEN STARTVALUE AND ENDVALUE
Here, you have TIME which is a NUMBER, and STARTVALUE and ENDVALUE which are strings (according to your comment). Therefore, an implicit conversion is done - i.e. your query is effectively:
SELECT SOMETHING
FROM MYTABLE
WHERE NAME = 'SOME-NAME'
AND TO_CHAR(TIME) BETWEEN STARTVALUE AND ENDVALUE
Unless you have a function-based index on TO_CHAR(TIME), it won't use an index.
Therefore, you must tell Oracle that you always expect the string parameters to be convertable to numbers, i.e.:
SELECT SOMETHING
FROM MYTABLE
WHERE NAME = 'SOME-NAME'
AND TIME BETWEEN TO_NUMBER(STARTVALUE) AND TO_NUMBER(ENDVALUE)
(It's always good practice to avoid implicit conversions, especially in queries, anyway)
I'm trying to understand how no_index actually speeds up a query and haven't been able to find documentation online to explain it.
For example I have this query that ran extremely slow
select *
from <tablename>
where field1_ like '%someGenericString%' and
field1_ <> 'someSpecificString' and
Action_='_someAction_' and
Timestamp_ >= trunc(sysdate - 2)
And one of our DBAs was able to speed it up significantly by doing this
select /*+ NO_INDEX(TAB_000000000019) */ *
from <tablename>
where field1_ like '%someGenericString%' and
field1_ <> 'someSpecificString' and
Action_='_someAction_' and
Timestamp_ >= trunc(sysdate - 2)
And I can't figure out why? I would like to figure out why this works so I can see if I can apply it to another query (this one a join) to speed it up because it's taking even longer to run.
Thanks!
** Update **
Here's what I know about the table in the example.
It's a 'partitioned table'
TAB_000000000019 is the table not a column in it
field1 is indexed
Oracle's optimizer makes judgements on how best to run a query, and to do this it uses a large number of statistics gathered about the tables and indexes. Based on these stats, it decides whether or not to use an index, or to just do a table scan, for example.
Critically, these stats are not automatically up-to-date, because they can be very expensive to gather. In cases where the stats are not up to date, the optimizer can make the "wrong" decision, and perhaps use an index when it would actually be faster to do a table scan.
If this is known by the DBA/developer, they can give hints (which is what NO_INDEX is) to the optimizer, telling it not to use a given index because it's known to slow things down, often due to out-of-date stats.
In your example, TAB_000000000019 will refer to an index or a table (I'm guessing an index, since it looks like an auto-generated name).
It's a bit of a black art, to be honest, but that's the gist of it, as I understand things.
Disclaimer: I'm not a DBA, but I've dabbled in that area.
Per your update: If field1 is the only indexed field, then the original query was likely doing a fast full scan on that index (i.e. reading through every entry in the index and checking against the filter conditions on field1), then using those results to find the rows in the table and filter on the other conditions. The conditions on field1 are such that an index unique scan or range scan (i.e. looking up specific values or ranges of values in the index) would not be possible.
Likely the optimizer chose this path because there are two filter predicates on field1. The optimizer would calculate estimated selectivity for each of these and then multiply them to determine their combined selectivity. But in many cases this will significantly underestimate the number of rows that will match the condition.
The NO_INDEX hint eliminates this option from the optimizer's consideration, so it essentially goes with the plan it thinks is next best -- possibly in this case using partition elimination based on one of the other filter conditions in the query.
Using an index degrades query performance if it results in more disk IO compared to querying the table with an index.
This can be demonstrated with a simple table:
create table tq84_ix_test (
a number(15) primary key,
b varchar2(20),
c number(1)
);
The following block fills 1 Million records into this table. Every 250th record is filled with a rare value in column b while all the others are filled with frequent value:
declare
rows_inserted number := 0;
begin
while rows_inserted < 1000000 loop
if mod(rows_inserted, 250) = 0 then
insert into tq84_ix_test values (
-1 * rows_inserted,
'rare value',
1);
rows_inserted := rows_inserted + 1;
else
begin
insert into tq84_ix_test values (
trunc(dbms_random.value(1, 1e15)),
'frequent value',
trunc(dbms_random.value(0,2))
);
rows_inserted := rows_inserted + 1;
exception when dup_val_on_index then
null;
end;
end if;
end loop;
end;
/
An index is put on the column
create index tq84_index on tq84_ix_test (b);
The same query, but once with index and once without index, differ in performance. Check it out for yourself:
set timing on
select /*+ no_index(tq84_ix_test) */
sum(c)
from
tq84_ix_test
where
b = 'frequent value';
select /*+ index(tq84_ix_test tq84_index) */
sum(c)
from
tq84_ix_test
where
b = 'frequent value';
Why is it? In the case without the index, all database blocks are read, in sequential order. Usually, this is costly and therefore considered bad. In normal situation, with an index, such a "full table scan" can be reduced to reading say 2 to 5 index database blocks plus reading the one database block that contains the record that the index points to. With the example here, it is different altogether: the entire index is read and for (almost) each entry in the index, a database block is read, too. So, not only is the entire table read, but also the index. Note, that this behaviour would differ if c were also in the index because in that case Oracle could choose to get the value of c from the index instead of going the detour to the table.
So, to generalize the issue: if the index does not pick few records then it might be beneficial to not use it.
Something to note about indexes is that they are precomputed values based on the row order and the data in the field. In this specific case you say that field1 is indexed and you are using it in the query as follows:
where field1_ like '%someGenericString%' and
field1_ <> 'someSpecificString'
In the query snippet above the filter is on both a variable piece of data since the percent (%) character cradles the string and then on another specific string. This means that the default Oracle optimization that doesn't use an optimizer hint will try to find the string inside the indexed field first and also find if the data it is a sub-string of the data in the field, then it will check that the data doesn't match another specific string. After the index is checked the other columns are then checked. This is a very slow process if repeated.
The NO_INDEX hint proposed by the DBA removes the optimizer's preference to use an index and will likely allow the optimizer to choose the faster comparisons first and not necessarily force index comparison first and then compare other columns.
The following is slow because it compares the string and its sub-strings:
field1_ like '%someGenericString%'
While the following is faster because it is specific:
field1_ like 'someSpecificString'
So the reason to use the NO_INDEX hint is if you have comparisons on the index that slow things down. If the index field is compared against more specific data then the index comparison is usually faster.
I say usually because when the indexed field contains more redundant data like in the example #Atish mentions above, it will have to go through a long list of comparison negatives before a positive comparison is returned. Hints produce varying results because both the database design and the data in the tables affect how fast a query performs. So in order to apply hints you need to know if the individual comparisons you hint to the optimizer will be faster on your data set. There are no shortcuts in this process. Applying hints should happen after proper SQL queries have been written because hints should be based on the real data.
Check out this hints reference: http://docs.oracle.com/cd/B19306_01/server.102/b14211/hintsref.htm
To add to what Rene' and Dave have said, this is what I have actually observed in a production situation:
If the condition(s) on the indexed field returns too many matches, Oracle is better off doing a Full Table Scan.
We had a report program querying a very large indexed table - the index was on a region code and the query specified the exact region code, so Oracle CBO uses the index.
Unfortunately, one specific region code accounted for 90% of the tables entries.
As long as the report was run for one of the other (minor) region codes, it completed in less than 30 minutes, but for the major region code it took many hours.
Adding a hint to the SQL to force a full table scan solved the problem.
Hope this helps.
I had read somewhere that using a % in front of query like '%someGenericString%' will lead to Oracle ignoring the INDEX on that field. Maybe that explains why the query is running slow.
To optimize SELECT queries, I run them both with and without an index and measure the difference. I run a bunch of different similar queries and try to select different data to make sure that caching doesn't throw off the results. However, on very large tables, indexes take a really long time to create, and I have several different ideas about what indexes would be appropriate.
Is it possible in Oracle (or any other database for that matter) to perform a query but tell the database to not use a certain index when performing the query? Or just turn off the index entirely, but be able to easily switch it back on without having to re-index the entire table? This would make it much easier to test, since I can create all the indexes I'm thinking about all at once, then try my queries using different ones.
Alternatively, is there any better way to go about optimizing queries on large tables and know which indexes would be best to create?
You can set index visibility in 11g -
ALTER INDEX idx1 [ INVISIBLE | VISIBLE ]
this makes it unusable by the optimizer, but oracle still updates the index when data is added or removed. This makes it easy to test performance with the index disabled without having to remove & rebuild the whole index.
See here for the oracle docs on index visibility
You can use the NO_INDEX hint in the queries to ignore the indexes - see docs for further details. The SQL Access Advisor is an Oracle utility that will recommend indexing strategies.
Well you can write the query in such a way that it wont use index(using expression instead of a value)
For example
Select * from foobar where column1 = 'result' --uses index on column1
To avoid using index for a number and varchar
Select * from foobar where column1 + 0 = 5 -- simple expression to disable the index
Select * from foobar where column1 || '' = 'result' --simple expression to disable the index
Or you can just use NVL to disable the index in the query without worrying about the column's data type
Select * from foobar where nvl(column1,column1) = 'result' --i love this way :D
Similarly you can use index hints
like /* Index(E employee_id) */ to use indexes.
P.S. This is all the paraphrased from Dan Tow's Book SQL Tuning. I started reading it a few days back :)