I have a massive table in which I can't do any more partitioning or sub-partitioning, nor am allowed to do any alter. I want to query its records by batches, and thought a good way would be using the last two digits from the account numbers (wouldn't have any other field splitting records as evenly).
I guess I'd need to at least index that somehow (remember I can't alter table to add a virtual column either).
Is there any kind of index to be used in such situation?
I am using Oracle 11gR2
You can use function based index:
create index two_digits_idx on table_name (substr(account_number, -2));
This index will work only in queries like that:
select ...
from table_name t ...
where substr(account_number, -2) = '25' -- or any other two digits
For using index, you need to use in a query the same expression like in an index.
Related
My sample oracle query structure is this:
SELECT <LIST_OF_COLUMNS> FROM <TABLE_NAME>
WHERE COLUMN_01 = <SOMETHING> AND COLUMN_02 = <SOMETHING> AND COLUMN_03 = <SOMETHING>
The table has over 1 Million records. I have indexed COLUMN_01, COLUMN_02 and COLUMN_03 separately. The above query is working fine and provide results as expected.
If I make COLUMN_01, COLUMN_02 and COLUMN_03 (all columns in WHERE clause) as composite index without changing existing indexes, will it improve performance? If so, is there an order for composite index columns?
If I use OR instead of AND like this query, will it improve performance?
SELECT <LIST_OF_COLUMNS> FROM <TABLE_NAME>
WHERE COLUMN_01 = <SOMETHING> OR COLUMN_02 = <SOMETHING> OR COLUMN_03 = <SOMETHING>
If you create a composite index based on the three columns AND
you never query the table using only one column in the WHERE
clause, you won't need the single column indexes. Otherwise, it
depends on what columns participate in a query. And this case should
be well analyzed and tested.
Columns order in a composite index does matter. Columns should be ordered by uniqueness where the least distinct column goes first. It helps trim down the number of rows matched the query predicate and thus speed up performance.
It also should be noticed that Oracle can use a composite index with queries that do not contain all the index columns in their predicates.
For example:
create index idx1 on table_name (col1, col2, col3);
/*In this query Oracle can use index idx1 as a standard one-column index,
because col1 is the first column in the index*/
select * from table_name where col1 = 'some_value';
/*Here Oracle can still use the composite index,
but in this case it will use INDEX SKIP SCAN (assuming col1 equals ANY value),
which reduces query performance comparing to an ordinary index*/
select * from table_name where col2 = 'some_value1' and col3 = 'some_value2';
OR or AND operators do not really matter here. What is more important is the number of rows which match the given predicate.
If I make COLUMN_01, COLUMN_02 and COLUMN_03 (all columns in WHERE clause) as composite index without changing existing indexes, will it improve performance?
Probably. One index which satisfies all WHERE criteria serves as a complete access path and hence is more effective than a single column index access path. The optimizer chooses one index, so it will index read all the rows matching (say) COLUMN_02 criterion and filter those rows using the other columns' criteria.
The price you pay for this improvement in performance is the overhead of maintaining an additional index. So you should consider whether you need all three single column indexes (for other queries).
is there an order for composite index columns?
Yes. Put them in ascending order of distinct values. The leading index column should be the least discriminating column. Having a unique key as the leading column is probably a disaster, although there are edge cases, so be sure to benchmark.
If I use OR instead of AND like this query, will it improve performance?
You're going to be returning more rows, which in itself is more work. It is also hard to use indexes in such a situation, so most likely you're facing a Full Table Scan. But why not try it and see what happens?
If I make COLUMN_01, COLUMN_02 and COLUMN_03 (all columns in WHERE clause) as composite index without changing existing indexes, will it improve performance?
For this query: likely. For INSERT/UPDATE/DELETE: the performance will deteriorate.
So you'll need to measure and see whether improvement in some queries justifies the deterioration in others.
If so, is there an order for composite index columns?
Not for this query. If you prefix-compress the index, you may choose the order that compresses the best, otherwise it shouldn't matter much.
However, there may be other queries that use only some of the indexed columns, in which case you'd want to make sure the columns that are actually used are at the leading edge of the index.
If I use OR instead of AND like this query, will it improve performance?
No. Separate indexes (that you already have) are what is needed in this case.
Which of the below would be the best way to create an Index, so that my query gives faster results?
My query is:
select emp_name,emp_last_name, salary
from employees_table
where salary <=2000;
1. create index emp_index on employees_table (salary);
2. create index emp_index on employees_table (emp_name,emp_last_name,salary);
In principle create index emp_index on employees_table (emp_name,emp_last_name,salary); would be the better one, because for your query Oracle has to read only the index but not any table data.
However, this means you create a separate index for each particular query which is certainly an overkill.
For 2 index Oracle may will be select table access full, if disctinct values in firsts columns of index to much, Oracle give you Table access full, if distinct values in first columns of index not much then you give index skip scan.
For first index you will see “index range scan” + “table access by rowid”....
but for both index’s - if sql returns >5-7% rows then Oracle maybe want to use “Table access full”...
All depends on your data in Table and count return rows
I have a large table of 7 column in Oracle 11G. Total size of the table is more than 3GB and total row in this table is 1876823. Query we are using
select doc_mstr_id from index_mstr where page_con1 like('%sachin%') it is taking almost a minute. please help me to optimize the query as well as proper indexing for this table. Please let me know if partitioned is required for this table.
Below are the column description
INDEX_MSTR_ID NUMBER
DOC_MSTR_ID NUMBER
PAGE_NO NUMBER
PAGE_PART NUMBER
PAGE_CON1 VARCHAR2(4000)
FILE_MODIFIED_DATE DATE
CREATED_DATE DATE
This query is always going to result in a full table scan. Your only filter cannot use a B-TREE index, due to the leading wildcard:
where page_con1 like('%sachin%')
If you want to do lots of queries of this nature you need to build a Text index on that column. From its datatype page_con1 appears to hold text fragments rather than full documents so you should use a CTXCAT index. This type of index has the advantage of being transactional, rather than requiring background maintenance. Find out more.
Your query would then look like this:
select doc_mstr_id from index_mstr
WHERE CATSEARCH(page_con1, 'sachin') > 0;
I have a table that I expect to get 7 million records a month on a pretty wide table. A small portion of these records are expected to be flagged as "problem" records.
What is the best way to implement the table to locate these records in an efficient way?
I'm new to Oracle, but is a materialized view an valid option? Are there such things in Oracle such as indexed views or is this potentially really the same thing?
Most of the reporting is by month, so partitioning by month seems like an option, but a "problem" record may be lingering for several months theorectically. Otherwise, the reporting shuold be mostly for the current month. Would you expect that querying across all month partitions to locate any problem record would cause significant performance issues compared to usinga single table?
Your general thoughts of where to start would be appreciated. I realize I need to read up and I'll do that but I wanted to get the community thought first to make sure I read the right stuff.
One more thought: The primary key is a GUID varchar2(36). In order of magnitude, how much of a performance hit would you expect this to be relative to using a NUMBER data type PK? This worries me but it is out of my control.
It depends what you mean by "flagged", but it sounds to me like you would benefit from a simple index, function based index, or an indexed virtual column.
In all cases you should be careful to ensure that all the index columns are NULL for rows that do not need to be flagged. This way your index will contain only the rows that are flagged (Oracle does not - by default - index rows in B-Tree indexes where all index column values are NULL).
Your primary key being a VARCHAR2 GUID should make no difference, at least with regards to the specific flagging of rows in this question, indexes will point to rows via Oracle internal ROWIDs.
Indexes support partitioning, so if your data is already partitioned, your index could be set to match.
Simple column index method
If you can dictate how the flagging works, or the column already exists, then I would simply add an index to it like so:
CREATE INDEX my_table_problems_idx ON my_table (problem_flag)
/
Function-based index method
If the data model is fixed / there is no flag column, then you can create a function-based index assuming that you have all the information you need in the target table. For example:
CREATE INDEX my_table_problems_fnidx ON my_table (
CASE
WHEN amount > 100 THEN 'Y'
ELSE NULL
END
)
/
Now if you use the same logic in your SELECT statement, you should find that it uses the index to efficiently match rows.
SELECT *
FROM my_table
WHERE CASE
WHEN amount > 100 THEN 'Y'
ELSE NULL
END IS NOT NULL
/
This is a bit clunky though, and it requires you to use the same logic in queries as the index definition. Not great. You could use a view to mask this, but you're still duplicating logic in at least two places.
Indexed virtual column
In my opinion, this is the best way to do it if you are computing the value dynamically (available from 11g onwards):
ALTER TABLE my_table
ADD virtual_problem_flag VARCHAR2(1) AS (
CASE
WHEN amount > 100 THEN 'Y'
ELSE NULL
END
)
/
CREATE INDEX my_table_problems_idx ON my_table (virtual_problem_flag)
/
Now you can just query the virtual column as if it were a real column, i.e.
SELECT *
FROM my_table
WHERE virtual_problem_flag = 'Y'
/
This will use the index and puts the function-based logic into a single place.
Create a new table with just the pks of the problem rows.
I'm using a function-based index with a user-defined function for the first time, and have stumbled across a performance problem when the index can't be used.
Internally, a function-based index seems to generate a hidden table column (of type varchar2(4000), since my function returns a varchar2), and indexes that. That works fine when the index is used, but sometimes we have to do a full table scan using the function as a filter, and in that case I see a performance degradation by a factor of 6. Seems in that case, Oracle does not use the hidden column, but recomputes the function for each row, make the query CPU-bound instead of IO-bound.
Is there a way to make Oracle use that hidden column also for filtering? I wonder if I'm missing some rewrite options or something along those lines.
If not, I'll have to define the column myself and using a trigger to keep it up to date. I'd prefer using the function-based index for transparency and easier maintanance.
Which version of Oracle are you using? If it's 11g you should try using a virtual column. This is a column whose value is derived from an expression or a literal. They're defined as part of the table, so they have a visibility in a table DESC (unlike a function-based index). We can build indexes on virtual columns. And they are maintained automatically, without the need for a trigger.
So you can add a virtual column to your table using the same expression as your function based index. Perhaps like this:
create table t23
(id number
, col_a varchar2(10)
, vcol_a as (upper(substr(col_a, 1, 1)))
)
/
Note that we cannot insert or update a virtual column. So you need to specify the projection of the insert statement:
insert into t23 (id, col_a) values (1, 'this is a test');
Then you can build a regular index on the virtual column:
create index t23_vc_i on t23(vcol_a)
/
Don't forget to drop your function based index!
There is no general way a function-based index can be used in a table scan.
The assumption I made in my question, namely "Internally, a function-based index seems to generate a hidden table column ...", is simply wrong: the results of the function are not stored in a table column, but only in the index.
So, unless there is a way to access the index when performing the scan (the only way I can think of is if it's a combined index starting with the key column(s)), the precomputed function result can't be used.
The 11g "virtual column" feature does not help either, as the column is not stored in the table, but computed on-the-fly, similar to using the function in a view.
In summary: if you can't rule out table scans, and your function call is expensive (slow), use a real column in combination with a "before insert or update" trigger. A function-based index won't do.
(Note: Added this answer because I did not want to let this question stand as unanswered. The credit for the answer belongs to thilo, who pointed out that the column is never materialized).