I have below query which deals with 10 million records in each ent_data and ent_details tables in Oracle DB.
select val1.type,count(val1.type)
from ent_data val1
inner join ent_details az
ON val1.ENT_ID=az.ent_id
and val1.ENT_TYPE=az.ent_type
and val1.user_key=87
group by val1.type;
----------------------------------------------------------------------------------------------------------------------------
Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------------------------------
HASH GROUP BY | | 3 | 126 | | 135K (1)| 00:27:10 |
FILTER | | | | | | |
HASH JOIN | | 10M| 413M| 305M| 134K (1)| 00:26:56 |
TABLE ACCESS FULL | ent_details | 10M| 187M| | 13116 (2)| 00:02:38 |
TABLE ACCESS FULL | ent_data | 10M| 226M| | 89074 (1)| 00:17:49 |
Executing above query takes around 27 seconds time for execution on SQL Developer but as the fetched data is used for display on UI 27 seconds is something I can't live with.
I have indexes on Category, ENT_ID and ent_type column of ent_data table. Also I have indexes on ent_id and ent_type column of ent_details table.
Will appreciate any guidance on how the query performance can be improved.
Related
Considering the execution plan for this query :
SQL_ID 1m5r644say02b, child number 0
-------------------------------------
select * from hr.employees where department_id = 80 intersect select *
from hr.employees where first_name like 'A%'
Plan hash value: 1738366820
------------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Starts | E-Rows | A-Rows | A-Time | Buffers | OMem | 1Mem | Used-Mem |
------------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | | 4 |00:00:00.01 | 8 | | | |
| 1 | INTERSECTION | | 1 | | 4 |00:00:00.01 | 8 | | | |
| 2 | SORT UNIQUE | | 1 | 34 | 34 |00:00:00.01 | 6 | 6144 | 6144 | 6144 (0)|
|* 3 | TABLE ACCESS FULL | EMPLOYEES | 1 | 34 | 34 |00:00:00.01 | 6 | | | |
| 4 | SORT UNIQUE | | 1 | 11 | 10 |00:00:00.01 | 2 | 2048 | 2048 | 2048 (0)|
| 5 | TABLE ACCESS BY INDEX ROWID BATCHED| EMPLOYEES | 1 | 11 | 10 |00:00:00.01 | 2 | | | |
|* 6 | INDEX SKIP SCAN | EMP_NAME_IX | 1 | 11 | 10 |00:00:00.01 | 1 | | | |
------------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - filter("DEPARTMENT_ID"=80)
6 - access("FIRST_NAME" LIKE 'A%')
filter("FIRST_NAME" LIKE 'A%')
The execution plan has both access and filter predicates with the same '%A' predicate here on the EMP_NAME_IX index. But shouldn't the access predicate be enough here, as they both will filter the same rows? Why did it perform the additional filter predicate?
Is there a general rule for when both access and filter are the same? Based on GV$SQL_PLAN, when an operation has either an access or a filter predicate, they are only equal about 1% of the time. And this situation only happens with operations and options like INDEX (FULL/RANGE/SKIP/UNIQUE) and SORT (JOIN/UNIQUE).
select *
from gv$sql_plan
where access_predicates = filter_predicates;
Presumably you have an index on hr.employees that includes the first_name column. But you are selecting * from hr.employees such that the rows obtained from the index would have to traced back (i.e. join) with the table.
For conceptual understanding it helps to think of indexes as plain tables with a foreign key to the original table's primary key. When usage of indexes helps, these two tables are joined. The index is used alone when it contains all needed columns.
In this case we assume a join is required since you are selecting *. When accessing the hr.employee table for the second query of the intersect, because its where clause filters on an index column, a join to the index is performed prior to filtering.
The first occurrence of "FIRST_NAME" LIKE 'A%' is the reason usage of the index is decided. The second occurrence, is then the actual filtering. Filtering happens only once, not twice.
These are listed as distinct operations as deciding to use the index (and therefore perform the join) has its own costs.
I have a SELECT statement
SELECT MIN(C_PRICE), MAX(C_PRICE)
FROM CAR;
I run the statement and create a processing plan to look at the cost.
-------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
-------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 3 | 12150 (1)| 00:00:01 |
| 1 | SORT AGGREGATE | | 1 | 3 | | |
| 2 | TABLE ACCESS FULL| CAR | 1800K| 5273K| 12150 (1)| 00:00:01 |
-------------------------------------------------------------------------------
I created an inmemory to this table car after setting the inmemory size to be 200M.
ALTER TABLE CAR INMEMORY;
The result
----------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
----------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 3 | 12150 (1)| 00:00:01 |
| 1 | SORT AGGREGATE | | 1 | 3 | | |
| 2 | TABLE ACCESS INMEMORY FULL| CAR | 1800K| 5273K| 12150 (1)| 00:00:01 |
----------------------------------------------------------------------------------------
My question is why isn't the query improved after altering the table to be in the inmemory? The SELECT statement clearly shows that it is accessing the table via inmemory. I thought inmemory creation will improve the query processing thus reducing the cost?
Two things you should look at. One, is the table populated in the IM column store (i.e. v$im_segments)? Second, what was the time difference between the two queries and where was the time spent? SQL Monitor active reports are excellent for determining this.
after research via playing with the queries and doing online research, I am turning to you for input. Looking forward to your replies! I am trying to write this in a general way as I am looking for general ideas on how to do this, not for exact statements. If this question is received poorly, I am happy to rework it, let me know. Here we go:
I have two tables:
Table 1 has 10 MIO records. It has only one column:
Column A: A unique ID (a text e.g. '5uz1g09aksmsd')
Table 2 has 300 MIO records. It has two columns:
Column A: A unique ID (a text e.g. '5uz1g09aksmsd')
Column B: A number (e.g. 32.5432)
There are no indices etc. yet, not even a PK as both tables have just been created via CTAS. Both tables have been analysed.
I am now looking for a fast query (the prep-work e.g. index creation can take time, no problem) to create a third table that contains the 10 MIO rows of Table 1, and Column B of Table 2 if a match is found via Column A (for 99% a match will be found). To make it clearer, a simple CTAS could be:
create table3 as
select t1.a,
(select t2.b from table2 t2 where t2.a = t1.a and rownum = 1)
-- the rownum = 1 is to show Oracle that there can only be one match
from table1 t1;
This is not as fast as it can be (I hope). What are your ideas to make it faster? Creating indices? Which ones? What kind of join would you want to see in the execution plan? A hash join? I already found
create table nologging
parallel query execution and parallel table creation
But I am interested in the specifics of how the perfect execution plan for this would look given that we are allowed to alter the system (e.g. create an index). In particular I am asking for 11gR2 but 12c comments are also very welcome.
Start with the simplest possibility using this query
create table c as
select /*+ parallel(6) */ a.a, b.b
from a
left outer join b
on a.a = b.a
;
It will use a hash join, adjust parallel degree based on your hardware setup.
The expected execution plan is as follows
--------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
--------------------------------------------------------------------------------------------------------------------------
| 0 | CREATE TABLE STATEMENT | | 261M| 6238M| 56799 (1)| 00:03:48 | | | |
| 1 | PX COORDINATOR | | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10002 | 261M| 6238M| 31984 (1)| 00:02:08 | Q1,02 | P->S | QC (RAND) |
| 3 | LOAD AS SELECT | C | | | | | Q1,02 | PCWP | |
|* 4 | HASH JOIN OUTER | | 261M| 6238M| 31984 (1)| 00:02:08 | Q1,02 | PCWP | |
| 5 | PX RECEIVE | | 7849K| 44M| 726 (1)| 00:00:03 | Q1,02 | PCWP | |
| 6 | PX SEND HASH | :TQ10000 | 7849K| 44M| 726 (1)| 00:00:03 | Q1,00 | P->P | HASH |
| 7 | PX BLOCK ITERATOR | | 7849K| 44M| 726 (1)| 00:00:03 | Q1,00 | PCWC | |
| 8 | TABLE ACCESS STORAGE FULL| A | 7849K| 44M| 726 (1)| 00:00:03 | Q1,00 | PCWP | |
| 9 | PX RECEIVE | | 261M| 4741M| 31149 (1)| 00:02:05 | Q1,02 | PCWP | |
| 10 | PX SEND HASH | :TQ10001 | 261M| 4741M| 31149 (1)| 00:02:05 | Q1,01 | P->P | HASH |
| 11 | PX BLOCK ITERATOR | | 261M| 4741M| 31149 (1)| 00:02:05 | Q1,01 | PCWC | |
| 12 | TABLE ACCESS STORAGE FULL| B | 261M| 4741M| 31149 (1)| 00:02:05 | Q1,01 | PCWP | |
--------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - access("A"."A"="B"."A"(+))
My test with synthetic data of your size is 35 seconds.
so I have this huge table SS(someID, someDate, ...). I need to join a subset of this table to the other table. The subset is determined by: select * from SS where someID in (select someID from SS where someDate is between date1 and date2).
When running this on Oracle XA data server in parallel, the execution takes a long time and TEMP space, even though Oracle can cell offloading efficiency of 99% on the SS table, but the subset query still bring back a large amount of data to the database server in joining with other table.
Is there anyway to make this more efficient? such as Oracle doesn't have to send back as much data and utilize more of the cell offloading efficiency?
Below is the query plan
PLAN_TABLE_OUTPUT
Plan hash value: 3198983388
---------------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
---------------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1044K| 589M| 46101 (1)| 00:01:33 | | | |
| 1 | PX COORDINATOR | | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10003 | 1044K| 589M| 46101 (1)| 00:01:33 | Q1,03 | P->S | QC (RAND) |
|* 3 | HASH JOIN BUFFERED | | 1044K| 589M| 46101 (1)| 00:01:33 | Q1,03 | PCWP | |
| 4 | PX RECEIVE | | | | | | Q1,03 | PCWP | |
| 5 | PX SEND HASH | :TQ10001 | | | | | Q1,01 | P->P | HASH |
| 6 | NESTED LOOPS | | | | | | Q1,01 | PCWP | |
| 7 | NESTED LOOPS | | 523K| 135M| 38264 (1)| 00:01:17 | Q1,01 | PCWP | |
| 8 | SORT UNIQUE | | 29402 | 401K| 13751 (1)| 00:00:28 | Q1,01 | PCWP | |
| 9 | PX RECEIVE | | 29402 | 401K| 13751 (1)| 00:00:28 | Q1,01 | PCWP | |
| 10 | PX SEND HASH | :TQ10000 | 29402 | 401K| 13751 (1)| 00:00:28 | Q1,00 | P->P | HASH |
| 11 | PX BLOCK ITERATOR | | 29402 | 401K| 13751 (1)| 00:00:28 | Q1,00 | PCWC | |
|* 12 | INDEX STORAGE FAST FULL SCAN| SUPERSET_IDX1 | 29402 | 401K| 13751 (1)| 00:00:28 | Q1,00 | PCWP | |
|* 13 | INDEX RANGE SCAN | XU_SUPERSET_01 | 18 | | 1 (0)| 00:00:01 | Q1,01 | PCWP | |
| 14 | TABLE ACCESS BY INDEX ROWID | SUPERSET | 18 | 4644 | 2 (0)| 00:00:01 | Q1,01 | PCWP | |
| 15 | PX RECEIVE | | 2886K| 880M| 7834 (2)| 00:00:16 | Q1,03 | PCWP | |
| 16 | PX SEND HASH | :TQ10002 | 2886K| 880M| 7834 (2)| 00:00:16 | Q1,02 | P->P | HASH |
| 17 | PX BLOCK ITERATOR | | 2886K| 880M| 7834 (2)| 00:00:16 | Q1,02 | PCWC | |
| 18 | TABLE ACCESS STORAGE FULL | POL_DTL | 2886K| 880M| 7834 (2)| 00:00:16 | Q1,02 | PCWP | |
---------------------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - access(SS.POL_ID=PD.POL_ID)
12 - storage(IMPT_DT<=TO_DATE(' 2014-11-20 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND IMPT_DT>=TO_DATE(' 2014-10-28
00:00:00', 'syyyy-mm-dd hh24:mi:ss'))
filter(IMPT_DT<=TO_DATE(' 2014-11-20 00:00:00', 'syyyy-mm-dd hh24:mi:ss') AND IMPT_DT>=TO_DATE(' 2014-10-28
00:00:00', 'syyyy-mm-dd hh24:mi:ss'))
13 - access(SS.POL_ID=POL_ID)
Note
-----
- Degree of Parallelism is 4 because of session
There may not be much you can do to improve this query. The execution plan looks pretty good:
Good objects The indexes seem to fit the query well, although it's hard to tell without the full definitions.
Good cardinality The estimated rows and actual rows are close. This strongly implies the optimizer is doing a good job and is picking a near-optimal plan. If it can estimate the number of rows correctly it will make wise decisions about the access paths, join methods, join order, etc. Even the time estimate is close, which is rare. It looks like there are good table and system statistics.
Cell offloading The storage predicates and the active report Cell offloading imply that cell offloading is working as expected, at least once.
Parallelism Large objects are already being processed in parallel. I don't see any obvious parallel problems.
Here are some ideas for improvement but don't expect drastic improvements:
Full table scan Force a full table scan instead of an index range scan with a hint like --+ no_index(superset XU_SUPERSET_01). With multiblock reads (use for full scans) and cell offloading (used for a direct path read for a full scan not used for an index range scan that uses the buffer cache), a full table scan reading all the data may be more efficient than an index range scan reading less data.
Covering index If the full table scan doesn't work, create a skinny version of the table with an index that includes all returned and queried columns. This gets the benefits of full scans (multiblock IO, cell offloading) but is smaller than the full table.
Larger DOP There's no magic number for the degree of parallelism (DOP). But in my experience the DOP sweet-spot is almost always larger than 4. This may improve performance but will use more resources.
Rewrite query? Re-writing the query may enable smart scan to process the join in the storage cells. Try changing
select * from SS where someID in
(select someID from SS where someDate is between date1 and date2)
to
select distinct ss1.*
from ss ss1
join ss ss2
on ss1.someID = ss2.someID
and ss2.someDate is between date1 and date2
This new version does extra work. The join returns more rows than necessary, and then they need to be made distinct. That extra work may be worth it if it means the join can happen in the storage cells. I can't find a great source for exactly what kind of processing can be offloaded, but at least some types of joins can be.
I have table say
TAB1
ID, TARGET, STATE, NEXT
Column ID is the primary key.
The query is that is showing deadlock is similar to this
SELECT *
FROM TAB1
WHERE NEXT = (SELECT MIN(NEXT) FROM TAB1 WHERE TARGET=? AND STATE=?) FOR UPDATE
I did an explain plan I see something like this:
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 8095 | 6 (0)| 00:00:01 |
| 1 | FOR UPDATE | | | | | |
| 2 | BUFFER SORT | | | | | |
|* 3 | TABLE ACCESS FULL | TAB1 | 1 | 8095 | 3 (0)| 00:00:01 |
| 4 | SORT AGGREGATE | | 1 | 2083 | | |
|* 5 | TABLE ACCESS FULL| TAB1 | 1 | 2083 | 3 (0)| 00:00:01 |
Since the query is doing TABLE ACCESS FULL twice, so I'm suspecting 2 session executing the same query will access the rows in different orders.
Can indexing of columns will help in preventing the deadlock? Say creating an index on NEXT??? Or by changing the PRIMARY to NON CLUSTERED KEY?? Note: Normally, the table will have max 1000 rows.
Addind a non clustered index on the NEXT column would indeed boost your performance and reduce your deadlock issues.