Oracle 19c performance issue - long operation - oracle

Question

I have an application that runs automated process against the database on a very regular basis. Unfortunately, the statement was crafted inside an ORM and can't be rewritten.
The statement is as follows:
SELECT t0.id FROM SCHEMA.CMTS t0, SCHEMA.CMNTSRCH t3, SCHEMA.CONT t1, SCHEMA.YU t2 WHERE (
(
t0.YUtypeyn = 0
OR
t0.YUtypeyn = 1
)
AND
(
(
t1.contcd IN ('FD')
OR
t1.contcd IS NULL
)
AND
(
t2.TRNSMDCD = 'RD'
AND
t2.spid = 1
AND
t0.compyn = NULL
AND
(
NOT
(
upper(t3.curryuloc) LIKE upper('ABC%') ESCAPE '\'
)
AND
NOT
(
upper(t3.curryuloc) LIKE upper('DEF%') ESCAPE '\'
)
AND
NOT
(
upper(t3.curryuloc) LIKE upper('GHI%') ESCAPE '\'
)
AND
NOT
(
upper(t3.curryuloc) LIKE upper('%JKL%') ESCAPE '\'
)
AND
NOT
(
upper(t3.curryuloc) LIKE upper('%MNO%') ESCAPE '\'
)
AND
NOT
(
upper(t3.curryuloc) LIKE upper('%PQR%') ESCAPE '\'
)
AND
NOT
(
upper(t3.curryuloc) LIKE upper('%STU%') ESCAPE '\'
)
AND
NOT
(
upper(t3.curryuloc) LIKE upper('%VWX%') ESCAPE '\'
)
AND
NOT
(
upper(t3.curryuloc) LIKE upper('%YZ%') ESCAPE '\'
)
)
AND
t0.cancelledyn = NULL
)
)
)
AND
t0.CMNTSRCHid = t3.id
AND
t0.contcd = t1.contcd
AND
t0.YUid = t2.id(+)
Performance issues have been noticed while this statement is running. When monitoring the v$session_longops, I noticed a full table scan against CMNTSRCH.
2640 seconds elapsed so far for a full table scan - which raised concerns.
Explain plan is as follows:
------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 59 | 0 (0)| |
|* 1 | FILTER | | | | | |
| 2 | NESTED LOOPS | | 1 | 59 | 26677 (1)| 00:00:02 |
| 3 | NESTED LOOPS | | 1 | 59 | 26677 (1)| 00:00:02 |
| 4 | MERGE JOIN CARTESIAN | | 1 | 27 | 26674 (1)| 00:00:02 |
| 5 | NESTED LOOPS | | 1 | 13 | 24369 (1)| 00:00:01 |
|* 6 | INDEX UNIQUE SCAN | CONT_PK | 1 | 5 | 0 (0)| 00:00:01 |
|* 7 | TABLE ACCESS FULL | CMNTSRCH | 1 | 8 | 24369 (1)| 00:00:01 |
| 8 | BUFFER SORT | | 36824 | 503K| 2305 (1)| 00:00:01 |
|* 9 | INDEX FAST FULL SCAN | IDX_YU20220408 | 36824 | 503K| 2305 (1)| 00:00:01 |
|* 10 | INDEX RANGE SCAN | CMNT_013 | 1 | | 2 (0)| 00:00:01 |
|* 11 | TABLE ACCESS BY INDEX ROWID| CMNT | 1 | 32 | 3 (0)| 00:00:01 |
------------------------------------------------------------------------------------------------------
Reviewing the statement, I decided to create a function based index on t3.id and t3.curryuloc and reviewed the explain plan - it appeared to improve CPU cost and now performs an Index Fast Full scan as opposed to a Table Scan.
New explain plan (after Function Based index creation):
----------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
----------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 59 | 0 (0)| |
|* 1 | FILTER | | | | | |
| 2 | NESTED LOOPS | | 1 | 59 | 3975 (1)| 00:00:01 |
| 3 | NESTED LOOPS | | 1 | 59 | 3975 (1)| 00:00:01 |
| 4 | MERGE JOIN CARTESIAN | | 1 | 27 | 3972 (1)| 00:00:01 |
| 5 | NESTED LOOPS | | 1 | 13 | 1667 (2)| 00:00:01 |
|* 6 | INDEX UNIQUE SCAN | CONT_PK | 1 | 5 | 0 (0)| 00:00:01 |
|* 7 | INDEX FAST FULL SCAN | IDX_CMNTSRCH_20220407 | 1 | 8 | 1667 (2)| 00:00:01 |
| 8 | BUFFER SORT | | 36824 | 503K| 2305 (1)| 00:00:01 |
|* 9 | INDEX FAST FULL SCAN | IDX_YU20220408 | 36824 | 503K| 2305 (1)| 00:00:01 |
|* 10 | INDEX RANGE SCAN | CMNT_013 | 1 | | 2 (0)| 00:00:01 |
|* 11 | TABLE ACCESS BY INDEX ROWID| CMNT | 1 | 32 | 3 (0)| 00:00:01 |
----------------------------------------------------------------------------------------------------------------
However, this operation is also taking an incredibly long time
Monitoring the v$session_longops, I can see when a new session runs the statement. SOFAR starts at about 4000 blocks, and gets slower and slower, almost grinding to a halt by the time it gets to 5500 blocks. Forcing the use of the Function based index doesn't appear to have made any difference to the time it takes for the operation to complete.
Oddly though, if I run the statement manually with the same arguments the app binds to the parameters, it take almost no time to execute (there are no rows returned).
What should be my next steps to troubleshoot this issue?

Answer 1

You have a whole mess of WHERE clauses like this:
AND (NOT(upper(t3.curryuloc) LIKE upper('ABC%') ESCAPE '\')
See the upper() function on the column CMNTSRCH.curryuloc? It can't random-access an ordinary index.
But Oracle has function indexes. You may be able to get better performance if you create an index like this:
CREATE INDEX whatever_1 ON CMNTSRCH (UPPER(curryuloc), id);
or, reversing the column order, like this:
CREATE INDEX whatever_2 ON CMNTSRCH (id, UPPER(curryuloc));
It's worth a try.
But, it has to be said, that cascade of AND...NOT...LIKE clauses isn't written for efficiency.