I have a block of sql that runs pretty smooth outside the procedure. The moment I put the sql block in the procedure to return the ref_cursor, the procedure takes quite a bit of long time to execute the ref_cursor.
With help from DBAs, we implemented DB profile and it worked great to speed up the procedure but then any minor change in that particular procedure make it go haywire. I am not sure what the problem.. I am running out of options. How should I go about troubleshooting this particular weird issue?
Thank you in advance.
Edit.. here is the query
with query_ownership as (SELECT leeo.legal_entity_id,
leeo.parent_le_id,
SUM(leeo.effective_ownership) ownership_percent
FROM data_ownership leeo
WHERE leeo.start_date <=
to_date('12/31/2012','mm/dd/yyyy')
AND ((leeo.end_date < &lvTaxYearDate and leeo.end_date > &lvTaxYearBeginDate)
to_date('12/31/2012','mm/dd/yyyy') OR
leeo.end_date IS NULL)
and leeo.stock_type in ('E')
GROUP BY leeo.legal_entity_id, leeo.parent_le_id
HAVING SUM(leeo.effective_ownership) > 0
),
query_branches as ( SELECT b.branch_id as legal_entity_id,
b.legal_entity_id as perent_le_id,
1.00 as ownership_percent
FROM company_branches b
WHERE b.tax_year = 2012),
child_query as (select * from query_ownership
UNION
select * from query_branches),
parent_query as (select * from query_ownership
UNION
select * from query_branches),
inner_query as (SELECT rownum as sortcode,
-level as lvl,
child_query.parent_le_id,
child_query.legal_entity_id,
child_query.ownership_percent
FROM child_query
START WITH child_query.legal_entity_id = 'AB1203'
CONNECT BY NOCYCLE PRIOR child_query.legal_entity_id =
child_query.parent_le_id
AND child_query.ownership_percent >= 0.01
and level = 0
UNION
SELECT rownum as sortcode,
level - 1 as lvl,
parent_query.parent_le_id,
parent_query.legal_entity_id,
parent_query.ownership_percent
FROM parent_query
START WITH parent_query.legal_entity_id = 'AB1203'
CONNECT BY NOCYCLE
PRIOR parent_query.parent_le_id =
parent_query.legal_entity_id
AND parent_query.ownership_percent >= 0.01)
,ownership_heirarchy as (
SELECT max(inner_query.sortcode) as sortcode,
max(inner_query.lvl) as lvl,
inner_query.parent_le_id,
inner_query.legal_entity_id,
inner_query.ownership_percent from inner_query
GROUP BY inner_query.parent_le_id,
inner_query.legal_entity_id,
inner_query.ownership_percent
)
,goldList as (
SELECT lem2.legal_entity_id from ownership_heirarchy,
company_entity_year lem1,
company_entity_year lem2
WHERE ownership_heirarchy.parent_le_id = lem2.legal_entity_id
AND lem2.tax_year = 2012
AND ownership_heirarchy.legal_entity_id = lem1.legal_entity_id
AND lem1.tax_year = 2012
AND lem1.legal_entity_type <> 'EXT'
AND lem1.non_legal_entity_flag is null
AND lem2.legal_entity_type <> 'EXT'
AND lem2.non_legal_entity_flag is null
and TRIM(lem2.alt_tax_type) is null
and UPPER(lem2.tax_type) in ('DC', 'DPS', 'TXN')
),
fulllist as (
select * from goldList
union
select gc.parent_le_id from company_entity_year e, consolidation_group gc
where e.LEGAL_ENTITY_ID = 'AB1203' and e.tax_year = 2012
and e.TAX_CONSOLIDATION_GRP = gc.group_id
union
select e.leid from vdst_entity e where e.TAX_YEAR = 2012
and e.ALT_TAX_TYPE in (3,8)
and e.LEID = 'AB1203'
)
select distinct dc.dcn_id as dcnId,
dc.dcn_name as dcnName,
dy.dcn_year_id dcnYearId,
ty.tax_year_id taxYearId,
ty.tax_year taxYear
from company_dcn dc, company_dcn_year dy, company_tax_year ty
where dc.dcn_id = dy.dcn_id
and dy.year_id = ty.tax_year_id
and ty.tax_year = 2012
and dc.leid in (
select * from fulllist
);
First, ensure that statistics are up-to-date by running DBMS_STATS.GATHER_TABLE_STATS for each table involved in the query. Next, obtain the plan for the query with different parameter values - it's entirely possible that a change in the parameters may make the plan better or worse. Given that you're showing us no information about the query, the procedure, and the tables involved there's no way to be more specific.
Best of luck.
Find out what part of the execution plan is causing problems. There are several ways to do this:
Use DBMS_XPLAN to find a good and bad plan. Use explain plan for ... and select * from table(dbms_xplan.display); to find the good plan in your session. Use dbms_xplan.display_cursor(sql_id => 'some sql_id') to find the bad plan. Compare the plans and look for differences. This can be very difficult because you can't usually tell which parts of the execution plan are slow. If you're lucky there will be only one difference and then obviously that difference is the problem.
Use DBMS_SQLTUNE.REPORT_SQL_MONITOR to find what part of the plan is bad. Run the bad query and use SQL Monitoring to find out which operation in the execution plan is bad. The report shows which operations take the longest, and which cardinality estimates are off by the most. Focus on the slow parts, and the first steps of the plan with a huge cardinality difference between estimated and actual.
Look at profile hints to find out how Oracle fixes the bad plan. Profiles are a collection of hints that help nudge the optimizer into making the correct decisions. Those hints might tell you what the problem is. For example, if one of the hints is OPT_ESTIMATE(JOIN (A B) SCALE_ROWS=100), the profile is telling the optimizer to increase the cardinality estimate by 100X. You may be able to recreate that same affect by either including that hint in the query or by creating and locking fake table statistics. Use this process from Kerry Osborne to find the profile hints.
Either way, this process can be difficult and time-consuming. Try to shrink the query as much as possible. Tuning a 97 line query can be almost impossible some times. It's possible there is only one root problem, but that problem changes so much of the execution plan that it looks like there are a dozen problems.
These steps only help you identify the problem. Fixing it may be a whole other question and answer.
Related
I am trying to create a sales table however after 8 hours it still has not run. I have attempted to speed up the query by adding the hints and reducing the timeframe to 2022 only however after 3 hours it is still running. Is there a way to optimise this query?
DROP TABLE BIRTHDAY_SALES;
CREATE TABLE BIRTHDAY_SALES AS
--(
SELECT /*+ parallel(32) */
DISTINCT T.CONTACT_KEY
, S.CAMPAIGN_NAME
, S.CONTROL_GROUP_FLAG
, S.SEGMENT_NAME
, count(distinct t.ORDER_NUM) as TRANS
, count(distinct case when p.store_key = '42381' then t.ORDER_NUM else NULL end) as TRANS_ONLINE
, count(distinct case when p.store_key != '42381' then t.ORDER_NUM else NULL end) as TRANS_OFFLINE
, sum(t.ITEM_AMT) as SALES
, sum(case when p.store_key = '42381' then t.ITEM_AMT else NULL end) as SALES_ONLINE
, sum(case when p.store_key != '42381' then t.ITEM_AMT else NULL end) as SALES_OFFLINE
, sum(case when t.item_quantity_val>0 and t.item_amt<=0 then 0 else t.item_quantity_val end) QTY
, sum(case when (p.store_key = '42381' and t.ITEM_QUANTITY_VAL>0 and t.ITEM_AMT>0) then t.ITEM_QUANTITY_VAL else null end) QTY_ONLINE
, sum(case when (p.store_key != '42381' and t.ITEM_QUANTITY_VAL>0 and t.ITEM_AMT>0) then t.ITEM_QUANTITY_VAL else null end) QTY_OFFLINE
FROM CRM_TARGET.B_TRANSACTION T
JOIN BDAY_PROG S
ON T.CONTACT_KEY = S.CONTACT_KEY
JOIN CRM_TARGET.T_ORDITEM_SD P
ON T.PRODUCT_KEY = P.PRODUCT_KEY
where t.TRANSACTION_TYPE_NAME = 'Item'
and t.BU_KEY = '15'
and t.TRANSACTION_DT_KEY >= '20220101'
and t.TRANSACTION_DT_KEY <= '20221231'
and t.member_sale_flag = 'Y'
and t.bu_key = '15'
and t.CONTACT_KEY != 0
group by
T.CONTACT_KEY
, S.CAMPAIGN_NAME
, S.CONTROL_GROUP_FLAG
, S.SEGMENT_NAME
-- )
;
Performance tuning is not something we can effectively deal with on a forum like this, as there are too many factors to consider. You will have to examine the explain plan and look at ASH data (v$active_session_history) to see what the predominant waits are and on what plan step. Only then can you determine what's wrong and take steps to fix it.
However, here are some obvious things to look for:
Make sure there are no many-to-many joins. I'm guessing B_TRANSACTION probably has many rows with the same CONTACT_KEY and many rows with the same PRODUCT_KEy. That's okay, but then you must ensure that CONTACT_KEY is unique within BDA_PROG and PRODUCT_KEY is unique within T_ORDITEM_SD. IF that's not the case, you will get a partial Cartesian product from the hidden many-to-many join and will spend a huge amount of time on reading/writing to temp.
Make sure no more than one of those joins is one-to-many. Multiple one-to-manies stemming off the same parent table will effectively give you a many-to-many between the children, with the same effect.
You are asking for a date range of a month. In most systems, you are better off doing a full table scan (with parallel query if you can) than using indexes to get a whole month's worth of transactional data. If it is using an index that can really mess you up. You can fix this with hints (see below)
It might be using nested loops joins when a reporting query like this is likely better off using hash joins. Again, I'm just guessing based on the names of your tables; only knowledge of your data can determine this for sure.
Ensure that the PGA workareas are of reasonable size. Ask your DBA to query v$pgastat and report the global memory bound. It should be at its max of 1G, but probably anything over 100M is reasonable. If it's less than that, you may need to ask the DBA to increase the pga_aggregate_target, or you can manually set your own sort_area_size/hash_area_size session parameters (not the best thing to do).
You are asking for DOP 32. That's pretty high. Ensure there are that many CPU cores on the database server, that parallel_max_servers > 64 and that you aren't getting downgraded to serial by anything. Ask your DBA what a reasonable DOP would be.
Do you really need COUNT(DISTINCT ... ) on ORDERNUM? If you are just counting # of transactions, it would be less work to simply say SUM(CASE (WHEN .... ) THEN 1 ELSE 0 END)
Remove the DISTINCT keyword. It's not doing anything - your GROUP BY will already result in the results being distinct.
Consult ASH (v$active_session_history) to see if you are actually blocked by something, showing some kind of concurrency wait. Your CTAS might not be doing anything at all because of some library cache lock or full tablespace if the database is configured to suspend until space is added.
Here's something to try - again, it's a long shot without knowing your data or table structure. But I've seen enough reports like this to make at least a somewhat educated guess:
SELECT /*+ USE_HASH(t s p) FULL(t) FULL(s) FULL(p) PARALLEL(8) */ t.contact_key . . .
Is there a hint to generate execution plan ignoring the existing one from the shared pool?
There is not a hint to create an execution plan that ignores plans in the shared pool. A more common way of phrasing this question is: how do I get Oracle to always perform a hard parse?
There are a few weird situations where this behavior is required. It would be helpful to fully explain your reason for needing this, as the solution varies depending why you need it.
Strange performance problem. Oracle performs some dynamic re-optimization of SQL statements after the first run, like adaptive cursor sharing and cardinality feedback. In the rare case when those features backfire you might want to disable them.
Dynamic query. You have a dynamic query that used Oracle data cartridge to fetch data in the parse step, but Oracle won't execute the parse step because the query looks static to Oracle.
Misunderstanding. Something has gone wrong and this is an XY problem.
Solutions
The simplest way to solve this problem are by using Thorsten Kettner's solution of changing the query each time.
If that's not an option, the second simplest solution is to flush the query from the shared pool, like this:
--This only works one node at a time.
begin
for statements in
(
select distinct address, hash_value
from gv$sql
where sql_id = '33t9pk44udr4x'
order by 1,2
) loop
sys.dbms_shared_pool.purge(statements.address||','||statements.hash_value, 'C');
end loop;
end;
/
If you have no control over the SQL, and need to fix the problem using a side-effect style solution, Jonathan Lewis and Randolf Geist have a solution using Virtual Private Database, that adds a unique predicate to each SQL statement on a specific table. You asked for something weird, here's a weird solution. Buckle up.
-- Create a random predicate for each query on a specific table.
create table hard_parse_test_rand as
select * from all_objects
where rownum <= 1000;
begin
dbms_stats.gather_table_stats(null, 'hard_parse_test_rand');
end;
/
create or replace package pkg_rls_force_hard_parse_rand is
function force_hard_parse (in_schema varchar2, in_object varchar2) return varchar2;
end pkg_rls_force_hard_parse_rand;
/
create or replace package body pkg_rls_force_hard_parse_rand is
function force_hard_parse (in_schema varchar2, in_object varchar2) return varchar2
is
s_predicate varchar2(100);
n_random pls_integer;
begin
n_random := round(dbms_random.value(1, 1000000));
-- s_predicate := '1 = 1';
s_predicate := to_char(n_random, 'TM') || ' = ' || to_char(n_random, 'TM');
-- s_predicate := 'object_type = ''TABLE''';
return s_predicate;
end force_hard_parse;
end pkg_rls_force_hard_parse_rand;
/
begin
DBMS_RLS.ADD_POLICY (USER, 'hard_parse_test_rand', 'hard_parse_policy', USER, 'pkg_rls_force_hard_parse_rand.force_hard_parse', 'select');
end;
/
alter system flush shared_pool;
You can see the hard-parsing in action by running the same query multiple times:
select * from hard_parse_test_rand;
select * from hard_parse_test_rand;
select * from hard_parse_test_rand;
select * from hard_parse_test_rand;
Now there are three entries in GV$SQL for each execution. There's some odd behavior in Virtual Private Database that parses the query multiple times, even though the final text looks the same.
select *
from gv$sql
where sql_text like '%hard_parse_test_rand%'
and sql_text not like '%quine%'
order by 1;
I think there is no hint indicating that Oracle shall find a new execution plan everytime it runs the query.
This is something we'd want for select * from mytable where is_active = :active, with is_active being 1 for very few rows and 0 for maybe billions of other rows. We'd want an index access for :active = 1 and a full table scan for :active = 0 then. Two different plans.
As far as I know, Oracle uses bind variable peeking in later versions, so with a look at the statistics it really comes up with different execution plans for different bind varibale content. But in older versions it did not, and thus we'd want some hint saying "make a new plan" there.
Oracle only re-used an execution plan for exactly the same query. It sufficed to add a mere blank to get a new plan. Hence a solution might be to generate the query everytime you want to run it with a random number included in a comment:
select /* 1234567 */ * from mytable where is_active = :active;
Or just don't use bind variables, if this is the problem you want to address:
select * from mytable where is_active = 0;
select * from mytable where is_active = 1;
So, I have been doing a fair amount of reading on this in various forums and resource sites but have not yet found found a solution I believe applies to my case. Also, I can't believe how difficult this is proving to be; I would think this kind of query would be fairly common.
Essentially what I am doing here is querying two historical tables (tbl_b and tbl_c), via union, for a specific milestone date - for which there may be multiple results... I then wish to find the most recent of these results, using max. This date is then returned as a column in the main query.
My problem is that, in the 3rd tier subquery, I need to reference an identifier value from the table in the top query (tbl_a).
I know that correlated queries only are able to reference their parent query - so, I am stuck.
Edit 1
The target date I am searching for will most likely, but not necessarily, be unique within the result set. It is a timestamp of the data record. I am looking for the most recent entry in the history that correlates to each column in tbl_a. Creating an SQL Fiddle for this.
See sample below:
select tbl_a.col_a,
tbl_a.col_b,
(
select max(target_date)
from
(
select tbl_b.target_date
from tbl_b
where tbl_b.tbl_a_id = tbl_a.id and
tbl_b.flag = 1 and
tbl_b.milestone_id = tbl_a.milestone_id
union
select tbl_c.target_date
from tbl_c
where tbl_c.tbl_a_id = tbl_a.id and
tbl_c.flag = 1 and
tbl_c.milestone_id = tbl_a.milestone_id
) most_recent_target_date
)
from tbl_a
Convert this query to a join, in this way:
select tbl_a.col_a,
tbl_a.col_b,
max(most_recent_target_date.target_date)
from tbl_a
join (
select tbl_b.target_date, tbl_b.date_id
from tbl_b
where tbl_b.flag = 1
union all
select tbl_c.target_date, tbl_c.date_id
from tbl_c
where tbl_c.flag = 1
) most_recent_target_date
ON tbl_a.date_id = most_recent_target_date.date_id
GROUP BY tbl_a.col_a,
tbl_a.col_b
I am using DBMS_PROFILER for basic profiling of my PL/SQL packages. I am also using it to get code coverage statistics using the following query:
SELECT EXEC.unit_name unitname,ROUND (EXEC.cnt/total.cnt * 100, 1) Code_coverage FROM
(SELECT u.unit_name, COUNT(1) cnt FROM plsql_profiler_data d, plsql_profiler_units u WHERE u.unit_number = d.unit_number GROUP BY u.unit_name) total,
(SELECT u.unit_name, COUNT(1) cnt FROM plsql_profiler_data d, plsql_profiler_units u WHERE u.unit_number = d.unit_number AND d.total_occur > 0 GROUP BY u.unit_name) EXEC
WHERE EXEC.unit_name = total.unit_name
I clear the plsql_profiler_data,plsql_profiler_units,plsql_profiler_runs tables before each profiler runs so that I need not know the run id each time.
This will give me Package wise information on the percentage of code that was covered during the profiling. Now I am trying to see if this can be built as a normal coverage report where I can know which line of code was covered and which one wasnt(say select lineOfCode, iscovered from...) so that I can built a report with html formatting to indicate if a line was covered or not.
I am not too proficient in Oracle table structures on where the functions and procedures get saved etc. (Got the above query from a blog and modified slightly to remove run id's)
Is this possible?
If so how can I achieve this?
I think this approaches what you're after:
-- View lines of code profiled, along with run times, next to the complete, ordered source..
-- Provides an annotated view of profiled packages, procs, etc.
-- Only the first line of a multiline SQL statement will register with timings.
SELECT u.UNIT_OWNER || '.' || u.UNIT_NAME AS "Unit"
, s.line
, CASE WHEN d.TOTAL_OCCUR >= 0 THEN 'C'
ELSE ' ' END AS Covered
, s.TEXT
, TO_CHAR(d.TOTAL_TIME / (1000*1000*1000), 'fm990.000009') AS "Total Time (sec)"
, CASE WHEN NVL(d.TOTAL_OCCUR, 1) > 0 THEN d.TOTAL_OCCUR ELSE 1 END AS "# Iterations"
, TO_CHAR(CASE WHEN d.TOTAL_OCCUR > 0 THEN d.TOTAL_TIME / (d.TOTAL_OCCUR * (1000*1000*1000))
ELSE NULL END, 'fm990.000009') AS "Avg Time (sec)"
FROM all_source s
LEFT JOIN plsql_profiler_units u ON s.OWNER = u.UNIT_OWNER
AND s.NAME = u.UNIT_NAME
AND s.TYPE = u.UNIT_TYPE
LEFT JOIN plsql_profiler_data d ON u.UNIT_NUMBER = d.UNIT_NUMBER
AND s.LINE = d.LINE#
AND d.RUNID = u.RUNID
WHERE u.RUNID = ? -- Add RUNID of profiler run to investigate here
ORDER BY u.UNIT_NAME
, s.LINE
There are few issues to keep in mind.
1) Many rows in the plsql_profiler_data table will NOT have accurate values in their TOTAL_TIME column because they executed faster than the resolution of the timer.
Ask Tom re: timings:
The timings are collected using some unit of time, typically only
granular to the HSECS.
That means that many discrete events that take less then 1/100th of a
second, appear to take ZERO seconds.
Many discrete events that take less then 1/100ths of a second may
appear to take 1/100th of a second.
2) Only the FIRST line in a multiline statement will show as covered. So if you split an INSERT or whatever across multiple lines, I don't know of any easy way to have every line of that statement to show as profiled in an Annotated Source style of report.
Also, check out Oracle's dbms_profiler documentation and this useful package reference for help crafting queries against the collected profiler data.
Actually there are some tools for PL/SQL that do code coverage. See the answers to this question for more information.
Said this, you can find information on user created data structure and code in following tables:
user_source: here you can find the source in the TEXT field typified by function, procedure, package, etc.
User_tables
user_indexes
user_types: if you use some kind of OO code.
Other tables beginning with user_ that you may need.
Basically you would need to check the result of your query against user_source and get extra information from the other tables.
I've got a table of a parameters, values, and times at which those values were recorded.
I've got a procedure which takes in a time, and needs to get the average result of each parameters value in the window of time that is -15/+5 seconds around that time frame. On top of that, I want to make sure that I take the no more than 15 records before the passed in time, and no more than 5 records after it.
For example, maybe I'm recording values of some parameters every second. If I passed in the time 21:30:30, I'd want to get the values between 21:30:15 and 21:30:35. But if I was recording every half second, I'd actually have more parameters that fit in that time frame than I want, and that's where my need to limit my results comes in.
I've read this question and this article which seem pretty related to what I'm trying to do, but unfortunately I'm dealing with Oracle and not MySQL, so I can't use "limit".
I've currently got something that looks like this:
std_values as
(
select
V.ParameterId,
V.NumericValue,
from
ValuesTable V
where
V.ValueSource = pValueSource
and V.Time >= pSummaryTime - 15/86400
and V.Time <= pSummaryTime + 5/86400
)
select
ParameterId,
avg(NumericValue) as NumericValue
from
std_values
group by
ParameterId
pValueSource is just something that lets me filter down which value types I'm looking at, and pSummaryTime is the input time that I'm basing my time frame around. The goal here is to get the 15 records before pSummaryTime that falls within that window, and the 5 after that falls within that window, and use those for the average. Currently I'm not limiting the number of "before" and "after" results though, so I'm ending up with the average of everything that falls into that time window. And without something like "limit", I'm not sure how to do this in Oracle.
Sounds like you want a moving window aggregate function. This is part of the Analytical functions feature of Oracle.
It's not my strong suit, and since you didn't include sample tables/data to build a test case, I'll just point you to the Oracle documentation, here:
http://docs.oracle.com/cd/B14117_01/server.101/b10736/analysis.htm#i1006709
You probably want something like:
AVG(NumericValue) over (order by pSummaryTime RANGE BETWEEN 15 PRECEDING AND 5 FOLLOWING)
but, like I said, not my strong suit, and totally untested, but, I hope it gets the idea across.
Hope that helps.
Thanks to Mark Bobak's answer getting me on the right track, I ended up with this solution.
with
values_before as
(
select
V.ParameterId,
V.NumericValue,
row_number() over (Partition by V.ParameterId order by V.Time desc) as RowNumber
from
ValuesTable V
where
V.ValueSource = pValueSource
and V.Time >= pSummaryTime - 15/86400
and V.Time <= pSummaryTime
),
values_after as
(
select
V.ParameterId,
V.NumericValue,
row_number() over (Partition by V.ParameterId order by V.Time desc) as RowNumber
from
ValuesTable V
where
V.ValueSource = pValueSource
and V.Time <= pSummaryTime + 5/86400
and V.Time > pSummaryTime
),
values_all as
(
select * from values_before where RowNumber <= 15
union all
select * from values_after where RowNumber <= 5
)
select ParameterId, avg(NumericValue) from values_all group by ParameterId
No doubt there's a better way to do this, but it at least seems to be giving the correct result. The key was using an analytical function to set the row number and order for the 15 before and 5 after, and then filtering my results down to just those.