I am using DBMS_PROFILER for basic profiling of my PL/SQL packages. I am also using it to get code coverage statistics using the following query:
SELECT EXEC.unit_name unitname,ROUND (EXEC.cnt/total.cnt * 100, 1) Code_coverage FROM
(SELECT u.unit_name, COUNT(1) cnt FROM plsql_profiler_data d, plsql_profiler_units u WHERE u.unit_number = d.unit_number GROUP BY u.unit_name) total,
(SELECT u.unit_name, COUNT(1) cnt FROM plsql_profiler_data d, plsql_profiler_units u WHERE u.unit_number = d.unit_number AND d.total_occur > 0 GROUP BY u.unit_name) EXEC
WHERE EXEC.unit_name = total.unit_name
I clear the plsql_profiler_data,plsql_profiler_units,plsql_profiler_runs tables before each profiler runs so that I need not know the run id each time.
This will give me Package wise information on the percentage of code that was covered during the profiling. Now I am trying to see if this can be built as a normal coverage report where I can know which line of code was covered and which one wasnt(say select lineOfCode, iscovered from...) so that I can built a report with html formatting to indicate if a line was covered or not.
I am not too proficient in Oracle table structures on where the functions and procedures get saved etc. (Got the above query from a blog and modified slightly to remove run id's)
Is this possible?
If so how can I achieve this?
I think this approaches what you're after:
-- View lines of code profiled, along with run times, next to the complete, ordered source..
-- Provides an annotated view of profiled packages, procs, etc.
-- Only the first line of a multiline SQL statement will register with timings.
SELECT u.UNIT_OWNER || '.' || u.UNIT_NAME AS "Unit"
, s.line
, CASE WHEN d.TOTAL_OCCUR >= 0 THEN 'C'
ELSE ' ' END AS Covered
, s.TEXT
, TO_CHAR(d.TOTAL_TIME / (1000*1000*1000), 'fm990.000009') AS "Total Time (sec)"
, CASE WHEN NVL(d.TOTAL_OCCUR, 1) > 0 THEN d.TOTAL_OCCUR ELSE 1 END AS "# Iterations"
, TO_CHAR(CASE WHEN d.TOTAL_OCCUR > 0 THEN d.TOTAL_TIME / (d.TOTAL_OCCUR * (1000*1000*1000))
ELSE NULL END, 'fm990.000009') AS "Avg Time (sec)"
FROM all_source s
LEFT JOIN plsql_profiler_units u ON s.OWNER = u.UNIT_OWNER
AND s.NAME = u.UNIT_NAME
AND s.TYPE = u.UNIT_TYPE
LEFT JOIN plsql_profiler_data d ON u.UNIT_NUMBER = d.UNIT_NUMBER
AND s.LINE = d.LINE#
AND d.RUNID = u.RUNID
WHERE u.RUNID = ? -- Add RUNID of profiler run to investigate here
ORDER BY u.UNIT_NAME
, s.LINE
There are few issues to keep in mind.
1) Many rows in the plsql_profiler_data table will NOT have accurate values in their TOTAL_TIME column because they executed faster than the resolution of the timer.
Ask Tom re: timings:
The timings are collected using some unit of time, typically only
granular to the HSECS.
That means that many discrete events that take less then 1/100th of a
second, appear to take ZERO seconds.
Many discrete events that take less then 1/100ths of a second may
appear to take 1/100th of a second.
2) Only the FIRST line in a multiline statement will show as covered. So if you split an INSERT or whatever across multiple lines, I don't know of any easy way to have every line of that statement to show as profiled in an Annotated Source style of report.
Also, check out Oracle's dbms_profiler documentation and this useful package reference for help crafting queries against the collected profiler data.
Actually there are some tools for PL/SQL that do code coverage. See the answers to this question for more information.
Said this, you can find information on user created data structure and code in following tables:
user_source: here you can find the source in the TEXT field typified by function, procedure, package, etc.
User_tables
user_indexes
user_types: if you use some kind of OO code.
Other tables beginning with user_ that you may need.
Basically you would need to check the result of your query against user_source and get extra information from the other tables.
Related
I am trying to create a sales table however after 8 hours it still has not run. I have attempted to speed up the query by adding the hints and reducing the timeframe to 2022 only however after 3 hours it is still running. Is there a way to optimise this query?
DROP TABLE BIRTHDAY_SALES;
CREATE TABLE BIRTHDAY_SALES AS
--(
SELECT /*+ parallel(32) */
DISTINCT T.CONTACT_KEY
, S.CAMPAIGN_NAME
, S.CONTROL_GROUP_FLAG
, S.SEGMENT_NAME
, count(distinct t.ORDER_NUM) as TRANS
, count(distinct case when p.store_key = '42381' then t.ORDER_NUM else NULL end) as TRANS_ONLINE
, count(distinct case when p.store_key != '42381' then t.ORDER_NUM else NULL end) as TRANS_OFFLINE
, sum(t.ITEM_AMT) as SALES
, sum(case when p.store_key = '42381' then t.ITEM_AMT else NULL end) as SALES_ONLINE
, sum(case when p.store_key != '42381' then t.ITEM_AMT else NULL end) as SALES_OFFLINE
, sum(case when t.item_quantity_val>0 and t.item_amt<=0 then 0 else t.item_quantity_val end) QTY
, sum(case when (p.store_key = '42381' and t.ITEM_QUANTITY_VAL>0 and t.ITEM_AMT>0) then t.ITEM_QUANTITY_VAL else null end) QTY_ONLINE
, sum(case when (p.store_key != '42381' and t.ITEM_QUANTITY_VAL>0 and t.ITEM_AMT>0) then t.ITEM_QUANTITY_VAL else null end) QTY_OFFLINE
FROM CRM_TARGET.B_TRANSACTION T
JOIN BDAY_PROG S
ON T.CONTACT_KEY = S.CONTACT_KEY
JOIN CRM_TARGET.T_ORDITEM_SD P
ON T.PRODUCT_KEY = P.PRODUCT_KEY
where t.TRANSACTION_TYPE_NAME = 'Item'
and t.BU_KEY = '15'
and t.TRANSACTION_DT_KEY >= '20220101'
and t.TRANSACTION_DT_KEY <= '20221231'
and t.member_sale_flag = 'Y'
and t.bu_key = '15'
and t.CONTACT_KEY != 0
group by
T.CONTACT_KEY
, S.CAMPAIGN_NAME
, S.CONTROL_GROUP_FLAG
, S.SEGMENT_NAME
-- )
;
Performance tuning is not something we can effectively deal with on a forum like this, as there are too many factors to consider. You will have to examine the explain plan and look at ASH data (v$active_session_history) to see what the predominant waits are and on what plan step. Only then can you determine what's wrong and take steps to fix it.
However, here are some obvious things to look for:
Make sure there are no many-to-many joins. I'm guessing B_TRANSACTION probably has many rows with the same CONTACT_KEY and many rows with the same PRODUCT_KEy. That's okay, but then you must ensure that CONTACT_KEY is unique within BDA_PROG and PRODUCT_KEY is unique within T_ORDITEM_SD. IF that's not the case, you will get a partial Cartesian product from the hidden many-to-many join and will spend a huge amount of time on reading/writing to temp.
Make sure no more than one of those joins is one-to-many. Multiple one-to-manies stemming off the same parent table will effectively give you a many-to-many between the children, with the same effect.
You are asking for a date range of a month. In most systems, you are better off doing a full table scan (with parallel query if you can) than using indexes to get a whole month's worth of transactional data. If it is using an index that can really mess you up. You can fix this with hints (see below)
It might be using nested loops joins when a reporting query like this is likely better off using hash joins. Again, I'm just guessing based on the names of your tables; only knowledge of your data can determine this for sure.
Ensure that the PGA workareas are of reasonable size. Ask your DBA to query v$pgastat and report the global memory bound. It should be at its max of 1G, but probably anything over 100M is reasonable. If it's less than that, you may need to ask the DBA to increase the pga_aggregate_target, or you can manually set your own sort_area_size/hash_area_size session parameters (not the best thing to do).
You are asking for DOP 32. That's pretty high. Ensure there are that many CPU cores on the database server, that parallel_max_servers > 64 and that you aren't getting downgraded to serial by anything. Ask your DBA what a reasonable DOP would be.
Do you really need COUNT(DISTINCT ... ) on ORDERNUM? If you are just counting # of transactions, it would be less work to simply say SUM(CASE (WHEN .... ) THEN 1 ELSE 0 END)
Remove the DISTINCT keyword. It's not doing anything - your GROUP BY will already result in the results being distinct.
Consult ASH (v$active_session_history) to see if you are actually blocked by something, showing some kind of concurrency wait. Your CTAS might not be doing anything at all because of some library cache lock or full tablespace if the database is configured to suspend until space is added.
Here's something to try - again, it's a long shot without knowing your data or table structure. But I've seen enough reports like this to make at least a somewhat educated guess:
SELECT /*+ USE_HASH(t s p) FULL(t) FULL(s) FULL(p) PARALLEL(8) */ t.contact_key . . .
I am facing some performance issue for below ORACLE query:
select fms.*,v.ph_typ_strng_id,v.ph_typ
from fct_miscdata fms -- (having 10 columns)
cross join lkp_lang LL -- (having 3 columns with 2 rows. we can't remove this cross join)
left outer join vw_dim_ph_typ v ON v.LANG_COD = LL.LANG_COD -- (PH_TYP_STRNG_ID, PH_TYP, Lang)
where v.lang_cod != '00_00' and fms.s_key = 0 and fms.t_key = 0
and v.ph_typ_strng_id = case
when exists (select 'x'
from vw_dim_excp_typ_inter m -- (misc_typSkey,src,tnt, misc_typId) -- only 3 miscTypeId : 0,98,99
where fms.in_misc_typ_skey = m.misc_typ_skey
and m.s_key = fms.s_key
and m.t_key = fms.t_key) then
case
when not exists (select 'x'
from vw_dim_excp_typ_inter m
where fms.out_misc_typ_skey = m.misc_typ_skey
and m.s_key = fms.s_key
and m.t_key = fms.t_key) then
0006
end
else 0004
end;
==> Its a materialized view in oracle which is refreshing in a Oracle procedure. Whenever I am trying to run this procedure, its taking long hours approx(1 hr) for run. Indexes are already created on tables.
--> Due to last code of lines CASE statement, its taking so much time to run. If I comment this part it'll run in few minutes. No. of records are 90772580.
--> Is there any alternate solution for this query to improve performance.
Please suggest how can I optimize/improve this query performance.
Thanks in advance
I have a block of sql that runs pretty smooth outside the procedure. The moment I put the sql block in the procedure to return the ref_cursor, the procedure takes quite a bit of long time to execute the ref_cursor.
With help from DBAs, we implemented DB profile and it worked great to speed up the procedure but then any minor change in that particular procedure make it go haywire. I am not sure what the problem.. I am running out of options. How should I go about troubleshooting this particular weird issue?
Thank you in advance.
Edit.. here is the query
with query_ownership as (SELECT leeo.legal_entity_id,
leeo.parent_le_id,
SUM(leeo.effective_ownership) ownership_percent
FROM data_ownership leeo
WHERE leeo.start_date <=
to_date('12/31/2012','mm/dd/yyyy')
AND ((leeo.end_date < &lvTaxYearDate and leeo.end_date > &lvTaxYearBeginDate)
to_date('12/31/2012','mm/dd/yyyy') OR
leeo.end_date IS NULL)
and leeo.stock_type in ('E')
GROUP BY leeo.legal_entity_id, leeo.parent_le_id
HAVING SUM(leeo.effective_ownership) > 0
),
query_branches as ( SELECT b.branch_id as legal_entity_id,
b.legal_entity_id as perent_le_id,
1.00 as ownership_percent
FROM company_branches b
WHERE b.tax_year = 2012),
child_query as (select * from query_ownership
UNION
select * from query_branches),
parent_query as (select * from query_ownership
UNION
select * from query_branches),
inner_query as (SELECT rownum as sortcode,
-level as lvl,
child_query.parent_le_id,
child_query.legal_entity_id,
child_query.ownership_percent
FROM child_query
START WITH child_query.legal_entity_id = 'AB1203'
CONNECT BY NOCYCLE PRIOR child_query.legal_entity_id =
child_query.parent_le_id
AND child_query.ownership_percent >= 0.01
and level = 0
UNION
SELECT rownum as sortcode,
level - 1 as lvl,
parent_query.parent_le_id,
parent_query.legal_entity_id,
parent_query.ownership_percent
FROM parent_query
START WITH parent_query.legal_entity_id = 'AB1203'
CONNECT BY NOCYCLE
PRIOR parent_query.parent_le_id =
parent_query.legal_entity_id
AND parent_query.ownership_percent >= 0.01)
,ownership_heirarchy as (
SELECT max(inner_query.sortcode) as sortcode,
max(inner_query.lvl) as lvl,
inner_query.parent_le_id,
inner_query.legal_entity_id,
inner_query.ownership_percent from inner_query
GROUP BY inner_query.parent_le_id,
inner_query.legal_entity_id,
inner_query.ownership_percent
)
,goldList as (
SELECT lem2.legal_entity_id from ownership_heirarchy,
company_entity_year lem1,
company_entity_year lem2
WHERE ownership_heirarchy.parent_le_id = lem2.legal_entity_id
AND lem2.tax_year = 2012
AND ownership_heirarchy.legal_entity_id = lem1.legal_entity_id
AND lem1.tax_year = 2012
AND lem1.legal_entity_type <> 'EXT'
AND lem1.non_legal_entity_flag is null
AND lem2.legal_entity_type <> 'EXT'
AND lem2.non_legal_entity_flag is null
and TRIM(lem2.alt_tax_type) is null
and UPPER(lem2.tax_type) in ('DC', 'DPS', 'TXN')
),
fulllist as (
select * from goldList
union
select gc.parent_le_id from company_entity_year e, consolidation_group gc
where e.LEGAL_ENTITY_ID = 'AB1203' and e.tax_year = 2012
and e.TAX_CONSOLIDATION_GRP = gc.group_id
union
select e.leid from vdst_entity e where e.TAX_YEAR = 2012
and e.ALT_TAX_TYPE in (3,8)
and e.LEID = 'AB1203'
)
select distinct dc.dcn_id as dcnId,
dc.dcn_name as dcnName,
dy.dcn_year_id dcnYearId,
ty.tax_year_id taxYearId,
ty.tax_year taxYear
from company_dcn dc, company_dcn_year dy, company_tax_year ty
where dc.dcn_id = dy.dcn_id
and dy.year_id = ty.tax_year_id
and ty.tax_year = 2012
and dc.leid in (
select * from fulllist
);
First, ensure that statistics are up-to-date by running DBMS_STATS.GATHER_TABLE_STATS for each table involved in the query. Next, obtain the plan for the query with different parameter values - it's entirely possible that a change in the parameters may make the plan better or worse. Given that you're showing us no information about the query, the procedure, and the tables involved there's no way to be more specific.
Best of luck.
Find out what part of the execution plan is causing problems. There are several ways to do this:
Use DBMS_XPLAN to find a good and bad plan. Use explain plan for ... and select * from table(dbms_xplan.display); to find the good plan in your session. Use dbms_xplan.display_cursor(sql_id => 'some sql_id') to find the bad plan. Compare the plans and look for differences. This can be very difficult because you can't usually tell which parts of the execution plan are slow. If you're lucky there will be only one difference and then obviously that difference is the problem.
Use DBMS_SQLTUNE.REPORT_SQL_MONITOR to find what part of the plan is bad. Run the bad query and use SQL Monitoring to find out which operation in the execution plan is bad. The report shows which operations take the longest, and which cardinality estimates are off by the most. Focus on the slow parts, and the first steps of the plan with a huge cardinality difference between estimated and actual.
Look at profile hints to find out how Oracle fixes the bad plan. Profiles are a collection of hints that help nudge the optimizer into making the correct decisions. Those hints might tell you what the problem is. For example, if one of the hints is OPT_ESTIMATE(JOIN (A B) SCALE_ROWS=100), the profile is telling the optimizer to increase the cardinality estimate by 100X. You may be able to recreate that same affect by either including that hint in the query or by creating and locking fake table statistics. Use this process from Kerry Osborne to find the profile hints.
Either way, this process can be difficult and time-consuming. Try to shrink the query as much as possible. Tuning a 97 line query can be almost impossible some times. It's possible there is only one root problem, but that problem changes so much of the execution plan that it looks like there are a dozen problems.
These steps only help you identify the problem. Fixing it may be a whole other question and answer.
I've got a table of a parameters, values, and times at which those values were recorded.
I've got a procedure which takes in a time, and needs to get the average result of each parameters value in the window of time that is -15/+5 seconds around that time frame. On top of that, I want to make sure that I take the no more than 15 records before the passed in time, and no more than 5 records after it.
For example, maybe I'm recording values of some parameters every second. If I passed in the time 21:30:30, I'd want to get the values between 21:30:15 and 21:30:35. But if I was recording every half second, I'd actually have more parameters that fit in that time frame than I want, and that's where my need to limit my results comes in.
I've read this question and this article which seem pretty related to what I'm trying to do, but unfortunately I'm dealing with Oracle and not MySQL, so I can't use "limit".
I've currently got something that looks like this:
std_values as
(
select
V.ParameterId,
V.NumericValue,
from
ValuesTable V
where
V.ValueSource = pValueSource
and V.Time >= pSummaryTime - 15/86400
and V.Time <= pSummaryTime + 5/86400
)
select
ParameterId,
avg(NumericValue) as NumericValue
from
std_values
group by
ParameterId
pValueSource is just something that lets me filter down which value types I'm looking at, and pSummaryTime is the input time that I'm basing my time frame around. The goal here is to get the 15 records before pSummaryTime that falls within that window, and the 5 after that falls within that window, and use those for the average. Currently I'm not limiting the number of "before" and "after" results though, so I'm ending up with the average of everything that falls into that time window. And without something like "limit", I'm not sure how to do this in Oracle.
Sounds like you want a moving window aggregate function. This is part of the Analytical functions feature of Oracle.
It's not my strong suit, and since you didn't include sample tables/data to build a test case, I'll just point you to the Oracle documentation, here:
http://docs.oracle.com/cd/B14117_01/server.101/b10736/analysis.htm#i1006709
You probably want something like:
AVG(NumericValue) over (order by pSummaryTime RANGE BETWEEN 15 PRECEDING AND 5 FOLLOWING)
but, like I said, not my strong suit, and totally untested, but, I hope it gets the idea across.
Hope that helps.
Thanks to Mark Bobak's answer getting me on the right track, I ended up with this solution.
with
values_before as
(
select
V.ParameterId,
V.NumericValue,
row_number() over (Partition by V.ParameterId order by V.Time desc) as RowNumber
from
ValuesTable V
where
V.ValueSource = pValueSource
and V.Time >= pSummaryTime - 15/86400
and V.Time <= pSummaryTime
),
values_after as
(
select
V.ParameterId,
V.NumericValue,
row_number() over (Partition by V.ParameterId order by V.Time desc) as RowNumber
from
ValuesTable V
where
V.ValueSource = pValueSource
and V.Time <= pSummaryTime + 5/86400
and V.Time > pSummaryTime
),
values_all as
(
select * from values_before where RowNumber <= 15
union all
select * from values_after where RowNumber <= 5
)
select ParameterId, avg(NumericValue) from values_all group by ParameterId
No doubt there's a better way to do this, but it at least seems to be giving the correct result. The key was using an analytical function to set the row number and order for the 15 before and 5 after, and then filtering my results down to just those.
I have a situation in my application for displaying the count of data which match different criterion. Since the performance of counting is degrading with respect to the growth of database, we decided to show only the availability information using the exists clause.
Below is my table structure
Table: DocInfo
---------------------------------------
DocId number
DocName varchar(250)
DocStatus number
SignedBy number
ForwardedBy number
ForwardCount number
DocOwner number
MgrID number
ProjectId number
The current query which does the counting is like this
SELECT NVL(SUM(CASE
WHEN (DocStatus IN (1150,1155,1170,1182,1190) AND
DocOwner=56366 AND
ForwardCount=0)
THEN 1
ELSE 0
END), 0) "ForReview",
NVL(SUM(CASE
WHEN (DocStatus IN (1200) And
MgrID = 56366 AND
ForwardCount = 0 )
THEN 1
ELSE 0
END), 0) "Accepted" ,
NVL(SUM(CASE
WHEN (DocStatus IN (1150,1155,1170,1182,1190) AND
DocOwner=56366 AND
MgrID = 0 )
THEN 1
ELSE 0
END), 0) "Waiting"
FROM DocInfo
WHERE ProjectId = 313 and
(DocOwner = 56366 or MgrID = 56366)
I need to change the counting to an exists clause so that i can show whether documents are available or not in each category.
Since this change is to improve the performance, running this as different queries is also not advisable. Please help me, I have ran out of my limited knowledge.
Sorry to miss the part which i have already tried.
I have changed the above query to a union with exists clause in each like below.
SELECT 'ForReview' AS A
FROM DUAL
WHERE EXISTS (SELECT NULL
FROM DocInfo
WHERE ProjectId = 313 and
(DocOwner = 56366 or MgrID = 56366) and
(DocStatus IN (1150,1155,1170,1182,1190) AND
DocOwner=56366 AND
ForwardCount=0))
UNION
SELECT 'Accepted' AS A
FROM DUAL
WHERE EXISTS (SELECT NULL
FROM DocInfo
WHERE ProjectId = 313 and
(DocOwner = 56366 or MgrID = 56366) and
(DocStatus IN (1200) And
MgrID = 56366 AND
ForwardCount = 0 ))
UNION
SELECT 'Waiting' AS A
FROM DUAL
WHERE EXISTS (SELECT NULL
FROM DocInfo
WHERE ProjectId = 313 and
(DocOwner = 56366 or MgrID = 56366) and
(DocStatus IN (1150,1155,1170,1182,1190) AND
DocOwner=56366 AND
MgrID = 0))
I have mentioned only 3 conditions, whereas my actual application has 8 different criteria to be added into this query. so when i have 8 Exists clauses, it runs internally as 8 different queries, and in effect it takes more time - single segment in the entire union query takes only 560 ms whereas all queries together takes around 7 seconds to generate the output.
Since my requirement is only to identify the Availability of any such record i do not want to navigate through the entire recordset and count it.
Is there anyway to optimize/rewrite this query
Thank You
"so when i have 8 Exists clauses, it runs internally as 8 different
queries, and in effect it takes more time - single segment in the
entire union query takes only 560 ms whereas all queries together
takes around 7 seconds to generate the output."
Surprise, surprise. Running what amounts to the same query eight times will not be faster than running that query once.
Now it is true that EXISTS can be faster, because it only needs to find a single row which matches the given criteria, rather than retrieving an entire data set. However you have just shifted the retrieved data into the WHERE clause so the database still has to do the same amount of work. In fact, it is apparently doing a lot more work, because 7s > (560ms * 8).
To solve your problem properly you need to understand how the database works and how to tune it. Find out more.
For a start, define a tuning goal. Your original query takes half a second to run: that's not lightning fast but it is pretty quick. Why is this a problem? How quickly do you want it to run?
Next, run an EXPLAIN PLAN. Is the query using indexes? How efficiently is its index usage> What percentage of the rows are being selected?
Now you also need to undersatnd your data. Is the selected data evenly distributed throughout the table or are there clusters? Do some projects, owners or managers have more records than others? How does that distribution effect performance?
Please bear in mind, tuning is a science and it is complicated: there are whole books on the subject and some people make very fine livings as performance troubleshooters. It requires a lot of information about your system, both knowledge of what your application does and low-level information on which activities your database is doing. We can help you in your quest to find a more performant solution but we cannot just look at a shonky query and tell you how to re-write so it runs quicker.