I have the following setup, which includes PARTITIONS. Is there a query I can use that will provide a count for each PARTITION within the table..
I prefer not to have a possible estimate by gathering statistics as opposed to the actual count(*). Note the PARTITION name can be renamed!!
Below is my test CASE. Thanks to all who answer.
ALTER SESSION SET NLS_DATE_FORMAT = 'MMDDYYYY HH24:MI:SS';
CREATE TABLE dts (
dt DATE
)
PARTITION BY RANGE (dt)
INTERVAL (NUMTODSINTERVAL(7,'DAY'))
(
PARTITION OLD_DATA values LESS THAN (TO_DATE('2022-01-01','YYYY-MM-DD'))
);
INSERT into dts(dt)
select to_date (
'01-08-2022','mm-dd-yyyy' ) +
( level / 24 ) dt
from dual
connect by level <= ( 24 + ( 24 *
(to_date('01-15-2022' ,'mm-dd-yyyy') - to_date('01-08-2022','mm-dd-yyyy') )
)
) ;
SELECT table_name,
partition_name,
num_rows
FROM user_tab_partitions
WHERE table_name not like 'BIN$%'
ORDER BY table_name, partition_name;
TABLE_NAME PARTITION_NAME NUM_ROWS
DTS OLD_DATA -
DTS SYS_P415755 -
DTS SYS_P415756 -
Try this one:
declare
c integer;
begin
for aPart in (select partition_name FROM user_tab_partitions where table_name = 'DTS') loop
execute immediate 'select count(*) from DTS PARTITION ('||aPart.partition_name||')' INTO c;
DBMS_OUTPUT.PUT_LINE(aPart.partition_name || ' ' || c || ' rows');
end loop;
end;
select table_name ,Partition_name, to_number(extractvalue(xmltype(dbms_xmlgen.getxml('select /*+ parallel(a,8) */
count(*) c from '||table_name||' partition ('||partition_name||') a ')),'/ROWSET/ROW/C')) as count
from user_tab_partitions
TABLE_NAME PARTITION_NAME COUNT
DTS OLD_DATA 0
DTS SYS_P415799 167
DTS SYS_P415800 25
Oracle provides a handy function PMARKER exact for this purpose
SELECT DBMS_MVIEW.PMARKER(p.rowid) PMARKER, count(*) cnt, min(dt), max(dt)
from dts p
group by DBMS_MVIEW.PMARKER(p.rowid)
order by 1;
PMARKER CNT MIN(DT) MAX(DT)
---------- ---------- ------------------- -------------------
74312 167 08.01.2022 01:00:00 14.01.2022 23:00:00
74313 25 15.01.2022 00:00:00 16.01.2022 00:00:00
Note that you need not know the partition name, the partition key column value lets you access the partition using the partition extended names:
Example for the first partition
select count(*) from dts partition for (DATE'2022-01-08');
COUNT(*)
----------
167
You can rely on optimizer statistics for a perfect count, as long as you're using the default sample size and algorithm.
begin
dbms_stats.gather_table_stats
(
ownname => user,
tabname => 'DTS',
estimate_percent => dbms_stats.auto_sample_size
);
end;
/
If you run the above PL/SQL block, your original query against USER_TAB_PARTITIONS will return the correct NUM_ROWS. Since version 11g, Oracle scans the entire table to calculate statistics. While it uses an approximation for counting things like non-distinct values and histograms, it's trivial for the algorithm to get a completely accurate row count.
The manual is not super clear about this behavior, but you can put it together from the manual and other articles that discuss how the new algorithm works. From the "Gathering Optmizer Statistics" chapter of the "SQL Tuning Guide":
To maximize performance gains while achieving necessary statistical
accuracy, Oracle recommends that the ESTIMATE_PERCENT parameter use
the default setting of DBMS_STATS.AUTO_SAMPLE_SIZE. In this case,
Oracle Database chooses the sample size automatically. This setting
enables the use of the following:
A hash-based algorithm that is much faster than sampling
This algorithm reads all rows and produces statistics that are nearly
as accurate as statistics from a 100% sample. The statistics computed
using this technique are deterministic.
Most likely, you don't even need to specify the ESTIMATE_PERCENT => DBMS_STATS.AUTO_SAMPLE_SIZE argument. It is extremely unlikely for someone to set that preference for a table or the system. You can use the below query to see how your statistics are typically gathered. Most likely the query will return "DBMS_STATS.AUTO_SAMPLE_SIZE":
select dbms_stats.get_prefs(pname => 'ESTIMATE_PERCENT', ownname => user, tabname => 'DTS')
from dual;
Related
My understanding of hard parse in Oracle is, when a SQL statement is processed, if no matching text is found it will be hard parsed. I am working on an assignment, where
they fire multiple SQL's from a service as below(around 50 variations executed multiple times)
select * from emp where emp_no in (:V1,:V2);
select * from emp where emp_no in (:V1,:V2,:V3);
select * from emp where emp_no in (:V1,:V2,V3,:V4);
This in turn generates multiple SQL_ID's with same PHV. My question is, does this create hard parse of each statement? Reason is
AWR report shows that only .10 seconds of total elapsed time of SQL is hard parsed time
V$SQL/v$SQLSTAS shows only a fraction of elapsed time is hard parse.
Yes.
When you submit a statement to the database, it hashes the text to produce a SQL_id. Meaning you have a cursor for each => there must have been a hard parse.
You can verify this by querying v$sql. This will have an entry for each version. You can also see the number of hard parses by checking the parse count (hard) stat:
var v1 number;
var v2 number;
var v3 number;
alter system flush shared_pool;
select sql_id, executions
from v$sql
where sql_text like 'select * from hr.employees%';
no rows selected
select n.display_name, s.value
from v$mystat s
join v$statname n
on s.statistic# = n.statistic#
where n.name like 'parse count%';
DISPLAY_NAME VALUE
parse count (total) 1682
parse count (hard) 449
parse count (failures) 4
parse count (describe) 0
select * from hr.employees
where employee_id in ( :v1 );
select * from hr.employees
where employee_id in ( :v1, :v2 );
select * from hr.employees
where employee_id in ( :v1, :v2, :v3 );
select sql_id, executions
from v$sql
where sql_text like 'select * from hr.employees%';
SQL_ID EXECUTIONS
63dqkasu1w4du 1
4kr9jqam2p6dw 1
8gbwr8cry9d84 1
select n.display_name, s.value
from v$mystat s
join v$statname n
on s.statistic# = n.statistic#
where n.name like 'parse count%';
DISPLAY_NAME VALUE
parse count (total) 1697
parse count (hard) 457
parse count (failures) 4
parse count (describe) 0
So what happens if we run the statements again?
Let's see:
select * from hr.employees
where employee_id in ( :v1 );
select * from hr.employees
where employee_id in ( :v1, :v2 );
select * from hr.employees
where employee_id in ( :v1, :v2, :v3 );
select sql_id, executions
from v$sql
where sql_text like 'select * from hr.employees%';
SQL_ID EXECUTIONS
63dqkasu1w4du 2
4kr9jqam2p6dw 2
8gbwr8cry9d84 2
select n.display_name, s.value
from v$mystat s
join v$statname n
on s.statistic# = n.statistic#
where n.name like 'parse count%';
DISPLAY_NAME VALUE
parse count (total) 1707
parse count (hard) 457
parse count (failures) 4
parse count (describe) 0
So:
The execution count for each SQL_id increased by 1
The count of hard parses stayed the same (457)
=> no new hard parses!
This goes part of the way to explaining why you are seeing such small values for hard parsing in AWR etc. Hopefully you're parsing each variation once, then executing it many, many times.
Also, while relatively expensive and something to avoid, hard parsing is still fast in absolute terms. Particularly for simple statements such as the above.
I have an interval partitioned table. I'd like to create a local index unusable by default to make all inserts work fast. I want manually turn indexes usable after insert. Currently every insert which creates new partition makes correspondend local index usable. How can I prevent it?
UPDATE: I'm working with Oracle 11.2
create table T (
part integer
,nm number
)
partition by range (part)
interval (1)
(partition empty values less than (1));
create index t_loc_idx on T (part, nm) local unusable
insert into T
select 1, rownum from user_tables
insert into T
select 2, rownum from user_tables
select index_name, partition_name, status from user_ind_partitions
where index_name = 'T_LOC_IDX'
What we can observe:
INDEX_NAME PARTITION_NAME STATUS
T_LOC_IDX EMPTY UNUSABLE
T_LOC_IDX SYS_P8744 USABLE
T_LOC_IDX SYS_P8745 USABLE
Keep interval partition local indexes unusable by creating a partial index and setting indexing off:
create table T (
part integer
,nm number
)
partition by range (part)
interval (1)
(partition empty values less than (1))
INDEXING OFF /* <-- New */;
create index t_loc_idx on T (part, nm) INDEXING PARTIAL /* <-- New */ local unusable ;
insert into T
select 1, rownum from user_tables where rownum <= 10;
insert into T
select 2, rownum from user_tables where rownum <= 10;
select index_name, partition_name, status from user_ind_partitions
where index_name = 'T_LOC_IDX';
New results:
INDEX_NAME PARTITION_NAME STATUS
========== ============== ======
T_LOC_IDX EMPTY UNUSABLE
T_LOC_IDX SYS_P643 UNUSABLE
T_LOC_IDX SYS_P644 UNUSABLE
This query will give me if a compression has been marked for compression
Select *
From All_Tab_Partitions
It came up in discussion that a Partition marked for compression can actually contain uncompressed data if it is loaded with FAST_LOAD.
I want to only issue compression commands for uncompressed partitions.
How can I find out if the data inside a partition is compressed or not?
Compression may be enabled on a partition but it is only used if data was created from a direct-path insert. Use sampling and the function
DBMS_COMPRESSION.GET_COMPRESSION_TYPE to estimate the percent of rows
compressed per partition.
Schema
create table compression_test
(
a number,
b varchar2(100)
) partition by range(a)
(
partition p_all_compressed values less than (2) compress,
partition p_none_compressed values less than (3) compress,
partition p_half_compressed values less than (4) compress
);
insert /*+ append */ into compression_test partition (p_all_compressed)
select 1, '0123456789' from dual connect by level <= 100000;
commit;
insert into compression_test partition (p_none_compressed)
select 2, '0123456789' from dual connect by level <= 100000;
commit;
insert /*+ append */ into compression_test partition (p_half_compressed)
select 3, '0123456789' from dual connect by level <= 50000;
commit;
insert into compression_test partition (p_half_compressed)
select 3, '0123456789' from dual connect by level <= 50000;
commit;
Estimate Code
--Find percent compressed from sampling each partition.
declare
v_percent_compressed number;
begin
--Loop through partitions.
for partitions in
(
select table_owner, table_name, partition_name
from dba_tab_partitions
--Enter owner and table_name here.
where table_owner = user
and table_name = 'COMPRESSION_TEST'
) loop
--Dynamic SQL to sample a partition and test for compression.
execute immediate '
select
round(sum(is_compressed)/count(is_compressed) * 100) percent_compressed
from
(
--Compression status of sampled rows.
--
--Numbers are based on constants from:
--docs.oracle.com/cd/E16655_01/appdev.121/e17602/d_compress.htm
--Assumption: Only basic compression is used.
--Assumption: Partitions are large enough for 0.1% sample size.
select
case dbms_compression.get_compression_type(
user,
'''||partitions.table_name||''',
rowid,
'''||partitions.partition_name||'''
)
when 4096 then 1
when 1 then 0
end is_compressed
from '||partitions.table_owner||'.'||partitions.table_name||'
partition ('||partitions.partition_name||') sample (0.1)
)
' into v_percent_compressed;
dbms_output.put_line(rpad(partitions.partition_name||':', 31, ' ')||
lpad(v_percent_compressed, 3, ' '));
end loop;
end;
/
Sample Output
P_ALL_COMPRESSED: 100
P_HALF_COMPRESSED: 55
P_NONE_COMPRESSED: 0
Wondering if someone can help point me in the right direction with this challenge, or tell me I'm crazy for trying this via sql. If sql would be too challenging, are there any free or inexpensive tools that would help me automate this?
I'm working on testing some data between an old and new Oracle database. What I'd like to do is be able to dynamically generate this query for all tables in a schema.
Select Column_1, Column_2 FROM Table_1
MINUS
Select Column_1, Column_2 FROM Table_1#"OLD_SERVER"
One catch is that the columns selected for each table should only be columns that do not begin with 'ETL' since those are expected to change with the migration.
To keep this dynamic, can I use the all_tab_columns to loop through each table?
So for a simplified example, let's say this query returned the following results, and you can expect the results from ALL_TAB_COLUMNS to be identical between the OLD and NEW database:
select TABLE_NAME, COLUMN_NAME from ALL_TAB_COLUMNS where owner = 'OWNER1'
TABLE_NAME, COLUMN_NAME
-----------------------
TABLE1, COLUMN_1
TABLE1, COLUMN_2
TABLE1, ETLCOLUMN_3
TABLE2, COLUMN_A
TABLE2, COLUMN_B
TABLE2, ETLCOLUMN_C
How would I write a query that would run a minus between the same table and columns (that do not begin with ETL) on the old and new database, and output the results along with the table name and the date ran, and then loop through to the next table and do the same thing?
First - check out this:
http://docs.oracle.com/cd/E11882_01/server.112/e41481/spa_upgrade.htm#RATUG210
Second - you would like to write a query that issues a query - The problem is that in user_tab_columns each column is a row.
for doing that I would recommend you reading this : http://www.dba-oracle.com/t_converting_rows_columns.htm
The source table for you is USER_TAB_COLUMNS, and when running the query you can add a where that says "where column_name not like 'ETL%' etc.
After that - the query would look something like:
select 'select '
|| listagg..... (from the link) || 'from table name' sql
from user_tab_columns
where column_name not like 'ETL%'
and table_name = 'table name'
group by table_name
and btw - you're not crazy - before changing a system you need to be able to sign the upgrade will succeed - this is the only way to do it.
btw - if you'll describe in more depth the system and the upgrade - I'm sure the community will be able to help you find ways to test it in more depth, and will point you out to things to test.
Testing only the output is not enough in many cases....
GOOD LUCK!
This testing can automated with SQL and PL/SQL. You're not crazy for doing this. Comparison systems like this can be incredibly helpful for testing changes to complex systems. It's not as good as automated unit tests but it can significantly enhance the typical database testing.
The code below is a fully working example. But in the real world there are many gotchas that could easily take several days to resolve. For example, dealing with CLOBs, large tables, timestamps and sequence-based values, etc.
Sample schemas and data differences
create user schema1 identified by schema1;
create user schema2 identified by schema2;
alter user schema1 quota unlimited on users;
alter user schema2 quota unlimited on users;
--Data in 1, not 2.
create table schema1.table1 as select 1 a, 1 b from dual;
create table schema2.table1(a number, b number);
--Data in 2, not 1.
create table schema1.table2(a number, b number);
create table schema2.table2 as select 1 a, 1 b from dual;
--Same data in both, excluding unused column.
create table schema1.table3 as select 1 a, 1 b, 'asdf' ETL_c from dual;
create table schema2.table3 as select 1 a, 1 b, 'fdsa' ETL_c from dual;
--Table DDL difference.
create table schema1.table4(a number);
create table schema2.table4(b number);
--Privileges can be tricky.
grant select on dba_tab_columns to <your schema>;
Procedure to print differences script
create or replace procedure print_differences(
p_old_schema in varchar2,
p_new_schema in varchar2) authid current_user
is
v_table_index number := 0;
v_row_count number;
begin
--Print header information.
dbms_output.put_line('--Comparison between '||p_old_schema||' and '||
p_new_schema||', at '||to_char(sysdate, 'YYYY-MM-DD HH24:MI')||'.'||chr(10));
--Create a SQL statement to return the differences for each table.
for differences in
(
--Return number of differences and SQL statements to view them.
select
'
with old_table as (select '||column_list||' from '||p_old_schema||'.'||table_name||')
, new_table as (select '||column_list||' from '||p_new_schema||'.'||table_name||')
select * from
(
select ''OLD'' old_or_new, old_table.* from old_table minus
select ''OLD'' old_or_new, new_table.* from new_table
)
union all
select * from
(
select ''NEW'' old_or_new, new_table.* from new_table minus
select ''NEW'' old_or_new, old_table.* from old_table
)
' difference_sql, table_name
from
(
select table_name
,listagg(column_name, ',') within group (order by column_id) column_list
from dba_tab_columns
where owner = p_old_schema
and column_name not like 'ETL%'
group by table_name
) column_lists
) loop
begin
--Print table information:
v_table_index := v_table_index+1;
dbms_output.put_line(chr(10)||'--'||lpad(v_table_index, 3, '0')||': '||differences.table_name);
--Count differences.
execute immediate 'select count(*) from ('||differences.difference_sql||')' into v_row_count;
--Print SQL statements to investigate differences.
if v_row_count = 0 then
dbms_output.put_line('--No differences.');
else
dbms_output.put_line('--Differences: '||v_row_count);
dbms_output.put_line(differences.difference_sql||';');
end if;
exception when others then
dbms_output.put_line('/*Error with this statement, possible DDL difference: '
||differences.difference_sql||dbms_utility.format_error_stack||
dbms_utility.format_error_backtrace||'*/');
end;
end loop;
end;
/
Running the procedure
begin
print_differences('SCHEMA1', 'SCHEMA2');
end;
/
Sample output
The procedure does not output the actual differences. If there are differences, it outputs a script that will display the differences. With a decent IDE this will be a much better way to view the data, and it also helps to further analyze the differences.
--Comparison between SCHEMA1 and SCHEMA2, at 2014-03-28 23:44.
--001: TABLE1
--Differences: 1
with old_table as (select A,B from SCHEMA1.TABLE1)
, new_table as (select A,B from SCHEMA2.TABLE1)
select * from
(
select 'OLD' old_or_new, old_table.* from old_table minus
select 'OLD' old_or_new, new_table.* from new_table
)
union all
select * from
(
select 'NEW' old_or_new, new_table.* from new_table minus
select 'NEW' old_or_new, old_table.* from old_table
)
;
--002: TABLE2
--Differences: 1
with old_table as (select A,B from SCHEMA1.TABLE2)
, new_table as (select A,B from SCHEMA2.TABLE2)
select * from
(
select 'OLD' old_or_new, old_table.* from old_table minus
select 'OLD' old_or_new, new_table.* from new_table
)
union all
select * from
(
select 'NEW' old_or_new, new_table.* from new_table minus
select 'NEW' old_or_new, old_table.* from old_table
)
;
--003: TABLE3
--No differences.
--004: TABLE4
/*Error with this statement, possible DDL difference:
with old_table as (select A from SCHEMA1.TABLE4)
, new_table as (select A from SCHEMA2.TABLE4)
select * from
(
select 'OLD' old_or_new, old_table.* from old_table minus
select 'OLD' old_or_new, new_table.* from new_table
)
union all
select * from
(
select 'NEW' old_or_new, new_table.* from new_table minus
select 'NEW' old_or_new, old_table.* from old_table
)
ORA-06575: Package or function A is in an invalid state
ORA-06512: at "JHELLER.PRINT_DIFFERENCES", line 48
*/
I am using an oracle 11 table with interval partitioning and list subpartitioning like this (simplified):
CREATE TABLE LOG
(
ID NUMBER(15, 0) NOT NULL PRIMARY KEY
, MSG_TIME DATE NOT NULL
, MSG_NR VARCHAR2(16 BYTE)
) PARTITION BY RANGE (MSG_TIME) INTERVAL (NUMTOYMINTERVAL (1,'MONTH'))
SUBPARTITION BY LIST (MSG_NR)
SUBPARTITION TEMPLATE (
SUBPARTITION login VALUES ('FOO')
, SUBPARTITION others VALUES (DEFAULT)
)
(PARTITION oldvalues VALUES LESS THAN (TO_DATE('01-01-2010','DD-MM-YYYY')));
How do I drop a specific subpartitition for a specific month without knowing the (system generated) name of the subpartition? There is a syntax "alter table ... drop subpartition for (subpartition_key_value , ...)" but I don't see a way to specify the month for which I am deleting the subpartition. The partition administration guide does not give any examples, either. 8-}
You can use the metadata tables to get the specific subpartition name:
SQL> insert into log values (1, sysdate, 'FOO');
1 row(s) inserted.
SQL> SELECT p.partition_name, s.subpartition_name, p.high_value, s.high_value
2 FROM user_tab_partitions p
3 JOIN
4 user_tab_subpartitions s
5 ON s.table_name = p.table_name
6 AND s.partition_name = p.partition_name
7 AND p.table_name = 'LOG';
PARTITION_NAME SUBPARTITION_NAME HIGH_VALUE HIGH_VALUE
--------------- ------------------ ------------ ----------
OLDVALUES OLDVALUES_OTHERS 2010-01-01 DEFAULT
OLDVALUES OLDVALUES_LOGIN 2010-01-01 'FOO'
SYS_P469754 SYS_SUBP469753 2012-10-01 DEFAULT
SYS_P469754 SYS_SUBP469752 2012-10-01 'FOO'
SQL> alter table log drop subpartition SYS_SUBP469752;
Table altered.
If you want to drop a partition dynamically, it can be tricky to find it with the ALL_TAB_SUBPARTITIONS view because the HIGH_VALUE column may not be simple to query. In that case you could use DBMS_ROWID to find the subpartition object_id of a given row:
SQL> insert into log values (4, sysdate, 'FOO');
1 row(s) inserted.
SQL> DECLARE
2 l_rowid_in ROWID;
3 l_rowid_type NUMBER;
4 l_object_number NUMBER;
5 l_relative_fno NUMBER;
6 l_block_number NUMBER;
7 l_row_number NUMBER;
8 BEGIN
9 SELECT rowid INTO l_rowid_in FROM log WHERE id = 4;
10 dbms_rowid.rowid_info(rowid_in =>l_rowid_in ,
11 rowid_type =>l_rowid_type ,
12 object_number =>l_object_number,
13 relative_fno =>l_relative_fno ,
14 block_number =>l_block_number ,
15 row_number =>l_row_number );
16 dbms_output.put_line('object_number ='||l_object_number);
17 END;
18 /
object_number =15838049
SQL> select object_name, subobject_name, object_type
2 from all_objects where object_id = '15838049';
OBJECT_NAME SUBOBJECT_NAME OBJECT_TYPE
--------------- --------------- ------------------
LOG SYS_SUBP469757 TABLE SUBPARTITION
As it turns out, the "subpartition for" syntax does indeed work, though that seems to be a secret Oracle does not want to tell you about. :-)
ALTER TABLE TB_LOG_MESSAGE DROP SUBPARTITION FOR
(TO_DATE('01.02.2010','DD.MM.YYYY'), 'FOO')
This deletes the subpartition that would contain MSG_TIME 2010/02/01 and MSG_NR FOO. (It is not necessary that there is an actual row with this exact MSG_TIME and MSG_NR. It throws an error if there is no such subpartition, though.)
Thanks for the post - it was very useful for me.
One observation though on the above script to identify the partition and delete it:
The object_id returned by dbms_rowid.rowid_info is not the object_id of the all_objects table. It is actually the data_object_id. It is observed that usually these ids match. However, after truncating the partitioned table several times, these ids diverged in my database. Hence it might be reasonable to instead use the data_object_id to find out the name of the partition:
select object_name, subobject_name, object_type
from all_objects where data_object_id = '15838049';
From the table description of ALL_OBJECTS:
OBJECT_ID Object number of the object
DATA_OBJECT_ID Object number of the segment which contains the object
http://docs.oracle.com/cd/B19306_01/appdev.102/b14258/d_rowid.htm
In the sample code provided in the above link, DBMS_ROWID.ROWID_OBJECT(row_id) is used instead to derive the same information that is given by dbms_rowid.rowid_info. However, the documentation around this sample mentions that it is a data object number from the ROWID.
Examples
This example returns the ROWID for a row in the EMP table, extracts
the data object number from the ROWID, using the ROWID_OBJECT function
in the DBMS_ROWID package, then displays the object number:
DECLARE object_no INTEGER; row_id ROWID; ... BEGIN
SELECT ROWID INTO row_id FROM emp
WHERE empno = 7499; object_no := DBMS_ROWID.ROWID_OBJECT(row_id); DBMS_OUTPUT.PUT_LINE('The obj. # is
'|| object_no); ...