I am doing a simple Select col1, col2, col22 from Table1 order by col1 and the same Select statement in Table2. Select col1, col2, col22 from Table2 order by col1.
I use Pentaho ETL tool to replicate data from Oracle 19c to SQL Server. Reading from Table1 is much much slower than reading from Table2. Both have almost the same number for columns and almost the same number for rows. Both exist in the same schema. Table1 is being read at 10 rows per sec while Table2 is being read at 1000 rows a sec.
What can cause this slowness?
Are the indexes the same on the two tables? It's possible Oracle is using a fast full index scan (like a skinny version of the table) if an index covers all the relevant columns in one table, or may be using a full index scan to pre-sort by COL1. Check the execution plans to make sure the statements are using the same access methods:
explain plan for select ...;
select * from table(dbms_xplan.display);
Are the table segment sizes the same? Although the data could be the same, occasionally a table can have a lot of wasted space. For example, if the table used to contain a billion rows, and then 99.9% of the rows were deleted, but the table was never rebuilt. Compare the segment sizes with a query like this:
select segment_name, sum(bytes)/1024/1024 mb
from all_segments
where segment_name in ('TABLE1', 'TABLE2')
It depends on many factors.
The first things I would check are the table indexes:
select
uic.table_name,
uic.index_name,
utc.column_name
from USER_TAB_COLUMNS UTC,
USER_IND_COLUMNS UIC
where utc.table_name = uic.table_name
and utc.column_name = uic.column_name
and utc.table_name in ('TABLE1', 'TABLE2')
order by 1, 2, 3;
Related
I’m joining 2 tables. Pruning is happening on table 1 but not on table 2 even though there is an outer join.
Example:
select *
from table1 t1, table2 t2
where t1.sk in (select sk from filter_table)
and t2.sk(+) = t1.sk
When I check the plan and noticed t1 table has KEY partition scan, but T2 is scanning all the partition(~4500). so the query is taking more than 4hrs just to pull 50 recs.
Is there any way to force the pruning on table 2 as well?
I am using Oracle 11g.
Without more data it is hard to say for sure what the problem can be. I have rewritten the query for clarity and with a simple test schema I get pruning for both tables with Oracle 12c (I don't have 11g handy). The first with key and the second with Bloom Filter (:BF0000 in the plan).
select t1.*, t2.*
from filter_table ft
join table1 t1 on t1.sk = ft.sk
left outer join table2 t2 on t2.sk = ft.sk;
Be sure to gather statistics for all three tables! Often when the optimizer seems to be stupid it is because the statistics are missing or not up to date.
I have a very large table in Hive, from which we need to load a subset of partitions. It looks something like this:
CREATE EXTERNAL TABLE table1 (
col1 STRING
) PARTITIONED BY (p_key STRING);
I can load specific partitions like this:
SELECT * FROM table1 WHERE p_key = 'x';
with p_key being the key on which table1 is partitioned. If I hardcode it directly in the WHERE clause, it's all good. However, I have another query which calculates which partitions I need. It's more complicated than this, but let's define it simply as:
SELECT DISTINCT p_key FROM table2;
So now I should be able to construct a dirty query like this:
SELECT * FROM table1
WHERE p_key IN (SELECT DISTINCT p_key FROM table2);
Or written as an inner join:
SELECT t1.* FROM table1 t1
JOIN (SELECT DISTINCT p_key FROM table2) t2 ON t1.p_key = t2.p_key
However, when I run this, it takes enough time to let me believe it's doing a full table scan. In the explain for the above queries, I can also see the result of the DISTINCT operation are used in the reducer, not the mapper, meaning it would be impossible for the mapper to know which partitions should be loaded or not. Granted, I'm not fully familiar with Hive explain output, so I may be overlooking something.
I found this page: MapJoin and Partition Pruning on the Hive wiki and the corrosponding ticket indicates it was released in version 0.11.0. So I should have it.
Is it possible to do this? If so, how?
I'm not sure how to help with MapJoin, but in the worst case you could dynamically create second query with something like:
SELECT concat('SELECT * FROM table1 WHERE p_key IN (',
concat_ws(',',collect_set(p_key)),
')')
FROM table2;
then execute obtained result. With this, query processor should be able to prune unneeded partitions.
I am constructing a test environment. Oracle 11g is my database. My goal is to place 80 million records in this database. I will start with 1 million records, which will be loaded into the partitioned table. Is there a way to duplicate the initial partition to create 80 partitions for a grand total of 80Meg records. The constraint is this process should take no longer than two hours to generate 80 million records.
After inserting the first partition, follow this principle:
INSERT INTO my_table (partition_column, col1, col2, col3, ...)
SELECT level, col1, col2, col3, ...
FROM my_table
CONNECT BY level < 80
The partition_column is just an assumption about your actual partitioning. You might have to change some values in the SELECT list for the new records to be put into different partitions. It helps to turn off constraints and indexes during this insert.
I currently have a Hive table that has 1.5 billion rows. I would like to create a smaller table (using the same table schema) with about 1 million rows from the original table. Ideally, the new rows would be randomly sampled from the original table, but getting the top 1M or bottom 1M of the original table would be ok, too. How would I do this?
As climbage suggested earlier, you could probably best use Hive's built-in sampling methods.
INSERT OVERWRITE TABLE my_table_sample
SELECT * FROM my_table
TABLESAMPLE (1m ROWS) t;
This syntax was introduced in Hive 0.11. If you are running an older version of Hive, you'll be confined to using the PERCENT syntax like so.
INSERT OVERWRITE TABLE my_table_sample
SELECT * FROM my_table
TABLESAMPLE (1 PERCENT) t;
You can change the percentage to match you specific sample size requirements.
You can define a new table with the same schema as your original table.
Then use INSERT OVERWRITE TABLE <tablename> <select statement>
The SELECT statement will need to query your original table, use LIMIT to only get 1M results.
This query will pull out top 1M rows and overwrite them in a new table.
CREATE TABLE new_table_name AS
SELECT col1, col2, col3, ....
FROM original_table
WHERE (if you want to put any condition) limit 100000;
I have a problem in indexing in Oracle. Will try to explain my problem with an instance as follows.
I have a table TABLE1 with columns A,B,C,D
another table TABLE2 with columns A,B,C,E,F,H
I have created Indexes for TABLE1
IX_1 A
IX_2 A,B
IX_3 A,C
IX_4 A,B,C
I have created Indexes for TABLE1
IY_1 A,B,C
IY_2 A
when i gave query similar to this
SELECT * FROM TABLE1 T1,TABLE2 T2
WHERE T1.A=T2.A
When i give Explain Plan i got its not getting IX_1 nor IY_2
Its taking IX_4 nor IY_1
why this is not picking right index?
EDITED:
Can anyone help me to know difference between INDEX RANGE SCAN,INDEX UNIQUE SCAN, INDEX SKIP SCAN
I guess SKIP SCAN means when a column is skipped in Composite Index by Oracle
what about others i dont have idea!
The best benefit of indexes is that you can select a few rows from a table without scanning the entire table.
If you ask for too many rows(let's say 30% - depends of many things) the engine will prefer to scan the entire table for those rows.
That's because reading a row using an index is gets an overhead : reading some index blocks, and after that reading table blocks.
In your case, in order to join tables T1 and T2, Oracle needs all the rows from those table. Reading(full) the index will be an unsefull operation, adding unnecesary cost.
UPDATE: A step forward: if you run:
SELECT T1.B, T2.B FROM TABLE1 T1,TABLE2 T2
WHERE T1.A=T2.A
Oracle probably will use the indexes(IX2, IY2), because it does not need to read anything from table, because the values T1.B, T2.B, are in indexes.