There are set of sql queries which I want to convert into procedure without intermediary tables creation and dropping. I have included the sample queries below. what could be best possible way for doing that? Since I am new to procedures can advice me how to do that.
Can we use cursor in procedures for doing that? In the last line of the query im joining 2 intermediary tables. instead of that Can we join two cursor? If so how can we do that? Is there any way to do that. Please suggest me.
CREATE TABLE A AS SELECT ID_LOAN FROM master_copy WHERE ZERO_BAL_CODE IN (1);
CREATE TABLE B AS SELECT master_copy.ID_LOAN,LOAN_AGE,master_copy.vintage,DELINQ_STATUS,ZERO_BAL_CODE,master_copy.ACTUAL_LOSS,current_upb FROM master_copy INNER JOIN A ON master_copy.ID_LOAN= A.ID_LOAN;
CREATE TABLE prepaidData AS SELECT ID_LOAN,max(to_number(DELINQ_STATUS)) as DELINQ_STATUS,max(loan_age) AS LOAN_AGE,max(ZERO_BAL_CODE) as ZERO_BAL_CODE,max(vintage) as vintage,
min(ACTUAL_LOSS) as actual_loss,MIN(NULLIF(current_upb,0)) as current_upb FROM B group by id_loan;
alter table prepaiddata add loan_type varchar2(255) default 'prepaid';
drop table a;
drop table b;
CREATE TABLE A AS SELECT ID_LOAN FROM master_copy WHERE ZERO_BAL_CODE IN (3);
CREATE TABLE B AS SELECT master_copy.ID_LOAN,LOAN_AGE,master_copy.vintage,DELINQ_STATUS,ZERO_BAL_CODE,master_copy.ACTUAL_LOSS,current_upb FROM master_copy INNER JOIN A ON master_copy.ID_LOAN= A.ID_LOAN;
CREATE TABLE DEFAULT_FORECLOSURE AS SELECT ID_LOAN,max(to_number(DELINQ_STATUS)) as DELINQ_STATUS,max(loan_age) AS LOAN_AGE,max(ZERO_BAL_CODE) as ZERO_BAL_CODE,max(vintage) as vintage,
min(ACTUAL_LOSS) as actual_loss,MIN(NULLIF(current_upb,0)) as current_upb FROM B group by id_loan;
alter table DEFAULT_FORECLOSURE add loan_type varchar2(255) default 'default_foreclosure';
drop table a;
drop table b;
create table aa_loan_type as(select * from prepaiddata union select * from DEFAULT_FORECLOSURE);
The best way to avoid intermediate tables is to use inline views. Your queries can be rewritten into this one SQL statement:
--aa_loan_type
--
--prepaidData
SELECT ID_LOAN,max(to_number(DELINQ_STATUS)) as DELINQ_STATUS,max(loan_age) AS LOAN_AGE,max(ZERO_BAL_CODE) as ZERO_BAL_CODE,max(vintage) as vintage, min(ACTUAL_LOSS) as actual_loss,MIN(NULLIF(current_upb,0)) as current_upb, 'PREPAID' LOAN_TYPE
FROM
(
--B
SELECT
master_copy.ID_LOAN,LOAN_AGE,master_copy.vintage,DELINQ_STATUS,ZERO_BAL_CODE,master_copy.ACTUAL_LOSS,current_upb
FROM master_copy
INNER JOIN
(
--A
SELECT ID_LOAN FROM master_copy WHERE ZERO_BAL_CODE IN (1)
) A
ON master_copy.ID_LOAN= A.ID_LOAN;
) B
GROUP BY ID_LOAN
union
--DEFAULT_FORECLOSURE
SELECT ID_LOAN,max(to_number(DELINQ_STATUS)) as DELINQ_STATUS,max(loan_age) AS LOAN_AGE,max(ZERO_BAL_CODE) as ZERO_BAL_CODE,max(vintage) as vintage, min(ACTUAL_LOSS) as actual_loss,MIN(NULLIF(current_upb,0)) as current_upb, 'default_foreclosure' loan_type
FROM
(
--B
SELECT master_copy.ID_LOAN,LOAN_AGE,master_copy.vintage,DELINQ_STATUS,ZERO_BAL_CODE,master_copy.ACTUAL_LOSS,current_upb
FROM master_copy
INNER JOIN
(
--A
SELECT ID_LOAN FROM master_copy WHERE ZERO_BAL_CODE IN (3)
) A
ON master_copy.ID_LOAN= A.ID_LOAN
) B
group by id_loan;
When built correctly, one large SQL statement is often much better than multiple small SQL statements. The overall code will be simpler (less objects, easier to debug in an IDE) and potentially much faster (no need to write data, gives the optimizer more chances to do something clever).
Building large SQL "correctly" is subjective, but it boils down to treating each inline view as a miniature program:
Keep each inline view simple, combine them in simple steps, and repeat.
Use good names and comments for each inline view. You'll probably want something better than "A" and "B".
Use Allman style bracket matching, with parentheses on each line. Inline views are important and deserve extra whitespace and aligned parentheses. Except for beginner developers, we don't need to align keywords like SELECT and FROM. We need to align important boundaries, like the parentheses of each inline view. This will help you debug by quickly highlighting and running code in an IDE.
Keep the subquery interfaces simple by using inline views instead of correlated subqueries or common table expressions. The beauty of inline views is their simplicity - relational data in, relational data out.
Related
I was working on task about optimization queries. One of the improvement ways was using WITH clause. I notice that it did very good job, and it lead to shorter time of execution, but i am not sure now, when should I use WITH clause and is there any risk of using it?
Here is one of the queries that I am working on :
WITH MY_TABLE AS
( SELECT PROD_KY,
sum(GROUPISPRIVATE) AS ISPRIVATE,
sum(GROUPISSHARED) AS ISSHARED
FROM
(
SELECT GRP_PROD_CUSTOMER.PROD_KY,
1 as ISPRIVATE,
0 as ISSHARED
FROM CUSTOMER
JOIN GRP_CUSTOMER ON GRP_CUSTOMER.CUST_KY = CUSTOMER.CUST_KY
JOIN GRP_PROD_CUSTOMER ON GRP_PROD_CUSTOMER.GRP_KY = GRP_CUSTOMER.GRP_KY
GROUP BY GRP_PROD_CUSTOMER.PROD_KY
)
GROUP BY PROD_KY
)
SELECT * FROM MY_TABLE;
is there any risk of using it?
Yes. Oracle may decide to materialize the subquery, which means writing its result set to disk and then reading it back (except it might not mean that in 12cR2 or later). That unexpected I/O could be a performance hit. Not always, and usually we can trust the optimizer to make the correct choice. However, Oracle has provided us with hints to tell the optimizer how to handle the result set: /*+ materialize */ to um materialize it and /*+ inline */ to keep it in memory.
I start with this potential downside because I think it's important to understand that the WITH clause is not a silver bullet and it won't improve every single query, and may even degrade performance. For instance I share the scepticism of the other commenters that the query you posted is in any way faster because you re-wrote it as a common table expression.
Generally, the use cases for the WITH clause are:
We want to use the result set from the subquery multiple times
with cte as
( select blah from meh )
select *
from t1
join t2 on t1.id = t2.id
where t1.col1 in ( select blah from cte )
and t2.col2 not in ( select blah from cte)
We want to be build a cascade of subqueries:
with cte as
( select id, blah from meh )
, cte2 as
( select t2.*, cte.blah
from cte
join t2 on t2.id = cte.id)
, cte3 as
( select t3.*, cte2.*
from cte2
join t3 on t3.col2 = cte2.something )
….
This second approach is beguiling and can be useful for implementing complex business logic in pure SQL. But it can lead to a procedural mindset and lose the power sets and joins. This too is a risk.
We want to use recursive WITH clause. This allows us to replace Oracle's own CONNECT BY syntax with a more standard approach. Find out more
In 12c and later we can write user-defined functions in the WITH clause. This is a powerful feature, especially for users who need to implement some logic in PL/SQL but only have SELECT access to the database. Find out more
For the record I have seen some very successful and highly performative uses of the second type of WITH clause. However I have also seen uses of WITH when it would have been just as easy to write an inline view. For instance, this is just using the WITH clause as syntactic sugar ...
with cte as
( select id, blah from meh )
select t2.*, cte.blah
from t2
join cte on cte.id = t2.id
… and would be clearer as ...
select t2.*, cte.blah
from t2
join ( select id, blah from meh ) cte on cte.id = t2.id
WITH clause is introduced in oracle to match SQL-99 standard.
The main purpose is to reduce the complexity and repetitive code.
Lets say you need to find the average salary of one department and then need to fetch all the department(d1) with more than average salary of that department(d1).
This can make multiple references to the subquery more efficient and readable.
The MATERIALIZE and INLINE optimizer hints can be used to influence the decision. The undocumented MATERIALIZE hint tells the optimizer to resolve the subquery as a global temporary table, while the INLINE hint tells it to process the query inline. Decision to use the hint is purely depends on logic that we are going to implement in query.
In oracle 12c, declaration of PL/SQL Block in WITH clause is introduced.
You must refer it from oracle documents.
Cheers!!
Your query is rather useless in terms of WITH statement (aka Common Table Expression, CTE)
Anyway, using the WITH clause brings several benefits:
The query is better readable (in my opinion)
You can use the same subquery several times in the main query. You can even cascade them.
Oracle can materialize the subquery, i.e. Oracle may create a temporary table and stores result of the subquery in it. This can give better performance.
The WITH clause may be processed as an inline view or resolved as a temporary table. The SQL WITH clause is very similar to the use of Global temporary tables. This technique is often used to improve query speed for complex subqueries and enables the Oracle optimizer to push the necessary predicates into the views.
The advantage of the latter is that repeated references to the subquery may be more efficient as the data is easily retrieved from the temporary table, rather than being requeried by each reference. You should assess the performance implications of the WITH clause on a case-by-case basis.
You can read more here:
http://www.dba-oracle.com/t_with_clause.htm
https://oracle-base.com/articles/misc/with-clause
one point to consider is, that different RDBMS handle the with clause - aka common table expressions (CTE) aka subquery factoring - differently:
Oracle may use a materialization or an inlining (as already explained in the answer provided by APC)
postgres always uses a materialization in releases up to 11 (so here a CTE is an optimization fence). In postgres 12 the behaviour changes and is similar to Oracles approach: https://info.crunchydata.com/blog/with-queries-present-future-common-table-expressions. You even have something that almost looks like a hint (though it is known that postgres does not use hints...)
in SQL Server currently a CTE is always inlined, as explained in https://erikdarlingdata.com/2019/08/what-would-materialized-ctes-look-like-in-sql-server/
So depending on the RDBMS you use and its version your mileage may vary.
Some tables' data need to be updated(or deleted , inserted) in my system.
But I want to know which data are updated,deleted and inserted.
So before the data are changed ,I will backup the table in different schema
just like this:
create table backup_table as select * from schema1.testtable
and after the data are changed,I want to find the difference between backup_table
and testtable ,and I want to save the difference into a table in the backup schema.
the sql I will run is like this:
CREATE TABLE TEST_COMPARE_RESULT
AS
SELECT 'BEFORE' AS STATUS, T1.*
FROM (
SELECT * FROM backup_table
MINUS
SELECT * FROM schema1.testtable
) T1
UNION ALL
SELECT 'AFTER' AS STATUS, T2.*
FROM (
SELECT * FROM schema1.testtable
MINUS
SELECT * FROM backup_table
) T2
What I am worried about is that I heared about that the minus operation will use
a lot of system resource.In my sysytem, some table size will be over 700M .So I want to
know how oracle will read the 700M data in memory (PGA??) or the temporary tablespace?
and How I should make sure that the resource are enough to to the compare operation?
Minus is indeed a resource intensive task. It need to read both tables and do sorts to compare the two tables. However, Oracle has advanced techniques to do this. It won't load the both tables in memory(SGA) if can't do it. It will use, yes, temporary space for sorts. But I would recommend you to have a try. Just run the query and see what happens. The database won't suffer and allways you can stop the execution of statement.
What you can do to improve the performance of the query:
First, if you have columns that you are sure that won't changed, don't include them.
So, is better to write:
select a, b from t1
minus
select a, b from t2
than using a select * from t, if there are more than these two columns, because the work is lesser.
Second, if the amount of data to compare si really big for your system(too small temp space), you should try to compare them on chunks:
select a, b from t1 where col between val1 and val2
minus
select a, b from t2 where col between val1 and val2
Sure, another possibility than minus is to have some log columns, let's say updated_date. selecting with where updated_date greater than start of process will show you updated records. But this depends on how you can alter the database model and etl code.
I am not an expert on TSQL but I wonder if such a thing is possible:
Imagine I have a select that joins to a table which is another query's result set:
SELECT *
FROM tProduct
JOIN (SELECT ProductId FROM ...... -- some other joins) tInlineQuey
ON tInlineQuery.ProductId = tProduct.Id
WHERE tInlineQuery. -- some econdition
Is it possible or meaningful to create an indedx on the tInlineQuery so that to apply a filtering on that result set can perform faster?
If so, how is it possible?
No, you could have suitable indexes on the objects within the subquery, but you can't add an temporary index to the subquery as you have it there. You can query hint the way in which the data is joined, e.g. nested loop, merge or hash join - but the optimizer tends to make the right decision.
An option to get that effect would be to select the results of that subquery into a temp table, and place an index on there, then join to that temp table.
To do this, you would need a stored procedure and to include the following code:
SELECT yourFields
INTO #TempTableName
JOIN SomeOtherTables
WHERE SomeField=SomeValue;
CREATE CLUSTERED INDEX SomeIndexName ON #TempTableName(SomeField,AnotherField);
SELECT *
FROM tProduct p
JOIN #TempTableName t ON t.SomeField = p.SomeField
...
DROP TABLE #TempTableName -- optional, the table will die when it goes out of scope at the end of the procedure.
The temp table index doesn't have to be clustered, thats down to your choice.
Andrew has an excellent answer, but if this is a subquery you will be using often, another option would be to create an indexed view. There are several good articles about that, including one I wrote at SqL Server Central titled On Indexes and Views .
Why inline views are used..??
There are many different reasons for using inline views. Some things can't be done without inline views, for example:
1) Filtering on the results of an analytic function:
select ename from
( select ename, rank() over (order by sal desc) rnk
from emp
)
where rnk < 4;
2) Using ROWNUM on ordered results:
select ename, ROWNUM from
( select ename
from emp
order by ename
);
Other times they just make it easier to write the SQL you want to write.
The inline view is a construct in Oracle SQL where you can place a query in the SQL FROM, clause, just as if the query was a table name.
Inline views provide
Bind variables can be introduced inside the statement to limit the data
Better control over the tuning
Visibility into the code
To get top N ordered rows.
SELECT name, salary,
FROM (SELECT name, salary
FROM emp
ORDER BY salary DESC)
WHERE rownum <= 10;
An inline view can be regarded as an intermediate result set that contributes to the required data set in some way. Sometimes it is entirely a matter of improving maintainability of the code, and sometimes it is logically neccessary.
From the Oracle Database Concepts document there are the inline view concept definition:
An inline view is not a schema object.
It is a subquery with an alias
(correlation name) that you can use
like a view within a SQL statement.
About the subqueries look in Using Subqueries from the Oracle SQL Reference manual. It have a very nice pedagogic information.
Anyway, today is preferred to use the Subquery Factoring Clause that is a more powerfull way of use inline views.
As an example of all together:
WITH
dept_costs AS (
SELECT department_name, SUM(salary) dept_total
FROM employees e, departments d
WHERE e.department_id = d.department_id
GROUP BY department_name),
avg_cost AS
SELECT * FROM dept_costs
WHERE dept_total >
(SELECT avg FROM (SELECT SUM(dept_total)/COUNT(*) avg
FROM dept_costs)
)
ORDER BY department_name;
There you can see one of all:
An inline view query: SELECT SUM...
A correlated subquery: SELECT avg FROM...
A subquery factoring: dept_costs AS (...
What are they used for?:
To avoid creating an intermediate view object: CREATE VIEW ...
To simplify some queries that a view cannot be helpfull. For instance, when the view need to filter from the main query.
You will often use inline views to break your query up into logical parts which helps both readability and makes writing more complex queries a bit easier.
Jva and Tony Andrews provided some good examples of simple cases where this is useful such as Top-N or Pagination queries where you may want to perform a query and order its results before using that as a part of a larger query which in turn might feed a query doing some other processing, where the logic for these individual queries would be difficult to achieve in a single query.
Another case they can be very useful is if you are writing a query that joins various tables together and want to perform aggregation on some of the tables, separating group functions and the processing into different inline views before performing the joins makes managing cardinality a lot easier. If you want some examples, I would be happy to provide them to make it more clear.
Factored subqueries (where you list your queries in the WITH clause at the start of the query) and inline views also often bring performance benefits. If you need to access the results of the subquery multiple times, you only need to run it once and it can be materialized as a Global Temporary Table (how the optimizer acts isn't totally black and white so I won't go into it here but you can do your own research - for example, see http://jonathanlewis.wordpress.com/2007/07/26/subquery-factoring-2/)
When joining across tables (as in the examples below), is there an efficiency difference between joining on the tables or joining subqueries containing only the needed columns?
In other words, is there a difference in efficiency between these two tables?
SELECT result
FROM result_tbl
JOIN test_tbl USING (test_id)
JOIN sample_tbl USING (sample_id)
JOIN (SELECT request_id
FROM request_tbl
WHERE request_status='A') USING(request_id)
vs
SELECT result
FROM (SELECT result, test_id FROM result_tbl)
JOIN (SELECT test_id, sample_id FROM test_tbl) USING(test_id)
JOIN (SELECT sample_id FROM sample_tbl) USING(sample_id)
JOIN (SELECT request_id
FROM request_tbl
WHERE request_status='A') USING(request_id)
The only way to find out for sure is to run both with tracing turned on and then look at the trace file. But in all probability they will be treated the same: the optimizer will merge all the inline views into the main statement and come up with the same query plan.
It doesn't matter. It may actually be WORSE since you are taking control away from the optimizer which generally knows best.
However, remember if you are doing a JOIN and only including a column from one of the tables that it is QUITE OFTEN better to re-write it as a series of EXISTS statements -- because that's what you really mean. JOINs (with some exceptions) will join matching rows which is a lot more work for the optimizer to do.
e.g.
SELECT t1.id1
FROM table1 t1
INNER JOIN table2 ON something = something
should almost always be
SELECT id1
FROM table1 t1
WHERE EXISTS( SELECT *
FROM table2
WHERE something = something )
For simple queries the optimizer may reduce the query plans into identical ones. Check it out on your DBMS.
Also this is a code smell and probably should be changed:
JOIN (SELECT request_id
FROM request_tbl
WHERE request_status='A')
to
SELECT result
FROM request
WHERE EXISTS(...)
AND request_status = 'A'
No difference.
You can tell by running EXPLAIN PLAN on both those statements - Oracle knows that all you want is the "result" column, so it only does the minimum necessary to get the data it needs - you should find that the plans will be identical.
The Oracle optimiser does, sometimes, "materialize" a subquery (i.e. run the subquery and keep the results in memory for later reuse), but this is rare and only occurs when the optimiser believes this will result in a performance improvement; in any case, Oracle will do this "materialization" whether you specified the columns in the subqueries or not.
Obviously if the only place the "results" column is stored is in the blocks (along with the rest of the data), Oracle has to visit those blocks - but it will only keep the relevant info (the "result" column and other relevant columns, e.g. "test_id") in memory when processing the query.