We have a requirement to get the records from the below tables with mentioned matching condition & limit the count of UNIQUE COL3 to 20K.
We have achieved this by using DENSE_RANK() logic but we observed extremely slowness in performance when we have implemented in live system.
For processing 20K UNIQUE COL3 & 60K total records, it is taking 14-15 hours to complete (The process is running over a daily JOB & select FROM view & performing DML operation to few tables based on View records).
SELECT COUNT (distinct COL3) CNT1,COUNT(1) CNT2 FROM VW_ENTP;
--20,000 60,000
Without the DENSE_RAN() logic it is finishing very fast but with DENSE_RAN() it extremely slow.
Request your help to share any recommendation to IMPROVE QUERY PERFORMANCE or any alternative approach. Thanks in advance !
Sample View Code
CREATE OR REPLACE VIEW VW_ENTP(COL1, COL2, COL3, COL4, COL5, COL6, COL7, COL8, COL9, COL10) AS
SELECT
COL1, COL2, COL3, COL4, COL5, COL6, COL7, COL8, COL9, COL10
FROM
(
SELECT
NP.NFCD COL1,
D.CLO2 COL2,
NP.COL3 COL3,
E.COL4 COL4,
EDT.COL5,
EDT.COL6,
NP. COL7,
NP. COL8,
EDT.COL9 COL9,
NP.COL10,
DENSE_RANK() OVER (ORDER BY NP.COL3) RANK
FROM ENP NP,EDTS EDT,EM E,EDT D,BDT BD
WHERE EDT.COL2=D.COL2
AND E.COL3=NP.COL3
AND E.COL7=NP.COL7
AND E.COL8=NP.COL8
AND E.APP=NP.APP
AND NP.STATUS='P'
AND NP.NFCD=D.COL1
AND E.SUSP !='YES'
AND NP.APP='EDOC'
AND TRUNC(NP.RC_DATE)<=TRUNC(BD.LAST_RUN_DATE)
)
WHERE RANK<=20000;
Note : The above used columns are indexed.
Generally if any join condition is missed it leads to poor performance. Also you're joining few other tables and any of the tables can be causing the performance issue. Suggestion for them can be given only after having look at those tables.
To confirm if dense rank is causing the issue, you can remove it and execute the select statement. If it takes less time, then you can conclude dense rank is causing the issue. But I highly suspect it'll happen.
You can achieve this by row_number() function only:
CREATE OR REPLACE VIEW VW_ENTP(COL1, COL2, COL3, COL4, COL5, COL6, COL7, COL8, COL9, COL10) AS
SELECT
COL1, COL2, COL3, COL4, COL5, COL6, COL7, COL8, COL9, COL10
FROM
(select a.*, row_number() over (order by col3) rn from (
SELECT
NP.NFCD COL1,
D.CLO2 COL2,
NP.COL3 COL3,
E.COL4 COL4,
EDT.COL5,
EDT.COL6,
NP. COL7,
NP. COL8,
EDT.COL9 COL9,
NP.COL10,
row_number() OVER (partition by np.col3 ORDER BY NP.COL3) RANK
FROM ENP NP,EDTS EDT,EM E,EDT D,BDT BD
WHERE EDT.COL2=D.COL2
AND E.COL3=NP.COL3
AND E.COL7=NP.COL7
AND E.COL8=NP.COL8
AND E.APP=NP.APP
AND NP.STATUS='P'
AND NP.NFCD=D.COL1
AND E.SUSP !='YES'
AND NP.APP='EDOC'
AND TRUNC(NP.RC_DATE)<=TRUNC(BD.LAST_RUN_DATE)
) a where rank > 1 and rank <= 20000)
;
Current Query results :
COL3 RANK
1 1
2 1
3 1
3 2
3 3
3 4
4 1
5 1
Requirement :
COL3 RANK
1 1
2 2
3 3
3 3
3 3
3 3
4 4
5 5
Related
I have a team leaderboard that I am improving. Here are the steps I am trying to streamline:
1. Import table using the ImportHTML function
2. Select needed columns using the Query function
3. Filter the information in the columns to only include the names that match the names on a specified pre-existing list using the Filter function.
4. Sort the data by score.
Here is a link to my example workbook: https://docs.google.com/spreadsheets/d/1F0w-7bW8Wbh-eJubyBubeMM_yMzzcSZw28R0OJ-l8q8/edit#gid=1646904068
You can see from the "IS Warm-Up" sheet the former way that this was being done. I am experimenting with streamlining it in "IS-Individual" before streamlining the other sheets as well.
The "Names" sheet contains the team members that I want to pull the results for.
=Sort(
Filter(
QUERY(
IMPORTHTML("https://ct.thecmp.org/app/v1/index.php?do=match&task=getMatchResultsDetail&MatchId=18095&EventId=7&AwardId=1","TABLE",1),
"SELECT Col1, Col2, Col3, Col4, Col5, Col6, Col7, Col8, Col9, Col10, Col11, Col12, Col13" & match(Names!A:A,COL1),
Match(Names!A:A,0)),2,True))
I've also tried:
=Sort(Filter(QUERY(IMPORTHTML("https://ct.thecmp.org/app/v1/index.php?do=match&task=getMatchResultsDetail&MatchId=18095&EventId=7&AwardId=1","TABLE",1),"SELECT Col1, Col2, Col3, Col4, Col5, Col6, Col7, Col8, Col9, Col10, Col11, Col12, Col13"),Match(QUERY(IMPORTHTML("https://ct.thecmp.org/app/v1/index.php?do=match&task=getMatchResultsDetail&MatchId=18095&EventId=7&AwardId=1","TABLE",1),"SELECT Col1, Col2, Col3, Col4, Col5, Col6, Col7, Col8, Col9, Col10, Col11, Col12, Col13"),Names!A:A,0)),2,True)
=SORT(FILTER(QUERY(IMPORTHTML(
"https://ct.thecmp.org/app/v1/index.php?do=match&task=getMatchResultsDetail&MatchId=18095&EventId=7&AwardId=1",
"TABLE", 1),
"SELECT Col1,Col2,Col3,Col4,Col5,Col6,Col7,Col8,Col9,Col10,Col11,Col12,Col13"),
MATCH(INDIRECT("Names!A1:A"&COUNTA(QUERY(IMPORTHTML(
"https://ct.thecmp.org/app/v1/index.php?do=match&task=getMatchResultsDetail&MatchId=18095&EventId=7&AwardId=1",
"TABLE", 1),
"SELECT Col1"))+1), 0)), 2, 1)
this would work too if the imported table won't change in size:
=SORT(FILTER(QUERY(IMPORTHTML(
"https://ct.thecmp.org/app/v1/index.php?do=match&task=getMatchResultsDetail&MatchId=18095&EventId=7&AwardId=1",
"TABLE", 1),
"SELECT Col1,Col2,Col3,Col4,Col5,Col6,Col7,Col8,Col9,Col10,Col11,Col12,Col13"),
MATCH(Names!A1:A89, 0)), 2, 1)
I've a SQL which creates few metrics in below format and I want to transpose them into rows at every 'n'-th column for example, cut every 4 columns and convert into rows. Also there can be 'n' number of columns i.e 12, 20, 40, etc (multiples of 4 in my case).
I need to load these rows into a table.
Find below example,
Sample data.
SELECT 1 AS col1, 'Total number of Customer' AS col2, 100 AS col3, NULL AS col4,
1.1 AS col5, 'Total active customers' AS col6, 50 AS col7, NULL AS col8,
1.2 AS col9, 'Total inactive customers' AS col10, 50 AS col11,ABC AS col12
FROM DUAL;
Note: There could be n number of columns. It can be 12, 40 (multiples of 4 in my case).
You could take a union of the groups of 4 columns:
SELECT Col1 AS "S.No", Col2 AS "Metric Name", Col3 AS Count, Col4 AS Example
FROM yourTable
UNION ALL
SELECT Col5, Col6, Col7, Col8
FROM yourTable
UNION ALL
SELECT Col9, Col10, Col11, Col12
FROM yourTable
ORDER BY 1
I wonder how you ended up with such a table, but in any case I recommend using the output of the above query as the new version of your data moving forward.
Here is one solution.
with cte as (
SELECT 1 AS col1, 'Total number of Customer' AS col2, 100 AS col3, NULL AS col4,
1.1 AS col5, 'Total active customers' AS col6, 50 AS col7, NULL AS col8,
1.2 AS col9, 'Total inactive customers' AS col10, 50 AS col11,ABC AS col12
FROM DUAL
)
select col1, col2, col3, col4 from cte
union all
select col5, col6, col7, col8 from cte
union all
select col9, col10, col11, col12 from cte
order by 1
;
However, this looks like the sort of format which should really be handled in the displaying client rather than in SQL.
" there could be 'n' number of columns. It can be 12, 40 (multiples of 4 in my case) "
Then it is even more the case that it should be handled in the client. Because the only way to implement an arbitrary projection is to use dynamic SQL: write a PL/SQL function to generate a bespoke SQL statement for a paramterised query and number of columns. The snag is, you will have to return a weak ref cursor, because the projection is unknown at compile time.
Hi have a table with 9 columns having the same datatype (all are percentage values). I'm trying to create one Select statement in a PL\SQL function to return tuple of 3, depending on an external parameter value.
Syntactically, something like this:
WITH tmp
AS (SELECT '1' col1,
'2' col2,
'3' col3,
'4' col4,
'5' col5,
'6' col6,
'7' col7,
'8' col8,
'9' col9
FROM DUAL
UNION
SELECT '10' col1,
'20' col2,
'30' col3,
'40' col4,
'50' col5,
'60' col6,
'70' col7,
'80' col8,
'90' col9
FROM DUAL
UNION
SELECT '100' col1,
'200' col2,
'300' col3,
'400' col4,
'500' col5,
'600' col6,
'700' col7,
'800' col8,
'900' col9
FROM DUAL)
SELECT CASE
WHEN externaparameter = 1 THEN (col1, col2, col3)
WHEN externaparameter = 2 THEN (col4, col5, col6)
WHEN externaparameter = 3 THEN (col7, col8, col9)
END
INTO var1, var2, var3
FROM tmp;
I have two solutions for this:
Implement CASE statements for each column. But it will create a large select statement and maybe confusing.
SELECT CASE
WHEN externaparameter = 1 THEN col1
WHEN externaparameter = 2 THEN col4
WHEN externaparameter = 3 THEN col7
END,
CASE
WHEN externaparameter = 1 THEN col2
WHEN externaparameter = 2 THEN col5
WHEN externaparameter = 3 THEN col8
END,
CASE
WHEN externaparameter = 1 THEN col3
WHEN externaparameter = 2 THEN col6
WHEN externaparameter = 3 THEN col9
END
INTO var1, var2, var3
FROM tmp;
Or, implement three select statement, with a union. But my original query have a couple of WHERE conditions and for that case I need to repeat them.
SELECT a, b, c
INTO var1, var2, var3
FROM (SELECT col1 a, col2 b, col3 c
FROM tmp
WHERE externalparameter = 1
UNION
SELECT col4 a, col5 b, col6 c
FROM tmp
WHERE externalparameter = 2
UNION
SELECT col7 a, col8 b, col9 c
FROM tmp
WHERE externalparameter = 3)
Did I have a more clean Select statement for this problem? And what are the advantages and drawbacks for each solutions?
If you can't selectively run one of three separate queries at run time based on externalparameter then option one using the case expressions is the cleaner solution.
Problems with the union solution:
Check the query plan generated to be sure, but I imagine that it will execute three separate select statements though they are mutually exclusive. (Unless 12c is much smarter than query optimizes I've word with to date.) This is because the plan might be reused so the plan has to include all three selects to have the correct one in all cases.
The union version includes an implicit distinct which either adds work if things are already distinct in the first version. Or changes the results versus the first version.
Dynamic SQL is sometimes used in cases like this, but I think three static statements run selectively would be better than building a dynamic SQL string.
I have a very complex query that includes a "With" clause. This query works fine when executed on the DB2 Client. But if the same query is used inside a For Loop Cursor of a PL SQL stored procedure it does not work. On trying to apply the stored procedure to the database, it gives a syntax error as shown below.
SQL0104N An unexpected token "AS" was found following "col5 )
The for loop is as shown below.
FOR records AS cursors CURSOR FOR
(
WITH
temp1
(
col1, col2, col3, col4, col5
)
AS
(
SELECT
col1, col2, col3, col4, col5
FROM
table1
)
WITH
temp2
(
col6, col7, col8, col9, col10
)
AS
(
SELECT
col6, col7, col8, col9, col10
FROM
table2
)
SELECT col1, col2, col3, col4, col5, col6, co7, col8, col9, col10
FROM temp1, temp2
)
DO
-- Do Something here.
END FOR;
Can you please help solve this problem. Thanks in advance.
You have two problems. First, the FOR statement is incorrect; it should refer to a previously declared cursor:
...
CURSOR mycur IS WITH ... SELECT ...;
...
FOR rec IN mycur LOOP ...
Second,the query
WITH temp1 ( col1, col2, col3, col4, col5 ) AS (
SELECT col1, col2, col3, col4, col5 FROM table1
)
WITH temp2 ( col6, col7, col8, col9, col10 ) AS (
SELECT col6, col7, col8, col9, col10 FROM table2
is invalid and would never run by itself, so your claim that it executes in the CLP is untrue.
Finally I got this thing working. The solution was fairly simple. It was to remove the braces enclosing the query as shown below.
FOR records AS cursors CURSOR FOR
WITH
temp1
(
col1, col2, col3, col4, col5
)
AS
(
SELECT
col1, col2, col3, col4, col5
FROM
table1
)
WITH
temp2
(
col6, col7, col8, col9, col10
)
AS
(
SELECT
col6, col7, col8, col9, col10
FROM
table2
)
SELECT col1, col2, col3, col4, col5, col6, co7, col8, col9, col10
FROM temp1, temp2
DO
-- Do Something here.
END FOR ;
I am not really sure why it happens like this. It works just fine for any other normal query without a WITH clause in it.
ist it possible to compare on insert and update if all values are different to each other (except NULL)? i have 10 columns with numbers and doesnt want to write for each possibility an if statement.
example:
column_1 | column_2 | column_3
--------------------------------
5 2 4 <- allowed to insert
1 2 1 <- forbidden to insert/update, because there are two '1' in a row
I think you're going to need to have the explicit comparisons. You can use a CHECK constraint to make the comparisons, as in
CREATE TABLE SOME_TABLE
(COL1 NUMBER,
COL2 NUMBER,
COL3 NUMBER,
COL4 NUMBER,
COL5 NUMBER,
COL6 NUMBER,
COL7 NUMBER,
COL8 NUMBER,
COL9 NUMBER,
COL10 NUMBER,
CONSTRAINT SOME_TABLE_CK1
CHECK(COL1 NOT IN (COL2, COL3, COL4, COL5, COL6, COL7, COL8, COL9, COL10) AND
COL2 NOT IN (COL3, COL4, COL5, COL6, COL7, COL8, COL9, COL10) AND
COL3 NOT IN (COL4, COL5, COL6, COL7, COL8, COL9, COL10) AND
COL4 NOT IN (COL5, COL6, COL7, COL8, COL9, COL10) AND
COL5 NOT IN (COL6, COL7, COL8, COL9, COL10) AND
COL6 NOT IN (COL7, COL8, COL9, COL10) AND
COL7 NOT IN (COL8, COL9, COL10) AND
COL8 NOT IN (COL9, COL10) AND
COL9 NOT IN (COL10)));
Share and enjoy.