While trying to run SQL query for PostgreSQL, instead of the column names from the table referred it it pushing down * to the database.
select
m.m_id,
cnt_c_no
from (
select
m_id
from pg_test_main.test1.table1
where
last_date >= '2019-01-01 00:00:00'
) as m
left join (
select
ci.m_id,
count(ci.c_no) as cnt_c_no
from (
select
m_id,
c_no
from pg_test.public.table2
) as ci
inner join (
select
c_no
from pg_test.public.table3
where
is_del = 'F'
) as c on ci.c_no = c.c_no
group by
ci.m_id
) as join1 on m.m_id = join1.m_id;
00-00 Screen
00-01 Project(m_id=[$0], cnt_c_no=[$1])
00-02 Project(m_id=[$0], cnt_c_no=[$2])
00-03 HashJoin(condition=[=($0, $1)], joinType=[left], semi-join: =[false])
00-05 Jdbc(sql=[SELECT "m_id" FROM "test1"."table1" WHERE "last_date" >= '2019-01-01 00:00:00' ])
00-04 Project(m_id0=[$0], cnt_c_no=[$1])
00-06 HashAgg(group=[{0}], cnt_c_no=[COUNT($1)])
00-07 Project(m_id=[$0], c_no=[$1])
00-08 HashJoin(condition=[=($1, $2)], joinType=[inner], semi-join: =[false])
00-10 Project(m_id=[$3], c_no=[$1])
00-12 Jdbc(sql=[SELECT * FROM "public"."table2" ])
00-09 Project(c_no0=[$0])
00-11 Project(c_no=[$0])
00-13 SelectionVectorRemover
00-14 Filter(condition=[=($60, 'F')])
00-15 Jdbc(sql=[SELECT * FROM "public"."table3" ])
As you can see, Jdbc Scan for table1 was using column names.
but, Jdbc Scan for table2 and table3 was not using column names. It pushed down * to the database.
How can I control jdbc scan so that it can push down colume names?
Apache Drill version is 1.16.0 (embedded-mode)
I tried to reproduce it with MySQL on both Drill 1.17 and Drill 1.15, but for the query, similar to the query you have specified, all the query is pushed into the JDBC storage:
SELECT m.person_id,
cnt_c_no
FROM
(SELECT person_id
FROM mysql.`drill_mysql_test1`.person1
WHERE date_field >= '2019-01-01 00:00:00') AS m
LEFT JOIN
(SELECT ci.person_id,
count(ci.last_name) AS cnt_c_no
FROM
(SELECT person_id,
last_name
FROM mysql.`drill_mysql_test`.person) AS ci
INNER JOIN
(SELECT last_name
FROM mysql.`drill_mysql_test`.person2
WHERE boolean_field = 'F' ) AS c ON ci.last_name = c.last_name
GROUP BY ci.person_id) AS join1 ON m.person_id = join1.person_id
The plan for this query:
00-00 Screen
00-01 Project(person_id=[$0], cnt_c_no=[$1])
00-02 Jdbc(sql=[SELECT `t0`.`person_id`, `t5`.`cnt_c_no` FROM (SELECT `person_id` FROM `drill_mysql_test1`.`person1` WHERE `date_field` >= '2019-01-01 00:00:00') AS `t0` LEFT JOIN (SELECT `t1`.`person_id`, COUNT(`t1`.`last_name`) AS `cnt_c_no` FROM (SELECT `person_id`, `last_name` FROM `drill_mysql_test`.`person`) AS `t1` INNER JOIN (SELECT `last_name` FROM `drill_mysql_test`.`person2` WHERE `boolean_field` = 'F') AS `t3` ON `t1`.`last_name` = `t3`.`last_name` GROUP BY `t1`.`person_id`) AS `t5` ON `t0`.`person_id` = `t5`.`person_id` ])
Could you please provide CTAS for Postgres tables, so I will try to reproduce it again with specific data types. Also, if possible, please check whether this issue is still reproduced on Drill 1.17.
UPD:
Comment under this answer helped to discover that this problem was caused by the following issue: https://issues.apache.org/jira/browse/DRILL-7340 and it will be resolved in Drill 1.18.0.
I have a query that doesn't work; can you help me with the transformation?
The original Informix query that I want to transform to Oracle.
SELECT DISTINCT table3.no_cev,
table1.literal,
table1.colid,
table2.repid,
table2.valor,
table2.indicador,
'',
'',
table2.origen,
table2.codi,
table2.no_cia,
table2.num_dcca,
table2.no_aprof,
table2.no_compta
FROM table1,
OUTER table2,
table3
WHERE ( table1.colid = table2.colid) and
( table1.grupid = table2.grupid) and
( table3.no_cev = table2.no_cev) and
( ( table1.grupid = 2) AND
( table2.cod_exp = 99609 ) AND
( table2.indicador = 'S' ) ) AND
( table3.num_dcca = 1);
( table3.codest = 76695);
My transformation of the query from Informix to Oracle — but it looks like it doesn't work:
SELECT DISTINCT table3.no_cev,
table1.literal,
table1.colid,
table2.repid,
table2.valor,
table2.indicador,
'',
'',
table2.origen,
table2.codi,
table2.no_cia,
table2.num_dcca,
table2.no_aprof,
table2.no_compta
FROM table1
LEFT OUTER JOIN (table2
RIGHT OUTER JOIN table3
ON table3.no_cev = table2.no_cev)
ON (( table1.colid = table2.colid)
AND ( table1.grupid = table2.grupid))
WHERE ( ( table1.grupid = '2' )
AND ( table2.cod_exp = '99609' )
AND ( table2.indicador = 'S' ) )
AND ( table3.num_dcca = '1')
AND ( table3.codest = '76695');
You have joined the table with ON clause at wrong place in the code.
Corrected your code now as following:
SELECT DISTINCT
TABLE3.NO_CEV,
TABLE1.LITERAL,
TABLE1.COLID,
TABLE2.REPID,
TABLE2.VALOR,
TABLE2.INDICADOR,
'',
'',
TABLE2.ORIGEN,
TABLE2.CODI,
TABLE2.NO_CIA,
TABLE2.NUM_DCCA,
TABLE2.NO_APROF,
TABLE2.NO_COMPTA
FROM
TABLE1
LEFT OUTER JOIN
-- ( -- removed this bracket
TABLE2 ON ( ( TABLE1.COLID = TABLE2.COLID )
AND ( TABLE1.GRUPID = TABLE2.GRUPID ) ) -- added this ON here
RIGHT OUTER JOIN TABLE3 ON TABLE3.NO_CEV = TABLE2.NO_CEV
-- ) -- removed this bracket
WHERE
TABLE1.GRUPID = '2'
AND TABLE2.COD_EXP = '99609'
AND TABLE2.INDICADOR = 'S'
AND TABLE3.NUM_DCCA = '1'
AND TABLE3.CODEST = '76695' ; -- no need of extra brackets
Cheers!!
It makes life unnecessarily difficult for people who would like to help you when you don't include a more or less minimal outline schema for the tables used in your query, and some sample data, and the expected results. Further, you seem to have converted numbers (integers) in the original Informix query into strings in the Oracle query. It is not clear why. Again, the schema would help explain what's going on.
As I noted in the comments, you should omit the two empty/null fields in the select-list; you could also drop a number of the columns from table2 — candidates for being dropped include all the columns not otherwise named in the query, such as repid, valor, origen, codi, no_cia, no_aprof, no_compta. Keep one or two of them; you don't really need more. However, I've preserved all the named columns in the sample data.
Schema and data
Here is some Informix SQL that appears to match the tables and columns in the query shown in the question. In case of doubt, the column was made into an INTEGER column. All the columns are qualified with NOT NULL.
DROP TABLE IF EXISTS table1;
DROP TABLE IF EXISTS table2;
DROP TABLE IF EXISTS table3;
CREATE TABLE table1
(
grupid INTEGER NOT NULL, -- 2
literal VARCHAR(32) NOT NULL,
colid INTEGER NOT NULL
);
CREATE TABLE table2
(
grupid INTEGER NOT NULL,
no_cev INTEGER NOT NULL,
colid INTEGER NOT NULL,
repid INTEGER NOT NULL,
valor INTEGER NOT NULL,
indicador CHAR(1) NOT NULL, -- 'S'
origen INTEGER NOT NULL,
codi INTEGER NOT NULL,
no_cia INTEGER NOT NULL,
num_dcca INTEGER NOT NULL,
no_aprof INTEGER NOT NULL,
no_compta INTEGER NOT NULL,
cod_exp INTEGER NOT NULL -- 99609
);
CREATE TABLE table3
(
no_cev INTEGER NOT NULL,
num_dcca INTEGER NOT NULL, -- 1
codest INTEGER NOT NULL -- 76695
);
LOAD FROM "table1.unl" INSERT INTO table1;
LOAD FROM "table2.unl" INSERT INTO table2;
LOAD FROM "table3.unl" INSERT INTO table3;
The annotations indicate the value specified in the query for that column; they helped guide the construction of the sample data.
Three sample data files in the Informix (pipe-separated values) UNLOAD format are:
table1.unl
2|Literal value 1|100
2|Literal value 2|123
2|Literal value 3|134
2|Literal value 4|145
table2.unl
2|2345|100|222|333|S|444|555|666|777|888|999|99609
2|2346|123|223|333|S|444|555|666|776|888|999|99609
2|2347|134|224|333|S|444|555|666|775|888|999|99609
2|2348|145|225|333|S|444|555|666|774|888|999|99609
1|2345|100|225|333|S|444|555|666|773|888|999|99609
2|2340|123|226|333|S|444|555|666|772|888|999|99609
3|2347|134|227|333|S|444|555|666|771|888|999|99609
2|2350|145|228|333|S|444|555|666|770|888|999|99609
table3.unl
2345|1|76695
2346|1|88776
2347|2|76695
2348|1|76695
Result of query using Informix-style OUTER join
Assuming that the stray early semicolon in the original query should be an AND (that matches what is written in the proposed Oracle query), removing the two empty string result columns, and removing the superfluous level of parentheses, then the original query looks like:
SELECT DISTINCT
table3.no_cev,
table1.literal,
table1.colid,
table2.repid,
table2.valor,
table2.indicador,
table2.origen,
table2.codi,
table2.no_cia,
table2.num_dcca,
table2.no_aprof,
table2.no_compta
FROM table1,
OUTER table2,
table3
WHERE (table1.colid = table2.colid) AND
(table1.grupid = table2.grupid) AND
(table3.no_cev = table2.no_cev) AND
(table1.grupid = 2) AND
(table2.cod_exp = 99609) AND
(table2.indicador = 'S') AND
(table3.num_dcca = 1) AND
(table3.codest = 76695);
On the sample data shown, using Informix 12.10.FC6 running on a MacBook Pro with macOS 10.14.6 Mojave (not that the o/s is likely to be a factor in the results), this produces:
2345|Literal value 1|100|222|333|S|444|555|666|777|888|999
2345|Literal value 2|123|||||||||
2345|Literal value 3|134|||||||||
2345|Literal value 4|145|||||||||
2348|Literal value 1|100|||||||||
2348|Literal value 2|123|||||||||
2348|Literal value 3|134|||||||||
2348|Literal value 4|145|225|333|S|444|555|666|774|888|999
Why, you ask? Good question! The Informix old-style OUTER join is a complex critter, and doesn't necessarily have a simple translation to modern standard SQL (and hence to Oracle, etc). You can find some description of the way it works at Complex Outer Joins.
There are two groups of tables — table1 and table3 are the dominant tables, and table2 is the only OUTER table here. This means that Informix processes table1 and table3 using inner join, and then outer joins the result with table2. Since there is no direct join between table1 and table3, the result is a cartesian product of the two tables — each of the 4 rows in table1 is joined with each of the 4 rows in table3, yielding 16 rows. However, the filter conditions eliminate the rows from table3 where no_cev is 2346 and 2347. All the remaining 8 rows will be preserved, regardless of the results of the outer join operation. Now the rows are outer joined with table2. The rows with (no_cev, colid) of (2345, 100) and (2348, 145) have matching rows in table3 where the data satisfies the conditions in the WHERE clause. The other rows don't have such matching rows so the columns from table2 for those rows are 'all NULL'. As I said, it is weird — contorted. And explaining is hard work!
A first approximation using standard SQL
This query is a moderate approximation to a direct translation of the Informix query:
SELECT DISTINCT
t3.no_cev,
t1.literal,
t1.colid,
t2.repid,
t2.valor,
t2.indicador,
t2.origen,
t2.codi,
t2.no_cia,
t2.num_dcca,
t2.no_aprof,
t2.no_compta
FROM table1 AS t1
INNER JOIN table3 AS t3 ON 1 = 1
LEFT JOIN table2 AS t2 ON t3.no_cev = t2.no_cev
AND t1.colid = t2.colid
AND t1.grupid = t2.grupid
WHERE t1.grupid = 2
AND t2.cod_exp = 99609
AND t2.indicador = 'S'
AND t3.num_dcca = 1
AND t3.codest = 76695;
The output is:
2345|Literal value 1|100|222|333|S|444|555|666|777|888|999
2348|Literal value 4|145|225|333|S|444|555|666|774|888|999
This is missing the rows with 'null values'.
Achieving the same result using standard INNER and OUTER joins
We can collect those rows by looking for rows where one of the columns in table2 is null (because they're either all null or none null — because the columns are qualified NOT NULL):
SELECT DISTINCT
t3.no_cev,
t1.literal,
t1.colid,
t2.repid,
t2.valor,
t2.indicador,
t2.origen,
t2.codi,
t2.no_cia,
t2.num_dcca,
t2.no_aprof,
t2.no_compta
FROM table1 AS t1
INNER JOIN table3 AS t3 ON 1 = 1
LEFT JOIN table2 AS t2 ON t3.no_cev = t2.no_cev
AND t1.colid = t2.colid
AND t1.grupid = t2.grupid
WHERE t1.grupid = 2
AND ((t2.cod_exp = 99609 AND t2.indicador = 'S') OR t2.cod_exp IS NULL)
AND t3.num_dcca = 1
AND t3.codest = 76695;
This yields the output:
2345|Literal value 1|100|222|333|S|444|555|666|777|888|999
2345|Literal value 2|123|||||||||
2345|Literal value 3|134|||||||||
2345|Literal value 4|145|||||||||
2348|Literal value 1|100|||||||||
2348|Literal value 2|123|||||||||
2348|Literal value 3|134|||||||||
2348|Literal value 4|145|225|333|S|444|555|666|774|888|999
This is the same as the original old-style Informix OUTER join query.
Tejash's proposed solution
The SQL in Tejash's answer (revision 1) yields, on the same data:
2345|Literal value 1|100|222|333|S|\ |\ |444|555|666|777|888|999
2348|Literal value 4|145|225|333|S|\ |\ |444|555|666|774|888|999
The backslash-space values correspond to the empty strings — it's Informix's slightly peculiar way of encoding a zero-length non-null string. It's an area where Oracle may well behave slightly differently, but it is tangential to the problem with the query.
Clearly, this is not the same result as the Informix query. It's probably more reasonable; it works out of the box (I simply did copy'n'paste, quoted numbers and all, and it worked with no editing needed).
I don't know about Informix OUTER syntax, so my answer may be wrong. The WHERE clause, however, lacking any relation between table1 and table3 suggests that this is just a cross join of table1 and table3 and then an outer join of table2.
One way to write this:
select t3.no_cev, t1.literal, t1.colid, t2.*
from table1 t1
cross join table3 t3
left join table2 t2 on t2.colid = t1.colid
and t2.grupid = t1.grupid
and t2.no_cev = t3.no_cev
and t2.cod_exp = 2
and t2.indicador = 'S'
where t1.grupid = 2
and t3.num_dcca = 1
and t3.codest = 76695;
Another is:
with t1 as (select * from table1 where grupid = 2)
, t2 as (select * from table1 where grupid = 2 and cod_exp = 2 and indicador = 'S')
, t3 as (select * from table3 where num_dcca = 1 and codest = 76695)
select t3.no_cev, t1.literal, t1.colid, t2.*
from t1
cross join t3
left join t2 on t2.colid = t1.colid and t2.no_cev = t3.no_cev;
Above queries are standard SQL and supported by Oracle as of version 9i I think.
My subquery returns more than one row and I need the first row and keep my max function
Select ...
Where GHTY_FIRME.FIRME_ID= (Select FIRME_ID
FROM imag_verification_tube
WHERE (numero_dossier = '12004' OR NUMERO_TIRE= '12004')
AND CREE_LE = (select max (CREE_LE)
from tableX where(numero_dossier ='12004' OR
NUMERO_PQDCS= '12004 ')));
Using the rownum pseudocolumn:
Select ...
Where GHTY_FIRME.FIRME_ID= (Select FIRME_ID
FROM imag_verification_tube
WHERE (numero_dossier = '12004' OR NUMERO_TIRE= '12004')
AND CREE_LE = (select max (CREE_LE)
from tableX
where(numero_dossier ='12004'
OR NUMERO_PQDCS= '12004 ')
and rownum < 2));
See also On ROWNUM and Limiting Results.
That said, you can try to optimize the sql. One of the first things I would do is to convert the inner SQLs to joins. Also, depending on your business logic, the or condition could also be elminated.
I am kind of new in Oracle.
I am trying to create a package that has several functions.
This is the pseudocode of what I want to do
function FunctionA(UserID, startdate, enddate)
/* Select TransactionDate, Amount
from TableA
where TransactionDate between startdate and enddate
and TableA.UserID = UserID */
Return TransactionDate, Amount
end FunctionA
function FunctionB(UserID, startdate, enddate)
/* Select TransactionDate, Amount
from TableB
where TransactionDate between startdate and enddate
and TableB.UserID = UserID */
Return TransactionDate, Amount
end FunctionA
TYPE TRANSACTION_REC IS RECORD(
TransactionDate DATE,
TransactionAmt NUMBER);
function MainFunction(startdate, enddate)
return TBL
is
vTrans TRANSACTION_REC;
begin
FOR rec IN
( Select UserID, UserName, UserStatus
from UserTable
where EntryDate between startdate and enddate )
LOOP
vTrans := FunctionA(rec.UserID, startdate, enddate)
if vTrans.TransactionDate is null then
vTrans := FunctionB(rec.UserID, startdate, enddate)
if vTrans.TransactionDate is null then
rec.UserStatus := 'Inactive'
endif;
endif;
END Loop;
PIPE ROW(USER_OBJ_TYPE(rec.UserID,
rec.UserName,
rec.UserStatus,
vTrans.TransactionDate,
vTtans.TransactionAmt));
end MainFunction
Running this kind of code takes a long time because TableA and TableB is a very large table, and I am only getting 1 entry per record from the tables.
I would want to create a temporary table (TempTableA, TempTableB) within the package that will temporarily store all records based on the startdate and enddate, so that when I try to retrieve the TransactionDate and Amount for each rec, I will only refer to the TempTables (which is smaller than TableA and TableB).
I also want to take into consideration if the UserID is not found in TableA and TableB. So basically, when there are no records found in TableA and TableB, I also want the entry in the output, but it is indicated that the user is inactive.
Thank you for all your help.
SQL is a set-based language. It is far more efficient to execute one statement which returns all the rows you need than to execute many statements which each return a single row.
Here is one way of getting all your rows at once. It uses a common table expression because you read the whole of the UserTable and you should only do that once.
with cte as
(select UserID
, UserStatus
from UserTable )
select cte.UserID
, cte.UserStatus
, TableA.TransactionDate
, TableA.Amount
from cte join TableA
on (cte.UserID = TableA.UserID)
where cte.UserStatus = 'A'
and TableA.TransactionDate between startdate and enddate
union
select cte.UserID
, cte.UserStatus
, TableB.TransactionDate
, TableB.Amount
from cte join TableB
on (cte.UserID = TableB.UserID)
where cte.UserStatus != 'A'
and TableB.TransactionDate between startdate and enddate
By the way, be careful with temporary tables. They aren't like temporary tables in T-SQL. They are permanent heap tables, it's just their data that's temporary. This means that populating a temporary table is an expensive process, because the database writes all those rows to disk. Consequently we need to be certain that the performance gain we get by reading a dataset from a temporary table is worth the overhead of all those writes.
That certainly would not be the case with your code. In fact, it is really pretty rare that the answer to a performance question turns out to be "Use a Global Temporary Table", at least not in Oracle. Better queries are the way to go, and in particular, embracing the Joy of Sets!
Probably better to do it in one query, e.g.:
Select UserTable.UserID, UserTable.UserName, UserTable.UserStatus
,TableA.TransactionDate AS ATransactionDate
,TableA.Amount AS AAmount
,TableB.TransactionDate AS BTransactionDate
,TableB.Amount AS BAmount
from UserTable
left join TableA
on (UserTable.UserID = TableA.UserID)
left join TableB
on (UserTable.UserID = TableB.UserID)
where UserTable.EntryDate between startdate and enddate
i have a Stored Procedure which returns output as a ref cursor. I would like to store the output in another table using the MERGE statement. I'm having problems however mixing all the statements together (WITH, USING, MERGE etc..).
Can somebody assist? Thanks!
This is the table i want the output in (STOP_TIME is left out on purpose):
TABLE: **USER_ALLOCATION**
START_TIME date NOT NULL Primary Key
USER_ID number NOT NULL Primary Key
TASK_ID number NULL
This is the SP:
create or replace
PROCEDURE REPORT_PLAN_AV_USER
(
from_dt IN date,
to_dt IN date,
sysur_key IN number,
v_reservations OUT INTRANET_PKG.CURSOR_TYPE
)
IS
BEGIN
OPEN v_reservations FOR
with
MONTHS as (select FROM_DT + ((ROWNUM-1) / (24*2)) as DT from DUAL connect by ROWNUM <= ((TO_DT - FROM_DT) * 24*2) + 1),
TIMES as (select DT as START_TIME,(DT + 1/48) as STOP_TIME from MONTHS where TO_NUMBER(TO_CHAR(DT,'HH24')) between 8 and 15 and TO_NUMBER(TO_CHAR(DT,'D')) not in (1,7))
select
TIMES.START_TIME,
TIMES.STOP_TIME,
T.TASK_ID,
sysur_key USER_ID,
from
TIMES
left outer join (ALLOCATED_USER u INNER JOIN REQUIRED_RESOURCE r ON u.AU_ID = r.RR_ID INNER JOIN TASK t ON r.TASK_ID = t.TASK_ID)
ON u.USER_ID = sysur_key AND t.PLAN_TYPE = 3 AND TIMES.start_time >= TRUNC30(t.START_DATE) AND TIMES.start_time < TRUNC30(t.FINISH_DATE)
where u.USER_ID is null OR u.USER_ID = sysur_key
order by START_TIME ASC;
END;
I don't think you can resuse the cursor unless you write a lot of procedural code.
Can't you write a single merge statement and drop procedure REPORT_PLAN_AV_USER?
If you still need procedure REPORT_PLAN_AV_USE you can create a view that you use in procedure REPORT_PLAN_AV_USER and in your merge statement (to prevent code duplication).