SQL Tuning for IN Clause - oracle

The below Teradata query is taking around 18 seconds to complete.
The highlighted values passed in IN clause is from another Oracle database so I am not able to implement a join with that table.
SELECT distinct sec.SerialNum esn, ef.EngineFamilyCd family, em.EngineModelCd model,
es.EngineSeriesCd series, sac.AircraftTailNum tailNumRef, sec.EnginePositionNum enginePosition,
o1.OrganizationId ownerOrgId, o2.OrganizationId operatorOrgId,
sec.EngineInstallationDttm installedDate, sec.EngineRemovalDttm removalDate,
sec.HardwareConfigNm hardwareConfig, sec.EngineControlNm engineControl,
sec.ApplicationSelectorNm appSelector, sec.EngineMonitorInd engineMonitorInd,
sec.EngineThrustRatingFctr engineThrustRating, sec.StatusDesc engineStatus, sec.n1modifiernum n1modifier
FROM DB_MASTER_BV.SZEngineCurrent sec,
DB_MASTER_BV.EngineSeries es,
DB_MASTER_BV.EngineModel em,
DB_Master_BV.EngineFamily ef,
DB_MASTER_BV.SZAircraftCurrent sac,
DB_MASTER_BV.Organization o1,
DB_MASTER_BV.Organization o2
WHERE sec.EngineSeriesCd = es.EngineSeriesCd
and es.EngineModelCd = em.EngineModelCd
and em.EngineFamilyCd = ef.EngineFamilyCd
and sec.MasterAircraftId = sac.MasterAircraftId
and o1.MasterOrganizationId = sec.OwnerMasterOrganizationId
and o2.MasterOrganizationId = sec.OperatorMasterOrganizationId
AND (sec.SerialNum in('733276','193283','690168','741471','876374','873383','193386','906397','804314','900116','785670','900399','724321','193488','811373','779917','193699','994688',
'779410','575169','A59299','900206','193297','575484','896359','367230','810105','876485','906385','876484','707149','811222','706801','193596','731949','697881',
'889697','804626','575194','707159','706129','900230','900231','706834','811352','900229','785748','193460','888221','906272','906266','906264','906263','994356',
'194431','731966','892417','811341','577413','741572','575564','889262','706956','876157','900257','900153','706958','706957','960436','892429','892427','900354',
'697138','645655','193352','994337','707189','697833','959190','900246','811317','577437','193643','697976','890692','193229','965579','900137','900135','894897',
'697723','193363','193367','785505','907077','959184','811311','706526','577302','706529','994332','702792','706663','779834','731931','960127','193371','876183',
'741563','193235','803843','577320','994318','907087','741460','907086','959170','994462','900464','193626','877503','643711','811202','811201','704585','193504',
'193500','875246','704876','725834','699783','699780','802380','900304','706885','906191','577773','959152','872574','811435','697388','699381','892485','577698',
'907035','811445','907039','894999','894857','894595','697273','894597','959139','577894','874898','706959','900424','193337','577697','907011','875696','699555',
'699554','575629','906149','906150','193452','962968','811264','811266','962970','875395','699543','575638','906153','857962','896247','858349','779746','906161',
'906928','802857','779640','193424','550309','424520','550305','575608','872517','906169','892196','811386','811385','906173','907220','959234','876666','959231',
'876662','893785','875914','802649','550218','550315','906111','741984','550319','906405','906501','550118','643371','785254','550116','550117','802946','906629',
'907145','550325','550324','906837','550320','906838','702591','550220','550227','906415','690289','906517','704416','731431','550125','959201','906413','994176',
'550333','550140','550337','891651','550141','550338','906746','907269','550132','550137','550138','892914','550342','906123','550153','550345','950923','906129',
'873188','906850','906953','690270','890713','645352','893127','697590','874826','424439','893126','907110','550144','856305','690269','892824','550256','550257',
'906867','907186','960852','720754','960851','906866','888607','805573','811530','960756','872352','550266','550267','550264','811518','888896','906730','994958',
'892247','960970','875186','906987','424124','550232','A59303','702660','875885','811609','888626','424219','906897','994981','731502','697496','695345','962996',
'894371','907153','805541','907154','424337','906613','906615','900512','906610','956141','994611','804582','994718','888648','575219','888756','896973','424395',
'872117','A59227','697616','731380','697614','900161','690410','994213','956155','956154','779492','994231','702876','577248','994727','193818','890879','722243',
'906499','577354','888560','645121','896972','960823','804279','900175','888853','193724','550285','550282','906469','994803','906466','888299','877141','890984',
'695688','994533','888327','A59348','A59346','994410','733116','550296','550290','550292','906478','731763','725658','896408','645145','994751','731654','740358',
'906441','550158','193849','906543','906448','994262','575824','424186','906345','643663','888305','906243','906244','702963','906453','906452','956119','906451',
'956116','950489','550166','906454','367457','896764','575833','994268','906252','994127','733236','906258','956123','550178','994777','956126','956127','956128',
'906786','906788','906687','643290','994631','956225','994632','888574','906365','804228','731599','643682','550182','804369','994784','550186','550183','888826',
'575127','906439','890482','906438','906691','890472','994509','193147','575718','804215','575276','994793','897257'))
and END(sec.EngineValidPd) is until_changed
order by esn
Also if there is more than 1000 records, I am implementing the IN clause as follows
AND (sec.SerialNum in( first 999 recods) OR sec.SerialNum in( next 999 recods)… OR sec.SerialNum in( remaining recods))
Please suggest solution which would be faster than the above query and which will not cause issue with more than 1000 records in IN clause


What is your Teradata release?
In TD14 there's a built-in table function to split a string of values, you can simply pass all values within a single string:
AND sec.SerialNum IN
(
SELECT token
FROM TABLE (STRTOK_SPLIT_TO_TABLE(1, '733276,193283,690168,741471,876374', ',')
RETURNS (outkey INTEGER,
tokennum INTEGER,
token VARCHAR(20) CHARACTER SET UNICODE)
) AS d
)

Related

Translate SQL's first_value and partition by into SAS

I have this code in SQL
SELECT acc_id,
time,
approved_amount,
balance,
coalesce(approved_amount,
first_value(balance) OVER (PARTITION BY acc_id
ORDER BY time)) orig_amount
FROM table;
Is it possible somehow to translate it into SAS? It is not working in proc sql step.

I don't use nor know SAS, however if it is something what does not support window functions, you can replace it by joins. I assume you want second argument of coalesce as the balance of oldest record of those in acc_id group, hence:
select acc_id,
time,
approved_amount,
balance,
coalesce(approved_amount, acc_id_to_balance.balance_fallback)
from table t
join (
select t.acc_id, t.balance as balance_fallback
from (
select acc_id, min(time) as min_time
from table
group by acc_id
) acc_id_to_min_time
join table t on acc_id_to_min_time.acc_id = t.acc_id and acc_id_to_min_time.min_time = t.time
) acc_id_to_balance on t.acc_id = acc_id_to_balance.acc_id
Just worked out in head, didn't try. Problems might appear in case of duplicate minimal time, which would require another level of grouping.

This is how you would do that in SAS since unlike SQL when you use a data step it will process the data in the order that it appears in the source dataset.
data want;
set table ;
by acc_id time;
if first.id then first_balance=balance;
retain first_balance;
orig_amount = coalesce(approved_amount,first_balance);
run;

Passing a parameter to a WITH clause query in Oracle

I'm wondering if it's possible to pass one or more parameters to a WITH clause query; in a very simple way, doing something like this (taht, obviously, is not working!):
with qq(a) as (
select a+1 as increment
from dual
)
select qq.increment
from qq(10); -- should get 11
Of course, the use I'm going to do is much more complicated, since the with clause should be in a subquery, and the parameter I'd pass are values taken from the main query....details upon request... ;-)
Thanks for any hint

OK.....here's the whole deal:
select appu.* from
(<quite a complex query here>) appu
where not exists
(select 1
from dual
where appu.ORA_APP IN
(select slot from
(select distinct slots.inizio,slots.fine from
(
with
params as (select 1900 fine from dual)
--params as (select app.ora_fine_attivita fine
-- where app.cod_agenda = appu.AGE
-- and app.ora_fine_attivita = appu.fine_fascia
--and app.data_appuntamento = appu.dataapp
--)
,
Intervals (inizio, EDM) as
( select 1700, 20 from dual
union all
select inizio+EDM, EDM from Intervals join params on
(inizio <= fine)
)
select * from Intervals join params on (inizio <= fine)
) slots
) slots
where slots.slot <= slots.fine
)
order by 1,2,3;
Without going in too deep details, the where condition should remove those records where 'appu.ORA_APP' match one of the records that are supposed to be created in the (outer) 'slots' table.
The constants used in the example are good for a subset of records (a single 'appu.AGE' value), that's why I should parametrize it, in order to use the commented 'params' table (to be replicated, then, in the 'Intervals' table.
I know thats not simple to analyze from scratch, but I tried to make it as clear as possible; feel free to ask for a numeric example if needed....
Thanks

Unable to get only first occurrence of each job

I am trying to query some jobs from a repo, however I only need the job with the latest start time. I have tried using ROW_NUMBER for this and select only row number 1 for each job, however it doesn't seem to fall through:
SELECT a.jobname||','||a.projectname||','||a.startdate||','||a.enddate||','||
ROW_NUMBER() OVER ( PARTITION BY a.jobname ORDER BY a.startdate DESC ) AS "rowID"
FROM taskhistory a
WHERE a.jobname IS NOT NULL AND a.startdate >= (SYSDATE-1))LIMIT 1 AND rowID = 1;
ERROR at line 7:
ORA-00932: inconsistent datatypes: expected ROWID got NUMBER
Can I please ask for some assistance?

You have aliased your concatenated string "rowID" which is a mistake because it clashes with the Oracle keyword rowid. This is a special datatype, which allows us to identify table rows by their physical location. Find out more.
When you reference the column alias you omitted the fouble quotes. Oracle therefore interprets it as the keyword, rowid, and expects an expression which can be converted to the ROWID datatype.
Double-quoted identifiers are always a bad idea. Avoid them unless truly necessary.
Fixing the column alias will reveal the logic bug in your code. You are concatenating a whole slew of columns together, including the ROW_NUMBER() function, and calling that string "rowID". Clearly that string is never going to equal one, so this will filter out all rows:
and "rowID" = 1
Also LIMIT is not valid in Oracle.
What you need to do is use a sub-query, like this
SELECT a.jobname||','
||a.projectname||','
||a.startdate||','
||a.enddate||','
||to_char(a.rn) as "rowID"
FROM (
SELECT jobname
, projectname
, startdatem
, enddate,
, ROW_NUMBER() OVER ( PARTITION BY jobname
ORDER BY startdate DESC ) AS RN
FROM taskhistory
WHERE jobname IS NOT NULL
AND a.startdate >= (SYSDATE-1)
) a
where a.RN = 1;
Concatenating the projection like that seems an odd thing to do but I don't understand your business requirements.

Oracle IN clause limitation

As we all know, the IN clause in Oracle is limited to 1000 elements.
One point I am not sure is, can I go beyond 1000 if they are the result of a query.
For instance, if we assume that :
SELECT TYPEID FROM SECONDTABLE WHERE DATE < YESTERDAY
assuming that query returns 4300 values, can we write:
SELECT
ID
FROM
FIRSTTABLE
WHERE
TYPEID IN (SELECT TYPEID FROM SECONDTABLE WHERE DATE < YESTERDAY)
? Since the IN contains ONE statement to begin with (but 4300 once "expanded").

That is no problem, since this is a totally different type of query. The in has a maximum of a thousand hard-coded values to check against.
This however, has nothing to do with that one. Oracle does not 'expand' the results of the sub-query the way you say it does.
You could also write this as a join, which is actually the same. This also doesn't limit to a specific number of rows.
How to use with join:
SELECT f.ID
FROM FIRSTTABLE f
JOIN SECONDTABLE s
ON f.TYPEID = s.TYPEID
WHERE s.DATE < s.YESTERDAY

You cannot have more than 1000 literals in an IN clause. You can,
however, have SELECT statements in your IN clause which can return an
unlimited number of elements
So you can use this
SELECT
ID
FROM
FIRSTTABLE
WHERE
TYPEID IN (SELECT TYPEID FROM SECONDTABLE WHERE DATE < YESTERDAY)
But why not JOIN?

How to put more than 1000 values into an Oracle IN clause [duplicate]

This question already has answers here:
SQL IN Clause 1000 item limit
(5 answers)
Closed 8 years ago.
Is there any way to get around the Oracle 10g limitation of 1000 items in a static IN clause? I have a comma delimited list of many of IDs that I want to use in an IN clause, Sometimes this list can exceed 1000 items, at which point Oracle throws an error. The query is similar to this...
select * from table1 where ID in (1,2,3,4,...,1001,1002,...)

Put the values in a temporary table and then do a select where id in (select id from temptable)

select column_X, ... from my_table
where ('magic', column_X ) in (
('magic', 1),
('magic', 2),
('magic', 3),
('magic', 4),
...
('magic', 99999)
) ...

I am almost sure you can split values across multiple INs using OR:
select * from table1 where ID in (1,2,3,4,...,1000) or
ID in (1001,1002,...,2000)

You may try to use the following form:
select * from table1 where ID in (1,2,3,4,...,1000)
union all
select * from table1 where ID in (1001,1002,...)

Where do you get the list of ids from in the first place? Since they are IDs in your database, did they come from some previous query?
When I have seen this in the past it has been because:-
a reference table is missing and the correct way would be to add the new table, put an attribute on that table and join to it
a list of ids is extracted from the database, and then used in a subsequent SQL statement (perhaps later or on another server or whatever). In this case, the answer is to never extract it from the database. Either store in a temporary table or just write one query.
I think there may be better ways to rework this code that just getting this SQL statement to work. If you provide more details you might get some ideas.

Use ...from table(... :
create or replace type numbertype
as object
(nr number(20,10) )
/
create or replace type number_table
as table of numbertype
/
create or replace procedure tableselect
( p_numbers in number_table
, p_ref_result out sys_refcursor)
is
begin
open p_ref_result for
select *
from employees , (select /*+ cardinality(tab 10) */ tab.nr from table(p_numbers) tab) tbnrs
where id = tbnrs.nr;
end;
/
This is one of the rare cases where you need a hint, else Oracle will not use the index on column id. One of the advantages of this approach is that Oracle doesn't need to hard parse the query again and again. Using a temporary table is most of the times slower.
edit 1 simplified the procedure (thanks to jimmyorr) + example
create or replace procedure tableselect
( p_numbers in number_table
, p_ref_result out sys_refcursor)
is
begin
open p_ref_result for
select /*+ cardinality(tab 10) */ emp.*
from employees emp
, table(p_numbers) tab
where tab.nr = id;
end;
/
Example:
set serveroutput on
create table employees ( id number(10),name varchar2(100));
insert into employees values (3,'Raymond');
insert into employees values (4,'Hans');
commit;
declare
l_number number_table := number_table();
l_sys_refcursor sys_refcursor;
l_employee employees%rowtype;
begin
l_number.extend;
l_number(1) := numbertype(3);
l_number.extend;
l_number(2) := numbertype(4);
tableselect(l_number, l_sys_refcursor);
loop
fetch l_sys_refcursor into l_employee;
exit when l_sys_refcursor%notfound;
dbms_output.put_line(l_employee.name);
end loop;
close l_sys_refcursor;
end;
/
This will output:
Raymond
Hans

I wound up here looking for a solution as well.
Depending on the high-end number of items you need to query against, and assuming your items are unique, you could split your query into batches queries of 1000 items, and combine the results on your end instead (pseudocode here):
//remove dupes
items = items.RemoveDuplicates();
//how to break the items into 1000 item batches
batches = new batch list;
batch = new batch;
for (int i = 0; i < items.Count; i++)
{
if (batch.Count == 1000)
{
batches.Add(batch);
batch.Clear()
}
batch.Add(items[i]);
if (i == items.Count - 1)
{
//add the final batch (it has < 1000 items).
batches.Add(batch);
}
}
// now go query the db for each batch
results = new results;
foreach(batch in batches)
{
results.Add(query(batch));
}
This may be a good trade-off in the scenario where you don't typically have over 1000 items - as having over 1000 items would be your "high end" edge-case scenario. For example, in the event that you have 1500 items, two queries of (1000, 500) wouldn't be so bad. This also assumes that each query isn't particularly expensive in of its own right.
This wouldn't be appropriate if your typical number of expected items got to be much larger - say, in the 100000 range - requiring 100 queries. If so, then you should probably look more seriously into using the global temporary tables solution provided above as the most "correct" solution. Furthermore, if your items are not unique, you would need to resolve duplicate results in your batches as well.

Yes, very weird situation for oracle.
if you specify 2000 ids inside the IN clause, it will fail.
this fails:
select ...
where id in (1,2,....2000)
but if you simply put the 2000 ids in another table (temp table for example), it will works
below query:
select ...
where id in (select userId
from temptable_with_2000_ids )
what you can do, actually could split the records into a lot of 1000 records and execute them group by group.

Here is some Perl code that tries to work around the limit by creating an inline view and then selecting from it. The statement text is compressed by using rows of twelve items each instead of selecting each item from DUAL individually, then uncompressed by unioning together all columns. UNION or UNION ALL in decompression should make no difference here as it all goes inside an IN which will impose uniqueness before joining against it anyway, but in the compression, UNION ALL is used to prevent a lot of unnecessary comparing. As the data I'm filtering on are all whole numbers, quoting is not an issue.
#
# generate the innards of an IN expression with more than a thousand items
#
use English '-no_match_vars';
sub big_IN_list{
#_ < 13 and return join ', ',#_;
my $padding_required = (12 - (#_ % 12)) % 12;
# get first dozen and make length of #_ an even multiple of 12
my ($a,$b,$c,$d,$e,$f,$g,$h,$i,$j,$k,$l) = splice #_,0,12, ( ('NULL') x $padding_required );
my #dozens;
local $LIST_SEPARATOR = ', '; # how to join elements within each dozen
while(#_){
push #dozens, "SELECT #{[ splice #_,0,12 ]} FROM DUAL"
};
$LIST_SEPARATOR = "\n union all\n "; # how to join #dozens
return <<"EXP";
WITH t AS (
select $a A, $b B, $c C, $d D, $e E, $f F, $g G, $h H, $i I, $j J, $k K, $l L FROM DUAL
union all
#dozens
)
select A from t union select B from t union select C from t union
select D from t union select E from t union select F from t union
select G from t union select H from t union select I from t union
select J from t union select K from t union select L from t
EXP
}
One would use that like so:
my $bases_list_expr = big_IN_list(list_your_bases());
$dbh->do(<<"UPDATE");
update bases_table set belong_to = 'us'
where id in ($bases_list_expr)
UPDATE

Instead of using IN clause, can you try using JOIN with the other table, which is fetching the id. that way we don't need to worry about limit. just a thought from my side.

Instead of SELECT * FROM table1 WHERE ID IN (1,2,3,4,...,1000);
Use this :
SELECT * FROM table1 WHERE ID IN (SELECT rownum AS ID FROM dual connect BY level <= 1000);
*Note that you need to be sure the ID does not refer any other foreign IDS if this is a dependency. To ensure only existing ids are available then :
SELECT * FROM table1 WHERE ID IN (SELECT distinct(ID) FROM tablewhereidsareavailable);
Cheers

Resources