How can I refactor the code with many unions [closed] - oracle

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
I have a package where I need either to do same (or similar) unions many times or not at all depending on parameters. The code below is simplified. But the real one would be a bit more complex. As I wish to avoid dynamic statements I though I create a temp view and do operations on this view.
PROCEDURE CREATE_VIEW( id IN BINARY_INTEGER, id2 IN BINARY_INTEGER, id3 IN BINARY_INTEGER) IS
sqlCommand VARCHAR2(32000);
BEGIN
sqlCommand :=
'CREATE OR REPLACE VIEW TMP_HELPER_VIEW AS
SELECT ID, IMPORT1_ID, IMPORT2_ID, PROD_ID
FROM
(
SELECT ID,
IMPORT1_ID,
-1 AS IMPORT2_ID,
PROD_ID
FROM TABLE1
WHERE IMPORT1> '|| id1;
IF id := 123 THEN
sqlCommand := sqlCommand||
'UNION
SELECT ID,
-1 AS IMPORT1_ID,
IMPORT2_ID,
PROD_ID
FROM TABLE2
WHERE IMPORT2> '|| id2;
END IF;
sqlCommand := sqlCommand||')' ;
EXECUTE IMMEDIATE sqlCommand;
END CREATE_VIEW;
With view I can do:
SELECT ID, IMPORT1_ID, IMPORT2_ID FROM TMP_HELPER_VIEW;
Instead of
sqlCommand :=
'SELECT ID,
IMPORT1_ID,
-1 AS IMPORT2_ID,
PROD_ID
FROM TABLE1
WHERE IMPORT1> '|| id1;
IF id := 123 THEN
sqlCommand := sqlCommand||
'UNION
SELECT ID,
-1 AS IMPORT1_ID,
IMPORT2_ID,
PROD_ID
FROM TABLE2
WHERE IMPORT2> '|| id2;
END IF;
So the code would be much better readable as I have many functions which uses similar statements. Statements are always bit different, so I need to write them every time (I can't write one in function and call evey time from there).
But I wonder if there is a better solution for this problem. Maybe someone can give me any clue.
Many thanks!

You're having a dynamic statement anyway, but in addition, you're storing that in a view. The use case is not very clear to me. If this is some sort of one time configuration to set up the view for a customer/installation, I think this could be fine.
If however, you're using this to generate different queries on the fly, I think it's risky. Two clients can be fighting to each generate their own version of the view, leading to unwanted, unpredictable results. The first client can create their version of the view, but by the time they want to use it, another client may have replaced it.
So for that, there are various better solutions, for example
Have the 'big' union of all tables in the view. Filter out unneeded rows from the results. This looks inefficient, but can be surprisingly fast, especially if you're using UNION ALL instead of UNION. The latter filters out duplicate rows, and it looks like your won't need that.
Create a pipelined table function. Open the needed queries one by one, and pipe their results to the output. You can use the function as a sort of parameterized view.
Create a function that generates the desired query, but instead of storing it as a view, return a cursor (sys_refcursor) based on the query. The caller can read the results of that cursor. Other clients can make consecutive calls and get their own cursor, without interfering with each other.
A mix between the two above would be to write a function that generates the query, bulk collects the results a table type and returns it. Very simple to implement, but may be less memory efficient, so less suitable for larger result sets.
If you're on one of the recent innovation versions, you can use SQL Macros to generate the SQL snippet you need.
It's hard to tell which one of this is best in your specific situation, but I've used each of these successfully in our application.
Example for 4.
create type IMPORT_R as object (
ID int,
IMPORT1_ID int,
IMPORT2_ID int,
PROD_ID int
);
create type IMPORT_T as table of IMPORT_R;
CREATE FUNCTION GetImport(
id IN BINARY_INTEGER,
id2 IN BINARY_INTEGER,
id3 IN BINARY_INTEGER)
return IMPORT_T
IS
sqlCommand VARCHAR2(32000);
v_Result IMPORT_T;
begin
sqlCommand :=
'SELECT ID, IMPORT1_ID, IMPORT2_ID, PROD_ID
FROM
(
SELECT ID,
IMPORT1_ID,
-1 AS IMPORT2_ID,
PROD_ID
FROM TABLE1
WHERE IMPORT1> '|| id1;
IF id := 123 THEN
sqlCommand := sqlCommand||
'UNION
SELECT ID,
-1 AS IMPORT1_ID,
IMPORT2_ID,
PROD_ID
FROM TABLE2
WHERE IMPORT2> '|| id2;
END IF;
sqlCommand := sqlCommand||')' ;
EXECUTE IMMEDIATE sqlCommand BULK COLLECT INTO v_Result;
return v_Result;
end;
/

Related

Create insert record dynamically by changing pk of existing record for passed in table

I want to pass a table name and schema into a procedure, and have it generate insert, update and delete statements for the particular table. This is part of an automated testing solution (in a development environment) in which I need to test some change data capture. I want to make this dynamic as it is going to be need to be done for lots of different tables over a long period of time, and I need to call it via a REST request through ORDS, so don't want to have to make an endpoint for every table.
Update and delete are fairly easy, however I am struggling with the insert statement. Some of the tables being passed in have hundreds of columns with various constraints, fks etc. so I think it makes sense to just manipulate an existing record by changing only the primary key. I need to be able to modify the primary key to a new value known to me beforehand (e.g. '-1').
Ideally I would create a dynamic rowtype, and select into where rownum = 1, then loop round the primary keys found from all_constraints, and update the rowtype.pk with my new value, before inserting this into the table. Essentially the same as this but without knowing the table in advance.
e.g. rough idea
PROCEDURE manipulate_records(p_owner in varchar2, p_table in varchar2)
IS
cursor c_pk is
select column_name
from all_cons_columns
where owner = p_owner
and constraint_name in (select constraint_name
from all_constraints
where table_name = p_table
and constraint_type = 'P');
l_row tbl_passed_in%ROWTYPE --(I know this isn't possible but ideally)
BEGIN
-- dynamic sql or refcursor to collect a record
select * into tbl_passed_in from tablename where rownum = 1;
-- now loop through pks and reassign their values to my known value
for i in c_pk loop
...if matches then reassign;
...
end loop;
-- now insert the record into the table passed in
END manipulate_records;
I have searched around but haven't found any examples which fit this exact use case, where an unknown column needs to be modified and insert into a table.
Depending on how complex your procedure is, you might be able to store it as a template in a CLOB. Then pull it in, replace table and owner, then compile it.
DECLARE
prc_Template VARCHAR2(4000);
vc_Owner VARCHAR2(0008);
vc_Table VARCHAR2(0008);
BEGIN
vc_Table := 'DUAL';
vc_Owner := 'SYS';
-- Pull code into prc_Template from CLOB, but this demonstrates the concept
prc_Template := 'CREATE OR REPLACE PROCEDURE xyz AS r_Dual <Owner>.<Table>%ROWTYPE; BEGIN NULL; END;';
prc_Template := REPLACE(prc_Template,'<Owner>',vc_Owner);
prc_Template := REPLACE(prc_Template,'<Table>',vc_Table);
-- Create the procedure
EXECUTE IMMEDIATE prc_Template;
END;
Then you have the appropriate ROWTYPE available:
CREATE OR REPLACE PROCEDURE xyz AS r_Dual SYS.DUAL%ROWTYPE; BEGIN NULL; END;
But you can't create the procedure and run it in the same code block.

Insert into not working on plsql in oracle

declare
vquery long;
cursor c1 is
select * from temp_name;
begin
for i in c1
loop
vquery :='INSERT INTO ot.temp_new(id)
select '''||i.id||''' from ot.customers';
dbms_output.put_line(i.id);
end loop;
end;
/
Output of select * from temp_name is :
ID
--------------------------------------------------------------------------------
customer_id
1 row selected.
I have customers table which has customer_id column.I want to insert all the customer_id into temp_new table but it is not being inserted. The PLSQL block executes successfully but the temp_new table is empty.
The output of dbms_output.put_line(i.id); is
customer_id
What is wrong there?
The main problem is that you generate a dynamic statement that you never execute; at some point you need to do:
execute immediate vquery;
But there are other problems. If you output the generated vquery string you'll see it contains:
INSERT INTO ot.temp_new(id)
select 'customer_id' from ot.customers
which means that for every row in customers you'll get one row in temp_new with ID set to the same fixed literal 'customer_id'. It's unlikely that's what you want; if customer_id is a column name from customers then it shouldn't be in single quotes.
As #mathguy suggested, long is not a sensible data type to use; you could use a CLOB but only really need a varchar2 here. So something more like this, where I've also switched to use an implicit cursor:
declare
l_stmt varchar2(4000);
begin
for i in (select id from temp_name)
loop
l_stmt := 'INSERT INTO temp_new(id) select '||i.id||' from customers';
dbms_output.put_line(i.id);
dbms_output.put_line(l_stmt);
execute immediate l_stmt;
end loop;
end;
/
db<>fiddle
The loop doesn't really make sense though; if your temp_name table had multiple rows with different column names, you'd try to insert the corresponding values from those columns in the customers table into multiple rows in temp_new, all in the same id column, as shown in this db<>fiddle.
I guess this is the starting point for something more complicated, but still seems a little odd.

Creating SQL-Injection proof dynamic where-clause from collection in PL/SQL

I need to execute a query where the where-clause is generated based on user input. The input consists of 0 or more pairs of varchar2s.
For example:
[('firstname','John')
,('lastname','Smith')
,('street','somestreetname')]
This would translate into:
where (attrib = 'firstname' and value = 'John')
and (attrib = 'lastname' and value = 'Smith')
and (attrib = 'street' and value = 'somestreetname')
This is not the actual data structure as there are several tables but for this example lets keep it simple and say the values are in 1 table. Also I know the parentheses are not necessary in this case but I put them there to make things clear.
What I do now is loop over them and concatinate them to the SQL string. I made a stored proc to generate this where-clause which might also not be very secure since I just concat to the original query.
Something like the following, where I try to get the ID's of the nodes that correspond with the requested parameters:
l_query := select DISTINCT n.id from node n, attribute_values av
where av.node_id = n.id ' || getWhereClause(p_params)
open l_rc
for l_query;
fetch l_rc bulk collect into l_Ids;
close l_rc;
But this is not secure so I'm looking for a way that can guaranty security and prevent SQL-Injection attacks from happening.
Does anyone have any idea on how this is done in a secure way? I would like to use bindings but I don't see how I can do this when you dont know the number of parameters.
DB: v12.1.0.2 (i think)
It's still a bit unclear and generalised, but assuming you have a schema-level collection type, something like:
create type t_attr_value_pair as object (attrib varchar2(30), value varchar2(30))
/
create type t_attr_value_pairs as table of t_attr_value_pair
/
then you can use the attribute/value pairs in the collection for the bind:
declare
l_query varchar2(4000);
l_rc sys_refcursor;
type t_ids is table of number;
l_ids t_ids;
l_attr_value_pairs t_attr_value_pairs;
-- this is as shown in the question; sounds like it isn't exactly how you have it
p_params varchar2(4000) := q'^[('firstname','John')
,('lastname','Smith')
,('street','somestreetname')]^';
begin
-- whatever mechanism you want to get the value pairs into a collection;
-- this is just a quick hack to translate your example string
select t_attr_value_pair(rtrim(ltrim(
regexp_substr(replace(p_params, chr(10)), '(.*?)(,|$)', 1, (2 * level) - 1, null, 1),
'[('''), ''''),
rtrim(ltrim(
regexp_substr(replace(p_params, chr(10)), '(.*?)(,|$)', 1, 2 * level, null, 1),
''''), ''')]'))
bulk collect into l_attr_value_pairs
from dual
connect by level <= regexp_count(p_params, ',') / 2 + 1;
l_query := 'select DISTINCT id from attribute_values
where (attrib, value) in ((select attrib, value from table(:a)))';
open l_rc for l_query using l_attr_value_pairs;
fetch l_rc bulk collect into l_ids;
close l_rc;
for i in 1..l_ids.count loop
dbms_output.put_line('id ' || l_ids(i));
end loop;
end;
/
although it doesn't need to be dynamic with this approach:
...
begin
-- whatever mechamism you want to get the value pairs into a collection
...
select DISTINCT id
bulk collect into l_ids
from attribute_values
where (attrib, value) in ((select attrib, value from table(l_attr_value_pairs)));
for i in 1..l_ids.count loop
dbms_output.put_line('id ' || l_ids(i));
end loop;
end;
/
or with a join to the table collection expression:
select DISTINCT av.id
bulk collect into l_ids
from table(l_attr_value_pairs) t
join attribute_values av on av.attrib = t.attrib and av.value = t.value;
Other collection types will need different approaches.
Alternatively, you could still build up your where clause with one condition per attribute/value pair, while still making them bind variables - but you would need two levels of dynamic SQL, similar to this.

PL/SQL Creating a procedure that contains result set joins

I want to create a procedure in PL/SQL that has 5 steps. Step 1 and 2 execute first and return an ID. In step 3, we have a SELECT statement that has a condition with that returned ID. I want then to take all of the results of that SELECT statement and use them in a JOIN in another SELECT statement and use THOSE results in a 3rd SELECT statement again using JOIN. From what I've seen, I can't use CURSOR in JOIN statements. Some of my co-workers have suggested that I save the results in a CURSOR and then use a loop to iterate through each row and use that data for the next SELECT. However since I'm going to do 2 selects this will create a huge fork of inside loops and that's exactly what I'm trying to avoid.
Another suggestion was to use Temprary Tables to store the data. However this procedure could be executed at the same time by many users and the table's data would conflict with each other. Right now I'm looking at LOCAL Temporary tables that supposedly filter the data according the the session but I'm not really sure I want to create dummy tables for my procedures since I want to avoid leaving trash in the schema (this procedure is for a custom part of the application). Is there a standard way of doing this? Any ideas?
Sample:
DECLARE
USERID INT := 1000000;
TEXT1 VARCHAR(100);
TEXT_INDEX INT;
CURSOR NODES IS SELECT * FROM NODE_TABLE WHERE DESCRIPTION LIKE TEXT || '%';
CURSOR USERS IS SELECT * FROM USERGROUPS JOIN NODES ON NODES.ID = USERGROUPS.ID;
BEGIN
SELECT TEXT INTO TEXT1 FROM TABLE_1 WHERE ID = USERID;
TEXT_INDEX = INSTR(TEXT, '-');
TEXT = SUBSTR(TEXT, 0, TEXT_INDEX);
OPEN NODES;
OPEN USERS;
END;
NOTE: This does NOT work. Oracle doesn't support joins between cursors.
NOTE2: This CAN be done in a single query but for the sake of argument (and in my real use case) I want to break those steps down in a procedure. The sample code is a depiction of what I'm trying to achieve IF joins between cursors worked. But they don't and I'm looking for an alternative.
I ended up using a function (although a procedure could be used as well) along with tables. Things I've learned and one should pay attention to:
PL/SQL functions can only return types that have been declared in the schema in advance and are clear. You can't create a function that returns something like MY_TABLE%ROWTYPE, even though it seems the type information is available it is not acceptable. You have to instead create a custom type of MY_TABLE%ROWTYPE is you want to return it.
Oracle treats tables of declared types differently from tables of %ROWTYPE. This confused the hell out of me at first but from what I've gathered this is how it works.
DECLARE TYPE MY_CUSTOM_TABLE IS TABLE OF MY_TABLE%ROWTYPE;
Declares a collection of types of MY_TABLE row. In order to add to this we must use BULK COLLECT INTO from an SQL statement that queries MY_TABLE. The resulting collection CANNOT be used in JOIN statements is not queryable and CANNOT be returned by a function.
DECLARE
CREATE TYPE MY_CUSTOM_TYPE AS OBJECT (COL_A NUMBER, COL_B NUMBER);
CREATE TYPE MY_CUSTOM_TABLE AS TABLE OF MY_CUSTOM_TYPE;
my_custom_tab MY_CUSTOM_TABLE;
This create my_custom_tab which is a table (not a collection) and if populated can be queried at using TABLE(my_custmo_tab) in the FROM statement. As a table which is declared in advance in the schema this CAN be returned from a function. However it CANNOT be populated using BULK COLLECT INTO since it is not a collection. We must instead use the normal SELECT INTO statement. However, if we want to populate it with data from an existing table that has 2 number columns we cannot simply do SELECT * INTO my_custom_tab FROM DOUBLE_NUMBER_TABLE since my_custom_tab hasn't been initialized and doesn't contain enough rows to receive the data. And if we don't know how many rows a query returns we can't initialize it. The trick into populating the table is to use the CAST command and cast our select result set as a MY_CUSTOM_TABLE and THEN add it.
SELECT CAST(MULTISET(SELECT COL_A, COL_B FROM DOUBLE_NUMBER_TABLE) AS MY_CUSTOM_TABLE) INTO my_custom_tab FROM DUAL
Now we can easily use my_custom_tab in queries etc through the use of the TABLE() function.
SELECT * FROM TABLE(my_custom_tab)
is valid.
You can do such decomposition in many ways, but all of them have a significant performance penalty in comaration with single SQL statement.
Maintainability improvement are also questionable and depends on specific situation.
To review all possibilities please look through documentation.
Below is some possible variants based on simple logic:
calculate Oracle user name prefix based on given Id;
get all users whose name starts with this prefix;
find all tables owned by users from step 2;
count a total number of found tables.
1. pipelined
Prepare types to be used by functions:
create or replace type TUserRow as object (
username varchar2(30),
user_id number,
created date
)
/
create or replace type TTableRow as object (
owner varchar2(30),
table_name varchar2(30),
status varchar2(8),
logging varchar2(3)
-- some other useful fields here
)
/
create or replace type TUserList as table of TUserRow
/
create or replace type TTableList as table of TTableRow
/
Simple function to find prefix by user id:
create or replace function GetUserPrefix(piUserId in number) return varchar2
is
vUserPrefix varchar2(30);
begin
select substr(username,1,3) into vUserPrefix
from all_users
where user_id = piUserId;
return vUserPrefix;
end;
/
Function searching for users:
create or replace function GetUsersPipe(
piNameStart in varchar2
)
return TUserList pipelined
as
vUserList TUserList;
begin
for cUsers in (
select *
from
all_users
where
username like piNameStart||'%'
)
loop
pipe row( TUserRow(cUsers.username, cUsers.user_id, cUsers.created) ) ;
end loop;
return;
end;
Function searching for tables:
create or replace function GetUserTablesPipe(
piUserNameStart in varchar2
)
return TTableList pipelined
as
vTableList TTableList;
begin
for cTables in (
select *
from
all_tables tab_list,
table(GetUsersPipe(piUserNameStart)) user_list
where
tab_list.owner = user_list.username
)
loop
pipe row ( TTableRow(cTables.owner, cTables.table_name, cTables.status, cTables.logging) );
end loop;
return;
end;
Usage in code:
declare
vUserId number := 5;
vTableCount number;
begin
select count(1) into vTableCount
from table(GetUserTablesPipe(GetUserPrefix(vUserId)));
dbms_output.put_line('Users with name started with "'||GetUserPrefix(vUserId)||'" owns '||vTableCount||' tables');
end;
2. Simple table functions
This solution use same types as a variant with pipelined functions above.
Function searching for users:
create or replace function GetUsers(piNameStart in varchar2) return TUserList
as
vUserList TUserList;
begin
select TUserRow(username, user_id, created)
bulk collect into vUserList
from
all_users
where
username like piNameStart||'%'
;
return vUserList;
end;
/
Function searching for tables:
create or replace function GetUserTables(piUserNameStart in varchar2) return TTableList
as
vTableList TTableList;
begin
select TTableRow(owner, table_name, status, logging)
bulk collect into vTableList
from
all_tables tab_list,
table(GetUsers(piUserNameStart)) user_list
where
tab_list.owner = user_list.username
;
return vTableList;
end;
/
Usage in code:
declare
vUserId number := 5;
vTableCount number;
begin
select count(1) into vTableCount
from table(GetUserTables(GetUserPrefix(vUserId)));
dbms_output.put_line('Users with name started with "'||GetUserPrefix(vUserId)||'" owns '||vTableCount||' tables');
end;
3. cursor - xml - cursor
It's is a specific case, which may be implemented without user-defined types but have a big performance penalty, involves unneeded type conversion and have a low maintainability.
Function searching for users:
create or replace function GetUsersRef(
piNameStart in varchar2
)
return sys_refcursor
as
cUserList sys_refcursor;
begin
open cUserList for
select * from all_users
where username like piNameStart||'%'
;
return cUserList;
end;
Function searching for tables:
create or replace function GetUserTablesRef(
piUserNameStart in varchar2
)
return sys_refcursor
as
cTableList sys_refcursor;
begin
open cTableList for
select
tab_list.*
from
(
XMLTable('/ROWSET/ROW'
passing xmltype(GetUsersRef(piUserNameStart))
columns
username varchar2(30) path '/ROW/USERNAME'
)
) user_list,
all_tables tab_list
where
tab_list.owner = user_list.username
;
return cTableList;
end;
Usage in code:
declare
vUserId number := 5;
vTableCount number;
begin
select count(1) into vTableCount
from
XMLTable('/ROWSET/ROW'
passing xmltype(GetUserTablesRef(GetUserPrefix(vUserId)))
columns
table_name varchar2(30) path '/ROW/TABLE_NAME'
)
;
dbms_output.put_line('Users with name started with "'||GetUserPrefix(vUserId)||'" owns '||vTableCount||' tables');
end;
Of course, all variants may be mixed, but SQL looks better at least for simple cases:
declare
vUserId number := 5;
vUserPrefix varchar2(100);
vTableCount number;
begin
-- Construct prefix from Id
select max(substr(user_list.username,1,3))
into vUserPrefix
from
all_users user_list
where
user_list.user_id = vUserId
;
-- Count number of tables owned by users with name started with vUserPrefix string
select
count(1) into vTableCount
from
all_users user_list,
all_tables table_list
where
user_list.username like vUserPrefix||'%'
and
table_list.owner = user_list.username
;
dbms_output.put_line('Users with name started with "'||vUserPrefix||'" owns '||vTableCount||' tables');
end;
P.S. All code only for demonstration purposes: no optimizations and so on.

can I use IF statement with more than one value?

I have a data base with daily tables. I want to write a procedure that will create a view to union all the tables for the days of the month. I want to catch months with 31 days or 30, and FEB with 29 days. Is it efficient to use if statement ?
Can I use if statement with more than one value as below?
if(upper(MON)) in ('JAN','MAR','MAY','JUL','AUG','OCT','DEC') THEN {DO SOMETHING}
Given how convoluted your code is, the efficiency of the IF is probably the least of your problems. I guess your company won't spring for the Partitioning licence. (Author's note: this sentence refers to the code posted in the original version of the query which the OP has since redacted.)
However, your approach won't work. You can't use an IF in DDL like that. You need to do something else.
Your view won't compile unless all the tables already exist. So I'm going to assume you have some form of scheduled job which creates those tables and then creates the view. In which case, you can drive the procedure off the data dictionary (Caution: untested code, you may need to fix typos).
CREATE OR REPLACE PROCEDURE monthly_view (
MON_P VARCHAR2,
YEAR_P VARCHAR2 DEFAULT TO_CHAR (SYSDATE, 'YY'))
IS
l_stmt varchar2(32767);
BEGIN
l_stmt := 'CREATE OR REPLACE VIEW monthly_view_'|| UPPER (MON_P)|| '_'|| YEAR_P||' as ';
for r in ( select table_name from user_tables
where table_name like 'LT_'|| UPPER (MON)||'____'|| YEAR_P||'_BCH'
)
loop
if r.table_name != 'LT_'|| UPPER (MON)||'_01_'|| YEAR_P||'_BCH'
then
l_stmt := l_stmt ||' UNION ALL ';
end if;
l_stmt := l_stmt ||' SELECT * FROM '||r.table_name;
end loop;
execute immediate l_stmt;
END;
/
This approach means only the routine which creates the tables needs to worry about the days of the month and whether it's a leap year.
Even better would be to not have separate tables for each month of each year.

Resources