clob datatype is causing performace issue - oracle

UPDATED: The code is working as expected but the performance is very slow. When I do a search without including CLOB data then the query runs very fast but if I include CLOB variable in my search the query is very slow. I am using CLOB to pass large string data('aaaaaaa,bbbb,c,ddddd...') and store those data in global table for better performance, I thought doing such will maximize query performance. How can I improve/utilize my CLOB variable for better perfomance? Please look at the code below for more information. Appreciated for any help. I am still struggling with performance can anyone help/provide any suggestions please.
GLOBAL TT GlobalTemp_EMP( //this already exists
emp_refno (30 byte);
)
Create or replace PROCEDURE Employee(
emp_refno IN CLOB
)
AS
Begin
OPEN p_resultset FOR
with inputs ( str ) as ( //red error line here
select to_clob(emp_refno )
from dual
),
prep ( s, n, token, st_pos, end_pos ) as (
select ',' || str || ',', -1, null, null, 1
from inputs
union all
select s, n+1, substr(s, st_pos, end_pos - st_pos),
end_pos + 1, instr(s, ',', 1, n+3)
from prep
where end_pos != 0
)
INSERT into GlobalTemp_EMP //red error line here
select token from prep;
select e.empname, e.empaddress, f.department
from employee e
join department f on e.emp_id = t.emp_id
and e.emp_refno in (SELECT emp_refno from GlobalTemp_EMP) //using GTT In subquery

put this code between BEGIN and OPEN p_resultset FOR : this might have some performance issue though.
INSERT into GlobalTemp_EMP
with inputs ( str ) as (
select to_clob(emp_refno )
from dual
),
prep ( s, n, token, st_pos, end_pos ) as (
select ',' || str || ',', -1, null, null, 1
from inputs
union all
select s, n+1, substr(s, st_pos, end_pos - st_pos),
end_pos + 1, instr(s, ',', 1, n+3)
from prep
where end_pos != 0
)
select token from prep where token is not NULL;

The below doesn't seem to be valid syntax:
GLOBAL TT GlobalTemp_EMP( //this already exists
emp_refno (30 byte);
)
I don't know the reason for using byte semantics, or whether you defined it as a clob or char or varchar2.
If it is currently a clob, then perhaps you could define the column as emp_refno varchar2(30 char) and add a unique index, changing the Employee procedure to only insert new IDs. An index would help the insertions more than when you read it out.
If you want to insert a huge amount of data into GlobalTemp_EMP faster, I would recommend making it a regular table, pre-processing the data (such as in Perl or other language) to split IDs outside Oracle and then use SQL*Loader. Or perhaps an external table.

I don't think using a global temporary table will improve your performance at all (at least without indexes). Are you sure these are CLOBs? At a glance, these seem to be varchars.
To compare CLOBs, you should be using dbms_lob.compare. I think = will do an implicit conversion to a varchar (and truncate), then do the comparison.

Related

Translate hierarchical Oracle query to DB2 query

I work primarily with SAS and Oracle and am still new to DB2. Im faced with needing a hierarchical query to separate a clob into chunks that can be pulled into sas. SAS has a limit of 32K for character variables so I cant just pull the dataset in normally.
I found an old stackoverflow question about the best way to pull a clob into a sas data set but it is written in Oracle.
Import blob through SAS from ORACLE DB
Since I am new to DB2 and the syntax for this type of join seems very different I was hoping to find someone that could help convert it and explain the syntax. I find the Oracle syntax to be much easier to understand. I'm not sure in DB2 if you would use a CTE recursion like this https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/apsg/src/tpc/db2z_xmprecursivecte.html or if you would use hierarchical queries like this https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_71/sqlp/rbafyrecursivequeries.htm
Here is the Oracle query.
SELECT
id
, level as chunk_id
, regexp_substr(clob_value, '.{1,32767}', 1, level, 'n') as clob_chunk
FROM (
SELECT id, clob_value
FROM schema.table
WHERE id = 1
)
CONNECT BY LEVEL <= regexp_count(clob_value, '.{1,32767}',1,'n')
order by id, chunk_id;
The table has two fields the id and the clob_value and would look like this.
ID CLOB_VALUE
1 really large clob
2 medium clob
3 another large clob
The thought is I would want this result. I would only ever be doing this one row at a time where id= which ever row I am processing.
ID CHUNK_ID CLOB
1 1 clob_chunk1of3
1 2 clob_chunk2of3
1 3 clob_chunk3of3
Thanks for any time spent reading and helping.
Here is a solution that should work in DB2 with few changes (but please be advised that I don't know DB2 at all; I am just using Oracle features that are in the SQL Standard, so they should be implemented identically - or almost so - in DB2).
Below I create a table with your sample data; then I show how to chunk it into substrings of length at most 8 characters. Although the strings are short, I defined the column as CLOB and I am using CLOB tools; this should work on much larger CLOBs.
You can make both the chunk size and the id into bind parameters, if needed. In my demo below I hardcoded the chunk size and I show the result for all IDs in the table. In case the CLOB is NULL, I do return one chunk (which is NULL, of course).
Note that touching CLOBs in a query is very expensive; so most of the work is done without touching the CLOBs. I only work on them as little as possible.
PREP WORK
drop table tbl purge; -- If needed
create table tbl (id number, clob_value clob);
insert into tbl (id, clob_value)
select 1, 'really large clob' from dual union all
select 2, 'medium clob' from dual union all
select 3, 'another large clob' from dual union all
select 4, null from dual -- added to check handling
;
commit;
QUERY
with
prep(id, len) as (
select id, dbms_lob.getlength(clob_value)
from tbl
)
, rec(id, len, ord, pos) as (
select id, len, 1, 1
from prep
union all
select id, len, ord + 1, pos + 8
from rec
where len >= pos + 8
)
select id, ord, dbms_lob.substr(clob_value, 8, pos)
from tbl inner join rec using (id)
order by id, ord
;
ID ORD CHUNK
---- ---- --------
1 1 really l
1 2 arge clo
1 3 b
2 1 medium c
2 2 lob
3 1 another
3 2 large cl
3 3 ob
4 1
Another option is to enable the Oracle compatibility in Db2 and just issue the hierarchical query.
This GitHub repository has background information on SQL recursion in DB2, including the Oracle-style syntax and a side by side example (both work against the Db2 sample database):
-- both queries are against the SAMPLE database
-- and should return the same result
SELECT LEVEL, CAST(SPACE((LEVEL - 1) * 4) || '/' || DEPTNAME
AS VARCHAR(40)) AS DEPTNAME
FROM DEPARTMENT
START WITH DEPTNO = 'A00'
CONNECT BY NOCYCLE PRIOR DEPTNO = ADMRDEPT;
WITH tdep(level, deptname, deptno) as (
SELECT 1, CAST( DEPTNAME AS VARCHAR(40)) AS DEPTNAME, deptno
FROM department
WHERE DEPTNO = 'A00'
UNION ALL
SELECT t.LEVEL+1, CAST(SPACE(t.LEVEL * 4) || '/' || d.DEPTNAME
AS VARCHAR(40)) AS DEPTNAME, d.deptno
FROM DEPARTMENT d, tdep t
WHERE d.admrdept=t.deptno and d.deptno<>'A00')
SELECT level, deptname
FROM tdep;

How to add new line separator into data for oracle

I'm working on displaying data from oracle.
is there a way to make the following data inside the table:
example :
'1.somedata, 2.somedata, 3.somedata, 4.somedata, 5.somedata'
to display like:
example:
'1. somedata
2. somedata
3. somedata
4. somedata
5. somedata'
on the interface?
do i add new line separator directly into the data?
or do i separator them into new line when i query it?
or is there any other simple way?
Thanks.
There are so many ways to do this, here is one if you are selecting from a column:
SELECT REPLACE ('1.somedata, 2.somedata, 3.somedata, 4.somedata, 5.somedata', ',', CHR (13) || CHR (10)) AS split
FROM DUAL;
1.somedata
2.somedata
3.somedata
4.somedata
5.somedata
I personally would use the listagg function and use '' as the delimiter.
SELECT LISTAGG(last_name, ' ')
WITHIN GROUP (ORDER BY hire_date, last_name) "Emp_list",
MIN(hire_date) "Earliest"
FROM employees
WHERE department_id = 30;
Remember that Apex is generating a web page, which means the end result is HTML. Apex, however, will also sometimes escape special HTML characters for you, like < and &. Since you're viewing a table, I assume the source of your data is a query and your "somedata" field is a single column. Try this:
SELECT REPLACE( somedata_column, ',', '<br />' )
FROM mytable
You don't say what version of Apex. In Apex 4.x, the column would need to be set to a Standard Report Column, which would stop Apex from the <br> elements. I forget what the column type is in Apex 5.x.
Check below sample query which converts coma separated list data into rows
SELECT substr( '1.AL,2.AL,3.AL,4.AL,5.AL,6.AL,',
( case when rownum = 1 then 1
else instr( '1.AL,2.AL,3.AL,4.AL,5.AL,6.AL,', ',', 1, rownum - 1 ) + 1
end ),
instr( substr( '1.AL,2.AL,3.AL,4.AL,5.AL,6.AL,',
( case when rownum = 1 then 1
else instr( '1.AL,2.AL,3.AL,4.AL,5.AL,6.AL,', ',', 1, rownum - 1 ) + 1
end )
), ',' ) - 1
) as data
FROM dual
CONNECT BY LEVEL <= length( '1.AL,2.AL,3.AL,4.AL,5.AL,6.AL,' ) - length ( replace('1.AL,2.AL,3.AL,4.AL,5.AL,6.AL,', ',') )
Hope this will help you!

PLSQL - Connect By Prior (many to many)

I have a query that uses a connect by statement to order the recursive data. The problem I have is that there is occasionally a one to many or a many to one relationship and I dont know how to deal with it.
SELECT *
FROM (SELECT * FROM bdTable WHERE parentek = t_parKey)
START WITH source is null
CONNECT BY PRIOR target = source
So to explain. I have a source and target columns. About 99% of the time these are a single unique ID. Unfortunately the other 1% of the time there are a grouping of IDs in one of the columns. This table is a flat representation of a flowchart type tool, so there are splits and decisions which can have many outputs and merges which can have many inputs.
To deal with this in loading the data for the table, the unique IDs are concatenated together using the listagg function. So I end up with a Target value of something like '1254143,2356334,6346436,3454363,3462354,442356'.
So when my connect by statement is executed, it works perfectly until it comes to one of these scenarios, at which point it just stops (which is expected of course).
I thought I might be able to use IN or INSTR in some way to get it working, but haven't had any luck yet and I can't find anything on it online.
Any help would be appreciated.....
If you want to join target and source with use of some logic, based on intersection of set of values, listed in each column, then most reliable way to do so is to split strings to collections and operate on collection from prior row and collection from current row to build a tree.
There are a number of techniques to build collection from separated string in Oracle, one of them illustrated in this answer to another question.
Create required collection type:
create or replace type TIdList as table of varchar2(100);
Inner select in your case would look like this:
SELECT
t.*,
(
cast(multiset( -- Convert set of values into collection
select -- Build list of values from separated string
substr(
source,
decode( level, 1, 1, instr(source,',',1,level-1)+1 ),
decode( instr(source,',',1,level), 0, length(source)+1, instr(source,',',1,level) )
-
decode( level, 1, 1, instr(source,',',1,level-1)+1 )
) code
from dual
start with source is not null
connect by instr(source,',',1,level-1) > 0
) as TIdList )
) source_id_list,
(
cast(multiset( -- Convert set of values into collection
select -- Build list of values from separated string
substr(
target,
decode( level, 1, 1, instr(target,',',1,level-1)+1 ),
decode( instr(target,',',1,level), 0, length(target)+1, instr(target,',',1,level) )
-
decode( level, 1, 1, instr(target,',',1,level-1)+1 )
) code
from dual
start with target is not null
connect by instr(target,',',1,level-1) > 0
) as TIdList )
) target_id_list
FROM bdTable t
WHERE t.parentek = t_parKey
Because I don't know which column (source or target) contains separated list, I include column for each.
After building collection(s) it's possible to use multiset operators and available test functions to match target with source. E.g.
with inner_query as (
SELECT
t.*,
(
cast(multiset( -- Convert set of values into collection
select -- Build list of values from separated string
substr(
source,
decode( level, 1, 1, instr(source,',',1,level-1)+1 ),
decode( instr(source,',',1,level), 0, length(source)+1, instr(source,',',1,level) )
-
decode( level, 1, 1, instr(source,',',1,level-1)+1 )
) code
from dual
start with source is not null
connect by instr(source,',',1,level-1) > 0
) as TIdList )
) source_id_list,
(
cast(multiset( -- Convert set of values into collection
select -- Build list of values from separated string
substr(
target,
decode( level, 1, 1, instr(target,',',1,level-1)+1 ),
decode( instr(target,',',1,level), 0, length(target)+1, instr(target,',',1,level) )
-
decode( level, 1, 1, instr(target,',',1,level-1)+1 )
) code
from dual
start with target is not null
connect by instr(target,',',1,level-1) > 0
) as TIdList )
) target_id_list
FROM bdTable t
WHERE t.parentek = t_parKey
)
select
level lvl,
tree_list.*
from
inner_query tree_list
start with
source is null
connect by
nvl(cardinality(prior target_id_list MULTISET INTERSECT source_id_list),0) > 0
If only one column can contain list of values, then MEMBER OF construct are useful.

PL/SQL query IN comma deliminated string

I am developing an application in Oracle APEX. I have a string with user id's that is comma deliminated which looks like this,
45,4932,20,19
This string is stored as
:P5_USER_ID_LIST
I want a query that will find all users that are within this list my query looks like this
SELECT * FROM users u WHERE u.user_id IN (:P5_USER_ID_LIST);
I keep getting an Oracle error: Invalid number. If I however hard code the string into the query it works. Like this:
SELECT * FROM users u WHERE u.user_id IN (45,4932,20,19);
Anyone know why this might be an issue?
A bind variable binds a value, in this case the string '45,4932,20,19'. You could use dynamic SQL and concatenation as suggested by Randy, but you would need to be very careful that the user is not able to modify this value, otherwise you have a SQL Injection issue.
A safer route would be to put the IDs into an Apex collection in a PL/SQL process:
declare
array apex_application_global.vc_arr2;
begin
array := apex_util.string_to_table (:P5_USER_ID_LIST, ',');
apex_collection.create_or_truncate_collection ('P5_ID_COLL');
apex_collection.add_members ('P5_ID_COLL', array);
end;
Then change your query to:
SELECT * FROM users u WHERE u.user_id IN
(SELECT c001 FROM apex_collections
WHERE collection_name = 'P5_ID_COLL')
An easier solution is to use instr:
SELECT * FROM users u
WHERE instr(',' || :P5_USER_ID_LIST ||',' ,',' || u.user_id|| ',', 1) !=0;
tricks:
',' || :P5_USER_ID_LIST ||','
to make your string ,45,4932,20,19,
',' || u.user_id|| ','
to have i.e. ,32, and avoid to select the 32 being in ,4932,
I have faced this situation several times and here is what i've used:
SELECT *
FROM users u
WHERE ','||to_char(:P5_USER_ID_LIST)||',' like '%,'||to_char(u.user_id)||',%'
ive used the like operator but you must be a little carefull of one aspect here: your item P5_USER_ID_LIST must be ",45,4932,20,19," so that like will compare with an exact number "',45,'".
When using it like this, the select will not mistake lets say : 5 with 15, 155, 55.
Try it out and let me know how it goes;)
Cheers ,
Alex
Create a native query rather than using "createQuery/createNamedQuery"
The reason this is an issue is that you cannot just bind an in list the way you want, and just about everyone makes this mistake at least once as they are learning Oracle (and probably SQL!).
When you bind the string '32,64,128', it effectively becomes a query like:
select ...
from t
where t.c1 in ('32,64,128')
To Oracle this is totally different to:
select ...
from t
where t.c1 in (32,64,128)
The first example has a single string value in the in list and the second has a 3 numbers in the in list. The reason you get an invalid number error is because Oracle attempts to cast the string '32,64,128' into a number, which it cannot do due to the commas in the string.
A variation of this "how do I bind an in list" question has come up on here quite a few times recently.
Generically, and without resorting to any PLSQL, worrying about SQL Injection or not binding the query correctly, you can use this trick:
with bound_inlist
as
(
select
substr(txt,
instr (txt, ',', 1, level ) + 1,
instr (txt, ',', 1, level+1) - instr (txt, ',', 1, level) -1 )
as token
from (select ','||:txt||',' txt from dual)
connect by level <= length(:txt)-length(replace(:txt,',',''))+1
)
select *
from bound_inlist a, users u
where a.token = u.id;
If possible the best idea may be to not store your user ids in csv! Put them in a table or failing that an array etc. You cannot bind a csv field as a number.
Please dont use: WHERE ','||to_char(:P5_USER_ID_LIST)||',' like '%,'||to_char(u.user_id)||',%' because you'll force a full table scan although with the users table you may not have that many so the impact will be low but against other tables in an enterprise environment this is a problem.
EDIT: I have put together a script to demonstrate the differences between the regex method and the wildcard like method. Not only is regex faster but it's also a lot more robust.
-- Create table
create table CSV_TEST
(
NUM NUMBER not null,
STR VARCHAR2(20)
);
create sequence csv_test_seq;
begin
for j in 1..10 loop
for i in 1..500000 loop
insert into csv_test( num, str ) values ( csv_test_seq.nextval, to_char( csv_test_seq.nextval ));
end loop;
commit;
end loop;
end;
/
-- Create/Recreate primary, unique and foreign key constraints
alter table CSV_TEST
add constraint CSV_TEST_PK primary key (NUM)
using index ;
alter table CSV_TEST
add constraint CSV_TEST_FK unique (STR)
using index;
select sysdate from dual;
select *
from csv_test t
where t.num in ( Select Regexp_Substr('100001, 100002, 100003 , 100004, 100005','[^,]+', 1, Level) From Dual
Connect By Regexp_Substr('100001, 100002,100003, 100004, 100005', '[^,]+', 1, Level) Is Not Null);
select sysdate from dual;
select *
from csv_test t
where ('%,' || '100001,100002, 100003, 100004 ,100005' || ',%') like '%,' || num || ',%';
select sysdate from dual;
select *
from csv_test t
where t.num in ( Select Regexp_Substr('100001, 100002, 100003 , 100004, 100005','[^,]+', 1, Level) From Dual
Connect By Regexp_Substr('100001, 100002,100003, 100004, 100005', '[^,]+', 1, Level) Is Not Null);
select sysdate from dual;
select *
from csv_test t
where ('%,' || '100001,100002, 100003, 100004 ,100005' || ',%') like '%,' || num || ',%';
select sysdate from dual;
drop table csv_test;
drop sequence csv_test_seq;
Solution from Tony Andrews works for me. The process should be added to "Page processing" >> "After submit">> "Processes".
As you are Storing User Ids as String so You can Easily match String Using Like as Below
SELECT * FROM users u WHERE u.user_id LIKE '%'||(:P5_USER_ID_LIST)||'%'
For Example
:P5_USER_ID_LIST = 45,4932,20,19
Your Query Surely Will return Any of 1 User Id which Matches to Users table
This Will Surely Resolve Your Issue , Enjoy
you will need to run this as dynamic SQL.
create the entire string, then run it dynamically.

How do I display a field's hidden characters in the result of a query in Oracle?

I have two rows that have a varchar column that are different according to a Java .equals(). I can't easily change or debug the Java code that's running against this particular database but I do have access to do queries directly against the database using SQLDeveloper. The fields look the same to me (they are street addresses with two lines separated by some new line or carriage feed/new line combo).
Is there a way to see all of the hidden characters as the result of a query?I'd like to avoid having to use the ascii() function with substr() on each of the rows to figure out which hidden character is different.
I'd also accept some query that shows me which character is the first difference between the two fields.
Try
select dump(column_name) from table
More information is in the documentation.
As for finding the position where the character differs, this might give you an idea:
create table tq84_compare (
id number,
col varchar2(20)
);
insert into tq84_compare values (1, 'hello world');
insert into tq84_compare values (2, 'hello' || chr(9) || 'world');
with c as (
select
(select col from tq84_compare where id = 1) col1,
(select col from tq84_compare where id = 2) col2
from
dual
),
l as (
select
level l from dual
start with 1=1
connect by level < (select length(c.col1) from c)
)
select
max(l.l) + 1position
from c,l
where substr(c.col1,1,l.l) = substr(c.col2,1,l.l);
SELECT DUMP('€ÁÑ', 1016)
FROM DUAL
... will print something like:
Typ=96 Len=3 CharacterSet=WE8MSWIN1252: 80,c1,d1

Resources