Related
UPDATED: The code is working as expected but the performance is very slow. When I do a search without including CLOB data then the query runs very fast but if I include CLOB variable in my search the query is very slow. I am using CLOB to pass large string data('aaaaaaa,bbbb,c,ddddd...') and store those data in global table for better performance, I thought doing such will maximize query performance. How can I improve/utilize my CLOB variable for better perfomance? Please look at the code below for more information. Appreciated for any help. I am still struggling with performance can anyone help/provide any suggestions please.
GLOBAL TT GlobalTemp_EMP( //this already exists
emp_refno (30 byte);
)
Create or replace PROCEDURE Employee(
emp_refno IN CLOB
)
AS
Begin
OPEN p_resultset FOR
with inputs ( str ) as ( //red error line here
select to_clob(emp_refno )
from dual
),
prep ( s, n, token, st_pos, end_pos ) as (
select ',' || str || ',', -1, null, null, 1
from inputs
union all
select s, n+1, substr(s, st_pos, end_pos - st_pos),
end_pos + 1, instr(s, ',', 1, n+3)
from prep
where end_pos != 0
)
INSERT into GlobalTemp_EMP //red error line here
select token from prep;
select e.empname, e.empaddress, f.department
from employee e
join department f on e.emp_id = t.emp_id
and e.emp_refno in (SELECT emp_refno from GlobalTemp_EMP) //using GTT In subquery
put this code between BEGIN and OPEN p_resultset FOR : this might have some performance issue though.
INSERT into GlobalTemp_EMP
with inputs ( str ) as (
select to_clob(emp_refno )
from dual
),
prep ( s, n, token, st_pos, end_pos ) as (
select ',' || str || ',', -1, null, null, 1
from inputs
union all
select s, n+1, substr(s, st_pos, end_pos - st_pos),
end_pos + 1, instr(s, ',', 1, n+3)
from prep
where end_pos != 0
)
select token from prep where token is not NULL;
The below doesn't seem to be valid syntax:
GLOBAL TT GlobalTemp_EMP( //this already exists
emp_refno (30 byte);
)
I don't know the reason for using byte semantics, or whether you defined it as a clob or char or varchar2.
If it is currently a clob, then perhaps you could define the column as emp_refno varchar2(30 char) and add a unique index, changing the Employee procedure to only insert new IDs. An index would help the insertions more than when you read it out.
If you want to insert a huge amount of data into GlobalTemp_EMP faster, I would recommend making it a regular table, pre-processing the data (such as in Perl or other language) to split IDs outside Oracle and then use SQL*Loader. Or perhaps an external table.
I don't think using a global temporary table will improve your performance at all (at least without indexes). Are you sure these are CLOBs? At a glance, these seem to be varchars.
To compare CLOBs, you should be using dbms_lob.compare. I think = will do an implicit conversion to a varchar (and truncate), then do the comparison.
I work primarily with SAS and Oracle and am still new to DB2. Im faced with needing a hierarchical query to separate a clob into chunks that can be pulled into sas. SAS has a limit of 32K for character variables so I cant just pull the dataset in normally.
I found an old stackoverflow question about the best way to pull a clob into a sas data set but it is written in Oracle.
Import blob through SAS from ORACLE DB
Since I am new to DB2 and the syntax for this type of join seems very different I was hoping to find someone that could help convert it and explain the syntax. I find the Oracle syntax to be much easier to understand. I'm not sure in DB2 if you would use a CTE recursion like this https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/apsg/src/tpc/db2z_xmprecursivecte.html or if you would use hierarchical queries like this https://www.ibm.com/support/knowledgecenter/en/ssw_ibm_i_71/sqlp/rbafyrecursivequeries.htm
Here is the Oracle query.
SELECT
id
, level as chunk_id
, regexp_substr(clob_value, '.{1,32767}', 1, level, 'n') as clob_chunk
FROM (
SELECT id, clob_value
FROM schema.table
WHERE id = 1
)
CONNECT BY LEVEL <= regexp_count(clob_value, '.{1,32767}',1,'n')
order by id, chunk_id;
The table has two fields the id and the clob_value and would look like this.
ID CLOB_VALUE
1 really large clob
2 medium clob
3 another large clob
The thought is I would want this result. I would only ever be doing this one row at a time where id= which ever row I am processing.
ID CHUNK_ID CLOB
1 1 clob_chunk1of3
1 2 clob_chunk2of3
1 3 clob_chunk3of3
Thanks for any time spent reading and helping.
Here is a solution that should work in DB2 with few changes (but please be advised that I don't know DB2 at all; I am just using Oracle features that are in the SQL Standard, so they should be implemented identically - or almost so - in DB2).
Below I create a table with your sample data; then I show how to chunk it into substrings of length at most 8 characters. Although the strings are short, I defined the column as CLOB and I am using CLOB tools; this should work on much larger CLOBs.
You can make both the chunk size and the id into bind parameters, if needed. In my demo below I hardcoded the chunk size and I show the result for all IDs in the table. In case the CLOB is NULL, I do return one chunk (which is NULL, of course).
Note that touching CLOBs in a query is very expensive; so most of the work is done without touching the CLOBs. I only work on them as little as possible.
PREP WORK
drop table tbl purge; -- If needed
create table tbl (id number, clob_value clob);
insert into tbl (id, clob_value)
select 1, 'really large clob' from dual union all
select 2, 'medium clob' from dual union all
select 3, 'another large clob' from dual union all
select 4, null from dual -- added to check handling
;
commit;
QUERY
with
prep(id, len) as (
select id, dbms_lob.getlength(clob_value)
from tbl
)
, rec(id, len, ord, pos) as (
select id, len, 1, 1
from prep
union all
select id, len, ord + 1, pos + 8
from rec
where len >= pos + 8
)
select id, ord, dbms_lob.substr(clob_value, 8, pos)
from tbl inner join rec using (id)
order by id, ord
;
ID ORD CHUNK
---- ---- --------
1 1 really l
1 2 arge clo
1 3 b
2 1 medium c
2 2 lob
3 1 another
3 2 large cl
3 3 ob
4 1
Another option is to enable the Oracle compatibility in Db2 and just issue the hierarchical query.
This GitHub repository has background information on SQL recursion in DB2, including the Oracle-style syntax and a side by side example (both work against the Db2 sample database):
-- both queries are against the SAMPLE database
-- and should return the same result
SELECT LEVEL, CAST(SPACE((LEVEL - 1) * 4) || '/' || DEPTNAME
AS VARCHAR(40)) AS DEPTNAME
FROM DEPARTMENT
START WITH DEPTNO = 'A00'
CONNECT BY NOCYCLE PRIOR DEPTNO = ADMRDEPT;
WITH tdep(level, deptname, deptno) as (
SELECT 1, CAST( DEPTNAME AS VARCHAR(40)) AS DEPTNAME, deptno
FROM department
WHERE DEPTNO = 'A00'
UNION ALL
SELECT t.LEVEL+1, CAST(SPACE(t.LEVEL * 4) || '/' || d.DEPTNAME
AS VARCHAR(40)) AS DEPTNAME, d.deptno
FROM DEPARTMENT d, tdep t
WHERE d.admrdept=t.deptno and d.deptno<>'A00')
SELECT level, deptname
FROM tdep;
I have a string as follows: ABCAPP9 Xore-Done-1. I want to chunk the string to get 4 elements separately at a given time in pl sql. Pls tell me the 4 different queries to get the following 4 results separately. Thanks
ABCAPP9
Xore
Done
1
REGEXP_SUBSTR ('ABCAPP9 Xore-Done-1', '[^[:space:]-]+', 1, n)
will give you the n-th part. Change n with the number you want. Here are all:
SELECT REGEXP_SUBSTR ('ABCAPP9 Xore-Done-1', '[^[:space:]-]+', 1, LEVEL)
FROM DUAL
CONNECT BY REGEXP_SUBSTR ('ABCAPP9 Xore-Done-1', '[^[:space:]-]+', 1, LEVEL) IS NOT NULL
This should be a comment really to #Mottor but due to no formatting in comments I need to make it here.
A word of warning. As long as all elements of your string will be present and the delimiters can NEVER be next to each other you will be ok. However, the regex format of '[^<delimiter>]+' commonly used for parsing strings will not return the correct value if there is a NULL element in the list! See this post for proof: https://stackoverflow.com/a/31464699/2543416. To test in your example, remove the substring "Xore", leaving the space and hyphen next to each other:
SQL> SELECT REGEXP_SUBSTR ('ABCAPP9 -Done-1', '[^[:space:]-]+', 1, LEVEL)
FROM DUAL
CONNECT BY REGEXP_SUBSTR ('ABCAPP9 -Done-1', '[^[:space:]-]+', 1, LEVEL) IS NOT NULL;
REGEXP_SUBSTR('
---------------
ABCAPP9
Done
1
The 2nd element should be NULL, but "Done" is returned instead! Not good if the position is important.
Use this format instead to handle NULLs and return the correct string element in the correct position (shown here with "Xore" removed and thus a NULL returned in that position to prove it handles the NULL):
SQL> with tbl(str) as (
select 'ABCAPP9 -Done-1' from dual
)
select regexp_substr(str, '(.*?)( |-|$)', 1, level, NULL, 1)
from tbl
connect by regexp_substr(str, '(.*?)( |-|$)', 1, level) is not null;
REGEXP_SUBSTR(S
---------------
ABCAPP9
Done
1
SQL>
I shudder to think of all the bad data being returned out there.
So user2153047, if you are still with me, for your need if you want the 3rd element (and handle the NULL) you would use:
SQL> select regexp_substr('ABCAPP9 -Done-1', '(.*?)( |-|$)', 1, 3, NULL, 1) "3rd"
from dual;
3rd
----
Done
The original problem was, that I was trying to find out if one of the points in set is inside given polygon. I was getting no results, so I've reduced it to a most simple case, but still got no results.
The polygon geometry looks like this(small square around the center of coordinates):
geom1 := SDO_GEOMETRY(2003, 8307, null, SDO_ELEM_INFO_ARRAY(1,1003,1),
SDO_ORDINATE_ARRAY(10,10, 10,-10, -10,-10, -10,10))
Now I was trying to find out, if point [0,0] is inside that polygon using SDO_RELATE.
sdo_relate(geom1,
SDO_GEOMETRY('MULTIPOINT((0 0))',8307),
'mask=anyinteract')
I've used anyinteract, because according to Oracle documentation:
ANYINTERACT: The objects are non-disjoint.
The point inside the polygon is clearly non-disjoint with it. So I've spend about an hour trying to initialize the points in different way and checking coordinates and everything, before I tried to put there CONTAINS instead of ANYINTERACT and finally got the desired output.
So my questions are:
Is this a bug? The contains parameter is clearly more strict, than anyinteract
What is the best way to check if any of the points in the set lies in the polygon? Contains doesn't help here, because if any of the points lies outside, the result od SDO_RELATE is false. There's no intersection type more fitting, than anyinteract, which doesn't work.
First of all your polygon is malformed: it does not close. For a polygon, the last point must match the first point. You can detect the error by doing:
SQL> select sdo_geom.validate_geometry_with_context (SDO_GEOMETRY(2003, 8307, null, SDO_ELEM_INFO_ARRAY(1,1003,1),SDO_ORDINATE_ARRAY(10,10, 10,-10, -10,-10, -10,10)), 0.05) from dual;
13348 [Element <1>] [Ring <1>]
1 row selected.
ORA-13348: polygon boundary is not closed
Once you correct that, you get a different error:
SQL> select sdo_geom.validate_geometry_with_context (SDO_GEOMETRY(2003, 8307, null, SDO_ELEM_INFO_ARRAY(1,1003,1),SDO_ORDINATE_ARRAY(10,10, 10,-10, -10,-10, -10,10, 10,10)), 0.05) from dual;
13367 [Element <1>] [Ring <1>]
1 row selected.
ORA-13367: wrong orientation for interior/exterior rings
For a polygon, the points must be in a counter-clockwise orientation. Once you correct that, the polygon is correct:
SQL> select sdo_geom.validate_geometry_with_context (SDO_GEOMETRY(2003, 8307, null, SDO_ELEM_INFO_ARRAY(1,1003,1),SDO_ORDINATE_ARRAY(10,10, -10,10, -10,-10, 10,-10, 10,10)), 0.05) from dual;
TRUE
1 row selected.
And queries work properly. Here is an example:
create table t1 (id number, note varchar2(20), geom sdo_geometry);
insert into t1 (id, note, geom)
values (
1,
'Not closed',
SDO_GEOMETRY(2003, 8307, null, SDO_ELEM_INFO_ARRAY(1,1003,1), SDO_ORDINATE_ARRAY(10,10, 10,-10, -10,-10, -10,10))
);
insert into t1 (id, note, geom)
values (
2,
'Wrong orientation',
SDO_GEOMETRY(2003, 8307, null, SDO_ELEM_INFO_ARRAY(1,1003,1), SDO_ORDINATE_ARRAY(10,10, 10,-10, -10,-10, -10,10, 10,10))
);
insert into t1 (id, note, geom)
values (
3,
'Valid',
SDO_GEOMETRY(2003, 8307, null, SDO_ELEM_INFO_ARRAY(1,1003,1), SDO_ORDINATE_ARRAY(10,10, -10,10, -10,-10, 10,-10, 10,10))
);
commit;
insert into user_sdo_geom_metadata (table_name, column_name, diminfo, srid)
values (
'T1',
'GEOM',
sdo_dim_array (
sdo_dim_element ('Long',-180,180,0.5),
sdo_dim_element ('Lat',-90,90,0.5)
),
8307
);
commit;
create index t1_sx on t1 (geom) indextype is mdsys.spatial_index;
The test table contains the three variants of your polygon. Let's try a query:
select *
from t1
where sdo_relate(
geom,
SDO_GEOMETRY('MULTIPOINT((0 0))',8307),
'mask=anyinteract'
) = 'TRUE';
which returns the correct result: only the valid polygon is correctly identified as containing your point:
ID NOTE GEOM(SDO_GTYPE, SDO_SRID, SDO_POINT(X, Y, Z), SDO_ELEM_INFO, SDO_ORDINATES)
---------- -------------------- -------------------------------------------------------------------------------
3 Valid SDO_GEOMETRY(2003, 8307, NULL, SDO_ELEM_INFO_ARRAY(1, 1003, 1), SDO_ORDINATE_ARRAY(10, 10, -10, 10, -10, -10, 10, -10, 10, 10))
1 row selected.
As for the difference between ANYINTERACT and CONTAINS for point-in-polygon it can impact those points that are on the boundary of the polygon (or more precisely within tolerance of the boundary). An ANYINTERACT search returns those. A CONTAINS/INSIDE search does not.
This can be important for example when you want to distribute points to polygons (like customer point locations in sales regions) and some customer location points happen to fall on the boundary between adjacent regions: an ANYINTERACT search will report that those locations are in both regions. An INSIDE search will report that they are in neither.
Finally, use the simpler syntax for your queries:
select *
from t1
where sdo_anyinteract(
geom,
SDO_GEOMETRY('MULTIPOINT((0 0))',8307)
) = 'TRUE';
Some info on how to get the message corresponding to an error code (like 13348). I usually use this technique:
set serveroutput on
exec dbms_output.put_line(sqlerrm(-13348))
That will print out the full error message. Notice that you must pass the error with a minus sign.
I am developing an application in Oracle APEX. I have a string with user id's that is comma deliminated which looks like this,
45,4932,20,19
This string is stored as
:P5_USER_ID_LIST
I want a query that will find all users that are within this list my query looks like this
SELECT * FROM users u WHERE u.user_id IN (:P5_USER_ID_LIST);
I keep getting an Oracle error: Invalid number. If I however hard code the string into the query it works. Like this:
SELECT * FROM users u WHERE u.user_id IN (45,4932,20,19);
Anyone know why this might be an issue?
A bind variable binds a value, in this case the string '45,4932,20,19'. You could use dynamic SQL and concatenation as suggested by Randy, but you would need to be very careful that the user is not able to modify this value, otherwise you have a SQL Injection issue.
A safer route would be to put the IDs into an Apex collection in a PL/SQL process:
declare
array apex_application_global.vc_arr2;
begin
array := apex_util.string_to_table (:P5_USER_ID_LIST, ',');
apex_collection.create_or_truncate_collection ('P5_ID_COLL');
apex_collection.add_members ('P5_ID_COLL', array);
end;
Then change your query to:
SELECT * FROM users u WHERE u.user_id IN
(SELECT c001 FROM apex_collections
WHERE collection_name = 'P5_ID_COLL')
An easier solution is to use instr:
SELECT * FROM users u
WHERE instr(',' || :P5_USER_ID_LIST ||',' ,',' || u.user_id|| ',', 1) !=0;
tricks:
',' || :P5_USER_ID_LIST ||','
to make your string ,45,4932,20,19,
',' || u.user_id|| ','
to have i.e. ,32, and avoid to select the 32 being in ,4932,
I have faced this situation several times and here is what i've used:
SELECT *
FROM users u
WHERE ','||to_char(:P5_USER_ID_LIST)||',' like '%,'||to_char(u.user_id)||',%'
ive used the like operator but you must be a little carefull of one aspect here: your item P5_USER_ID_LIST must be ",45,4932,20,19," so that like will compare with an exact number "',45,'".
When using it like this, the select will not mistake lets say : 5 with 15, 155, 55.
Try it out and let me know how it goes;)
Cheers ,
Alex
Create a native query rather than using "createQuery/createNamedQuery"
The reason this is an issue is that you cannot just bind an in list the way you want, and just about everyone makes this mistake at least once as they are learning Oracle (and probably SQL!).
When you bind the string '32,64,128', it effectively becomes a query like:
select ...
from t
where t.c1 in ('32,64,128')
To Oracle this is totally different to:
select ...
from t
where t.c1 in (32,64,128)
The first example has a single string value in the in list and the second has a 3 numbers in the in list. The reason you get an invalid number error is because Oracle attempts to cast the string '32,64,128' into a number, which it cannot do due to the commas in the string.
A variation of this "how do I bind an in list" question has come up on here quite a few times recently.
Generically, and without resorting to any PLSQL, worrying about SQL Injection or not binding the query correctly, you can use this trick:
with bound_inlist
as
(
select
substr(txt,
instr (txt, ',', 1, level ) + 1,
instr (txt, ',', 1, level+1) - instr (txt, ',', 1, level) -1 )
as token
from (select ','||:txt||',' txt from dual)
connect by level <= length(:txt)-length(replace(:txt,',',''))+1
)
select *
from bound_inlist a, users u
where a.token = u.id;
If possible the best idea may be to not store your user ids in csv! Put them in a table or failing that an array etc. You cannot bind a csv field as a number.
Please dont use: WHERE ','||to_char(:P5_USER_ID_LIST)||',' like '%,'||to_char(u.user_id)||',%' because you'll force a full table scan although with the users table you may not have that many so the impact will be low but against other tables in an enterprise environment this is a problem.
EDIT: I have put together a script to demonstrate the differences between the regex method and the wildcard like method. Not only is regex faster but it's also a lot more robust.
-- Create table
create table CSV_TEST
(
NUM NUMBER not null,
STR VARCHAR2(20)
);
create sequence csv_test_seq;
begin
for j in 1..10 loop
for i in 1..500000 loop
insert into csv_test( num, str ) values ( csv_test_seq.nextval, to_char( csv_test_seq.nextval ));
end loop;
commit;
end loop;
end;
/
-- Create/Recreate primary, unique and foreign key constraints
alter table CSV_TEST
add constraint CSV_TEST_PK primary key (NUM)
using index ;
alter table CSV_TEST
add constraint CSV_TEST_FK unique (STR)
using index;
select sysdate from dual;
select *
from csv_test t
where t.num in ( Select Regexp_Substr('100001, 100002, 100003 , 100004, 100005','[^,]+', 1, Level) From Dual
Connect By Regexp_Substr('100001, 100002,100003, 100004, 100005', '[^,]+', 1, Level) Is Not Null);
select sysdate from dual;
select *
from csv_test t
where ('%,' || '100001,100002, 100003, 100004 ,100005' || ',%') like '%,' || num || ',%';
select sysdate from dual;
select *
from csv_test t
where t.num in ( Select Regexp_Substr('100001, 100002, 100003 , 100004, 100005','[^,]+', 1, Level) From Dual
Connect By Regexp_Substr('100001, 100002,100003, 100004, 100005', '[^,]+', 1, Level) Is Not Null);
select sysdate from dual;
select *
from csv_test t
where ('%,' || '100001,100002, 100003, 100004 ,100005' || ',%') like '%,' || num || ',%';
select sysdate from dual;
drop table csv_test;
drop sequence csv_test_seq;
Solution from Tony Andrews works for me. The process should be added to "Page processing" >> "After submit">> "Processes".
As you are Storing User Ids as String so You can Easily match String Using Like as Below
SELECT * FROM users u WHERE u.user_id LIKE '%'||(:P5_USER_ID_LIST)||'%'
For Example
:P5_USER_ID_LIST = 45,4932,20,19
Your Query Surely Will return Any of 1 User Id which Matches to Users table
This Will Surely Resolve Your Issue , Enjoy
you will need to run this as dynamic SQL.
create the entire string, then run it dynamically.