I have to use Hibernate and retrieve data from Oracle but the problem is, that the number of parameters passed to the query is not always the same.
For the sake of simplicity let's consider the following query:
select COL_1, COL_2, ..., COL_N from TAB_1 where COL_1 in (?, ?, ... ?)
The number of parameters passed to in clause is between 1 and 500. If the number is about 1-50 it works quite fast, but for 200 it takes a few seconds to execute the query (parsing, creating explain plan, executing the query). Indexes are created and used - it was checked.
The query is created dynamicly so I use Hibernate Criteria API. For the first query (with > 100 parameters) it takes 3-5 seconds, but for the next one it works faster (even if the number of parameters varies). I would like to improve the response time for the first query. What can I do in that case assuming that Hibernate is a must?
I though about removig this dynamic query, creating a few static queries as named queries in xml file (in that case those queries will be precompiled at the beginning) For example
1) one query if the number of parameters is less then 50.
In this case if we have 30 parameters than the query would look like:
select COL_1, COL_2, ..., COL_N from TAB_1 where COL_1 in (PAR_1, PAR_2, ..., PAR_30, -1, -1 , ..., -1 ?)
2) the second one if the number is between 50 and 100 etc.
The problem is that it's not so simple using named queries and HQL (in JDBC it would be straighforward). In HQL we passed only a list and we don't specify a number of parameters in that list i.e. In fact there is only one query
'from Person where id in (:person_list)'
myQuery.setParameterList("person_list", myList)
Is there any option to solve that?
By the way, I thought that the explain plan is executed for each new query so for example:
(a) select COL_1, COL_2, ..., COL_N from TAB_1 where COL_1 in (?, ?, ..., ?) <100> - explain plan must be created
(b) select COL_1, COL_2, ..., COL_N from TAB_1 where COL_1 in (?, ?, ..., ?) <100> - explain plan won't be created because it already exists in cache
(c) select COL_1, COL_2, ..., COL_N from TAB_1 where COL_1 in (?, ?, ..., ?) <120> - explain plan should be created (there is no explain plan for a query with 120 parameters) but it takes less time in comparison with (a), almost the same as (b) so probably Oracle can create this plan faster if a similar query was executed before
What is the reason for that?
There are a couple of things here. First of all, you cannot bind an IN list, at least I am pretty sure you cannot. I suspect Hibernate is using some sort of trick you put your array contents into a static inlist Oracle can use.
Secondly if this query is executed with lots of different parameters, you must you bind variables or the performance of the entire database will suffer.
That said, there is a way to bind an IN list using a 'trick' which Tom Kyte describes on his blog -
http://tkyte.blogspot.com/2006/01/how-can-i.html
The code in there looks like:
ops$tkyte#ORA10GR2> with bound_inlist
2 as
3 (
4 select
5 substr(txt,
6 instr (txt, ',', 1, level ) + 1,
7 instr (txt, ',', 1, level+1) - instr (txt, ',', 1, level) -1 )
8 as token
9 from (select ','||:txt||',' txt from dual)
10 connect by level <= length(:txt)-length(replace(:txt,',',''))+1
11 )
12 select *
13 from all_users
14 where user_id in (select * from bound_inlist);
USERNAME USER_ID CREATED
------------------------------ ---------- ---------
SYSTEM 5 30-JUN-05
OPS$TKYTE 104 20-JAN-06
The part:
12 select *
13 from all_users
14 where user_id in (select * from bound_inlist);
Is basically where your query goes. The bit above is the trick which splits the comma separated string into a list of values. Instead of binding a list into the :txt placeholder, you would need to convert the list to a string and just bind that.
Are you sure the difference in query times isn't due to caching or load variations on the machine? Parsing the query will take a little time, but several seconds is a long time.
I've worked with IN(...) queries that had up to a 1000 of ids in that list; I can guarantee you that it does not take several seconds to parse / prepare / cache a statement.
Hibernate does indeed auto-expand the parameter list you specify using the actual number of elements in the list you pass, so if you really wanted to keep it "fixed" at a certain level all you need to do is to append enough -1s to the end. However, this is most certainly not the problem especially since we're talking about speeding up the first query run - no statements have been prepared / cached yet anyway.
Did you look at the execution plans for your queries? Both via explain plan and autotrace enabled? Do they differ when you have 30 elements and 120 elements in your list? Does your actual query really look like "select ... from table where id in (...)" you've posted or is it more complex? I'm willing to bet that somewhere between 30 and 120 elements Oracle decides (perhaps mistakenly) that it'll be faster not to use an index, which is why you're seeing the time increase.
Related
This question already has answers here:
SQL IN Clause 1000 item limit
(5 answers)
how to select a list of 10,000 unique ids from dual in oracle SQL
(4 answers)
Closed 2 years ago.
When i pass value more than 1000 in In clause and try to Run Query in Oracle i found error that is "ORA-01795" that maximum number of expressions in a list is 1000 , so let me know how to solve this
A simple way is to put those values into a table and use it (the table) as a subquery, e.g.
select *
from your_table
where id in (select id from your_new_table)
There are other options, but - from my point of view - this is the simplest and easiest to maintain.
The "new table" can even be a global (or private, depending on your database version) temporary table so you'd fill it at the beginning and use throughout the session (or transaction, depending on how you created the GTT). Once you're done, its contents is "lost" and table doesn't occupy space.
The restriction only applies to single column comparisons. A strange looking workaround is to use a two-column comparison with one constant:
So instead of
select *
from some_table
where id in (1, 2, 3, .... 1001);
you can use:
select *
from some_table
where (id, -42) in (1, -42), (2, -42), ... (1001, -42)
Not sure about the performance implications with regards to index usage though (but then a IN clause with more than 1000 values isn't like to use an index to begin with)
Those 3 queries return the same result set but use 2 different technics. Is there an advantage using one over the other? The plan and the execution time are in the same cost range.
Select *
from user_tab_columns
order by data_length desc,table_name, column_name
fetch first 5 rows only
Select *
from user_tab_columns
order by data_length desc,table_name, column_name
fetch next 5 rows only
select *
from (
Select *
from user_tab_columns
order by data_length desc,table_name, column_name
)
where rownum <=5
The keywords first and next as used in the fetch clause are perfect substitutes for each other, they can be used interchangeably - this is stated clearly in the documentation. So you really only have two queries there, not three. (The first two are really identical.)
The first query is easier to write and maintain than the last query. On the other hand, it is only available in Oracle 12.1 and later versions; in Oracle 11.2 and earlier, the only option is your last query.
The fetch clause is more flexible, for example it allows you to specify with ties (to include more than 5 rows if rows with rownum 4, 5, 6 and 7 are tied on the order by criteria, for example).
In my company's application there is a query in oracle using parallel execution (configured to 4 servers), it wasn't me who built it, but the developer put it that way for performance.
The query makes joins between views and tables and the weirdest thing is: sometimes it returns 11k results (incorrect), sometimes 27k results (correct).
After much research I found out that if I removed this parallel thing, it always returns the correct number: 27k. And if I increase the number of server to 6 or 7, it always returns the incorrect number: 11k.
The layout of the query is like this:
SELECT /*+ PARALLEL(NAME, 4) */ * FROM(
SELECT DISTINCT COLUMNS
FROM VIEW
JOIN TABLE1 ON (....)
JOIN TABLE2 ON (....)
JOIN TABLE3 ON (....)
ORDER BY 3
) NAME
Anyone has any idea why? I don't know much about this subject.
In my program there are a lot of situation when i need to get additional information about knowing ids. So i have list of ids, which length may be very long (for example 100000 elements in it).
How i can use this list and transfer in oracle for getting sql without using temp tables?
No i try to use collection:
CREATE TYPE TEST_VARRAY IS VARRAY(5000) OF NUMBER(18);
SELECT G.ID, G.NAME FROM ANY_TABLE G
WHERE G.ID IN
(
SELECT COLUMN_VALUE FROM TABLE(
NEW TEST_VARRAY
(0,1,2,3... and so on ...,995,996,997,998,999)
)
);
there are 1000 numbers. And when I try execute this query the error ORA-00907: missing right parenthesis tips is appeared! But if i delete first 0 (so we have 999 numbers) the sql is executed ok.
What is problem here?
There is a limit in Oracle IN clause.
A comma-delimited list of expressions can contain no more than 1000
expressions. A comma-delimited list of sets of expressions can contain
any number of sets, but each set can contain no more than 1000
expressions.
Read here or here or here
In my opinion, you are misusing collections, at least I am not sure something like you did is good.
As far as I understand you generate this query before run, so what is the problem to do like that?
with ids as (select /*+ materialize */ 1 id from dual union all
select 2 from dual union all
select 3 from dual union all
select 4 from dual union all
/* repeat with the ids you need */
select 9999 from dual)
select *
from yourTable, ids
where yourTable.id = ids.id;
And that's it! Without any limitations, with pure SQL only. I have added materialize hint to ensure it is not performance relevant, but I think it can be skipped.
No temporary tables, no collections, nothing to create and support. Just SQL.
If you will put ids out of with into from clause it will work in any RDBMS (I guess).
I have a quite complicated view (using several layers of views across several database links) which takes a second to return all of it's rows. But, when I ask for distinct rows, it takes considerably more time. I stopped waiting after 4 minutes.
To make my self as clear as possible:
select a, b from compicated_view; -- takes 1 sec (returns 6 rows)
select distinct a, b from compicated_view; -- takes at least 4 minutes
I find that pretty weird, but hey, that's how it is. I guess Oracle messed something up when planing that query. Now, is there a way to force Oracle to first finish the select without distinct, and then do a "select distinct *" on the results? I looked into optimizer hints, but I can't find anything about hinting the order in which distinct is applied (this is first time I'm optimizing a query, obviously :-/).
I'm using Oracle SQl Developer on Oracle 10g EE.
Try:
SELECT DISTINCT A,B FROM (
SELECT A,B FROM COMPLICATED_VIEW
WHERE rownum > 0 );
this forces to materialize the subquery and prevents from view merging/predicate pushing, and likely from changing the original plan of the view.
You may also try NO_MERGE hint:
SELECT /*+ NO_MERGE(alias) */
DISTINCT a,b
FROM (
SELECT a,b FROM COMPLICATED_VIEW
) alias
Since you haven't posted details... try the following:
SELECT DISTINCT A,B
FROM
(SELECT A,B FROM COMPLICATED_VIEW);