I have a vey huge query. It is rather large, so i will not post it here(it has 6 levels of nested queries with ordering and grouping). Query has 2 parameters that are passed to it via PreparedStatement.setString(index, value). When I execute my query through SQL Developer(replacing query parameters to actual values before it by hand) the query runs about 10 seconds and return approximately 15000 rows. But when I try to run it through java program using PreparedStament with varibales it fails with ORA-01652(unable to extend temp segment). I have tried to use simple Statement from java program - it works fine. Also when I use preparedStatement without variables(don't use setString(), but specify parameters by hand) it works fine too.
So, I suspect that problem is in PreparedStatemnt parameters.
How does the mechanism of that parameters work? Why simple statement works fine but prepared one fails?
You're probably running into issues with bind variable peeking.
For the same query, the best plan can be significantly different depending on the actual bind variables. In 10g, Oracle builds the execution plan based on the first set of bind variables used. 11g mostly fixed this problem with adaptive cursor sharing, a feature that creates multiple plans for different bind variables.
Here are some ideas for solving this problem:
Use literals This isn't always as bad as people assume. If the good version of your query runs in 10 seconds, the overhead of hard-parsing the query will be negligible. But you may need to be careful to avoid SQL injection.
Force a hard-parse There are a few ways to force Oracle to hard-parse every query. One method is to call DBMS_STATS with NO_INVALIDATE=>FALSE on one of the tables in the query.
Disable bind-variable peeking / hints You can do this by removing the relevant histograms, or using one of the parameters in the link provided by OldProgrammer. This will stabilize your plan, but will not necessarily pick the correct plan. You may also need to use hints to pick the right plan. But then you may not have the right plan for every combination of inputs.
Upgrade to 11g This may not be an option, but this issue is another good reason to start planning an upgrade.
Related
I have an Oracle bind query that is extremely slow (about 2 minutes) when it executes in my C# program but runs very quickly in SQL Developer. It has two parameters that hit the tables index:
select t.Field1, t.Field2
from theTable t
where t.key1=:key1
and t.key2=:key2
Also, if I remove the bind variables and create dynamic sql, it runs just like it does in SQL Developer.
Any suggestion?
BTW, I'm using ODP.
If you are replacing the bind variables with static varibles in sql developer, then you're not really running the same test. Make sure you use the bind varibles, and if it's also slow you're just getting bit by a bad cached execution plan. Updating the stats on that table should resolve it.
However if you are actually using bind variables in sql developers then keep reading. The TLDR version is that parameters that ODP.net run under sometimes cause a slightly more pessimistic approach. Start with updating the stats, but have your dba capture the execution plan under both scenarios and compare to confirm.
I'm reposting my answer from here: https://stackoverflow.com/a/14712992/852208
I considered flagging yours as a duplicate but your title is a little more concise since it identifies the query does run fast in sql developer. I'll welcome advice on handling in another manner.
Adding the following to your config will send odp.net tracing info to a log file:
This will probably only be helpful if you can find a large gap in time. Chances are rows are actually coming in, just at a slower pace.
Try adding "enlist=false" to your connection string. I don't consider this a solution since it effecitively disables distributed transactions but it should help you isolate the issue. You can get a little bit more information from an oracle forumns post:
From an ODP perspective, all we can really point out is that the
behavior occurs when OCI_ATR_EXTERNAL_NAME and OCI_ATR_INTERNAL_NAME
are set on the underlying OCI connection (which is what happens when
distrib tx support is enabled).
I'd guess what you're not seeing is that the execution plan is actually different (meaning the actual performance hit is actually occuring on the server) between the odp.net call and the sql developer call. Have your dba trace the connection and obtain execution plans from both the odp.net call and the call straight from SQL Developer (or with the enlist=false parameter).
If you confirm different execution plans or if you want to take a preemptive shot in the dark, update the statistics on the related tables. In my case this corrected the issue, indicating that execution plan generation doesn't really follow different rules for the different types of connections but that the cost analysis is just slighly more pesimistic when a distributed transaction might be involved. Query hints to force an execution plan are also an option but only as a last resort.
Finally, it could be a network issue. If your odp.net install is using a fresh oracle home (which I would expect unless you did some post-install configuring) then the tnsnames.ora could be different. Host names in tnsnams might not be fully qualified, creating more delays resolving the server. I'd only expect the first attempt (and not subsequent attempts) to be slow in this case so I don't think it's the issue but I thought it should be mentioned.
Are the parameters bound to the correct data type in C#? Are the columns key1 and key2 numbers, but the parameters :key1 and :key2 are strings? If so, the query may return the correct results but will require implicit conversion. That implicit conversion is like using a function to_char(key1), which prevents an index from being used.
Please also check what is the number of rows returned by the query. If the number is big then possibly C# is fetching all rows and the other tool first pocket only. Fetching all rows may require many more disk reads in that case, which is slower. To check this try to run in SQL Developer:
SELECT COUNT(*) FROM (
select t.Field1, t.Field2
from theTable t
where t.key1=:key1
and t.key2=:key2
)
The above query should fetch the maximum number of database blocks.
Nice tool in such cases is tkprof utility which shows SQL execution plan which may be different in cases above (however it should not be).
It is also possible that you have accidentally connected to different databases. In such cases it is nice to compare results of queries.
Since you are raising "Bind is slow" I assume you have checked the SQL without binds and it was fast. In 99% using binds makes things better. Please check if query with constants will run fast. If yes than problem may be implicit conversion of key1 or key2 column (ex. t.key1 is a number and :key1 is a string).
I'm in the need to rewrite some query made by OBIEE
My need is to change all the literal value in bind variable but I didn't find how to use bind variables
Someone can help me?
Thanks
First, a statement:
If you are utilizing ADF (Oracle Application Development Framework) with OBIEE, there is a setting in the ADF layer which allows us to specify how the View Criteria and the WHERE clauses used in the queries fired on the VOs should be handled.
By default the setting useBindVarsForViewCriteriaLiterals is set to "False" in adf-config.xml. If this setting is False, then ADF will generate SQLs with literals for the view criteria and this can cause contention in the shared pool area of the database.
If we change the setting in adf-config.xml to "True" ADF generates SQL with bind variables for all view criterias.
However this setting should not be changed for BI (OBIEE), as BI does not support Bind variables in the queries. If we see any queries (or its related logs) which are using Bind variables in OBIEE queries/reports, then its possible that it is due to the above setting.
For BI, this setting should be left to the default value, i.e. "False"
Quick answer: It would seem that OBIEE is not capable of using bind variables in place of literals.
Now, the reasoning behind the restriction on bind variables:
In a data warehouses, instead of running say 1,000 statements
per second, they do something like take an average of 100 seconds to run a single query.
In these systems, the queries are few but big (they ask large questions). Here, the
overhead of the parse time is a tiny fraction of the overall execution time. Even if you
have thousands of users, they are not waiting behind each other to parse queries, but
rather are waiting for the queries to finish getting the answer.
In these systems, using bind variables may be counterproductive. Here, the runtimes for
the queries are lengthy-in seconds, minutes, hours, or more. The goal is to get the best
query optimization plan possible to reduce the runtime, not to execute as many of OLTP,
one-tenth-second queries as possible. Since the optimizer's goal is different, the rules
change.
Sometimes using a bind variable forces the optimizer to come up with the best generic plan, which actually might be the worst plan for the specific query. In a system where the queries take considerable time to execute, bind variables remove information the optimizer could have used to come up with a superior plan. In fact, some data warehouse-specific features are defeated by using bind variables. For example, Oracle supports a star transformation feature for data warehouses that can greatly reduce the time a query takes. However, one restriction that precludes star transformation is having queries that contain bind variables.
I have an Oracle 9 database from which my Delphi 2006 application reads data into a TSimpleDataSet using a SQL statement like this one (in reality it is more complex, of course):
select * from myschema.mytable where ID in (1, 2, 4)
My applications starts up and executes this query quite often during the course of the day, each time with different values in the in clause.
My DBAs have notified me that this is creating execessive load on the database server, as the query is re-parsed on every run. They suggested to use bind variables instead of building the SQL statement on the client.
I am familiar with using parameterized queries in Delphi, but from the article linked to above I get the feeling that is not exactly what bind variables are. Also, I would need theses prepared statements to work across different runs of the application.
Is there a way to prepare a statement containing an in clause once in the database and then have it executed with different parameters passed in from a TSimpleDataSet so it won't need to be reparsed every time my application is run?
My answer is not directly related to Delphi, but this problem in general. Your problem is that of the variable-sized in-list. Tom Kyte of Oracle has some recommendations which you can use. Essentially, you are creating too many unique queries, causing the database to do a bunch of hard-parsing. This will spike the CPU consumption (and DBA blood pressures) unnecessarily.
By making your query static, it can get by with a soft-parse or perhaps no parse at all! The DB can then cache the execution plan, the DBAs can deal with a more "stable" SQL, and overall performance should be improved.
I'm trying to make sure I have a good understanding regarding the relationship between CURSOR_SHARING, bind variables, bind variable peeking and histograms as most sources cover these topics is different sections.
Ok so here's what I've gathered so far, feel free too correct me if I got anything wrong:
CURSOR_SHARING
1. = EXACT (default)
1.1. if SQL statement uses literals: the optimizer will generate a new execution plan for every combination of literals - optimizer will not replace literals with binds. A new parent cursor is generated for every literal combination.
1.2. if SQL statement uses bind variables: first time the statment is run, the optimizer will peek at the value of the bind variables and use those specific values to generate an execution plan - all future statements with those bind variables will use that same plan (even if the plan is suboptimal for other values of the bind variable).
2. = FORCE
2.1. optimizer will replace all literals with binds - and will basically use the same algorithm as scenario 1.2
3. = SIMILAR
3.1. no histogram: optimizer replaces all literals with binds -> same final effect as with 1.2 and 2.1
3.2. with histogram: optmizer replaces all literals with binds, but peeks at the bind variable EVERY time the statement is run (as opposed to just on the first run through) to see if there is a more optimal execution plan for that specific value of the bind variable (based on histogram statistics). Therefore, a new child cursor is effectively created for every distinct value of the bind variable that the optimizer encounters.
Questions:
From my understanding, isn't using CUSOR_SHARING = EXACT + writing SQL statments with bind-variables (1.2) lead to the exact same outcome as setting CURSOR_SHARING = FORCE (2.1)? In both cases, the optimizer will only peek at the bind variable on the first run to generate the execution plan and then reuse that plan no matter what the values of the bind variables on subsequent runs? If so, then why do most sources recommend using bind variables? this seems like it could have a significant impact on performance.
Is the histogram used in the initial bind variable peek for 1.2 and 2.1? As in, the first time that SQL statment is run and the optimizer peeks at the bind variable, does it use the histogram (if there is one) to determine if full-table scan or index scan is used? "Oracle Database 11g, Performance Tuning Recipes" seems to indicate that histograms are relevant only when CURSOR_SHARING = SIMILAR but some other sources indicate that the histogram is used in all the other CURSOR_SHARING settings as well.
In case 1.1, would the optimizer make use the histogram to determine the best execution plan? Basically I just want to know when the histogram is used. Is it only when CURSOR_SHARING = SIMILAR or for other CURSOR_SHARING settings are well?
Adpative Cursor Sharing - this feature will only take place if there are bind variables (either from user query or system-generated (by literal replacements)). Therefore it only takes place in 1.2, 2.1, 3.1 and 3.2? but since SIMILAR has been deprecated, does this mean that ACS only occurs in 1.2 and 2.1?
Hopefully, I'm not too far off base right now but if I made any mistakes please do correct me
Thanks!
Edited by: BYS2 on Dec 20, 2011 12:11 PM
The difference between (a) using FORCE and (b) using EXACT and coding bind variable yourself, is that in the latter case you control when bind variables are used. So if you can see that in a particular case bind variables are hurting performance, or aren't necessary, you can change that query. With FORCE, you're stuck. The reason that bind variables are recommended for OLTP type queries is that parsing is a highly serialized process that can become a big bottleneck. In OLTP systems you tend to see lots of queries that should always be using the same execution plan run with different values, so re-parsing them all the time is a waste. Any good source will also recommend that you consider when to not use bind variables -- for instance, if you only have a few possible values that can appear at a particular position in a query, and one or more of those values might benefit from a different execution plan, it may be better overall to use literals since you can parse each variant once and then reuse the cached plan.
(Another benefit to using bind variables is that it leaves you less open to SQL injection.)
2 & 3. Histograms are used in general when creating execution plans for queries, and in more ways than are obvious. Yes, in the case of a standard bind variable peek with EXACT setting, the histogram is (or at least, may be) used by the optimizer in determining the execution plan. This can be a good thing or a bad thing, depending on the skew and what particular value you have for the bind. I think the point your source is making about histograms and the SIMILAR setting is that in that case, the presence of the histogram is one of the triggers that will cause a new execution plan to be created.
(I would highly recommend Jonathan Lewis's "Cost-Based Oracle Fundamentals" for all the information you could want about when and how histograms are used.)
4.. I believe that Adaptive Cursor Sharing is essentially an enhanced version of the logic that was previously implemented for CURSOR_SHARING=SIMILAR. The optimizer will consider creating new plans based on bind variable peeking, in all circumstances. SIMILAR appears to still exist as an option. This post may provide some further helpful info.
I have a report engine, performing PreparedStatements on Oracle 11, that is a highly prioritized task.
What I see is that first query invocation usually performs much much longer than the same query afterwards (query has different parameters and return different data).
I suppose this is due to hard parsing done by Oracle, on first query invocation.
I wonder, is there a way of hinting to Oracle, that this query is highly prioritized query which would be performed often, and which performance is critical, so it should remain in shared pool, no matter what?
I know that I can fix execution plan in Oracle 11, but I don't want to fix it, I want Oracle still to be able to change it, as system changes, all I want is to exclude query hard parsing.
Perhaps you should change your "I suppose..." into a "I tested and have determined..." :)
The query performance may be affected by more than just parsing; when it executes it has to fetch blocks from disk into the buffer cache - subsequent executions quite possibly are taking advantage of the blocks being found in memory and so are faster.
EDIT: to answer your immediate question - a workaround may be to have a job run periodically that parses the query but doesn't execute it. You might even be able to use this to determine whether parsing or fetching is the locus of the problem.
You can try pinning to shared pool using dbms_shared_pool.keep
But I would first make sure that you have an aging out problem first
Anton,
if your query is using bind variables it will be re-used. The cursor will be cached and as long as it is re-used, it will remain in the cursor cache. Make sure that it uses bind variables. This increases re-usability and scalability.
If you don't trust the rdbms you can pin it using dbms_shared_pool.keep.
See http://psoug.org/reference/dbms_shared_pool.html
You need to find your cursor in order to do so.
Normally there is an other problem that should be fixed.
Ronald.
http://ronr.blogspot.com