A SELECT query in my stored procedure takes 3 seconds to execute when the table queried has no indexes. This is true in both when executing the query in Toad Editor and when calling the stored procedure. The Explain Plan shows that a full table scan is done.
When an index is added, the same query in Toad Editor returns results instantaneously (just a few milliseconds). The Explain Plan shows that the index is used. However, even when the index is present, the query still takes 3 seconds in the stored procedure. It looks like the query uses a full table scan when executed in stored procedure despite having an index that can speed it up. Why?
I have tried with indexes on different columns with different orders. The same results persist in all cases.
In the stored procedure, the results of the query are collected using BULK COLLECT INTO. Does this make a difference? Also, The stored procedure is inside a package.
The query is a very simple SELECT statement, like this:
SELECT MY_COL, COUNT (MY_COL)
/* this line is only in stored proc */ BULK COLLECT INTO mycollection
FROM MY_TABLE
WHERE ANOTHER_COL = '123' /* or ANOTHER_COL = filterval (which is type NUMBER) */
GROUP BY MY_COL
ORDER BY MY_COL
Without source code we can only guess...
So I suspect it's because in Toad you get just first 500 rows (500 is default buffer size in Toad) but in stored proc you fetch ALL rows into collection. So fetching probably takes most of 3 sec time. Especially if there are nested loops iny our query.
Update: It might also be implicit type conversion in where condition
Related
Please see image below:
As per that the IN query got converted to Exists in explain plan. Any reason for that? does it mean Oracle automatically converts IN to Exists?
Also any suggestion to reduce the cost? this statement is a part of a SP and it receives ~ separated string ('123') for example (63278~63282~63285~63288~63291~63296~63299~63302~63305~63308~63311~63314~63319~63322~63325~63329~63332~63253~63256~63260~63264~63267~63272~63275~63279~63283~63286~63289~63292~63297~63300~63303~63306~63309~63312~63315~63320~63323~63326~63330~63333~63269~63258~63277~63294~63317~63262~63270~63281~63295~63318~63328~63254~63257~63261~63265~63268~63273~63276~63280~63284~63287~63290~63293~63298~63301~63304~63307~63310~63313~63316~63321~63324~63327~63331~63334) in query. It takes around 10 to 15 mins to execute.
How can we generate explain plan for entire stored proc? We are using Oracle 19.
Thank you in advance.
IN clause retrieves all records which match with the given set of values. It acts as multiple OR conditions. IN clause scans all rows fetched from the inner query.
But, EXISTS is a Boolean operator that returns either True or False. Its used in combination to a sub-query. If the subquery returns any row, it returns True else False. If the result of data large inside the IN clause, then not recommended using IN. For getting high performance most time uses EXISTS vs IN. For that Oracle and PostgreSQL converts your IN to EXISTS
Since you are doing the job in a PL/SQL procedure you could create (out of the procedure) a GLOBAL TEMPORARY TABLE with DELETE ON COMMIT, in the procedure you INSERT in this table the result of the sub select with the CONNECT BY, then your replace the SELECT ... CONNECT BY by a SELECT in the temporary table. The temp table will be emptied at the end of the procedure and this method is session safe. And you have benefit of the index and probably a better plan. You could also compare the UPDATE with 2 ones: splitting the OR condition on 2 statements.
I have a query that updates a set of records based on specific criteria. I want to get columns of the result set of that update statement and pass it back in a refcursor.
I can get the result set by using RETURNING INTO, or in my case, RETURNING myrows BULK COLLECT INTO .... However, I'm not sure how to make this work with a cursor - you can't do an OPEN cursor FOR with an update statement.
I'm guessing there's a way to get the results of a RETURNING statement into my cursor. How can I do this?
Assuming that you have a SQL collection defined (rather than a PL/SQL collection), you should be able to
RETURNING my_column
BULK COLLECT INTO my_collection;
and then
OPEN p_rc
FOR SELECT *
FROM TABLE( my_collection );
Though that works, there are some caveats. If you expect the UPDATE to modify a large number of rows (or you expect many sessions to be running this code), storing all this data in a collection may consume a large amount of space in the PGA which may negatively impact performance. Reading a bunch of data into a collection just to send it all back to the SQL engine also tends to be a bit inelegant. And, as I said initially, this assumes that your collection is declared at the SQL level rather than being declared in PL/SQL.
I want to write pl/sql code which utilizes a Cursor and Bulk Collect to retrieve my data. My database has rows in the order of millions, and sometimes I have to query it to fetch nearly all records on client's request. I do the querying and subsequent processing in batches, so as to not congest the server and show incremental progress to the client. I have seen that digging down for later batches takes considerably more time, which is why I am trying to do it by way of cursor.
Here is what should be simple pl/sql around my main sql query:
declare
cursor device_row_cur
is
select /my_query_details/;
type l_device_rows is table of device_row_cur%rowtype;
out_entries l_device_rows := l_device_rows();
begin
open device_row_cur;
fetch device_row_cur
bulk collect into out_entries
limit 100;
close device_row_cur;
end;
I am doing batches of 100, and fetching them into out_entries. The problem is that this block compiles and executes just fine, but doesn't return the data rows it fetched. I would like it to return those rows just the way a select would. How can this be achieved? Any ideas?
An anonymous block can't return anything. You can assign values to a bind variable, including a collection type or ref cursor, inside the block. But the collection would have to be defined, as well as declared, outside the block. That is, it would have to be a type you can use in plain SQL, not something defined in PL/SQL. At the moment you're using a PL/SQL type that is defined within the block, and a variable that is declared within the block too - so it's out of scope to the client, and wouldn't be a valid type outside it either. (It also doesn't need to be initialised, but that's a minor issue).
Dpending on how it will really be consumed, one option is to use a ref cursor, and you can declare and display that through SQL*Plus or SQL Developer with the variable and print commands. For example:
variable rc sys_refcursor
begin
open :rc for ( select ... /* your cursor statement */ );
end;
/
print rc
You can do something similar from a client application, e.g. have a function returning a ref cursor or a procedure with an out parameter that is a ref cursor, and bind that from the application. Then iterate over the ref cursor as a result set. But the details depend on the language your application is using.
Another option is to have a pipelined function that returns a table type - again defined at SQL level (with create type) not in PL/SQL - which might consume fewer resources than a collection that's returned in one go.
But I'd have to question why you're doing this. You said "digging down for later batches takes considerably more time", which sounds like you're using a paging mechanism in your query, generating a row number and then picking out a range of 100 within that. If your client/application wants to get all the rows then it would be simpler to have a single query execution but fetch the result set in batches.
Unfortunately without any information about the application this is just speculation...
I studied this excellent paper on optimizing pagination:
http://www.inf.unideb.hu/~gabora/pagination/article/Gabor_Andras_pagination_article.pdf
I used technique 6 mainly. It describes how to limit query to fetch page x and onward. For added improvement, you can limit it further to fetch page x alone. If used right, it can bring a performance improvement by a factor of 1000.
Instead of returning custom table rows (which is very hard, if not impossible to interface with Java), I eneded up opening a sys_refcursor in my pl/sql which can be interfaced such as:
OracleCallableStatement stmt = (OracleCallableStatement) connection.prepareCall(sql);
stmt.registerOutParameter(someIndex, OracleTypes.CURSOR);
stmt.execute();
resultSet = stmt.getCursor(idx);
I have an SSIS package that fills a SQL Server 2008 R2 DataWarehouse, and when it recreates the DW from scratch, it does several million calls to a stored procedure that does the heavy lifting in terms of calculations.
The problem is that the SSIS package takes days to run and shouldn't take that long. The key seems to be that when the SSIS package calls the SProc, it takes about 2 minutes for the SProc to return the results. But if I recreate the call manually (on the same database) it takes <1 sec to return the result, which is what I'd expect.
See this screen shot, in the top is the SQL Profiler Trace showing the call by the SSIS Package taking 130 seconds, and in the bottom is my recreation of the call, taking <1 sec.
http://screencast.com/t/ygsGcdBV
The SProc queries the database, iterates through the results with a cursor, does a lot of calculations on pairs of records, and amalgamates the numbers into 2 results which get returned.
However the timing of the manual call, suggests to me that it's not an issue with the SProc itself, or any indexing issue with the database itself, so why would the SSIS package be taking so much longer than a manual call?
Any hints appreciated.
Thanks,
I don't have a solution for you, but I do have a suggestion and a procedure that can help you get more information for a solution.
Suggestion: a stored proc that gets called millions of times should not be using a cursor. Usually a cursor can be replaced by a few statements and a temp table or two. For temp tables with more than 10k rows or so, index it.
Procedure: At key places in your stored proc put a statement
declare #timer1 datetime = GetDate()
(some code)
declare #timer2 datetime = GetDate()
(more code)
declare #timer3 datetime = GetDate()
then, at the end:
select
datediff(ss,#timer1,#timer2) as Action1,
datediff(ss,#timer2,#timer3) as Action2
At that point, you'll know which part of your query is working differently. I'm guessing you've got an "Aha!" coming.
I suspect the real problem is in your stored procedure, but I've included some basic SSIS items as well to try to fix your problem:
Ensure connection managers for OLE DB sources are all set toDelayValidation ( = True).
Ensure that ValidateExternalMetadata is set to false
DefaultBufferMaxRows and DefaultBufferSize to correspond to the table's row sizes
DROP and Recreate your destination component is SSIS
Ensure in your stored procedure that SET ANSI_NULLS ON
Ensure that the SQL in your sproc hits an index
Add the query hint OPTION (FAST 10000) - This hint means that it will choose a query which will optimise for the first 10,000 rows – the default SSIS buffer size
Review your stored procedure SQL Server parameter sniffing.
Slow way:
create procedure GetOrderForCustomers(#CustID varchar(20))
as
begin
select * from orders
where customerid = #CustID
end
Fast way:
create procedure GetOrderForCustomersWithoutPS(#CustID varchar(20))
as
begin
declare #LocCustID varchar(20)
set #LocCustID = #CustID
select * from orders
where customerid = #LocCustID
end
It could be quite simple. Verify the indexes in the table. It could be similiar/conflicting indexes and the solution could be to drop one of them.
With the SQL query in SSMS have a look at the Execution plan and which index object is used. Is it the same for the slow SP?
If they use different indexes try to use the fast one in SP. Example how to:
SELECT *
FROM MyTable WITH (INDEX(IndexName))
WHERE MyIndexedColumn = 0
I need to write a sproc which performs some INSERTs on a table, and compile a list of "statuses" for each row based on how well the INSERT went. Each row will be inserted within a loop, the loop iterates over a cursor that supplies some values for the INSERT statement. What I need to return is a resultset which looks like this:
FIELDS_FROM_ROW_BEING_INSERTED.., STATUS VARCHAR2
The STATUS is determined by how the INSERT went. For instance, if the INSERT caused a DUP_VAL_ON_INDEX exception indicating there was a duplicate row, I'd set the STATUS to "Dupe". If all went well, I'd set it to "SUCCESS" and proceed to the next row.
By the end of it all, I'd have a resultset of N rows, where N is the number of insert statements performed and each row contains some identifying info for the row being inserted, along with the "STATUS" of the insertion
Since there is no table in my DB to store the values I'd like to pass back to the user, I'm wondering how I can return the info back? Temporary table? Seems in Oracle temporary tables are "global", not sure I would want a global table, are there any temporary tables that get dropped after a session is done?
If you are using Oracle 10gR2 or later then you should check out DML error logging. This basically does what you want to achieve, that is, it allows us to execute all the DML in a batch process by recording any errors and pressing on with the statements.
The principle is that we create an ERROR LOG table for each table we need to work with, using a PL/SQL built-in package DBMS_ERRLOG. Find out more. There is a simple extension to the DML syntax to log messages to the error log table. See an example here. This approach doesn't create any more objects than your proposal, and has the merit of using some standard Oracle functionality.
When working with bulk processing (that is, when using the FORALL syntax) we can trap exceptions using the built-in SQL%BULK_EXCEPTIONS collection. Check it out. It is possible to combine Bulk Exceptions with DML Error Logging but that may create problems in 11g. Find out more.
"Global" in the case of temporary tables just means they are permanent, it's the data which is temporary.
I would define a record type that matches your cursor, plus the status field. Then define a table of that type.
TYPE t_record IS
(
field_1,
...
field_n,
status VARCHAR2(30)
);
TYPE t_table IS TABLE OF t_record;
FUNCTION insert_records
(
p_rows_to_insert IN SYS_REFCURSOR
)
RETURN t_table;
Even better would be to also define the inputs as a table type instead of a cursor.