I have been trying to address the issue of how Oracle processes ROWNUM and SELECT ... FOR UPDATE SKIP LOCKED while trying to return several rows that aren't locked. I have tried a number of the solutions from the following: Force Oracle to return TOP N rows with SKIP LOCKED, as well as several other examples that look very similar to the ones found on that question. I know Oracle AQ is probably the best solution to this, but we have very little control over the databases and I have met with considerable resistance to the idea.
The problem I am running into is trying to get the results back to Java using JDBC. I have tried setFetchSize(20), but I run into the issue where only the top 20 rows are distributed to the clients. I usually see one processing agent getting all 20 rows or a few processors getting some rows, all of them adding up to 20. This is very much like the behavior one would see with using ROWNUM in conjunction with SELECT ... FOR UPDATE SKIP LOCKED.
The most promising solution I have tried is the following function:
create type IND_ID as object
(
ID varchar2(200)
);
create type IND_ID_TABLE as table of IND_ID;
create or replace function SELECTIDS return IND_ID_TABLE
pipelined is
ST_CURSOR SYS_REFCURSOR;
ID_REC IND_ID := IND_ID(null);
begin
open ST_CURSOR for
select ID
from TABLE
/* where clause */
for update SKIP LOCKED;
loop
fetch ST_CURSOR
into ID_REC.ID;
exit when ST_CURSOR%rowcount > 20 or ST_CURSOR%notfound;
pipe row(ID_REC);
end loop;
close ST_CURSOR;
return;
end;
However, when I try invoking it like so:
select * from table(SELECTIDS)
I get an ORA-14551: cannot perform a DML operation inside a query error, which I now understand is an issue with transactions. Removing the locks causes the function to return rows.
How can I get multiple rows out of this function into JDBC while preserving the locks?
This not gonna work. You calling a pl/sql function as part of a select statement and try to start a transaction in that function. I think the error is pretty clear.
Related
I have a requirement to return a subset of rows from a table using FOR UPDATE SKIP LOCKED. Based on application parameters this subset may or may not be ordered by a column. I can't use ROWNUM since the numbers are assigned before SKIP LOCKED happens, so using cursors and FETCH ... LIMIT seems to be the way to go.
That works using an anonymous PL/SQL block, but I need to expose the data back to the java application. The most straightforward way would be to use a table function, so I can just do SELECT * FROM table(my_function(<params>)).
I tried a standard table function returning a collection first, but I got the error [902] ORA-00902: invalid datatype. This is roughly what I had in place:
Package specification:
CREATE OR REPLACE PACKAGE ACTIVITY_UTILS AS
TYPE ActivityList IS TABLE OF ACTIVITY_TABLE%ROWTYPE;
FUNCTION activity_batch(batch_size IN INTEGER, order_by_source IN VARCHAR2)
RETURN ActivityList;
END ACTIVITY_UTILS;
Package body:
CREATE OR REPLACE PACKAGE BODY ACTIVITY_UTILS AS
FUNCTION activity_batch(batch_size IN INTEGER, order_by_source IN VARCHAR2)
RETURN ActivityList
IS
batch ActivityList := ActivityList();
selectStatement VARCHAR2(200);
TYPE CursorType IS REF CURSOR;
activitiesCursor CursorType;
BEGIN
IF UPPER(order_by_source) = 'TRUE' THEN
selectStatement := 'SELECT * FROM ACTIVITY_TABLE ORDER BY source FOR UPDATE SKIP LOCKED';
ELSE
selectStatement := 'SELECT * FROM ACTIVITY_TABLE FOR UPDATE SKIP LOCKED';
OPEN activitiesCursor FOR selectStatement;
FETCH activitiesCursor BULK COLLECT INTO batch LIMIT batch_size;
CLOSE activitiesCursor;
RETURN batch;
END activity_batch;
While debugging the ORA-00902 error I ran into this question:
Return collection from packaged function for use in select
My (limited) understanding was I was trying to use a PL/SQL type on plain SQL, which is not allowed. I tried using a pipelined table function, as mentioned in the answer, but then I got the error ORA-14551: cannot perform a DML operation inside a query.
This seemed odd, is SELECT ... FOR UPDATE considered DML? At any rate, I noticed I could workaround this by using pragma autonomous_transaction, but that defeats the purpose of having FOR UPDATE SKIP LOCKED.
My question is, is this requirement achievable at all using functions, or would I have to use a procedure with an OUT parameter?
Option 1: create a function (or procedure) that returns a cursor and let your java application fetch it normally.
Option 2: Use Implicit statement results: in this case your java application can run something like call proc() where proc returns implicit statement results.
PS. It's not a good idea to hide DML under SQL select...
I am doing data insert into a table in Oracle which is having a sequence set to it in one of the columns say Id column. I would like to know how to do data loads into such tables.
I followed the below link -
It's possible to use OleDbConnections with the Script Component?
and tried to create a function to get the .nextval from the Oracle table but I am getting the following error -
Error while trying to retrieve text for error ORA-01019
I realized that manually setting the value via the package i.e. by using the Script task to enumerate the values but is not incrementing the sequence and that is causing the problem. How do we deal with it? Any links that can help me solve it?
I am using SSIS-2014 but I am not able to tag it as I don't due to paucity of reputation points.
I created a workaround to cater to this problem. I have created staging tables of the destination without the column that takes the Sequence Id. After the data gets inserted, I am then calling SQL statement to get the data into the main tables from staging table and using the .nextval function. Finally truncating/dropping the table depending on the need. It would still be interesting to know how this same thing can be handled via script rather having this workaround.
For instance something like below -
insert into table_main
select table_main_sequence_name.nextval
,*
from (
select *
from table_stg
)
ORA-01019 may be related to fact you have multiple Oracle clients installed. Please check ORACLE_HOME variable if it contains only one client.
One workaround I'm thinking about is creating two procedures for handling sequence. One to get value you start with:
create or replace function get_first from seq as return number
seqid number;
begin
select seq_name.nexval into seqid from dual;
return seqid;
end;
/
Then do your incrementation in script. And after that call second procedure to increment sequence:
create or replace procedure setseq(val number) as
begin
execute immediate 'ALTER SEQUENCE seq_name INCREMENT BY ' || val;
end;
/
This is not good approach but maybe it will solve your problem
I want to write pl/sql code which utilizes a Cursor and Bulk Collect to retrieve my data. My database has rows in the order of millions, and sometimes I have to query it to fetch nearly all records on client's request. I do the querying and subsequent processing in batches, so as to not congest the server and show incremental progress to the client. I have seen that digging down for later batches takes considerably more time, which is why I am trying to do it by way of cursor.
Here is what should be simple pl/sql around my main sql query:
declare
cursor device_row_cur
is
select /my_query_details/;
type l_device_rows is table of device_row_cur%rowtype;
out_entries l_device_rows := l_device_rows();
begin
open device_row_cur;
fetch device_row_cur
bulk collect into out_entries
limit 100;
close device_row_cur;
end;
I am doing batches of 100, and fetching them into out_entries. The problem is that this block compiles and executes just fine, but doesn't return the data rows it fetched. I would like it to return those rows just the way a select would. How can this be achieved? Any ideas?
An anonymous block can't return anything. You can assign values to a bind variable, including a collection type or ref cursor, inside the block. But the collection would have to be defined, as well as declared, outside the block. That is, it would have to be a type you can use in plain SQL, not something defined in PL/SQL. At the moment you're using a PL/SQL type that is defined within the block, and a variable that is declared within the block too - so it's out of scope to the client, and wouldn't be a valid type outside it either. (It also doesn't need to be initialised, but that's a minor issue).
Dpending on how it will really be consumed, one option is to use a ref cursor, and you can declare and display that through SQL*Plus or SQL Developer with the variable and print commands. For example:
variable rc sys_refcursor
begin
open :rc for ( select ... /* your cursor statement */ );
end;
/
print rc
You can do something similar from a client application, e.g. have a function returning a ref cursor or a procedure with an out parameter that is a ref cursor, and bind that from the application. Then iterate over the ref cursor as a result set. But the details depend on the language your application is using.
Another option is to have a pipelined function that returns a table type - again defined at SQL level (with create type) not in PL/SQL - which might consume fewer resources than a collection that's returned in one go.
But I'd have to question why you're doing this. You said "digging down for later batches takes considerably more time", which sounds like you're using a paging mechanism in your query, generating a row number and then picking out a range of 100 within that. If your client/application wants to get all the rows then it would be simpler to have a single query execution but fetch the result set in batches.
Unfortunately without any information about the application this is just speculation...
I studied this excellent paper on optimizing pagination:
http://www.inf.unideb.hu/~gabora/pagination/article/Gabor_Andras_pagination_article.pdf
I used technique 6 mainly. It describes how to limit query to fetch page x and onward. For added improvement, you can limit it further to fetch page x alone. If used right, it can bring a performance improvement by a factor of 1000.
Instead of returning custom table rows (which is very hard, if not impossible to interface with Java), I eneded up opening a sys_refcursor in my pl/sql which can be interfaced such as:
OracleCallableStatement stmt = (OracleCallableStatement) connection.prepareCall(sql);
stmt.registerOutParameter(someIndex, OracleTypes.CURSOR);
stmt.execute();
resultSet = stmt.getCursor(idx);
I am having some trouble with a procedure; when run for “big” sets (800+ parents, 1300+ children), it is very slow (30 - 60 secs).
Basic idea is to fetch all parent records (and their respective children) fitting a certain search criteria, along with 3 additional pieces of information that will have to be computed.
My approach to the problem was
to create a custom record type with additional fields for the computed values.
A reference to this record type can then be passed around to each function, controlled by a main processing function.
As a value is computed for each parent record, tack it onto the record.
Each procedure GET_PARENT_RECORDS and GET_CHILD_RECORDS are called once per search, and each computing functions are run N times (where N is the number of parent and/or child records).
Question 1: Is this the right approach? (weakly typed cursors, pipelined functions) If not, then how should I have approached the problem, assuming I can have a re-do?
Question 2: Barring a complete rewrite, is there anything obvious that can be improved in the code provided?
Question 3: Or could something else be wrong, as I notice that the same slow query returned in 20 secs when I've run the procedures a few times?
Package definition
create or replace
PACKAGE THIS_PKG AS
Type parentCursor IS REF CURSOR;
Type childCursor IS REF CURSOR;
Type ParentRecordType IS RECORD (
other_columns,
Extra_column_A,
Extra_column_B,
Extra_column_C,
Row_num);
--associative array
TYPE ParentArray IS TABLE OF ParentRecordType;
FUNCTION processParents(
p IN THIS_PKG. parentCursor
) RETURN ParentArray
PIPELINED
;
FUNCTION countSomething(some params…)
RETURN INT;
FUNCTION checkCondX (SomeParent IN ParentRecordType)
RETURN VARCHAR2;
FUNCTION checkCondY (SomeParent IN ParentRecordType)
RETURN VARCHAR2;
PROCEDURE GET_PARENT_RECORDS( other_parameters, Parents OUT THIS_PKG.parentCursor);
PROCEDURE GET_CHILD_RECORDS( other_parameters, Children OUT THIS_PKG.childCursor);
END THIS_PKG;
Package Body
-- omitted
FUNCTION processParents(
p IN THIS_PKG.parentCursor
) RETURN ParentArray
PIPELINED
IS
out_rec ParentArray;
someParent ParentRecordType;
BEGIN
LOOP
FETCH p BULK COLLECT INTO out_rec LIMIT 100;
FOR i IN 1 .. out_rec.COUNT
LOOP
out_rec(i).extra_column_A := countSomething (out_rec(i).field1, out_rec(i).field2);
out_rec(i).extra_column_B := checkCondX(out_rec(i));
out_rec(i).extra_column_C := checkCondY(out_rec(i));
pipe row(out_rec(i));
END LOOP;
EXIT WHEN p%NOTFOUND;
END LOOP;
RETURN;
END processParents;
PROCEDURE GET_PARENT_RECORDS(
some_columns,
Parents OUT THIS_PKG. parentCursor) IS
BEGIN
OPEN Parents FOR
SELECT *
FROM TABLE(processParents (CURSOR(
SELECT *
FROM (
--some select statement with quite a few where clause
--to simulate dynamic search (from pre-canned search options)
)
))) abc
WHERE abc.extra_column_C like '%xyz%' --(xyz is a user given value)
;
END GET_PARENT_RECORDS;
Update
Did some exploring yesterday and came across the Quest Batch SQL Optimizer (from Toad). I plugged in the package and here's what I got.
Batch Optimizer results
Complex query
Problematic query
What is happening in the row processing section? A lot of time may be spent in these countSomething, checkCondX/Y functions. Are they also making SQL calls? I'd check the performance of the table function without the additional predicates first. It may be better to simply create a query which does this all in SQL rather than functions - if you can do this it will be very much quicker than calling out to a function for every row.
out_rec(i).extra_column_A := countSomething (out_rec(i).field1, out_rec(i).field2);
out_rec(i).extra_column_B := checkCondX(out_rec(i));
out_rec(i).extra_column_C := checkCondY(out_rec(i));
Also the explain plan you provided is interesting as the optimiser believes there is only 1 row being returned from all tables (cardinality 1). If this isn't the case then the query plan will not be optimal. It may be necessary to gather statistics, use dynamic sampling or cardinality hints on the table function.
Finally, look into DBMS_SQLTUNE.REPORT_SQL_MONITOR which provides a detailed report on your sql. Unless the query is dynamically identified as needing monitoring your need to add the /*+ MONITOR */ hint. This provides more details like number of rows returned, execution count and other interesting tidbits not available in an explain plan.
SELECT /*+ MONITOR */
FROM slow_query;
-- then run sqltune to get a report
SELECT *
FROM TABLE(DBMS_SQLTUNE.REPORT_SQL_MONITOR());
Quest Batch SQL Optimizer (from Toad) or any other tool will be unable to help you taking into consideration that they don't understand what you do inside the functions. The problem is in "FETCH p BULK COLLECT INTO out_rec LIMIT 100;". The quality of the query that is passed into p actually defines the final execution plan and run times. Pipelining is not the reason for slowness. When you run your procedure several times, Oracle uses cached data. My best advice is: use Java instead of PL/SQL for this particular purpose, it will be more simpler for understanding.
I need to write a sproc which performs some INSERTs on a table, and compile a list of "statuses" for each row based on how well the INSERT went. Each row will be inserted within a loop, the loop iterates over a cursor that supplies some values for the INSERT statement. What I need to return is a resultset which looks like this:
FIELDS_FROM_ROW_BEING_INSERTED.., STATUS VARCHAR2
The STATUS is determined by how the INSERT went. For instance, if the INSERT caused a DUP_VAL_ON_INDEX exception indicating there was a duplicate row, I'd set the STATUS to "Dupe". If all went well, I'd set it to "SUCCESS" and proceed to the next row.
By the end of it all, I'd have a resultset of N rows, where N is the number of insert statements performed and each row contains some identifying info for the row being inserted, along with the "STATUS" of the insertion
Since there is no table in my DB to store the values I'd like to pass back to the user, I'm wondering how I can return the info back? Temporary table? Seems in Oracle temporary tables are "global", not sure I would want a global table, are there any temporary tables that get dropped after a session is done?
If you are using Oracle 10gR2 or later then you should check out DML error logging. This basically does what you want to achieve, that is, it allows us to execute all the DML in a batch process by recording any errors and pressing on with the statements.
The principle is that we create an ERROR LOG table for each table we need to work with, using a PL/SQL built-in package DBMS_ERRLOG. Find out more. There is a simple extension to the DML syntax to log messages to the error log table. See an example here. This approach doesn't create any more objects than your proposal, and has the merit of using some standard Oracle functionality.
When working with bulk processing (that is, when using the FORALL syntax) we can trap exceptions using the built-in SQL%BULK_EXCEPTIONS collection. Check it out. It is possible to combine Bulk Exceptions with DML Error Logging but that may create problems in 11g. Find out more.
"Global" in the case of temporary tables just means they are permanent, it's the data which is temporary.
I would define a record type that matches your cursor, plus the status field. Then define a table of that type.
TYPE t_record IS
(
field_1,
...
field_n,
status VARCHAR2(30)
);
TYPE t_table IS TABLE OF t_record;
FUNCTION insert_records
(
p_rows_to_insert IN SYS_REFCURSOR
)
RETURN t_table;
Even better would be to also define the inputs as a table type instead of a cursor.