Right now I'm importing and transforming data into an Oracle database as follows:
A program regularly polls specific folders, once a file is found it executes a batch file which does some light transformation in Python & bash and then calls SQL*Loader to load the CSV file into a staging table.
Then, the batch script calls an SQL script (via SQLPlus) to do the final transformation and insert the transformed data into master tables for their respective staging table.
The problem with this method is there's no error-handling on the SQLPlus side, eg. if an 'insert into' statement fails because of a violated constraint (or any other reason), it will still continue to execute the rest of the statements contained in the SQL script.
Ideally, if any exception occurs, I'd prefer all changes to be rolled back and details of the exception inserted into an etl log table.
Stored procedures seem to be a good fit as exception handling is built-in. However, I'm struggling with the syntax - specifically how I can take my big SQL scripts (which are just a combination of INSERT INTO, UPDATE, CREATE, DROP, DELETE, etc statements) and throw them into a stored procedure with some very basic error handling.
What I'm hoping for is either:
a quick & dirty dummy's guide to taking my depressing blob of PL/SQL and get it to execute within a stored procedure OR
Any alternative (if a stored proc isn't appropriate) which offers the same functionality, ie. a way to execute a bunch of SQL statements and rollback if any of these statements throw an exception.
About my attempts - I've tried copying portions of my SQL scripts into a stored procedure but they always fail to compile with the error 'PLS-00103 Encountered the symbol when expecting one of the following'. eg.
CREATE OR REPLACE PROCEDURE ETL_2618A AS
BEGIN
DROP SEQUENCE "METER_REPORTING"."SEQ_2618";
CREATE SEQUENCE SEQ_2618;
END ETL_2618A;
Oracle documentation isn't terribly accessible and I've not had much luck with googling/searching StackOverflow, but I apologise if I've missed something obvious.
To do DDL in PL/SQL you need to use dynamic sql.
CREATE OR REPLACE PROCEDURE testProc IS
s_sql VARCHAR2(500);
BEGIN
s_sql := 'DROP SEQUENCE "METER_REPORTING"."SEQ_2618"';
EXECUTE IMMEDIATE s_sql;
s_sql := 'CREATE SEQUENCE "METER_REPORTING"."SEQ_2618"';
EXECUTE IMMEDIATE s_sql;
EXCEPTION
WHEN OTHERS THEN
NULL;
end testProc;
/
If you run the script in sqlplus you can use:
whenever sqlerror
to control what should happen when an error occurs.
http://docs.oracle.com/cd/B19306_01/server.102/b14357/ch12052.htm
Adding exception handling to a PL/SQL proc or script isn't difficult, but of course some coding is required. Here's your procedure dressed up slightly with some very basic error reporting added:
CREATE OR REPLACE PROCEDURE ETL_2618A AS
nCheckpoint NUMBER;
BEGIN
nCheckpoint := 1;
EXECUTE IMMEDIATE 'DROP SEQUENCE "METER_REPORTING"."SEQ_2618"';
nCheckpoint := 2;
EXECUTE IMMEDIATE 'CREATE SEQUENCE SEQ_2618';
RETURN;
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE('ETL_2618A failed at checkpoint ' || nCheckpoint ||
' with error ' || SQLCODE || ' : ' || SQLERRM);
RAISE;
END ETL_2618A;
Not tested on animals - you'll be first! :-)
Related
I need to delete one or more row from list of tables stored in a table, and commit only if all deletion succeed.
So I wrote something like this (as part of a bigger procedure):
BEGIN
SAVEPOINT sp;
FOR cur_table IN (SELECT * FROM TABLE_OF_TABLES)
LOOP
EXECUTE IMMEDIATE 'DELETE FROM ' || cur_table.TABNAME || ' WHERE ID = :id_bind'
USING id;
END LOOP;
EXCEPTION
WHEN OTHERS THEN
ROLLBACK TO SAVEPOINT sp;
END;
I know this couldn't work, because of the "execute immediate".
So, what is the correct way to do that?
Dynamic SQL (Execute Immediate) doesn't commit the transaction. You have to commit explicitly. Your code is fine but it doesn't record/log the errors in case if they occur.
Log the errors in the exception handler section.
Honestly, this sounds like a bad idea. Since you have the tables stored in the database, why not just list them out in your procedure? Surely, you're not adding tables that often that this procedure would need to be updated often?
As pointed out in the comments, this will work fine, since EXECUTE IMMEDIATE does not automatically commit.
Don't forget to add a RAISE at the end of your exception block, or you'll never know that an error happened.
I have the function in which I need to drop and create tables. In the example below I try to create the table but it fails
CREATE OR REPLACE FUNCTION DEVTEST
RETURN NUMBER
IS
COMMAND VARCHAR2(256);
ID VARCHAR2(128);
NAME VARCHAR2(128);
TMP_LIST VARCHAR2(128);
BEGIN
ID := '12345';
NAME := 'ABCdef';
TMP_LIST := 'tmpTest';
command := 'create table ' || TMP_LIST || ' ( USER_ID VARCHAR2(11), USER_NAME VARCHAR2(36))';
DBMS_OUTPUT.PUT_LINE('command = ' || command);
EXECUTE IMMEDIATE command;
return 0;
END;
I call the function:
select NSB_DEVTEST() from dual
And get the error:
ORA-14552: cannot perform a DDL, commit or rollback inside a query or DML ORA-06512: at "DEV1_SERVER.DEVTEST", line 15
How do I correct this to create/drop a table inside a function?
My server details:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bi
PL/SQL Release 10.2.0.5.0 - Production
CORE 10.2.0.5.0 Production
TNS for Solaris: Version 10.2.0.5.0 - Production
NLSRTL Version 10.2.0.5.0 - Production
The problem is not with the function but with it being called from a SQL statement rather than from a pl/sql block.
A SELECT statement in SQL is equivalent to a READ operation that comes with read consistency. It cannot make any changes to the database. The database should always be the same before and after the "READ" operation completed, otherwise it would be a WRITE operation and the entire database consistency would go havoc.
Also, like the error says, DDL operations do a COMMIT behind the scenes before they start. Any read consistent operation should never do any COMMITS and write to the database without the user knowing.
You can instead call the function from pl/sql like this -
DECLARE
l_result NUBMER;
BEGIN
l_result := DEVTEST;
DBMS_OUTPUT.PUT_LINE(l_number);
END;
Still I would prefer writing a procedure for this, so others don't get confused by why this can't be called from SQL. The general rule that I follow for myself is that - Functions "get" things and Procedures "do" things (like DML).
The answer to your question is: don't. Production code, on the whole, shouldn't be creating tables on the fly. If you need a table to hold data temporarily, then create a Global Temporary Table (GTT) once and have your code refer to it.
The reason why you're getting that error (apart from it being self-evident from the error message) is because you're calling the function from within a SQL statement. You can't do that; you'd have to call it directly in PL/SQL.
I'm curious as to why you think this approach is a good, feasible approach, and also what you're going to be doing with the table once you've created it.
Your code is perfect no problem in the code . the problem is while you try to execute
you can only execute a pure function in select statement, which means a function without ddl & dml . ( if you use pragma autonomous_transaction while performing dml inside a function then you can use it in select statement ). When function has DDL command you can never ever execute it in select statement , but instead you can only execute it in PLSQL block like this
declare
a number;
begin
a:= devtest;
end;
/
and you can check your table
select * from tmptest;
on one of our servers, while compiling:
CREATE OR REPLACE PROCEDURE PROC_KO
IS
TABLE_SUFFIX_ VARCHAR2 (100);
QUERY_DROP_ VARCHAR2 (1000);
BEGIN
TABLE_SUFFIX_ := 'TABLE_SUFFIX';
QUERY_DROP_ := 'DROP TABLE ' || 'TMP_' || TABLE_SUFFIX_;
END;
I get the following error:
I/O Error: Connection reset
What could be the cause of this?
Its possible that in your case you're disallowed to create a procedure by a DDL trigger.
CREATE OR REPLACE TRIGGER bcs_trigger
BEFORE CREATE
ON SCHEMA
DECLARE
oper ddl_log.operation%TYPE;
BEGIN
-- in some case
DBMS_SERVICE.DISCONNECT_SESSION(my_session);
END bcs_trigger;
Sounds like I had the same, or at least a similar, problem. I had already successfully created/compiled the procedure and later returned to make some changes using SQL Developer (17.2.0.188). When attempting to compile the procedure I received the same error (Connection Reset).
It was only after creating a new procedure by copying blocks of code was I able to isolate the 2-3 lines that seemed to be causing the problem. A slight rewrite of one of the lines (by using different variable names) seemed to do the trick.
There was nothing about the lines in question that seemed to stand out. Simple for loop using a SQL statement.
Still don't know the root cause of the problem, but making minor changes to the code seemed to allow it to compile.
We have a stored procedure that inserts data into a table from some temporary tables in oracle database. There is an update statement after the insert that updates a flag in the same table based on certain checks. At the end of the stored procedure commit happens.
The problem is that the update works on 95% cases but in some cases it fails to update. When we try to run it again without changing anything, it works. Even trying to execute the same stored procedure on the same data at some other time, works perfectly. I haven't found any issues in the logic in the stored procedure. I feel there is some database level issue which we are not able to find (maybe related to concurrency). Any ideas on this would be very helpful.
Without seeing the source code we will just be guessing. The most obvious suggestion that I can think of is that it hits an exception somewhere along the way in some cases and never gets as far as the commit. Another possibility is that there is a lock on the table during execution when it fails.
Probably the best thing to investigate further would be to add an exception handler that writes the exceptions to some table or file and see what error is raised e.g.
-- create a logging table
create table tmp_error_log (timestamp timestamp(0), Error_test varchar2(1000));
-- add a variable to your procedure declaration
v_sql varchar2(1000);
-- add an exception handler just before the final end; statement on your procedure
exception
when others then
begin
v_sql := 'insert into tmp_error_log values(''' ||
to_char(sysdate, 'DD-MON-YYYY HH24:MI:SS') || ''', ''' || SQLERRM || ''')';
dbms_output.put_line(v_sql);
execute immediate v_sql;
commit;
end;
-- see what you get in the table
select * from tmp_error_log;
I have a wrapper procedure(proc_main) that calls some procedures within.
create or replace Procedure proc_main
as
begin
proc_child1;
proc_child2;
proc_child3;
proc_compile_invalids; -- This invokes "alter procedure <procedure_name> compile" statement for all the invalids.
end;
/
proc_child procedures apply some processing logic that involves some steps to rename the tables within.
This invalidates the procedures which is the reason why I have the proc_compile_invalids procedure to set them to a valid state again.
My problem is: when I execute the proc_main procedure, it invalidates the main procedure along with the inner child ones.
Hence, When the proc_compile_invalids is called as a last step, it hangs as it is trying to recompile the main calling procedure.
Obviously, it is not an issue if i remove the last step and execute it separately.
I know I could separate them out as 2 different calls by commenting the compile proc and executing it as a stand alone.
And i also am aware it is a cosmetic step as oracle would try to compile a procedure before executing the next time. So, the invalids become valid anyway.
But, at the end of the execution for that day, they all are in an invalid state and I get questioned by the powers be if it can be avoided !
So, just wanted to know if I can avoid separating the calls and still retain it as a last step in the main procedure.
Any thoughts/pointers much appreciated.
You can use dynamic SQL to break the dependency:
CREATE OR REPLACE PROCEDURE proc_main AS
BEGIN
EXECUTE IMMEDIATE 'BEGIN proc_child1; END;';
EXECUTE IMMEDIATE 'BEGIN proc_child2; END;';
EXECUTE IMMEDIATE 'BEGIN proc_child3; END;';
proc_compile_invalids; -- This invokes
-- "alter procedure <procedure_name> compile"
-- statement for all the invalids.
END;
Oracle 11g onward
You can use compile_schema procedure of dbms_utility package instead of proc_compile_ivalids in your main procedure to recompile all invalid procedures, functions, packages, and triggers in the specified schema
create or replace Procedure proc_main
as
begin
Proc_child1;
proc_child2;
proc_child3;
dbms_utility.compile_schema(schema, false);
end;