My requirement is copy data from Oracle to SQL Server. Before copying from Oracle database, I need to update the Oracle table using procedure which has some logic.
How do I execute Oracle stored procedure from Azure datafactory?
I referred to this thread
if I use EXECUTE PROC_NAME (PARAM); in preCopy script it's failing with following error
Failure happened on 'Source' side.
ErrorCode=UserErrorOdbcOperationFailed,
Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException
Message=ERROR [42000] [Microsoft][ODBC Oracle Wire Protocol driver]
[Oracle]ORA-00900: invalid SQL statement
Source=Microsoft.DataTransfer.ClientLibrary.Odbc.OdbcConnector,
Type=System.Data.Odbc.OdbcException
Message=ERROR [42000] [Microsoft][ODBC Oracle Wire Protocol driver]
[Oracle]ORA-00900: invalid SQL statement,Source=msora28.dll
Could anyone help on this?
Note: I am using self-hosted runtime environment for data factory
thanks!!
I used a Lookup Activity and a SELECT statement of DUAL TABLE. Due to the stored procedures can not be call from a statement SELECT. I created an oracle function and the function calls the stored procedure. The function returns a value and this value is received by the lookup activity.
When you define the function, you have to add the statement PRAGMA AUTONOMOUS_TRANSACTION. This is because Oracle does not allow to execute DML instructions with a SELECT statement by default. Then, you need to define that DML instructions in the Stored Procedure will be an autonomous transaction.
--Tabla
CREATE TABLE empleados(
emp_id NUMBER(9),
nombre VARCHAR2(100),
CONSTRAINT empleados_pk PRIMARY KEY(emp_id),
);
create or replace procedure insert_empleado (numero in NUMBER, nombre in VARCHAR2) is
begin
INSERT INTO empleados (emp_id, nombre)
Values(numero, nombre);
COMMIT;
end;
create or replace function funcinsert_empleado (numero in NUMBER, nombre in VARCHAR2)
return VARCHAR2
is
PRAGMA AUTONOMOUS_TRANSACTION;
begin
insert_empleado (numero, nombre);
return 'done';
end;
--statement in query of lookup
SELECT funcinsert_empleado ('1', 'Roger Federer')
FROM DUAL;
Example lookup
This is example in Spanish. https://dev.to/maritzag/ejecutar-un-stored-procedure-de-oracle-desde-data-factory-2jcp
In Oracle, EXECUTE X(Y) is a SQL*Plus-specific command shortcut for the PL/SQL statement BEGIN X(Y); END;. Since you are not using SQL*Plus, try the BEGIN/END syntax.
In case you only want to execute the DML query using the Azure Data Factory without procedure on oracle database :-
I have another solution where you can use the copy activity with the pre-copy feature of sink in-spite of lookup activity.
For this approach just follow the below steps :-
Keep both the source table and sink table as same ( Let say table A ) using the same linked service.
In sink use the pre-copy script feature and keep the DML (Insert/Update/Delete ) query that you want to perform over the table B.( This table is not necessary to be same as table A )
In case you want to avoid the copy of data to same table you can select query option in the source part and provide a where clause which is not going to satisfy and hence no copy of data will happen .
or you can create a table temp with one column and one row .
I have tested both the options and it works ... good part of above solution is you can avoid the procedure or function creation and maintenance .
Related
I see the concept of temporary table in oracle is quite different from other databases like SQL Server. In Oracle, we have a concept of global temporary table and we create it only once and in each session we fill it with data which is not the same in other databases.
In 18c, oracle has introduced the concept of private temporary tables which states that upon successful usage, tables can be dropped like in other databases. But how do we use it in a PL/SQL block?
I tried using it using dynamic SQL - EXECUTE IMMEDIATE. But it is giving me table must be declared error. what do I do here?
But how do we use it in a PL/SQL block?
If what you mean is, how can we use private temporary tables in a PL/SQL program (procedure or function) the answer is simple: we can't. PL/SQL programs need to be compiled before we can call them. This means any table referenced in the program must exist at compilation time. Private temporary tables don't change that.
The private temporary table is intended for use in ad hoc SQL work. It allows us to create a data structure we can use in SQL statements for the duration of a session, to make life easier for ourselves.
For instance, suppose I have a massive table of sales data - low level transactions - and my task is to investigate monthly trends. So I only need the total sales by month. Unfortunately, there is no materialized view providing this summary. I don't want to include the aggregating query in my select statements. In previous versions I would have had to create a permanent table (and had to remember to drop it afterwards) but in 18c I can use a private temporary table to stage my summary just for the session.
create private temporary table ora$ptt_sales_summary (
sales_month date
, total_value number )
/
insert into ora$ptt_sales_summary
select trunc(sales_date, 'MM')
, sum (qty*price)
from massive_sales_table
group by trunc(sales_date, 'MM')
/
select *
from ora$ptt_sales_summary
order by sales_month
/
Obviously we can write anonymous PL/SQL blocks in our session but let's continue assuming that's not what you need. So what is the equivalent of a private temporary table in a permanent PL/SQL program? Same as it's been for several versions now: a PL/SQL collection or a SQL nested table type.
Private temporary tables (Available from Oracle 18c ) are dropped at the end of the session/transaction depending on the definition of PTT.
The ON COMMIT DROP DEFINITION option creates a private temporary table that is transaction-specific. At the end of the transaction,
Oracle drops both table definitions and data.
The ON COMMIT PRESERVE DEFINITION option creates a private temporary table that is session-specific. Oracle removes all data and
drops the table at the end of the session.
You do not need to drop it manually. Oracle will do it for you.
CREATE PRIVATE TEMPORARY TABLE ora$ptt_temp_table (
......
)
ON COMMIT DROP DEFINITION;
-- or
-- ON COMMIT PRESERVE DEFINITION;
Example of ON COMMIT DROP DEFINITION (table is dropped after COMMIT is executed)
Example of ON COMMIT PRESERVE DEFINITION (table is retained after COMMIT is executed but it will be dropped at the end of the session)
Note: I don't have access to 18c DB currently and db<>fiddle is facing some issue so I have posted images for you.
Cheers!!
It works with dynamic SQL:
declare
cnt int;
begin
execute immediate 'create private temporary table ora$ptt_tmp (id int)';
execute immediate 'insert into ora$ptt_tmp values (55)';
execute immediate 'insert into ora$ptt_tmp values (66)';
execute immediate 'insert into ora$ptt_tmp values (77)';
execute immediate 'select count(*) from ora$ptt_tmp' into cnt;
dbms_output.put_line(cnt);
execute immediate 'delete from ora$ptt_tmp where id = 66';
cnt := 0;
execute immediate 'select count(*) from ora$ptt_tmp' into cnt;
dbms_output.put_line(cnt);
end;
Example here:
https://livesql.oracle.com/apex/livesql/s/l7lrzxpulhtj3hfea0wml09yg
I'm trying to execute a Merge procedure Iv'e created but I encounter the following error:
Could not get schema Object:java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist
I read in forum about the error and I made sure I'm on the right scheme.
In addition, the merge query works properly.
any suggestion?
procedure
create or replace procedure insert_or_update_Account
IS
BEGIN
MERGE INTO CRMAS_ODS_RAW_DATA.ACCOUNT T
USING CRMAS_ODS_RAW_DATA.ACCOUNT_TMP S
ON ( T.TCCID=S.TCCID)
WHEN MATCHED THEN
UPDATE
SET
T.BATCH_ID =S.BATCH_ID ,
T.SOURCE_SYSTEM =S.SOURCE_SYSTEM ,
T.UPDATE_DATE =S.UPDATE_DATE ,
T.MLI_LOCALID =S.MLI_LOCALID
WHERE
T.BATCH_ID <>S.BATCH_ID OR
T.SOURCE_SYSTEM <>S.SOURCE_SYSTEM OR
T.MLI_LOCALID <>S.MLI_LOCALID
WHEN NOT MATCHED THEN
INSERT (T.BATCH_ID,T.SOURCE_SYSTEM,T.UPDATE_DATE,T.TCCID,T.MLI_LOCALID,T.MLI_LOCALSYSTEMNAME)
VALUES (S.BATCH_ID,S.SOURCE_SYSTEM,S.UPDATE_DATE,S.TCCID,S.MLI_LOCALID,S.MLI_LOCALSYSTEMNAME);
commit;
END insert_or_update_Account;
The procedure owner should be granted privileges on CRMAS_ODS_RAW_DATA.ACCOUNT table directly, not via role.
The fact that MERGE works in SQL (but not in PL/SQL) sounds exactly like that.
I have the function in which I need to drop and create tables. In the example below I try to create the table but it fails
CREATE OR REPLACE FUNCTION DEVTEST
RETURN NUMBER
IS
COMMAND VARCHAR2(256);
ID VARCHAR2(128);
NAME VARCHAR2(128);
TMP_LIST VARCHAR2(128);
BEGIN
ID := '12345';
NAME := 'ABCdef';
TMP_LIST := 'tmpTest';
command := 'create table ' || TMP_LIST || ' ( USER_ID VARCHAR2(11), USER_NAME VARCHAR2(36))';
DBMS_OUTPUT.PUT_LINE('command = ' || command);
EXECUTE IMMEDIATE command;
return 0;
END;
I call the function:
select NSB_DEVTEST() from dual
And get the error:
ORA-14552: cannot perform a DDL, commit or rollback inside a query or DML ORA-06512: at "DEV1_SERVER.DEVTEST", line 15
How do I correct this to create/drop a table inside a function?
My server details:
Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bi
PL/SQL Release 10.2.0.5.0 - Production
CORE 10.2.0.5.0 Production
TNS for Solaris: Version 10.2.0.5.0 - Production
NLSRTL Version 10.2.0.5.0 - Production
The problem is not with the function but with it being called from a SQL statement rather than from a pl/sql block.
A SELECT statement in SQL is equivalent to a READ operation that comes with read consistency. It cannot make any changes to the database. The database should always be the same before and after the "READ" operation completed, otherwise it would be a WRITE operation and the entire database consistency would go havoc.
Also, like the error says, DDL operations do a COMMIT behind the scenes before they start. Any read consistent operation should never do any COMMITS and write to the database without the user knowing.
You can instead call the function from pl/sql like this -
DECLARE
l_result NUBMER;
BEGIN
l_result := DEVTEST;
DBMS_OUTPUT.PUT_LINE(l_number);
END;
Still I would prefer writing a procedure for this, so others don't get confused by why this can't be called from SQL. The general rule that I follow for myself is that - Functions "get" things and Procedures "do" things (like DML).
The answer to your question is: don't. Production code, on the whole, shouldn't be creating tables on the fly. If you need a table to hold data temporarily, then create a Global Temporary Table (GTT) once and have your code refer to it.
The reason why you're getting that error (apart from it being self-evident from the error message) is because you're calling the function from within a SQL statement. You can't do that; you'd have to call it directly in PL/SQL.
I'm curious as to why you think this approach is a good, feasible approach, and also what you're going to be doing with the table once you've created it.
Your code is perfect no problem in the code . the problem is while you try to execute
you can only execute a pure function in select statement, which means a function without ddl & dml . ( if you use pragma autonomous_transaction while performing dml inside a function then you can use it in select statement ). When function has DDL command you can never ever execute it in select statement , but instead you can only execute it in PLSQL block like this
declare
a number;
begin
a:= devtest;
end;
/
and you can check your table
select * from tmptest;
It appears as though there's an implementation restriction that forbids the use of forall .. insert on Oracle, when used over a database link. This is a simple example to demonstrate:
connect schema/password#db1
create table tmp_ben_test (
a number
, b number
, c date
, constraint pk_tmp_ben_test primary key (a, b)
);
Table created.
connect schema/password#db2
Connected.
declare
type r_test is record ( a number, b number, c date);
type t__test is table of r_test index by binary_integer;
t_test t__test;
cursor c_test is
select 1, level, sysdate
from dual
connect by level <= 10
;
begin
open c_test;
fetch c_test bulk collect into t_test;
forall i in t_test.first .. t_test.last
insert into tmp_ben_test#db1
values t_test(i)
;
close c_test;
end;
/
Very confusingly this fails in 9i with the following error:
ERROR at line 1: ORA-01400: cannot insert NULL into
("SCHEMA"."TMP_BEN_TEST"."A") ORA-02063: preceding line from DB1
ORA-06512: at line 18
If was only after checking in 11g that I realised this was an implementation restriction.
ERROR at line 18: ORA-06550: line 18, column 4: PLS-00739: FORALL
INSERT/UPDATE/DELETE not supported on remote tables
The really obvious way round this is to change forall .. to:
for i in t_test.first .. t_test.last loop
insert into tmp_ben_test#db1
values t_test(i);
end loop;
but, I'd rather keep it down to a single insert if at all possible. Tom Kyte suggests the use of a global temporary table. Inserting the data into a GTT and then over a DB link seems like massive overkill for a set of data that is already in a user-defined type.
Just to clarify this example is extremely simplistic compared to what is actually happening. There is no way we will be able to do a simple insert into and there is no way all the operations could be done on a GTT. Large parts of the code have to be done in user-defined type.
Is there another, simpler or less DMLy, way around this restriction?
What restrictions do you face on the remote database? If you can create objects there you have a workaround: on the remote database create the collection type and a procedure which takes the collection as a parameter and executes the FORALL statement.
If you create the t__test/r_test type in db2 and then create a public synonym for them on db1 then you should be able to call a procedure from db1 to db2 filling the t_table and returning in to db1. Then you should be able to insert into your local table.
I'm assuming you would use packaged types and procedures in the real world, not anonymous blocks.
Also, it would not be the ideal solution for big datasets, then GTT or similar would be better.
I am trying to move some data over a dblink and one of the columns is an XMLType column. The code looks like this:
begin
delete from some_schema.some_remote_tab#src_2_trg_dblink;
INSERT INTO some_schema.some_remote_tab#src_2_trg_dblink(id, code, gen_date, xml_data)
SELECT id, code, gen_date, xml_data
FROM local_table;
end;
Oracle returns these errors:
ORA-02055: distributed update operation failed; rollback required
ORA-22804: remote operations not permitted on object tables or user-defined type columns
Some research on ORA-22804 shows that I am probably getting this error because of the XMLType column, but I am not sure how to resolve this.
(Oracle 10g)
We get ORA-22804 because every instance of a Type in our Oracle database has an OID, which is unique within the database. We cannot transfer that OID to another database; this has caused me grief before when trying to import schemas which have User-Defined Types. I hadn't realised that it also affected XMLType, but it is an Object so it is not surprising.
The solution is icky: you will have to unload the XML into text on your local database and then convert it back into XML in the remote database.
I don't have a distributed DB set-up to test this right now, but if you're lucky it may work:
INSERT INTO some_schema.some_remote_tab#src_2_trg_dblink(id, code, gen_date, xml_data)
SELECT id, code, gen_date, xmltype ( xml_data.asClobVal() )
FROM local_table;
If the asClobVal() method doesn't work you may need to use the SQL function XMLSERIALIZE() instead.
XMLSerialize(DOCUMENT xml_data AS CLOB)
If you're really unlucky you won't be able to do this in a single SQL statement, and you'll have to solve it using PL/SQL. To a certain extent this will depend on which version of the database you are using; the more recent the version, the more likely you'll be able to it in SQL rather than PL/SQL.
Try to do this the other way around. That is log into the remote db, create a dblink to the local db, and do an insert like this
INSERT INTO remote_schema.some_remote_tab(id, code, gen_date, xml_data)
SELECT id, code, gen_date, xml_data
FROM local_table#dblink_to_local_db;
Instead Perform a Data PULL.
create the data pull procedure at Remote database B.
create synonyms and provide grants to the dblink user.
Call the Remote procedure from Database A (Source) Perform a commit at Database A(source).
(Meanwhile .. wait for oracle to find some solution to perform the PUSH of XML over dblink in the future)
Create a procedure at Remote site Database B
CREATE OR REPLACE PROCEDURE PR_REMOTE(OP_TOTAL_COUNT OUT NUMBER) IS
BEGIN
INSERT /*+ DRIVING_SITE(src) */
INTO REMOTE_TABLE TGT_B
(XMLDATA_COL)
SELECT SRC.XMLDATA FROM LOCAL_TABLE#TGT2SRC_DBLINK SRC;
OP_TOTAL_COUNT := SQL%ROWCOUNT;
END;
Call the procedure from Database A
DECLARE
V_COUNT NUMBER := 0;
BEGIN
PR_REMOTE(V_COUNT);
COMMIT;
END;
I was facing the same issue with an heterogeneous DB link to SQL server.
Ended up using xmltype.getStringVal() to insert in a VARCHAR column on SQL Server side as the data was under 4000 characters.
There is also xmltype.getClobVal() if over 4000 characters but I haven't tested it.
The "xml->text->xml" chain might be complicated, but could help in some cases (for example when inserting is not on option but updating only).
You can try with "n" peaces of varchar columns (in the destination table or in a differnet one, perheaps in different schema on the remote DB), where "n" is:
ceil(max(dbms_lob.getlength(MyXmlColumn)) / 4000)
Then you can transfer these fragments to remote temporary fields:
insert into RemoteSchema.MyTable(Id, XmlPart1, XmlPart2,...)
(select 1 /*some Id*/,
dbma_lob.substr(MyXmlColumn.getclobval(), 4000, 1),
dbma_lob.substr(MyXmlColumn.getclobval(), 4000, 4001),
...
from LocalSchema.MyTable
XmlType can be re-composed from fragments like this:
create or replace function concat_to_xml(p_id number)
return xmltype
is
xml_lob clob;
xml xmltype;
begin
dbms_lob.createtemporary(xml_lob, true);
for r in (select XmlPart1, XmlPart2, ... from RemoteSchema.MyTable where Id = p_id)
loop
if r.XmlPart1 is not null then
dbms_lob.writeappend(xml_lob, length(r.XmlPart1), r.XmlPart1);
end if;
if r.XmlPart2 is not null then
dbms_lob.writeappend(xml_lob, length(r.XmlPart2), r.XmlPart2);
end if;
...
end loop;
xml := xmltype(xml_lob);
dbms_lob.freetemporary(xml_lob);
return xml;
end;
Finally use the result to update any other table in the remothe schema like:
update RemoteSchema.MyTable2 t2 set t2.MyXmlColumn = concat_to_xml(1 /*some Id*/);