in ODI 12c LKM is used in a mapping in order to load the data from the source to the staging area, but i do not need an IKM which to inset the data from the staging to the target, can ODI mapping do the first phase only that is the LKM pahse, as doing the 2 KM in my case doubles the time.
That's possible but you'd need to use LKM that is written that way.
I don't think there is one OOB there but you should be able to easily write your own.
The main thing there is in the java bean shell code (see A. Substitution API Reference) you would need to change the call from Collection Table:
…INTO TABLE <%=snpRef.getTable("L", "COLL_NAME", "W")%>
to Target Table:
…INTO TABLE <%=snpRef.getTable("L", "TARG_NAME", "A")%>
That's the main thing. You would also need to adjust the fields etc… The post here ODI - Load data directly from the Source to the Target without creating any Temporary table describes steps in more details, but once you get the idea how powerful API is you can pretty much do anything.
Related
I am working on ODI 10 project which has 153 interfaces divided in a few packages. What I want to do is create a PL/SQL procedure with INSERT statements instead of having 153 interfaces. These interfaces are more or less similar i.e they have the same source table and same target (in my case target is a Essbase Hyperion cube), the transformations & filters are different. So anytime I have to update something like a column value , I have to open 153 interfaces and update in each and every one of them. In a procedure, I could do this so easily, I can just replace all values.
So I feel that its best that I create a PL/SQL procedure, as I can maintain the code better that way.
Is there a way to convert the interface into a SQL query?. I want a direct data dump, I don't want to do an complex incremental load. I am just looking to truncate the table and load the data.
Thanks in advance.
It is possible to get the SQL code generated by ODI from the Operator in the log tables. It can also be retrieved in the repository.
Here is an example of a query for ODI 12c (10g being out of support for a long time now) :
SELECT s.sess_no, s.nno, step_name, scen_task_no, def_txt
FROM SNP_SESS_STEP s, SNP_SESS_TASK_LOG t
WHERE s.sess_no = t.sess_no
AND s.nno = t.nno;
Starting with ODI 11g, it is also possible to simulate the execution instead of doing an actual execution. This functionality will just display the code generated in ODI Studio instead of running it.
Finally, upgrading to a more recent of ODI would allow to use the ODI SDK. With it you could programmatically do changes to all the mappings in one go. Reusable mappings could also help as it sounds that some logic is implemented multiple times. That would enable to ease these kind of changes while keeping the benefits of an ELT tool (scheduling, monitoring, visual representation of flows, cross-technology, ...).
Disclaimer : I'm an Oracle employee
I want to migrate a subset of customer data from one shared database environment to another shared database environment. I use hibernate and have quite a few ID and FK_ID columns which are auto generated from an oracle sequence.
I have a liquibase change log that I exported from jailer which has the customer specific data.
I want to be able to rewrite all of the sequence ID columns so that they don't clash with what's already in the target database.
I would like to avoid building something that my company has to manage, and would prefer to upstream this to liquibase.
Is anyone aware of anything within liquibase that might be a good place to start.
I would like to either do this on the liquidbase xml before passing it to 'update' command, or as part of the update command itself. Ideally as part of the update command itself.
I am aware that I would need to make liquibase aware of which columns are PK sequence columns and the related FK columns. The database structure does have this all well defined, so I should be able to read this into the update process.
Alternatively I had thought I could use the extraction model csv from jailer
Jailer - http://jailer.sourceforge.net/
I would suggest that for one-time data migrations like this, Liquibase is not the best tool. It is really better for schema management rather than data management. I think that an ETL tool such as Pentaho would be a better solution.
I actually managed to figure it out for myself with the command line 'update' command of liquibase by using a custom change exec listener.
1) I pushed a MR to liquibase to allow registration of a change exec listener
2) I implemented my own change exec listener that intercepts each insert statement and rewrites each FK and PK field to one that is not as yet allocated in the target database. I achieve this by using a oracle sequence. In order to avoid having to go back to the database each time for a new sequence, I implemented my own version of the hibernate sequence caching
https://github.com/liquibase/liquibase/pull/505
https://github.com/pellcorp/liquibase-extensions
This turned out to be quite a generic solution and in concert with some fixes upstreamed to jailer to improve the liquibase export support its a very viable and reusable solution.
Basic workflow is:
1) Export a subset of data from source db using jailer to liquibase xml
2) Run the liquibase update command, with the custom exec change listener against the target.
3) TODO Run the jailer export on the target db and compare with the original source data.
As an organization we are moving towards the purchase of ODI as an ELT tool.
We have plenty of PLSQL resource but I have heard ODI is powerful enough at data manipulation to replace much of what was previously done in PLSQL.
What are its strengths? And weaknesses?
And can it completely do away with coding the data transformation in PLSQL?
No, it doesn't however you might be 99% correct here.
It's actually a tricky question as PL/SQL might be submitted by ODI too.
I would reserve it (PL/SQL) for defining functions/procedures (if you REALLY need to) to be later called by ODI.
This should NEVER be something immediately related to ETL like INSERT INTO … SELECT … FROM … - that's where ODI fits the bill perfectly.
The only justified cases, I came across during my ODI experience (9yrs) were:
- creating PL/SQL function to authenticate (and later authorize through OBIEE) an LDAP/AD user
- creating helper functions to be later called by ODI DQ(CKM) modules like is_number, is_date
- creating XML files directly by DB (even with never ODI XML driver you might still find it's best to use native DB XML API/functionality to produce XML) - for performance reasons. Other direct file operations (load/unload) could be done in this way.
- creating my own (optimized) hierarchy traversal query for performance reasons (beaten the standard Oracle SQL 'Recursive Subquery Factoring' feature to about 1000:1)
It's up to you if you want to make a reusable piece of logic by using PL/SQL and call it from ODI or code it from ODI directly (in the PL/SQL form)
For a project we need to investigate an existing installation of IBM Data Stage, doing a whole lot of ETL in loads of jobs.
The job flow diagrams contain lots of tables being used a source (both in MSSQL as well as Oracle), as well as a target (mostly in Oracle).
My question is now
How can I find all database tables used by all jobs in a certain Data Stage Project ?
I looked in Tools - Advanced Find, and there I can see all "table definitions". BUT, most of the tables actually used in jobs do not show up there, as they are defined as what Data Stage calls "Parallel Jobs" which in effect are SQL queries against database tables.
I am particularly interested in locating TARGET tables which are being loaded by a job.
So to put it bluntly, I want to be able to answer the question "Which job loads table XY ?".
If that is not possible, an automated means of extracting all the SQL statements used by the jobs would be an alternative.
We have access to IBM Websphere Data Stage and Quality Stage Designer 8.1
Exporting the jobs creates a text file that details what the job does. Open the export file in a text editor and you should be able to find SQL inserts with a simple search. Start with searching for SQL keywords like 'INTO' and 'FROM'.
Edit: Alternatively, if every table that was used was defined by importing table definitions, you should be able to find the table definition in the folder for its type. This however, will not make it apparent where and how the table was used (which job, insert or select from?), so I would recommend the first method of searching the Export files.
I have a to develop a PL/SQL procedure that dumps the content of a table when an error occurs during an application transaction, the content of the dump must match the content of the table before the ROLLBACK of the transaction.
I thought about using external table as the dump format of the table (TYPE ORACLE_DATAPUMP). After going through the Oracle documentation, I found that the only way to that is by executing:
CREATE TABLE tabtest_test (
F1 NUMBER,
F2 CHAR(10))
ORGANIZATION EXTERNAL (
TYPE ORACLE_DATAPUMP
DEFAULT DIRECTORY USER_DUMP_DEST
LOCATION ('tabtest.dmp’));
The problem is that by executing the “CREATE TABLE”, Oracle performs an implicit commit within our failed transaction which needs to be rolled back after the dump of the table.
I thought about using the “PRAGMA AUTONOMOUS_TRANSACTION;” to execute the “CREATE TABLE”, but it doesn’t really fit our need as it dumps the content of the table outside our application transaction.
My question: is there a way to get the 'tabtest.dmp’ without doing a “CREATE TABLE” ? for example by accessing directly the Oracle API responsible for this.
Regards.
How about creating the external table once, as part of your application setup process?
Failing that, you could create it at the beginning of the transaction that might need it. If there is an error, populate it; if the transaction finishes successfully, drop it.
If (and it's a big IF) you can use AUTONOMOUS_TRANSACTIONS to create the table in a separate transaction, I think this is what you need to do. If you manage to create the table within the scope of your current transaction, and write your data to that newly-created table, that data should, by all rights, disappear as soon as you do your ROLLBACK.
The problems you're experiencing here are a subset of the large class of issues known as "Problems Which Occur When Trying To Treat A Relational Database As A Flat File". Relational databases are great when used AS DATABASES, but are really bad at being flat files. It's kind of like animals on the farm - sheep are great AS SHEEP, but make lousy cows. Cows make lousy goats. Goats - great animals - intelligent, affectionate (yep), low-maintenance, won't hear a word spoken against 'em - but NOT what you want in a draft animal - use a horse, ox, or mule for that. Basically, you should pick horses for courses (pardon the expression). A database makes a crappy flat file, and vice versa. Use what's appropriate.
IMO you'd be better off writing your data to a flat file, and perhaps this file could be mapped in as an external table. You might want to write the file in something like CSV format that lots of other tools can process. YMMV.
Share and enjoy.
Why do you need to use external tables? You could just read the file using UTL_FILE.