I am using talend for ETL I don't have enough experience in this, I am having two tables for example- account and account_roles account table is having id, name, password etc fields and account_roles table is having account_id which is f.k to account table's pk. and one more field.
Both the fields in account_roles are having duplicates, I want to save account_roles in destination with update and insert logic using talend.
But I am getting error as I don't have any table that can be treated as primary key in the account_roles table, so talend can't update or insert it.
How I deal with this situation I tried tDBOutput advance option use_field_option but still it requires unique entries.
Is there any possible solution to this issue, I also want to know if I can make table Foreign key in the account_roles table will it work then? If yes then How to make F.k in talend OPen studio is my second question.
Attaching Snapshots of my tables and tMap below -
I want to know the way I can put my tables into database if I don't have any primary key! Kindly help me.
First question
I think you should place the primary key in the physical account_roles table. Talend will use the key indication of the dbOutput component, and the physical key of the table.
In order to delete duplicates rows, you can also use a tUniqRow before the dbOutput: The key you indicate in the UniqRow is not directly linked to the database; this is only the key on which tUniqRow will be based.
Second question
It's not possible to delegate the f.k. verification to Talend. But you can do this verification in your database by placing foreign keys in your table. If an id is not present in the reference table, the database returns an error that is returned by Talend.
Related
There is a name field in the UI which is also the primary key column in the underlying table. There is a requirement to make that field editable in the UI. There should be an ID which should serve as the primary key, but there isn't and now it is not feasible to introduce any ID column.
Is there any alternate design idea which can be used in such a scenario ?
The UI is in Swing and DB is Oracle.
First of all, I don't know, who thinks Name field can be Primary Key. That's the wrong database design ever.
Yes, you better change it to some ID column as Primary Key and that shouldn't be updated in future. Since, you can't have multiple Primary Key. So, you need to perform some circus here.
You need to drop existing Primary Key first. Since, you can't have multiple Primary Key in single table.
Create your ID column and allow NULL
Then, update this column with sequence.
Once your ID column gets populated, you need to create Primary Key on this column.
You can only have one primary key, but you can have any number of unique indexes on a table. So let the existing primary key be the immutable primary key and have the application use this key internally for everything. Add another column to the table and create a unique index on it. Let the users modify this other field.
Another alternative would be to declare all child tables with foreign keys ON UPDATE CASCADE. That way, any update to the primary key will cascade to the child tables. Once implemented in production, quit the company and run fast in the other direction and write an article about how you were the first person ever to use ON UPDATE CASCADE in a production setting.
I have a table which needs to be ingested from Oracle source to Greenland target using ETL tool talend. The table is huge , hence we want to load the data on daily basis incrementally. The table doesn't have any primary or unique key.
Table has date column, I am able to get both inserted/updated records from last update date but to insert that data, we need a primary key.
Any solution on how to load the data without using a primary key?
You need to define your key in talend in the schema of the component that insert into your target table, like this :
And you can use this key to update your table, in the advanced settings of the same component, activate the check box use fields optins and select your key :
This is tested and worked fine against Oracle table that does not have primary key, and it should work for you.
The diagram has over 40 tables, most of them have a primary key defined.
For some reason there is this one table, which has a primary key defined, but that's being ignored when I export the model to a DDL script.
This is the "offending" key (even though it's checked it is nowhere to be found on the generated DDL script):
Has anybody had the same problem? Any ideas on how to solve it?
[EDIT] This is where the key is defined:
And this is the DDL preview (yes, the primary key shows up there):
This is what happens if I try to generate the DDL for just that table (primary key still not generated):
I was finally able to identify and reproduce the problem.
It was a simple conflict of constraints.
Table MIEMBROS had a mandatory 1 to n relationship (foreign key) from another table on its primary key column and vice-versa (there was a foreign key on MIEMBROS against the other table's primary key).
This kind of relationship between two tables makes it impossible to add a record to any of them: The insert operation will return an error complaining about the foreign key restriction pointing the other table.
Anyway I realized that one of the relationships was 0 to n so I simply unchecked the "mandatory" checkbox on the foreign key definition and everything went fine.
So, in a nutshell: The Data Modeler "fails" silently if you are defining a mutual relationship (two foreign keys, one on each table against the other table) on non nullable unique columns, by not generating the primary key of one of the tables.
Such an odd behavior, if you ask me!
"This kind of relationship between two tables makes it impossible to add a record to any of them: The insert operation will return an error complaining about the foreign key restriction pointing the other table."
Actually, if you have deferred constraints, this is not impossible. The constraints can be enforced, for example, at commit time rather than immediately at insert time.
From the Data Modeler menu under File, I used Export -> DDL File. The keys appeared in the DDL, then when I went back to the diagram and did DDL Preview, it showed all the missing stuff.
I need to update the primary key of a large Index Organized Table (20 million rows) on Oracle 11g.
Is it possible to do this using multiple UPDATE queries? i.e. Many smaller UPDATEs of say 100,000 rows at a time. The problem is that one of these UPDATE batches could temporarily produce a duplicate primary key value (there would be no duplicates after all the UPDATEs have completed.)
So, I guess I'm asking is it somehow possible to temporarily disable the primary key constraint (but which is required for an IOT!) or alter the table temporarily some other way. I can have exclusive and offline access to this table.
The only solution I can see is to create a new table and when complete, drop the original table and rename the new table to the original table name.
Am I missing another possibility?
You can't disable / drop the primary key constraint from an IOT, since it is a unique index by definition.
When I need to change an IOT like this, I either do a CTAS (create table as) for a new plain heap table, do my maintenance, and then CTAS a new IOT.
Something like:
create table t_temp as select * from t_iot;
-- do maintenance
create table t_new_iot as select * from t_temp;
If, however, you need to simply add or join a new field to the existing key, you can do this in one step by creating the new IOT structure, then populating directly from the old IOT with a query.
Unfortunately, this is one of the downsides to IOTs.
I would recommend following method:
Create new IOT table partitioned by system with single partition
with exactly same structure as current one.
Lock current IOT table to prevent any DML.
insert into new table as select from current table changing PK values in select. This step
could be repeated several times if needed. In this case it's better
to do it in another session to keep lock on original table.
Exchange partition of new table with original table.
I need to insert records into a table that has no primary key using LINQ to SQL. The table is poorly designed; I have NO control over the table structure. The table is comprised of a few varchar fields, a text field, and a timestamp. It is used as an audit trail for other entities.
What is the best way to accomplish the inserts? Could I extend the Linq partial class for this table and add a "fake" key? I'm open to any hack, however kludgey.
LINQ to SQL isn't meant for this task, so don't use it. Just warp the insert into a stored procedure and add the procedure to your data model. If you can't do that, write a normal function with a bit of in-line SQL.
Open your DBML file in the designer, and give the mapping a key, whether your database has one or not. This will solve your problem. Just beware, however, that you can't count on the column being used for identity or anything else if there isn't a genuine key in the database.
I was able to work around this using a composite key.
I had a similar problem with a table containing only two columns: username, role.
This table obviously does not require an identity column. So, I created a composite key with username and role. This enabled me to use LINQ for adding and deleting entries.
You might use the DataContext.ExecuteCommand method to run your own custom insert statement.
Or, you might add a primary key to a column, this will allow the objects to be tracked for inserts/updates/deletes by the datacontext. This will work even if the column isn't really an enforced primary key in the database (how would linq know?). If you're only doing inserts and never re-use a primary key value in the same datacontext, you'll be fine.