I'm using the Common Data Service for Apps connector in Azure Data Factory to load data into Dynamics 365
I've done this successfully before using the entity key. See this question: Loading records into Dynamics 365 through ADF
Now I'm trying to use an alternate key to Upsert records into the account entity. (In this case insert)
In Dynamics
I've created two custom attributes fields in account:
Field name Data Type Field Type Max Length
=======================================================
xyz_srcsystem Single Line Simple 50
xyz_srccode Single Line Simple 50
Then I created a Key on account which contains these fields:
xyz_alternatekeyaccount
In ADF
Then I used a Copy Data activity in ADF to copy data from a SQL view into the account entity, using the CDS connector as a target.
This my source SQL statement:
SELECT
CAST(NULL as uniqueidentifier) as accountid,
'ADFTest1' as accountnumber, 'ADF Test 1' as [description],
'nmcdermaid#xyz.com.au' as emailaddress1,
CAST('TST' AS NVARCHAR(50)) as xyz_srcsystem,
CAST('1' AS NVARCHAR(50)) as xyz_srccode
In the target, in the Alternate key name field I entered the alternate key name: xyz_alternatekeyaccount
The error I get when I run the pipeline is
Invalid type for entity id value
Some test to rule out edge cases:
if I put a dummy alternate key in, I get Cannot retrieve key information of alternate key 'xyz_alternatekeyaccountx' for entity 'account'. This implies it is finding the alternate key correctly
If I remove the alternate key from the connector, it drops back to the other usual set of errors that I see
When I pull the entity into SQL using the CDM connector, the custom attributes arrive as NVARCHAR(MAX)
I've tried casting to these data types: NVARCHAR(MAX) NVARCHAR(50) VARCHAR(MAX) VARCHAR(50)
If I use the normal key (not an alternate key), and get the datatype wrong (anything other than GUID), I'll get the same error
Also see this Doco GitHub I raised:
https://github.com/MicrosoftDocs/azure-docs/issues/59028
When I changed the source SQL to this, it worked:
SELECT
'ADFTest1' as accountnumber, 'ADF Test 1' as [description],
'nmcdermaid#xyz.com.au' as emailaddress1,
CAST('TST' AS NVARCHAR(50)) as xyz_srcsystem,
CAST('1' AS NVARCHAR(50)) as xyz_srccode
Note: the difference is I did not include the true primary key in the source dataset.
Not that if you want to UPSERT a new record (INSERT) and this isn't based on an alternate key, you have to include a NULL primary key
Related
I'm using EF Core to connect to an Oracle11g database (using the Oracle.EntityFrameworkCore v2.19.90 provider). It's a code first scenario, the tables are created successfully and everything is as expected.
The problem is, when I try to insert something into the database, for example:
_context.Roles.Add(new ApplicationRole()
{
Name = "FOO",
DisplayName = "Foo"
});
_context.SaveChanges();
I get an error:
OracleException: ORA-01400: cannot insert NULL into ("SCHEMA"."AppRole"."Id")
The column Id is indeed non-nullable. When I use the SQL Server provider, everything is fine, the SQL Server automatically chooses an id for my entity.
Is there any way to get Oracle to set an Id for me? Or could it be done in another way?
I don't want to use Oracle triggers and the solution should be full code first.
As you're on Oracle 11g, then you have to use a trigger along with a sequence which will populate ID column in the background.
Another option is to, obviously, provide ID value during insert.
If you were on 12c or above, you could have used identity column. As you're not, your options are listed above.
One option may be usage of SEQUENCE and default value:
CREATE TABLE AppRole(
Id INT NOT NULL,
Name VARCHAR2(100),
DisplayName VARCHAR2(100)
);
CREATE SEQUENCE seq;
ALTER TABLE AppRole MODIFY Id DEFAULT seq.NEXTVAL;
INSERT INTO AppRole(Name, DisplayName) VALUES ('Foo','Foo');
db<>fiddle demo
Default with sequence is supported from Oracle 12c.
There should exist syntax in EntityFramework core that allow to do the following without relying on triggers(raw SQL query as last resort):
INSERT INTO AppRole(Id, Name, DisplayName) VALUES (seq.NextVal, 'Foo','Foo');
Sequences
Basic usage
You can set up a sequence in the model, and then use it to generate
values for properties: C#
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
modelBuilder.HasSequence<int>("OrderNumbers");
modelBuilder.Entity<Order>()
.Property(o => o.OrderNo)
.HasDefaultValueSql("NEXT VALUE FOR shared.OrderNumbers");
}
Note that the specific SQL used to generate a value from a sequence is
database-specific; the above example works on SQL Server but will fail
on other databases. Consult your specific database's documentation for
more information.
Oracle syntax is sequence_name.NEXTVAL.
I am using slick 2.1.0. Oracle doesn't have a notion of auto increment attribute for a column, so how can I manage an insert via slick using a sequence.
e.g. I have a table & sequence as follows :
CREATE TABLE USER
( "USER_ID" NUMBER NOT NULL ENABLE,
"NAME" VARCHAR2(100) NOT NULL ENABLE,
"ADDRESS" VARCHAR2(1000) NOT NULL ENABLE
);
CREATE SEQUENCE USER_ID_SEQ MINVALUE 1 MAXVALUE 99999999999999 INCREMENT BY 2;
How can I use this sequence to set my USER_ID? Also setting autoIncLastAsOption = true in Slicks's SourceCodeGenerator doesnt seem to help. My IDs are still not an Option[].
Here are some of the options suggested by Typesafe Developer:
If you don’t mind letting Slick manage the DDL, you can use O.AutoInc with OracleDriver. It will automatically create a backing sequence for the generated identity values. The way this works is by installing a trigger that automatically populates the ID from the sequence. Here’s the code that Slick generates for an AutoInc column on Oracle:
create sequence $seq start with 1 increment by 1;
create or replace trigger $trg before insert on $tab referencing new as new for each row when (new.$col is null) begin select $seq.nextval into :new.$col from sys.dual;
end;
where $seq, $trg, $col and $tab are the names of the sequence, trigger, identity column and table.
There is no special code being run during an actual insert operation. So if you already have a database schema with an identity sequence, you can manually create a trigger as shown above and mark the column as O.AutoInc in Slick to get the standard handling for auto-incrementing columns.
If you want a solution without a trigger, you could you insertExpr for inserting in Slick. This allows computed expressions, like using Slick’s own sequence API (which is supported by OracleDriver), but unlike a normal insert you do not get all features, convenience and performance (e.g. batch inserts and pre-compiled inserts).
The downside is that this can’t be precompiled (but compiling a simple expression of a few scalar values should be relatively cheap) and you can’t just insert a mapped cased class that way without some extra mapping boilerplate.
Another option would be to first get a new id (or even multiple ids for a batch insert) from the sequence with one query, put them into the data transfer objects, and then insert those normally with the ids in place. This requires one extra query per batch (for first fetching the ids) but you can easily use mapped objects and precompile everything.
I have a Form Applet and it has the Field SERIAL_NUM From the S_CASE Table , but whan i wont to insert a new record i get the Field Empty , how can i ganarate a SERIAL_NUM to insert to the table like the ones in the S_CASE table , is there and Predefult expretion that dose it ?
Best Regards
In the Siebel Object Type Business Component Field
has attribute named
Predefault Value
you can provide Siebel valid expression to populate value when creating new record using expression
Field: 'Id'
or
Expr: 'RowIdToRowIdNum ([Id])'.
Alternatively you can also write Server side script on
Siebel Object Type Business Component
in the method
BusComp_PreNewRecord
to populate customized value when creating new record.
Alternatively you can also use
Database Trigger
which can also populate customized value to columns and will be available in Business Component Field value.
I am working on an SSIS package to transfer data from an ERP system (Microsoft Dynamics AX) to a custom CRM solution. One of the things I need to transfer/update is customers. In the CRM system, there is a field called CustomerID that is unique to the CRM environment and must be set when the data is transferred from the ERP system to the CRM system.
The issue is that I must call a stored procedure to get the CustomerID for the next customer to be inserted. How can I do this from within SSIS? I am currently using a Data Flow to transfer data and the only place I can see to called the stored procedure is from within the ADO NET Source task as a SQL Command query and each time the procedure is called it increments the ID so every time the SSIS package is run it updates the CustomerID field for all customers in the CRM database.
The stored procedure resides on the custom CRM application. The stored procedure does not accept any input parameters. It simply outputs a scalar value like CT000001, CT000002, etc. every time it is called.
What are my other options for doing this properly?
You need to call the stored procedure for each row using Script Component and add the value to the pipeline downstream to the destination.
Sample SSIS 2012 package that illustrates a sample:
The sample uses the following database objects.
SQL Server 2012 database named PracticeDB. In this scenario both the CRM and ERP tables are in the same database to keep it simple.
Table dbo.CRM will contain the destination CRM data.
Table dbo.ERP holds the source ERP data
Table dbo.CustomerId is a hypothetical table that is in CRM system which will be used to increment the customer number.
Stored procedure dbo.GetCustomerId increments the customer number whenever it is called.
Script to create the above mentioned database objects:
Below script will create the above mentioned objects and will insert some rows into the source ERP table.
USE PracticeDB;
GO
CREATE TABLE dbo.CRM(
CRMId int IDENTITY(1,1) NOT NULL,
CustomerId int NOT NULL,
CustomerName varchar(50) NULL
);
GO
CREATE TABLE dbo.ERP(
ERPId int IDENTITY(1,1) NOT NULL,
CustomerName varchar(50) NOT NULL
);
GO
CREATE TABLE dbo.CustomerId(
Id int NOT NULL
);
GO
CREATE PROCEDURE dbo.GetCustomerId
AS
BEGIN
SET NOCOUNT ON;
IF NOT EXISTS(SELECT Id FROM dbo.CustomerId)
INSERT INTO dbo.CustomerId (Id) VALUES(1);
ELSE
INSERT INTO dbo.CustomerId (Id)
SELECT MAX(Id) + 1 FROM dbo.CustomerId;
SELECT MAX(Id) AS CustomerId FROM dbo.CustomerId;
END;
GO
INSERT INTO dbo.ERP (CustomerName) VALUES
('John Doe')
, ('Jane Doe')
, ('Stephen Smith')
, ('Dean Jones')
;
GO
Create an SSIS package and create an ADO.NET connection manager to connect to the appropriate data sources.
Drag and drop a Data Flow Task onto the Control Flow tab
Within the data flow task, drag and drop the components ADO.NET Source, Script Component (Transformation) and ADO.NET Destination.
ADO.NET Source will read data from table dbo.ERP
ADO.NET Destination will insert data into table dbo.CRM
Script Component will involve the following configuration.
Double-click the Script Component to open the Script Component Editor.
Click Connection Managers tab page
Select the ADO.NET Connection Manager ADONET_PracticeDB and give it a name like PracticeDB. This connection manager will be used inside the script component code.
Click Inputs and Outputs tab page
Rename Input to ERP and Output to CRM
Expand CRM, select Output Columns and click Add Column.
Rename the column name to CustomerID. The example uses the column of type integer. You need to change the data type according to your needs.
Click Script tab page.
On the script tab page, click Edit Script....
Type the code shown in the below script component section. The script initiates the connection using the ADO.NET Connection Manager available on the package and then initiates the command object within the PreExecute method.
The stored procedure is then called within the ERP_ProcessInputRow method for every row in the pipeline to generate new customer ids.
Script component code in C#
#region Namespaces
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Data.OleDb;
using System.Data.SqlClient;
#endregion
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
IDTSConnectionManager100 manager;
SqlConnection connection = null;
SqlCommand command = null;
public override void PreExecute()
{
manager = base.Connections.PracticeDB;
connection = new SqlConnection();
connection = (SqlConnection)manager.AcquireConnection(null);
command = new SqlCommand();
command.Connection = connection;
command.CommandType = CommandType.StoredProcedure;
command.CommandText = "dbo.GetCustomerId";
base.PreExecute();
}
public override void PostExecute()
{
base.PostExecute();
}
public override void ERP_ProcessInputRow(ERPBuffer Row)
{
Row.CustomerID = Convert.ToInt32(command.ExecuteScalar());
}
}
Close the Script component
Executed the below query against data to check the rows before executing the package.
Table select queries:
SELECT * FROM dbo.ERP;
SELECT * FROM dbo.CRM;
SELECT * FROM dbo.CustomerId;
After first execution of the package, you can notice that 4 rows have been inserted and also the stored procedure was successfully called to add the customer Ids.
After second execution of the package, you can notice that 4 more rows have been inserted and the stored procedure has successfully incremented the customer Ids.
Hope that gives you an idea.
I have a - for me unknown - issue and I don't know what's the logic/cause behind it. When I try to insert a record in a table I get a DB2 error saying:
[SQL0803] Duplicate key value specified: A unique index or unique constraint *N in *N
exists over one or more columns of table TABLEXXX in SCHEMAYYY. The operation cannot
be performed because one or more values would have produced a duplicate key in
the unique index or constraint.
Which is a quite clear message to me. But actually there would be no duplicate key if I inserted my new record seeing what records are already in there. When I do a SELECT COUNT(*) from SCHEMAYYY.TABLEXXX and then try to insert the record it works flawlessly.
How can it be that when performing the SELECT COUNT(*) I can suddenly insert the records? Is there some sort of index associated with it which might give issues because it is out of sync? I didn't design the data model, so I don't have deep knowledge of the system yet.
The original DB2 SQL is:
-- Generate SQL
-- Version: V6R1M0 080215
-- Generated on: 19/12/12 10:28:39
-- Relational Database: S656C89D
-- Standards Option: DB2 for i
CREATE TABLE TZVDB.PRODUCTCOSTS (
ID INTEGER GENERATED BY DEFAULT AS IDENTITY (
START WITH 1 INCREMENT BY 1
MINVALUE 1 MAXVALUE 2147483647
NO CYCLE NO ORDER
CACHE 20 )
,
PRODUCT_ID INTEGER DEFAULT NULL ,
STARTPRICE DECIMAL(7, 2) DEFAULT NULL ,
FROMDATE TIMESTAMP DEFAULT NULL ,
TILLDATE TIMESTAMP DEFAULT NULL ,
CONSTRAINT TZVDB.PRODUCTCOSTS_PK PRIMARY KEY( ID ) ) ;
ALTER TABLE TZVDB.PRODUCTCOSTS
ADD CONSTRAINT TZVDB.PRODCSTS_PRDCT_FK
FOREIGN KEY( PRODUCT_ID )
REFERENCES TZVDB.PRODUCT ( ID )
ON DELETE RESTRICT
ON UPDATE NO ACTION;
I'd like to see the statements...but since this question is a year old...I won't old my breath.
I'm thinking the problem may be the
GENERATED BY DEFAULT
And instead of passing NULL for the identity column, you're accidentally passing zero or some other duplicate value the first time around.
Either always pass NULL, pass a non-duplicate value or switch to GENERATED ALWAYS
Look at preceding messages in the joblog for specifics as to what caused this. I don't understand how the INSERT can suddenly work after the COUNT(*). Please let us know what you find.
Since it shows *N (ie n/a) as the name of the index or constraing, this suggests to me that is is not a standard DB2 object, and therefore may be a "logical file" [LF] defined with DDS rather than SQL, with a key structure different than what you were doing your COUNT(*) on.
Your shop may have better tools do view keys on dependent files, but the method below will work anywhere.
If your table might not be the actual "physical file", check this using Display File Description, DSPFD TZVDB.PRODUCTCOSTS, in a 5250 ("green screen") session.
Use the Display Database Relations command, DSPDBR TZVDB.PRODUCTCOSTS, to find what files are defined over your table. You can then DSPFD on each of these files to see the definition of the index key. Also check there that each of these indexes is maintained *IMMED, rather than *REBUILD or *DELAY. (A wild longshot guess as to a remotely possible cause of your strange anomaly.)
You will find the DB2 for i message finder here in the IBM i 7.1 Information Center or other releases
Is it a paging issue? we seem to get -0803 on inserts occasionally when a row is being held for update and it locks a page that probably contains the index that is needed for the insert? This is only a guess but it appears to me that is what is happening.
I know it is an old topic, but this is what Google shown me on the first place.
I had the same issue yesterday, causing me a lot of headache. I did the same as above, checked the table definitions, keys, existing items...
Then I found out the problem was with my INSERT statement. It was trying to insert to identical records at once, but as the constraint prevented the commit, I could not find anything in the database.
Advice: review your INSERT statement carefully! :)