I'm using the Dynamics connector in Azure Data Factory.
TLDR
Does this connector support loading child records which need a parent record key passed in? For example if I want to create a contact and attach it to a parent account, I upsert a record with a null contactid, a valid parentcustomerid GUID and set parentcustomeridtype to 1 (or 2) but I get an error.
Long Story
I'm successfully connecting to Dynamics 365 and extracting data (for example, the lead table) into a SQL Server table
To test that I can transfer data the other way, I am simply loading the data back from the lead table into the lead entity in Dynamics.
I'm getting this error:
Failure happened on 'Sink' side. ErrorCode=DynamicsMissingTargetForMultiTargetLookupField,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=,Source=,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot find the target column for multi-target lookup field: 'ownerid'.
As a test I removed ownerid from the list of source columns it loads OK.
This is obviously a foreign key value.
It raises two questions for me:
Specifically with regards to the error message: If I knew which lookup it needed to use, how can I specify which lookup table it should validate against? There's no settings in the ADF connector to allow me to do this.
This is obviously a foreign key value. If I only had the name (or business key) for this row, how can I easily lookup the foreign key value?
How is this normally done through other API's, i.e. the web API?
Is there an XRMToolbox addin that would help clarify?
I've also read some posts that imply that you can send pre-connected data in an XML document so perhaps that would help also.
EDIT 1
I realised that the lead.ownertypeid field in my source dataset is NULL (that's what was exported). It's also NULL if I browse it in various Xrmtoolbox tools. I tried hard coding it to systemuser (which is what it actually is in the owner table against the actual owner record) but I still get the same error.
I also notice there's a record with the same PK value in systemuser table
So the same record is in two tables, but how do I tell the dynamics connector which one to use? and why does it even care?
EDIT 2
I was getting a similar message for msauto_testdrive for customerid.
I excluded all records with customerid=null, and got the same error.
EDIT 2
This link appears to indicate that I need to set customeridtype to 1 (Account) or 2 (Contact). I did so, but still got the same error.
Also I believe I have the same issue as this guy.
Maybe the ADF connector suffers from the same problem.
At the time of writing, #Arun Vinoth was 100% correct. However shortly afterwards there was a documentation update (in response to a GitHub I raised) that explained how to do it.
I'll document how I did it here.
To populate a contact with against a parent account, you need the parent accounts GUID. Then you prepare a dataset like this:
SELECT
-- a NULL contactid means this is a new record
CAST(NULL as uniqueidentifier) as contactid,
-- the GUID of the parent account
CAST('A7070AE2-D7A6-EA11-A812-000D3A79983B' as uniqueidentifier) parentcustomerid,
-- customer id is an account
'account' [parentcustomerid#EntityReference],
'Joe' as firstname,
'Bloggs' lastname,
Now you can apply the normal automapping approach in ADF.
Now you can select from this dataset and load into contact. You can apply the usual automapping approach, this is: create datasets without schemas. Perform a copy activity without mapping columns
This is the ADF limitation with respect to CDS polymorphic lookups like Customer and Owner. Upvote this ADF idea
Workaround is to use two temporary source lookup fields (owner team and user in case of owner, account and contact in case of customer) and with parallel branch in a MS Flow to solve this issue. Read more, also you can download the Flow sample to use.
First, create two temporary lookup fields on the entity that you wish to import Customer lookup data into it, to both the Account and Contact entities respectively
Within your ADF pipeline flow, you will then need to map the GUID values for your Account and Contact fields to the respective lookup fields created above. The simplest way of doing this is to have two separate columns within your source dataset – one containing Account GUID’s to map and the other, Contact.
Then, finally, you can put together a Microsoft Flow that then performs the appropriate mapping from the temporary fields to the Customer lookup field. First, define the trigger point for when your affected Entity record is created (in this case, Contact) and add on some parallel branches to check for values in either of these two temporary lookup fields
Then, if either of these conditions is hit, set up an Update record task to perform a single field update, as indicated below if the ADF Account Lookup field has data within it
Related
Background
Each Dataverse table contains a primary name column. When displayed in a subgrid, clicking on the primary name column will navigate to the form so that the user can edit that row. Most subgrids in my application work this way.
The Problem
I have a Course form with a list of participants displayed in a subgrid. The subgrid displays each student's name (as a link) and the grade received in the course. There is no appropriate primary name column for this Participant table. To edit the participant record, the user must select the row in the subgrid, then click the subgrid's Edit button. As a result, this UI is different from all other subgrids in the application and I know that user's will click the student name to try to edit the participant record and be confused when they are presented with the student record.
Am I missing something? Is there a better way to handle this?
It's a common problem I face quite often. Here is usually what I would do.
Make sure the Primary Name Column always contains relevant information to the user to be able to quickly identify a record. Sometimes it requires copying information from one or multiple other columns into the primary column.
In your case that would probably means concatenating the student's name and grade.
How to do that?
Common to all solutions below
Use one of the following solution to copy the content of one or several fields into the primary column.
Make sure the solution you select also updates the content of the primary name column when one of the copied field is updated.
Remove or hide the primary column from the form, the name of the record will be displayed at the top of the form anyway and you probably don't want users to play with it.
Display the primary name column in every subgrid.
I would recommend not adding the fields copied into the primary column in the subgrids to avoid confusion.
Solution 1 - Classic Workflow
Create a classic workflow that runs when a record is created / updated
Pros:
Very quick to put in place
Runs synchronously (users will see the name updated in real-time)
Cons:
Not very practical if you need to add business logic (using different fields as source depending on a certain condition for example)
Solution 2 - Power Automate
Create a Flow that runs when a record is created / updated
Pros:
You can implement complex business logic in your Flow
Cons:
Runs asynchronously (users will have to refresh the page after the creation of a record to see the record's name)
According to Power Automate licensing that flow would certainly be considered as an "enterprise flow" and you are supposed to pay 100$ / month. That specific point must be taken with a grain of salt. I had several discussions with Microsoft about it and they haven't given me a clear answer about what would be considered an enterprise flow.
Solution 3 - Plugin
Create a plugin that executes when a record is created / updated
Pros:
You can implement very complex business logic in your Flow
It can run synchronously
Cons:
Pro-code (I put it as a con since Model-Driven App is a low-code / no-code approach but there is nothing wrong about pro-code per say)
Developing a new plugin for each entity where you need this logic is kind of overkill in my opinion. I would consider developing something very generic that would only require some sort of configuration when the logic needs to be applied to a new table.
I'm basically trying to create a primary ID between CRM and QuickBooks. Figured I'd just use the existing PK in CRM for the lookup. I'd like the PK to visible to the user, but not editable in CRM.
This has presented several problems in that you can't do that out of the box. I thought I read somewhere you could either via business rule or calculated field, but I haven't had luck with that.
It sounds like it would require web resources if I were to go this route.
The other option would be to just generate unique values for every record in Accounts and Contacts.
Does this automatically populate existing records or just new records? How do I get it to populate existing records?
You can use Auto number manager for configuring an auto-number attribute in every entity. This seeds a number based on configured format for new records. Uniqueness assured by SQL sequence feature & no need of any extra plugin/workflow.
For existing records - you can design a workflow along with a temp entity to assign auto-number. Read more.
Otherwise you can use SSIS + Kingswaysoft package to generate auto-number & assign for existing records.
I am suggesting you to create a new text field on the entity and create a pre plugin that will get the record primary GUID id from context and will set this GUID into the newly added attribute. You can set this field as read-only of form as well.
OR you can generate new GUID as well into the plugin.
Let me start by saying that I am indeed self-taught and I have no business trying to do what I am trying to do...However, I am combining two Access databases that track different information about the same projects (one tracks the proposal information for projects considered for selection and the other tracks budget information for those that are selected). The same projects have different IDs in each database. Each database has lookup fields (yeah, yeah, I know lookup fields are evil, but that is what I have). I want to update the values in the lookup fields for the related tables in the Budget DB to the ID fields for the same projects in the Project DB. Is there a way to write an update query to change the value in the project ID field of one table to the ID fields from another table?
Details: ProjectListTbl contains ProjectID field and the BudgetID field which refers to project ID from the Budget database (BDBID). I want the Reporting table values to change from the Budget Database ID (BDBProjectID) to the ProjectID from the ProjectListTbl. I have tried
UPDATE Reporting INNER JOIN ProjectListTbl ON ReportingTbl.BDBProjectID = ProjectListTbl.ProjectID SET ReportingTbl.BDBProjectID = [ProjectListTbl].[ProjectID], WHERE (((ReportingTbl.BDBProjectID)=DLookUp([ProjectListTbl].[ProjectID],[ProjectListTbl],[BudgetID]=[ReportingTbl]![BDBProjectID])));
Simple parent/child scenario like Order and OrderLineItems. I am inserting a new Order, the OrderID is an identity column (sql server). I'm also inserting OrderLineItems in the same SaveChanges transaction. I need to get the new OrderID into the OrderLineItems, but not sure how to do it. I have the appropriate FK relationships setup properly. When I save, I get an error that OrderID is a required field in OrderLineItems.
Will I have to split this out into 2 server calls? First to insert the Order, which will return the OrderID. And then another to insert the OrderLineItems?
The Breeze documentation discusses this topic (key generation) at several points including but not limited to: http://www.breezejs.com/documentation/save-changes, http://www.breezejs.com/documentation/extending-entities and http://www.breezejs.com/documentation/add-new-entity.
The basic idea idea is that providing that your model and metadata are set up properly, breeze can assign a temporary id in place of the identity column for use in linking your order and orderlineitem entities prior to being saved. As part of the save process, Breeze updates these temporary keys to their "real" key values and updates the local cache as well upon successful completion of the save.
I am creating an informatica workflow which can extract data from two tables of DB2 database and load into a temporary table. Suppose the two source tables name are Account (Parent) and Activities (Child). They have 1:M relationship. Means an Account can have many Activities (Account.PK = Activities.FK). Activities table has two columns- first 'Type' whose value could be 'Paid', 'Will-Pay' or 'Not-Paid'.And second column is 'Created_Date' datetime datatype, whenever you create new activity record, date and time would get stamp in this field. Now, condition to load data in temporary table is - "For an Account record, it would 1st check in Activities table for today's Paid activities (Type = Paid). If it finds more than one paid activities, then it would pick the Latest created one (Created_Date column) out of them. If there is no Paid activity record for the Account, then it would pick latest created 'Will-Pay' activity." Means, it should pick latest Paid activity for today (Sysdate) for an Account, if it is not present then only It will pick latest Will-pay activity for today. Please help me to understand how I can implement this logic in Informatica workflow and which transformations I should use and how? Thanks alot. Kindly help.
Best way to do it on SQL cause realize business logic on ETL it's not good. But if you insist it can be created by many ways. As example:
With SQL override
You can create 3 lookup transformation for Activities table with overrided SQL (and columns too) and one expression transformation for condition.
Lookup to find more than one 'paid' activities accounts
Lookup to find last 'paid' activity per account
Lookup to find last 'will pay' activity per account
Expression to return correct Activities key based from 1-3 lookup results
Without SQL override you need to recreate similar logic with filter, aggregator, joiner transformations.