Using Azure Data Factory to Sink to Dynamics Common Data Model.
Source:
Col1: CustomerName
Col2: CustomerGroup
Destination - CDM
Col1: CustomerName
Col2: CustomerGroup (this is Lookup Column in Dynamics)
Fails when mapping in ADF is:
Src.CustomerName -----------> Dst.CustomerName
Src.CustomerGroup ----------> Dst.CustomerGroup
followed the docs # https://learn.microsoft.com/en-gb/azure/data-factory/connector-dynamics-crm-office-365#writing-data-to-a-lookup-field and updated the mappings as below:
Src.CustomerName -----------> Dst.CustomerName
Src.CustomerGroup ----------> Dst.CustomerGroup**#EntityReference**
This works, but didn't populate the column Dst.CustomerGroup with the value
Any Clues..
As we know the sink column is a virtual column representing the entity reference. A Lookup may refer to more than one entity type.
As the document said:
What about adding an additional column in the copy activity source?
Related
I have a flow-files with the below structure
{
"PN" : "U0-WH",
"INPUT_DATE" : "44252.699895833335",
"LABEL" : "Marker",
"STATUS" : "Approved",
}
and I need to execute an update statement using some fields
update table1 set column1 = 'value' where pn=${PN}
I found convertJsonToSQL but am not sure how to use it in this case
You can use a processor namely ConvertjSONToSQL. Using this you can convert your json into an update query.
ConvertjSONToSQL Description
It takes the following parameters :
1. JDBC Connection Pool : Create a JDBC pool which takes DB connection information as input.
2. Statement Type : Here you need to provide type of statement you want to create. In your case its 'UPDATE'.
3. Table Name : Name of the table for which update query needed to be created
4. Schema Name : Name of the schema of your database.
5. Translate Field Names : If true, the Processor will attempt to translate JSON field names into the appropriate column names for the table specified. If false, the JSON field names must match the column names exactly, or the column will not be updated
6. Unmatched Field Behaviour : if an incoming JSON element has a field that does not map to any of the database table's columns, this property specifies how to handle the situation
7. Unmatched Column Behaviour : If an incoming JSON element does not have a field mapping for all of the database table's columns, this property specifies how to handle the situation
8. Update Keys : A comma-separated list of column names that uniquely identifies a row in the database for UPDATE statements. If the Statement Type is UPDATE and this property is not set, the table's Primary Keys are used. In this case, if no Primary Key exists, the conversion to SQL will fail if Unmatched Column Behaviour is set to FAIL. This property is ignored if the Statement Type is INSERT
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Read the description above and try to use the properties given. Detailed description of the processor is given in the link.
ConvertjSONToSQL Description
I'm using Amplify from AWS to build a small ecommerce project using React as frontend.
I'd like to know how I should write the "Product" and "Order" types in the schema in order to be able to write productId's to a product array in the Order table when users complete a purchase.
My schema.graphql file:
type Product #model {
id: ID!
name: String!
price: Int!
category: String!
images: [String]!
}
type Order #model {
id: ID!
products: [Product] #connection
}
My question is about the last line, do I need to define that [Product] connection there or I can use [String] to store product id's in a simple string array?
Point 1: In dynamoDB, you only need to define the data type of your partition key and sort key, and these can be string, number etc. For all the other attributes, you don't need to define anything.
Point 2: The dynamoDB designers prefer using a single table per application, unless it's impossible to manage data without multiple tables. Keeping this in mind, your table can be something like this.
Please observe: Only Id aka partition key and Sk aka sort key column is fixed here, all other columns can be anything per item. This is the beauty of DynamoDB. Refer to this document for dynamoDB supported data types.
I'm creating a Grails 3 app with some tables from my Oracle 12c database scaffolded and while so far everything went fast, I came across one problematic table which doesn't have an ID column. It's just four VARCHAR2's. Also, in the Constraints tab of the Oracle SQL Developer, I don't see any of them defined as the Primary Key. How should one progress in such a scenario to successfully create a CRUD for this class?
I've tried telling Grails to use the row id as an ID but this only results in a "getLong not implemented for class oracle.jdbc.driver.T4CRowidAccessor" error when I try accessing the (to-be) scaffolded page. My code looks like this:
class AliasFrequencyDict {
String frequency
String unit
String description
String lang
static constraints = {
frequency maxSize: 10, sqlType: 'VARCHAR2'
unit maxSize: 1, sqlType: 'VARCHAR2'
description maxSize: 30, sqlType: 'VARCHAR2'
lang maxSize: 2, sqlType: 'VARCHAR2'
}
static mapping = {
sort 'frequency'
version false
id column: 'ROWID'
}
}
How should I solve this if I don't have an explicit ID column and I am not allowed to create one?
EDIT: So the only way I was able to progress so far was to use a composite key consisting of all the columns and then manually change the index view so that instead of this:
<f:table collection="${aliasFrequencyDict}" />
we now have this:
<f:table collection="${aliasFrequencyDictList}" properties="['frequency','unit','description','lang']"/>
This, however, doesn't let me access any of the existing entries or add new ones, as I guess I'd have to manually specify these properties there, too. It doesn't seem like it's nearly as easy to explicitly state them in the edit view, for example, as it is in the index, though (or to make the editing actually work, on top of that).
EDIT2: Also, from what I gathered, using ROWID isn't a good idea anyway. Oracle docs state:
Although you can use the ROWID pseudocolumn in the SELECT and WHERE
clause of a query, these pseudocolumn values are not actually stored
in the database. You cannot insert, update, or delete a value of the
ROWID pseudocolumn.
Thus, I'm out of ideas about how to progress :( Any help?
I am trying to migrate my database from MySQL to Cassasndra. The problem I am facing is with one of the column type defined as Enum (enum('GP','NGP','PGP','PAGP')). Cassandra does not support Enum data types (it supports collections though). Is there a way to implement Enum data type in Cassandra, so that the value of a column should be restricted from a set of values? I am using Apache Cassandra version 2.0.7.
See datastax cassandra Object-mapping API,
http://docs.datastax.com/en/developer/java-driver/2.1/java-driver/reference/crudOperations.html
enum Gender { FEMALE, MALE };
// FEMALE will be persisted as 'FEMALE'
#Enumerated(EnumType.STRING)
private Gender gender;
// FEMALE will be persisted as 0, MALE as 1
#Enumerated(EnumType.ORDINAL)
private Gender gender
for cassandra 3.0
enum State {INIT, RUNNING, STOPPING, STOPPED}
cluster.getConfiguration().getCodecRegistry()
.register(new EnumNameCodec<State>(State.class));
// schema: create table name_example(id int PRIMARY KEY, state text)
session.execute("insert into name_example (id, state) values (1, ?)", State.INIT);
// state is saved as 'INIT'
http://docs.datastax.com/en/developer/java-driver/3.1/manual/custom_codecs/extras/
As far I know and after reading the documentation about cql types, you can not use directly enum in cql statements (I check this for the java clients).
So the option you have is convert the Enum to String to include the field in a cql statement. BY this way all your application use the Enum but in the backend layer use the string representation for the enum.
I was facing the same issue with an integer enum... here's what I did:
MappingConfiguration.Global.Define(
new[] {
new Map<Login>()
.TableName("logins")
.PartitionKey(el => el.UserId)
.Column(el => el.UserId, cm => cm.WithName("user_id")),
.Column(el => el.Gender, cm => cm.WithName("gender_id").WithDbType<int>()),
});
Using C# driver 2.5 and DSE 4.7.
there is more or less native support of enums in cassandra
http://docs.datastax.com/en/developer/java-driver/2.1/java-driver/reference/crudOperations.html
As far as I know you can write your own custom serializers etc for cassandra and it will be able to understand your specific enum. But the those jars should be places in cassandra folder.
You can also store it as String or ordinal int value
I have a JSF datatable, it has three columns, those are:Work_Type_Desc, Project_Phase, and Activity_Desc. Those columns comes from 2 different database tables, the relationship of those two tables are one-to-many.
The 1st Table name is Work_Type. It has 1) Work_Type_Cd, 2)Work_Type_Desc, 3)Created_By_Name, 4)Created_DT, 5)Updated_By_Name, 6) Updated_DT
The 2nd Table name is Activity_Type. It has 1)Activity_Cd,2) Work_Type_Cd,3)Project_Phase,4)Activity_Desc, 5)Created_By_Name, 6)Created_DT, 7)Updated_By_Name, 8) Updated_DT.
I use Hibernate+Spring+JSF, my question is how to show those three column records in JSF datatable, do I need to create a new model domain class to store this two tables properties? If so, how to handle the PK and FK in the new model class. Thanks!
Just let your service layer return a List<ActivityType>. The WorkType is already referenced by #ManyToOne property in ActivityType, right?
<h:dataTable value="#{bean.activityTypes}" var="activityType">
<h:column>#{activityType.workType.workTypeDesc}</h:column>
<h:column>#{activityType.projectPhase}</h:column>
<h:column>#{activityType.activityDesc}</h:column>
</h:dataTable>
It's generally unnecessary to create another mapping layer for that.