Am new to nifi(1.14) and using JOLTTransform processor to transfer the input json (flatten) to nested json. I could not able to fix with the JOLT Schema. kindly help me on this.
The input JSON is :
[
{
"executionTime": "2244",
"processorId": "3ef03f9c-b42c-4c4e-9d5f-d14878bb2c84",
"filename": "Trading",
"processStartTime": "2021-07-26 16:31:19",
"processSQL": "TradingDetails.sql",
"executionFetchTime": "2049",
"sourceCountValue": "5076",
"startTime": "1627297279651",
"processGroupName": "Trading"
},
{
"ProcessEndTime": "2021-07-26 16:31:29",
"completedTime": "1627297289412",
"processDuration": "0",
"destCountValue": "5076",
"tableName": "TRADINGDETAILS"
},
{
"ProcessEndTime": "2021-07-26 16:31:29",
"completedTime": "1627297289412",
"processDuration": "0",
"destCountValue": "5076",
"tableName": "CUSTTRADINGDETAILS"
}
]
and the expected JSON output is
{
"Trading": {
"Source": {
"executionTime": "2244",
"processorId": "3ef03f9c-b42c-4c4e-9d5f-d14878bb2c84",
"filename": "Trading",
"processStartTime": "2021-07-26 16:31:19",
"processSQL": "TradingDetails.sql",
"executionFetchTime": "2049",
"sourceCountValue": "5076",
"startTime": "1627297279651",
"processGroupName": "Trading"
},
"Destination": {
"TRADINGDETAILS": {
"ProcessEndTime": "2021-07-26 16:31:29",
"completedTime": "1627297289412",
"processDuration": "10",
"destCountValue": "5076",
"tableName": "TRADINGDETAILS"
},
"CUSTTRADINGDETAILS": {
"ProcessEndTime": "2021-07-26 16:31:29",
"completedTime": "1627297289412",
"processDuration": "10",
"destCountValue": "5076",
"tableName": "CUSTTRADINGDETAILS"
}
}
}
}
Thanks in Advance.
You can use successive shift transformations; enumarate the keys of the objects within the first one, and then rename them while nesting each under the related keys such as
[
{
"operation": "shift",
"spec": {
"*": "&"
}
},
{
"operation": "shift",
"spec": {
"0": "Trading.Source",
"*": "Trading.Destination.#(0,tableName)"
}
}
]
Related
Hi pls help with this jolt transformation.
Note:
If Environment exists then add new field numid and value is 1
If costcenter exists then add new field numid and the value is 2
Input:
[
{
"Environment": "net",
"BillingProfileId": 863,
"ms-resource-usage": "azure-cloud-shell"
},
{
"CostCenter": "check",
"BillingAccountName": "HCL Technologies Ltd.",
"ServiceFamily": "Compute"
}
]
Expected output:
[
{
"Environment": "net",
"numid": "1"
"BillingProfileId": 863,
"ms-resource-usage": "azure-cloud-shell"
},
{
"CostCenter": "check",
"numid" : "2",
"BillingAccountName": "HCL Technologies Ltd.",
"ServiceFamily": "Compute"
}
]
Thanks
You can use this spec:
[
{
"operation": "shift",
"spec": {
"*": {
"#": "[&1]",
"Environment": {
"#1": "[&2].numid"
},
"CostCenter": {
"#2": "[&2].numid"
}
}
}
}
]
Can we use string manipulation functions like below in JoltTransformJSON
${BIG:padRight(5, '#')}
Link:
https://jolt-demo.appspot.com/#modify-stringFunctions
Expected Output: BIG##
Like we use
"small_toUpper": "=toUpper(#(1,small))",
"BIG_toLower": "=toLower(#(1,BIG))",
I am trying but it's not giving any error, but not giving the desired result as well. What would be other alternative for this.
Input JSON:
{
"x": [
3,
2,
1,
"go"
],
"small": "small",
"BIG": "BIG",
"people": [
{
"firstName": "Bob",
"lastName": "Smith",
"address": {
"state": null
}
},
{
"firstName": "Sterling",
"lastName": "Archer"
}
]
}
Spec:
[
{
"operation": "modify-default-beta",
"spec": {
"y": "=join(',',#(1,x))",
"z": "=join(' ',#(1,x))",
"small_toUpper": "=toUpper(#(1,small))",
"BIG_toLower": "=toLower(#(1,BIG))",
"BIG_padding": "=padRight(#(5,BIG))"
}
}
]
You can use rightPad function along with modify-xXx-beta (xXx:default or overwrite) transformation such as
[
{
"operation": "modify-default-beta",
"spec": {
"BIG_padding": "=rightPad(#(1,BIG),5,'#')"
}
}
]
I want to copy all they keys in the json except one which i want to transform.
ex.
Input JSON
{
"ts": "20200420121222",
"name": "broker",
"city": "queensland",
"age": 21,
"gender": "male"
"characteristics": {
"Card Id": "63247354",
"Termination Plan": "paid"
}
}
Output JSON
{
"ts": "20200420121222",
"name": "broker",
"city": "queensland",
"age": 21,
"gender": "male"
"characteristics": {
"card_id": "63247354", // change here
"termination_plan": "paid" // change here
}
}
Is there any better way via which i can just change the following above keys and copy the rest
You can use the "*": "&" construct to include all other fields that have not yet been matched:
[
{
"operation": "shift",
"spec": {
"characteristics": {
"Card Id": "characteristics.card_id",
"Termination Plan": "characteristics.termination_plan"
},
"*": "&"
}
}
]
I have Json array with each of its object having dynamic attributes. I have to merge all the object into one Json object in nifi with common attribute's value concatenated and other attributes as it is.
I have already tried following spec, and it is giving common attributes as array list. But I want that common attribute's value to be concatenated and I don't know how to do that. Any suggestion would be really helpful. Thanks in advance.
[
{
"operation": "shift",
"spec": {
"*": {
"*": "&"
}
}
}
]
The input is like( NOTE: The attributes are dynamic so not sure always going to have these attributes)
[
{
"firstName": "Sam1",
"lastName": "Achom1",
"place": "Silchar1"
},
{
"firstName": "Saya2",
"lastName": "Singh2",
"place": "Macherial2",
"Second1stAttribute": "SomeValue"
},
{
"firstName": "Sam3",
"lastName": "Achom3",
"place": "Silchar3",
"Third2ndAttribute": "SomeValue2"
}
]
The output I am getting with before mentioned spec
{
"firstName" : [ "Sam1", "Saya2", "Sam3" ],
"lastName" : [ "Achom1", "Singh2", "Achom3" ],
"place" : [ "Silchar1", "Macherial2", "Silchar3" ],
"Second1stAttribute" : "SomeValue",
"Third2ndAttribute" : "SomeValue2"
}
And Expecting the output as
{ "firstName" : "Sam1 Saya2 Sam3",
"lastName" : "Achom1 Singh2 Achom3",
"place" : "Silchar1 Mancherial2 Silchar3",
"Second1stAttribute" : "SomeValue",
"Third2ndAttribute" : "SomeValue2"
}
You can "join" the items from your arrays. This spec will do the trick:
[
{
"operation": "shift",
"spec": {
"*": {
"*": "&"
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=join(' ',#(1,&))"
}
}
]
I'm new with Jolt and I need to have two field having different name but same value coming from the input JSON.
INPUT:
{
"date": 15746555589,
"sensorid": "23r098hd20c8jd02hd0h02300000000000"
}
DESIDERED OUTPUT:
{
"lastseen": 15746555589,
"firstseen": 15746555589,
"sensorid": "23r098hd20c8jd02hd0h02300000000000"
}
This spec should work:
[
{
"operation": "shift",
"spec": {
"date": "lastseen",
"#(1,date)": "firstseen",
"*": "&"
}
}
]