Unable to transform below json using jolt transformation - etl

I have a JSON which have studentId's who play cricket and I want a student list for each object of the group array. But the output is getting merged into the same student list. I tried iterating each and every studentId and getting the below output. And I wish to get the output like in the below expected/desired format. Any help?
Input:
{
"studentEligibility": {
"sportsEligibility": {
"cricketEligibility": [
{
"group": [
{
"multiPlayGame": [
{
"studentId": "2aefcb01-fe81-4760-b531-9767c2e6d322_209537"
},
{
"studentId": "2aefcb01-fe81-4760-b531-9767c2e6d322_217649"
}
]
}
]
},
{
"group": [
{
"multiPlayGame": [
{
"studentId": "2aefcb01-fe81-4760-b531-9767c2e6d322_217609"
}
]
},
{
"multiPlayGame": [
{
"studentId": "2aefcb01-fe81-4760-b531-9767c2e6d322_216386"
}
]
}
]
},
{
"group": [
{
"multiPlayGame": [
{
"studentId": "2aefcb01-fe81-4760-b531-9767c2e6d322_217008"
},
{
"studentId": "2aefcb01-fe81-4760-b531-9767c2e6d322_217628"
}
]
}
]
}
]
}
}
}
Jolt Spec:
[
{
"operation": "shift",
"spec": {
"studentEligibility": {
"sportsEligibility": {
"cricketEligibility": {
"*": {
"group": {
"*": {
"multiPlayGame": {
"*": {
"studentId": "team[&5].players[]"
}
}
}
}
}
}
}
}
}
}
]
Current Ouptut:
{
"team": [
{
"players": [
"2aefcb01-fe81-4760-b531-9767c2e6d322_209537",
"2aefcb01-fe81-4760-b531-9767c2e6d322_217649"
]
},
{
"players": [
"2aefcb01-fe81-4760-b531-9767c2e6d322_217609",
"2aefcb01-fe81-4760-b531-9767c2e6d322_216386"
]
},
{
"players": [
"2aefcb01-fe81-4760-b531-9767c2e6d322_217008",
"2aefcb01-fe81-4760-b531-9767c2e6d322_217628"
]
}
]
}
Expected / Desired output:
{
"team": [
{
"players": [
"2aefcb01-fe81-4760-b531-9767c2e6d322_209537",
"2aefcb01-fe81-4760-b531-9767c2e6d322_217649"
]
},
{
"players": [
"2aefcb01-fe81-4760-b531-9767c2e6d322_217609"
]
},
{
"players": [
"2aefcb01-fe81-4760-b531-9767c2e6d322_216386"
]
},
{
"players": [
"2aefcb01-fe81-4760-b531-9767c2e6d322_217008",
"2aefcb01-fe81-4760-b531-9767c2e6d322_217628"
]
}
]
}

Jolt Spec Solution
[
{
"operation": "shift",
"spec": {
"studentEligibility": {
"sportsEligibility": {
"cricketEligibility": {
"*": {
"group": {
"*": "teams[].players[]"
}
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"teams": {
"*": {
"players": {
"*": {
"multiPlayGame": "team.players[]"
}
}
}
}
}
},
{
"operation": "shift",
"spec": {
"team": {
"players": {
"*": {
"*": {
"studentId": "team[&2].players[]"
}
}
}
}
}
}
]

Related

Need Nested sorting elastic search query

Sample Result : need to sort this (name will always be constant, only value will change)
{
"sampleJson": { "loop1": {
"userId": "1",
"loop2": {
"loop3": [
{
"name": "hindi",
"value": "abc"
},
{
"name": "telugu",
"value": "xyz"
}
]
}
},
"loop1": {
"userId": "2",
"loop2": {
"loop3": [
{
"name": "hindi",
"value": "def"
},
{
"name": "telugu",
"value": "ghi"
}
]
}
},
"loop1": {
"userId": "1",
"loop2": {
"loop3": [
{
"name": "hindi",
"value": "jkl"
},
{
"name": "telugu",
"value": "mno"
}
]
}
}
}
}
After performing elastic Sorting, result should look like below:
{
"sampleJson": {
"loop1": {
"userId": "1",
"loop2": {
"loop3": [
{
"name": "hindi",
"value": "jkl"
},
{
"name": "telugu",
"value": "mno"
}
]
}
},
"loop1": {
"userId": "1",
"loop2": {
"loop3": [
{
"name": "hindi",
"value": "abc"
},
{
"name": "telugu",
"value": "xyz"
}
]
}
}
}
}
below Elastic sorting is not working: not sure what is wrong with this. any help that would be great.
{
"from": 0,
"size": 25,
"query": {
"bool": {
"must": [
{
"match": {
"userId": "1"
}
}
]
}
},
"sort": [
{
"sampleJson.loop1.loop2.loop3.value": {
"order":"asc",
"nested": {
"path": "sampleJson",
"filter": {
"terms" : {
"sampleJson.loop1.loop2.loop3.name": [
"telugu"
] }
}
}
}
}
]
}

Jolt transform convert large array to smaller array

My input JSON:
[
{
"label": [
{
"name": "abc"
}
]
},
{
"label": [
{
"name": "xyz"
}
]
}
]
My spec:
[{
"operation": "shift",
"spec": {
"*": {
"label": {
"0": {
"name": "name"
}
}
}
}
}]
Expected output:
[{
"name": "abc"
}, {
"name": "xyz"
}]
Generated output:
{
"name" : [ "abc", "xyz" ]
}
How do I not combine the array in the spec?
Try this spec,
[
{
"operation": "shift",
"spec": {
"*": {
"label": {
"*": {
"name": "[&3].name"
}
}
}
}
}
]

Jolt Transform to Array with Parent Attributes

I am using NiFi Jolt Processor to transform some JSON data.
My JSON field genre contains shared attributes and an array that contains the list of actual genre names.
I needed to transform the "genre" attribute to become an array of "genres" containing the list of the common attributes and the different genre names.
I have the following input JSON:
{
"programIdentifier": "9184663",
"programInstance": {
"genre": {
"source": "GN",
"locked": false,
"lastModifiedDate": 1527505462094,
"lastModifiedBy": "Some Service",
"genres": [
"Miniseries",
"Drama"
]
}
}
}
I have tried the following spec:
[{
"operation": "shift",
"spec": {
"programIdentifier": ".&",,
"genre": {
"source": "genres[].source.value",
"locked": "genres[].locked",
"lastModifiedDate": "genres[].lastModifiedDate",
"lastModifiedBy": "genres[].lastModifiedBy",
"genres": {
"*": "genres[&0].name"
}
}
}]
This my expected output:
{
"programIdentifier": "9184663",
"programInstance": {
"genres": [
{
"source": {
value: "GN"
}
"locked": false,
"lastModifiedDate": 1527505462094,
"lastModifiedBy": "Some Service",
"name": "Miniseries"
},
{
"source": {
value: "GN"
}
"locked": false,
"lastModifiedDate": 1527505462094,
"lastModifiedBy": "Some Service",
"name": "Drama"
}
]
}
}
But it's coming out as:
{
"programIdentifier": "9184663",
"programInstance": {
"genres": [
{
"source": {
"value": "GN"
},
"name": "Miniseries"
}, {
"locked": false,
"name": "Drama"
}, {
"lastModifiedDate": 1527505462094
}, {
"lastModifiedBy": "Some Service"
}],
}
}
Is it what you want to achieve?
[
{
"operation": "shift",
"spec": {
"programIdentifier": ".&",
"programInstance": {
"genre": {
"genres": {
"*": {
"#2": {
"source": "programInstance.genres[&2].source[]",
"locked": "programInstance.genres[&2].locked",
"lastModifiedDate": "programInstance.genres[&2].lastModifiedDate",
"lastModifiedBy": "programInstance.genres[&2].lastModifiedBy"
},
"#": "programInstance.genres[&1].name"
}
}
}
}
}
}
]

I need to transform below input to some out put using JOLT Transformation

I need to transform below input to exact output using JOLT transformation.
Input:
{
"First_Name": "Test",
"Last_Name": "User",
"allpointofInterest": "someText",
"campaign_id": "123456789",
"lead_id": "123456789"
}
Output:
"input" : {
"tokens" : [ {
"name" : "{{my.First_Name}}",
"value" : "Test"
}, {
"name" : "{{my.Last_Name}}",
"value" : "User"
}, {
"name" : "{{my.allpointofInterest}}",
"value" : "someText"
} ]
},
"leads": [{
"id": "123456789"
}]
}
I tried with below spec but not driven to exact output what i needed.
spec:
[
{
"operation": "remove",
"spec": {
"campaign_id": ""
}
}, {
"operation": "shift",
"spec": {
"lead_id": {
"#": "input.leads[#1].id"
},
"*": "temptoken.&"
}
}, {
"operation": "shift",
"spec": {
"*": "&",
"temptoken": {
"*": {
"$": "input.tokens[#2].tmpname",
"#": "input.tokens[#2].value"
}
}
}
}, {
"operation": "modify-overwrite-beta",
"spec": {
"input": {
"leads": {
"*": {
"id": ["=toInteger", 0]
}
}
}
}
}, {
"operation": "modify-default-beta",
"spec": {
"input": {
"tokens": {
"*": {
"name": "=concat('{{my.',#(1,tmpname), '}}')"
}
}
}
}
}, {
"operation": "remove",
"spec": {
"input": {
"tokens": {
"*": {
"tmpname": ""
}
}
}
}
}
]
Any help would be much appreciated. Thanks
Does sth like that suits you?
[
{
"operation": "shift",
"spec": {
"#{{my.First_Name}}": "input.tokens[0].name",
"First_Name": "input.tokens[0].value",
"#{{my.Last_Name}}": "input.tokens[1].name",
"Last_Name": "input.tokens[1].value",
"#{{my.allpointofInterest}}": "input.tokens[2].name",
"allpointofInterest": "input.tokens[2].value",
"lead_id": "leads[#].id"
}
}
]

How to add indices in an array in jolt transformation

I am trying to modify my input and add an id for each object with '-indexNum' but this seems not working.Here I m posting my code.
My input,
{
"employees": [{
"f_name": "tom",
"l_name": "smith"
},
{
"f_name": "don",
"l_name": "jones"
}
]
}
Expected o/p:
{
"employees": [{
"firstName": "tom",
"lastName": "smith",
"test": "emp-"
}, {
"firstName": "don",
"lastName": "jones",
"test": "emp-"
}]
}
My spec,
[{
"operation": "shift",
"spec": {
"employees": {
"*": {
"f_name": "employees[&1].firstName",
"l_name": "employees[&1].lastName"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"employees": {
"*": {
"test": "=concat('emp-',)"
}
}
}
}]
I am trying to modify my input and add an id for each object with '-indexNum'.Can anyone please suggest me help.Thanks.
Spec
[
{
"operation": "shift",
"spec": {
"employees": {
"*": {
"$": "employees[&1].employeeNum",
"f_name": "employees[&1].firstName",
"l_name": "employees[&1].lastName"
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"employees": {
"*": {
"employeeNum": "=concat('emp-',#(1,employeeNum))"
}
}
}
}
]

Resources