I have a JoltTransformJSON processor. I want to use the FlowFile attribute (containing the Jolt spec) as the specification.
When I attempt to do it, there is a validation error "JSON Spec provided is not valid JSON format".
I use NiFi 1.6
Is it possible to specify jolt spec as an attribute?
Jolt Specification supports NiFi Expression Language but only for substituting the values inside the spec. That is:
Say you have a NiFi attribute : jolt.operation: shift you can refer that inside your spec as:
[
{
"operation" : ${jolt.operation},
...
...
}
]
The entire spec cannot be sent as a flowfile attribute, at least as of now.
Related
SplitJson processor is throwing warning message when the json path does not exists. We wants to ignore this messages on our prod environment.
Example:
{"Payload":{"ProfileScores":{"scoreList":[{"Profile":"Value 1","MatchedPatternId":"Value 2"}]}}}
Now based on above json, if we use the SplitJson processor with $.Payload.ProfileScores.scoreList[*] as JsonPath Expression then everything works fine because the path is accessible.
BUT in some cases ProfileScores object is null like this: {"Payload":{"ProfileScores":""}}.
Now as ProfileScores object is empty or null, then SplitJson throws the warning message for that particular flowfile.
Now we want to remove those warning message as it is clogging up the logs area in prod environment.
Is there any way to check if $.Payload.ProfileScores is null/empty or exists or not before sending the flowfile to SplitJson processor?
Solutions i have tried:
https://community.cloudera.com/t5/Support-Questions/Apache-nifi-Split-json-error-when-an-array-has-only-one/td-p/236040
"Path not found behavior" is not available on SplitJson processor, so could not use that.
here is an expression you could use:
$.Payload[?(#.ProfileScores nin [null,''])].ProfileScores.scoreList[*]
?(expression) - where
# - current element
A nin [] - A not in array on the right
more explanation and examples here: https://github.com/json-path/JsonPath#path-examples
i used your file from question and this jsonpath works fine in nifi 1.16.3
I need to fetch the second attribute from a json file. and have tried to use 'evaluateJsonPath' processor and/or 'attributetoJson' processor, in both of which case i always get the whole line instead of single attribute.
in the json processor I added the property as 'bouncers_mobile_social' and value as '$.bouncers_mobile_social' yet getting the full line.
the sample json is
{"mtimespent_other": 0.4, "bouncers_mobile_social": 45, "numvisitors_mobile_search": 647}
Please suggest
I created a template demonstrating that a GenerateFlowFile processor which creates the JSON you provided and is passed to an EvaluateJSONPath processor with the property you provided works. Please check the configuration, typos, unexpected input data, etc.
I want to create Jolt Specification in Update Attribute processor and use in JoltTransformer processor as an attribute like below.
I get this error in JoltTransformer:
. "is invalid because of the specification not valid for the selected transformation.
Reason being Jolt Specification will set during runtime.
Thanks for the help!
Ani
In NiFi-1.6.0 JoltTransformJson processor not accepting attributes in jolt specification property value.
We still need to keep literal jolt specification in property value.
Example:
But Starting from NiFi-1.7.0 JoltTransformJson processor will be evaluated using flow file attributes and variable registry
we can use ${mapping} starting from NiFi-1.7.0
I am trying to insert the whole content of a row of an XML file into a JSON attribute (I am a newbie).
I am doing it this way (tell me if there is an easier way, it's good to now):
I have configured Extract text this way:
And to finish, I configure the Replace Text, giving a JSON format:
But he result appears to be wrong (doesn't work like a normal JSON file, for example if I a try to do a httpPost):
How can I fix this problem?
cheers
If you are concern regards to new lines and json key/values then use NiFi expression language functions on the extracted attribute(data).
ReplaceText Configs:
Replacement value:
{"name" : "user1","time" : "${now()}","data" : "${data:replaceAll('\s',''):escapeJson()}"}
Use escapeJson and replaceAll function to replace all spaces,newlines with ''
Replacement Strategy as Always Replace
(or)
Another way of preparing json message is by using AttributesToJson processor.
if we are using this processor then we need to prepare attributes/values before AttributesToJson processor by using UpdateAttribute processor
Flow:
1.SplitXml
2.ExtractText //add data property to extract content to flowfile attribute
3.UpdateAttribute //add name property -> user1
add time property -> ${now()}
add data property -> ${data:replaceAll('\s',''):escapeJson()}}
4.AttributeToJson //Attributes List -> name,time,data
Destination -> flowfile content
include core attributes -> false
Please guide me the right component for converting string to json
using appropriate Nifi processor component.
Input is a string of content type text/plain
{ productName : "tv", locationName: " chennai"}
Output of EvaluateJsonPath is still the same as I am unable to evaluate json property based on json path due to wrong content type sent as input.
{
productName : "tv",
locationName: " chennai"
}
Note: Tried SplitText, AttirtubesToJson processors not able to achieve desired conversion.
This is because the input data is not valid JSON. I recreated this flow locally and the error from EvaluateJsonPath is
2017-08-22 10:20:21,079 ERROR [Timer-Driven Process Thread-5] o.a.n.p.standard.EvaluateJsonPath EvaluateJsonPath[id=0aec27af-015e-1000-fac5-4e0f455a10fe] FlowFile StandardFlowFileRecord[uuid=b903eeb0-8985-4517-910f-5e3bbbccb8dc,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1503421928125-1, container=default, section=1], offset=376, length=47],offset=0,name=91708717370829,size=47] did not have valid JSON content.
Which condenses to [flowfile] did not have valid JSON content. The processor uses a strict validator and the input you're providing is not valid JSON. You'll need to use text manipulation or regexes to update this content to the following:
{"productName":"tv", "locationName":"chennai"}
Once you have accomplished this (via ReplaceText, etc.), the EvaluateJsonPath processor will work properly.
Also, to be clear, EvaluateJsonPath is designed to execute JSONPath expressions to extract JSON values to flowfile attributes. It is not designed to manipulate arbitrary text into JSON format.
Update
There is no universal process to convert arbitrary data to JSON. Given the specific input you provided, the following values for ReplaceText will convert this to valid JSON:
Search Value: (?<!\")(\w+)(?=[\s:])
Replacement Value: "$1"
Replacement Strategy: Regex Replace
Evaluation Mode: Entire text
If you get incoming data that is invalid in some other way, you'll have to modify this process. You may be interested in something like JSONLint to validate and format your incoming data.