I want to create Jolt Specification in Update Attribute processor and use in JoltTransformer processor as an attribute like below.
I get this error in JoltTransformer:
. "is invalid because of the specification not valid for the selected transformation.
Reason being Jolt Specification will set during runtime.
Thanks for the help!
Ani
In NiFi-1.6.0 JoltTransformJson processor not accepting attributes in jolt specification property value.
We still need to keep literal jolt specification in property value.
Example:
But Starting from NiFi-1.7.0 JoltTransformJson processor will be evaluated using flow file attributes and variable registry
we can use ${mapping} starting from NiFi-1.7.0
Related
Update Attribute configuration
It is hard to tell from the screenshots provided, but it looks like the fields you want to remove are part of the content of the flow file, which is different then the attributes of the flow file. UpdateAttribute can only remove attributes, not anything in the content.
In order to modify the content you would need to use a processor specific to the type of content being processed. In your case it looks like JSON, so you could use a ConvertRecord processor with a JsonTreeReader and JsonRecordSetWriter, and configure the writer to have a different schema then the reader. Basically read in all the fields, but only write out the fields you want.
There is an UpdateRecord processor too, but it doesn't currently have the ability to remove fields.
To delete a FlowFile's attribute, you can use UpdateAttribute and a property named Delete Attributes Expression. You just need to fill it with a regular expression that matches the attributes you want to remove.
But as #Bryan Bende said, it doesn't look like you're trying to remove FlowFile's attributes, but content..
If you are willing to remove JSON attributes from your content, you can use JoltTransformJSON and Jolt Transformation DSL of Remove. Then just use specification of the attributes you are willing to remove. For example, I want to delete from this JSON the attribute t1:
{
"t1": "test",
"t2": "test2",
"t3": "test3"
}
So, my specification would be:
{
"t1": ""
}
You can read more about it here.
I have JSON objects coming into Nifi via MQTT from two different inputs - for instance, let's say one is from a top sensor, and one is from a bottom sensor. Each of the sensors has its own MQTT topic, so I am using two different ConsumeMQTT Processors to ingest this data into my Nifi Flow.
JSON Object for top sensor is {"Top_Data": "value"}
JSON Object for bottom sensor is {"Bottom_Data": "value"}
I am currently using two separate EvaluateJsonPath Processors to store either the value of Top_Data or Bottom_Data in an attribute called sensorData.
How can I use some kind of if/or statement to only use one processor to EvaluateJsonPath for both of the JSON objects I could get from MQTT? Basically, I want to have an expression that says "If my JSON object has a property called Top_Data, use its value for the attribute sensorData, otherwise, use the value from the property Bottom_Data."
Example of my EvaluateJsonPath Processor
maybe try JSONPath expression
$[Top_Data,Bottom_Data]
in the single EvaluateJSONPathProcessor.
According to https://goessner.net/articles/JsonPath/ there is a possibility to use alternate operator [,]:
[,] Union operator in XPath results in a combination of node sets. JSONPath allows alternate names or array indices as a set.
I have tested the expression using http://jsonpath.com/ and it should work.
Let us know if that helps.
You could try extracting them both using EvaluateJsonPath(property 1: top: $['top'], property 2: bottom: $['bottom']) and of course don't forget to set Destination to flowfile-attribute.
Then, transfer to UpdateAttribute and set property finalData as ${top:isEmpty():ifElse(${bottom}, ${top})}.
If EvaluateJsonPath won't find a full element, then it will set it as empty string, so all you need to do is check if either of them is empty and if it is, set the final data as the other one.
I need to fetch the second attribute from a json file. and have tried to use 'evaluateJsonPath' processor and/or 'attributetoJson' processor, in both of which case i always get the whole line instead of single attribute.
in the json processor I added the property as 'bouncers_mobile_social' and value as '$.bouncers_mobile_social' yet getting the full line.
the sample json is
{"mtimespent_other": 0.4, "bouncers_mobile_social": 45, "numvisitors_mobile_search": 647}
Please suggest
I created a template demonstrating that a GenerateFlowFile processor which creates the JSON you provided and is passed to an EvaluateJSONPath processor with the property you provided works. Please check the configuration, typos, unexpected input data, etc.
I have a JoltTransformJSON processor. I want to use the FlowFile attribute (containing the Jolt spec) as the specification.
When I attempt to do it, there is a validation error "JSON Spec provided is not valid JSON format".
I use NiFi 1.6
Is it possible to specify jolt spec as an attribute?
Jolt Specification supports NiFi Expression Language but only for substituting the values inside the spec. That is:
Say you have a NiFi attribute : jolt.operation: shift you can refer that inside your spec as:
[
{
"operation" : ${jolt.operation},
...
...
}
]
The entire spec cannot be sent as a flowfile attribute, at least as of now.
I have a common process group that will infer avro schema based on the file i supplied. But I want to set the Avro Record Name to a name corresponding to the filename i am supplying. So I used ${filename}. But the InferAvroSchema got error saying the record name is empty. Note that before this, I already set the property "filename" to the flowfile attribute and it has a value since i tested it using ReplaceText to see if there's value for ${filename}
Unfortunately this looks like a bug in InferAvroSchema. Many of the properties support expression language, but then the processor doesn't evaluate them against the incoming flow file. So it ends up only being able to use a value typed directly into the property (non-EL), or a value from system or environment properties which doesn't really make sense for a lot of these properties.
I created this JIRA for the issue:
https://issues.apache.org/jira/browse/NIFI-2465
The fix is that all of the calls to evaluateAttributeExpressions() should be passing in a flow file like:
context.getProperty(CSV_HEADER_DEFINITION).evaluateAttributeExpressions(inputFlowFile).getValue()