SplitJson processor is throwing warning message when the json path does not exists. We wants to ignore this messages on our prod environment.
Example:
{"Payload":{"ProfileScores":{"scoreList":[{"Profile":"Value 1","MatchedPatternId":"Value 2"}]}}}
Now based on above json, if we use the SplitJson processor with $.Payload.ProfileScores.scoreList[*] as JsonPath Expression then everything works fine because the path is accessible.
BUT in some cases ProfileScores object is null like this: {"Payload":{"ProfileScores":""}}.
Now as ProfileScores object is empty or null, then SplitJson throws the warning message for that particular flowfile.
Now we want to remove those warning message as it is clogging up the logs area in prod environment.
Is there any way to check if $.Payload.ProfileScores is null/empty or exists or not before sending the flowfile to SplitJson processor?
Solutions i have tried:
https://community.cloudera.com/t5/Support-Questions/Apache-nifi-Split-json-error-when-an-array-has-only-one/td-p/236040
"Path not found behavior" is not available on SplitJson processor, so could not use that.
here is an expression you could use:
$.Payload[?(#.ProfileScores nin [null,''])].ProfileScores.scoreList[*]
?(expression) - where
# - current element
A nin [] - A not in array on the right
more explanation and examples here: https://github.com/json-path/JsonPath#path-examples
i used your file from question and this jsonpath works fine in nifi 1.16.3
Related
while using RouteOnAttribute nifi processor , i have input of json data
[{"dev":"xyz","detail":"abc"}] which i got from convertRecord processor
Routing Strategy :Route to Property name
ifmatch: ${dev:equals( "xyz" )}
I tried ${dev:matches( "xyz")} in both single quotes and double quotes still am not getting flowfile redirecting towards "ifmatch" . its redirecting to unmatched
is there anyway to resolve this i tried many other option
The flowfile content is different from attributes. Content is arbitrary -- could be empty, text, KB of XML, GB of video or binary. Each flowfile also has attributes which are key/value pairs of Strings kept in memory.
If you want to route on this piece of data, you have multiple options:
Use RouteOnText or RouteOnContent to use the actual flowfile content directly.
Extract it to an attribute using EvaluateJsonPath and then route on that attribute.
The Apache NiFi User Guide and In-Depth provide more information around this distinction.
I have JSON objects coming into Nifi via MQTT from two different inputs - for instance, let's say one is from a top sensor, and one is from a bottom sensor. Each of the sensors has its own MQTT topic, so I am using two different ConsumeMQTT Processors to ingest this data into my Nifi Flow.
JSON Object for top sensor is {"Top_Data": "value"}
JSON Object for bottom sensor is {"Bottom_Data": "value"}
I am currently using two separate EvaluateJsonPath Processors to store either the value of Top_Data or Bottom_Data in an attribute called sensorData.
How can I use some kind of if/or statement to only use one processor to EvaluateJsonPath for both of the JSON objects I could get from MQTT? Basically, I want to have an expression that says "If my JSON object has a property called Top_Data, use its value for the attribute sensorData, otherwise, use the value from the property Bottom_Data."
Example of my EvaluateJsonPath Processor
maybe try JSONPath expression
$[Top_Data,Bottom_Data]
in the single EvaluateJSONPathProcessor.
According to https://goessner.net/articles/JsonPath/ there is a possibility to use alternate operator [,]:
[,] Union operator in XPath results in a combination of node sets. JSONPath allows alternate names or array indices as a set.
I have tested the expression using http://jsonpath.com/ and it should work.
Let us know if that helps.
You could try extracting them both using EvaluateJsonPath(property 1: top: $['top'], property 2: bottom: $['bottom']) and of course don't forget to set Destination to flowfile-attribute.
Then, transfer to UpdateAttribute and set property finalData as ${top:isEmpty():ifElse(${bottom}, ${top})}.
If EvaluateJsonPath won't find a full element, then it will set it as empty string, so all you need to do is check if either of them is empty and if it is, set the final data as the other one.
I need to fetch the second attribute from a json file. and have tried to use 'evaluateJsonPath' processor and/or 'attributetoJson' processor, in both of which case i always get the whole line instead of single attribute.
in the json processor I added the property as 'bouncers_mobile_social' and value as '$.bouncers_mobile_social' yet getting the full line.
the sample json is
{"mtimespent_other": 0.4, "bouncers_mobile_social": 45, "numvisitors_mobile_search": 647}
Please suggest
I created a template demonstrating that a GenerateFlowFile processor which creates the JSON you provided and is passed to an EvaluateJSONPath processor with the property you provided works. Please check the configuration, typos, unexpected input data, etc.
I am testing a flow in NIFI that checks for specific value of a counter using REST API. I am able to extract it correct value from REST response. However, when I check the condition in Route on Attribute processor, the expected matched condition's results are routing to unmatched processor.
Attached is the :
Flow and configuration
I have already checked my response to be "1". But its routing to unmatched branch.
Is there something wrong with NIFI expression language I am using?
Jasim,
Initial setup check counter attribute in which value is 1.
And modify the expression language like ${counter:equals('1')} or ${counter:matches('1')} instead of contains.
because contains not suitable for your scanerio.
Hope this helpful for you.
I am using PutElasticsearch5 processor to index documents into ES.My workflow has couple of other processors before PutElasticsearch5 which converts avro to json.
I am getting the below given error when I run the workflow.
java.lang.IllegalArgumentException: Validation Failed: 1: content type is missing;2: content type is missing;
I coudlnt find any other relvant information to troubleshoot this. There is no setting for "Content Type" under Putelasticsearch5 configuration
I'm also having this issue, like user2297083 said if you are sending a batched JSON file into the PutElasticsearch5 then it will throw this exception and move the file into the FAILED relationship. The processor seems like it only handles one JSON object written into a file at a time that cannot be surrounded by array brackets. So if you have a file with content such as:
[{"key":"value"}]
then the processor will fail however if you send the same document as:
{"key":"value"}
then the processor will index successfully, considering your other configurations are correct.
One solution can be such that if you don't want to send everything through a splitter before coming to the PutElasticsearch5 processor, then use a splitter processor that works off of the FAILURE relationship to the PutElasticsearch5 and sends data back into the same PutElasticsearch5. More FlowFiles means more IO in your node, so I'm actively looking for a way to have the PutElasticsearch5 processor handle a batched JSON document. I feel like there's got to be a way without writing a custom iteration of it or creating a ton of new FlowFiles.
EDIT:
Actually, it does answer the question. His question is:
I am using PutElasticsearch5 processor to index documents into ES.My workflow has couple of other processors before PutElasticsearch5 which converts avro to json.
I am getting the below given error when I run the workflow.
java.lang.IllegalArgumentException: Validation Failed: 1: content type is missing;2: content type is missing;
which is exactly the exception message that is given by the PutElasticsearch5 processor when passing a JSON file that is not formatted correctly. His question is why this is happening.
My answer states why it's happening (one possible use case) and how to work around it by giving a solution that does work.
In this case, correctly formatted JSON means a FlowFile that has a single JSON object as it's content as I have shown above.
Further looking into this though, it makes sense that the processor only takes a single JSON document FlowFile at a time because you can use FlowFile attributes to specify the "id" of the indexed document. If the uuid of the FlowFile is used and it was a batched JSON i.e.
[{"one":1},{"two":2},{"three":3}]
then each JSON object would be indexed in elasticsearch using the same "index","type", and "id" (id being the FlowFile uuid), and this would not be desired.