I need to fetch the second attribute from a json file. and have tried to use 'evaluateJsonPath' processor and/or 'attributetoJson' processor, in both of which case i always get the whole line instead of single attribute.
in the json processor I added the property as 'bouncers_mobile_social' and value as '$.bouncers_mobile_social' yet getting the full line.
the sample json is
{"mtimespent_other": 0.4, "bouncers_mobile_social": 45, "numvisitors_mobile_search": 647}
Please suggest
I created a template demonstrating that a GenerateFlowFile processor which creates the JSON you provided and is passed to an EvaluateJSONPath processor with the property you provided works. Please check the configuration, typos, unexpected input data, etc.
Related
while using RouteOnAttribute nifi processor , i have input of json data
[{"dev":"xyz","detail":"abc"}] which i got from convertRecord processor
Routing Strategy :Route to Property name
ifmatch: ${dev:equals( "xyz" )}
I tried ${dev:matches( "xyz")} in both single quotes and double quotes still am not getting flowfile redirecting towards "ifmatch" . its redirecting to unmatched
is there anyway to resolve this i tried many other option
The flowfile content is different from attributes. Content is arbitrary -- could be empty, text, KB of XML, GB of video or binary. Each flowfile also has attributes which are key/value pairs of Strings kept in memory.
If you want to route on this piece of data, you have multiple options:
Use RouteOnText or RouteOnContent to use the actual flowfile content directly.
Extract it to an attribute using EvaluateJsonPath and then route on that attribute.
The Apache NiFi User Guide and In-Depth provide more information around this distinction.
I have JSON objects coming into Nifi via MQTT from two different inputs - for instance, let's say one is from a top sensor, and one is from a bottom sensor. Each of the sensors has its own MQTT topic, so I am using two different ConsumeMQTT Processors to ingest this data into my Nifi Flow.
JSON Object for top sensor is {"Top_Data": "value"}
JSON Object for bottom sensor is {"Bottom_Data": "value"}
I am currently using two separate EvaluateJsonPath Processors to store either the value of Top_Data or Bottom_Data in an attribute called sensorData.
How can I use some kind of if/or statement to only use one processor to EvaluateJsonPath for both of the JSON objects I could get from MQTT? Basically, I want to have an expression that says "If my JSON object has a property called Top_Data, use its value for the attribute sensorData, otherwise, use the value from the property Bottom_Data."
Example of my EvaluateJsonPath Processor
maybe try JSONPath expression
$[Top_Data,Bottom_Data]
in the single EvaluateJSONPathProcessor.
According to https://goessner.net/articles/JsonPath/ there is a possibility to use alternate operator [,]:
[,] Union operator in XPath results in a combination of node sets. JSONPath allows alternate names or array indices as a set.
I have tested the expression using http://jsonpath.com/ and it should work.
Let us know if that helps.
You could try extracting them both using EvaluateJsonPath(property 1: top: $['top'], property 2: bottom: $['bottom']) and of course don't forget to set Destination to flowfile-attribute.
Then, transfer to UpdateAttribute and set property finalData as ${top:isEmpty():ifElse(${bottom}, ${top})}.
If EvaluateJsonPath won't find a full element, then it will set it as empty string, so all you need to do is check if either of them is empty and if it is, set the final data as the other one.
I am trying to insert the whole content of a row of an XML file into a JSON attribute (I am a newbie).
I am doing it this way (tell me if there is an easier way, it's good to now):
I have configured Extract text this way:
And to finish, I configure the Replace Text, giving a JSON format:
But he result appears to be wrong (doesn't work like a normal JSON file, for example if I a try to do a httpPost):
How can I fix this problem?
cheers
If you are concern regards to new lines and json key/values then use NiFi expression language functions on the extracted attribute(data).
ReplaceText Configs:
Replacement value:
{"name" : "user1","time" : "${now()}","data" : "${data:replaceAll('\s',''):escapeJson()}"}
Use escapeJson and replaceAll function to replace all spaces,newlines with ''
Replacement Strategy as Always Replace
(or)
Another way of preparing json message is by using AttributesToJson processor.
if we are using this processor then we need to prepare attributes/values before AttributesToJson processor by using UpdateAttribute processor
Flow:
1.SplitXml
2.ExtractText //add data property to extract content to flowfile attribute
3.UpdateAttribute //add name property -> user1
add time property -> ${now()}
add data property -> ${data:replaceAll('\s',''):escapeJson()}}
4.AttributeToJson //Attributes List -> name,time,data
Destination -> flowfile content
include core attributes -> false
I am using PutElasticsearch5 processor to index documents into ES.My workflow has couple of other processors before PutElasticsearch5 which converts avro to json.
I am getting the below given error when I run the workflow.
java.lang.IllegalArgumentException: Validation Failed: 1: content type is missing;2: content type is missing;
I coudlnt find any other relvant information to troubleshoot this. There is no setting for "Content Type" under Putelasticsearch5 configuration
I'm also having this issue, like user2297083 said if you are sending a batched JSON file into the PutElasticsearch5 then it will throw this exception and move the file into the FAILED relationship. The processor seems like it only handles one JSON object written into a file at a time that cannot be surrounded by array brackets. So if you have a file with content such as:
[{"key":"value"}]
then the processor will fail however if you send the same document as:
{"key":"value"}
then the processor will index successfully, considering your other configurations are correct.
One solution can be such that if you don't want to send everything through a splitter before coming to the PutElasticsearch5 processor, then use a splitter processor that works off of the FAILURE relationship to the PutElasticsearch5 and sends data back into the same PutElasticsearch5. More FlowFiles means more IO in your node, so I'm actively looking for a way to have the PutElasticsearch5 processor handle a batched JSON document. I feel like there's got to be a way without writing a custom iteration of it or creating a ton of new FlowFiles.
EDIT:
Actually, it does answer the question. His question is:
I am using PutElasticsearch5 processor to index documents into ES.My workflow has couple of other processors before PutElasticsearch5 which converts avro to json.
I am getting the below given error when I run the workflow.
java.lang.IllegalArgumentException: Validation Failed: 1: content type is missing;2: content type is missing;
which is exactly the exception message that is given by the PutElasticsearch5 processor when passing a JSON file that is not formatted correctly. His question is why this is happening.
My answer states why it's happening (one possible use case) and how to work around it by giving a solution that does work.
In this case, correctly formatted JSON means a FlowFile that has a single JSON object as it's content as I have shown above.
Further looking into this though, it makes sense that the processor only takes a single JSON document FlowFile at a time because you can use FlowFile attributes to specify the "id" of the indexed document. If the uuid of the FlowFile is used and it was a batched JSON i.e.
[{"one":1},{"two":2},{"three":3}]
then each JSON object would be indexed in elasticsearch using the same "index","type", and "id" (id being the FlowFile uuid), and this would not be desired.
I need to perform an HTTP Post from NiFi, but I don't want/need the request to carry all of the FlowFile's content.
Is there some way to pass attributes of a FlowFile but not the full content?
If the request body of your Http Post is JSON, you can use the AttributesToJSON Processor which allows you to pick which attributes you want to include in the resulting JSON. You can then configure the processor so the resulting JSON overwrites the existing flowfile content.
Keep in mind that the resulting JSON will be flat so you may need to transform it to the expected format. For that, you can use the JoltTransformJSON Processor.
Below is an example of what your dataflow might look like. I hope this helps!