NiFi Version 1.8.0
I'm trying to build our my json, and one of my fields needs to be an array. I thought I could simply use the UpdateAttribute Processor to set my attribute to '["arrayItem1", "arrayItem2"]' and then I could used AttributeToJSON to convert the attribute to JSON and it would convert to an array. Unfortunately, it simply turns into a string.
In the simplest way, how can I set an attribute to be an array so my final JSON (when using AttributeToJSON) field has the specific array?
EDIT 1
I will have a few SyslogListeners, I want to set an attribute so I know what data came from where. I want to be able to tag this data, so I though of adding an UpdateAttribute to set my attribute. I would like this to be an array. So the tag for:
SyslogListener1 will be ["tag1", "tag2"]
SyslogListener2 will be ["tag3", "tag4"]
SyslogListener3 will be ["tag1", "tag3"]
I thought of just having my flow look like this: SyslogListener -> UpdateAttribute -> Then all the data is now in the main flow -> AttributeToJSON. However, when I look at my JSON, my field is a string, not an array. How can I make this field to be an array? What I used to do, was use ReplaceText , the only problem with this is I didn't want to create a ReplaceText for ever single instance. Is there a single processor that could handle this?
Does your incoming flow file have any existing content? If not, you can use ReplaceContent to set the content to ["arrayItem1", "arrayItem2"] or whatever you wish the JSON to look like.
If the incoming flow file has existing JSON content, you can add the field explicitly (without attributes) using JoltTransformJSON or UpdateRecord.
Not my ideal solution, but I simply added a ReplaceText for each instance I would need. In my case, it was 7 different tag formations. So my nifi looks a little ugly. I was hoping for a single processor solution where I could tell it my JSON field and make it an array. So my pipeline is:
SyslogListener -> UpdateAttribute (creates our tags attribute with the string tag1, tag2 and the other tag combinations because I have 7 total SyslogListeners with their own UpdateAttribute) -> Data is now in the main pipeline, and some Other processing stuff happens here -> AttributeToJSON (setting our json with some attributes including our tags attribute) -> My 7 ReplaceTexts (which checks to see if our tags field has "tag1, tag2" and then replaces it with ["tag1", "tag2"], I do this for all 7 cases) -> PutElasticSearchHttp
So ingesting rsyslog messages, doing a bit of enriching, making my data into a JSON, then saving it to ES.
If anyone knows a single processor solution to this, so I don't need to have 7 unique ReplaceTexts (and more if I need new tags).
Related
while using RouteOnAttribute nifi processor , i have input of json data
[{"dev":"xyz","detail":"abc"}] which i got from convertRecord processor
Routing Strategy :Route to Property name
ifmatch: ${dev:equals( "xyz" )}
I tried ${dev:matches( "xyz")} in both single quotes and double quotes still am not getting flowfile redirecting towards "ifmatch" . its redirecting to unmatched
is there anyway to resolve this i tried many other option
The flowfile content is different from attributes. Content is arbitrary -- could be empty, text, KB of XML, GB of video or binary. Each flowfile also has attributes which are key/value pairs of Strings kept in memory.
If you want to route on this piece of data, you have multiple options:
Use RouteOnText or RouteOnContent to use the actual flowfile content directly.
Extract it to an attribute using EvaluateJsonPath and then route on that attribute.
The Apache NiFi User Guide and In-Depth provide more information around this distinction.
I'm using Nifi, and it looks like ReplaceText doesn't work properly. I want to perform a text search and replace with these criteria:
I want to store a regex of my search string in a variable, not directly in a processor's property.
I want to store a regex of my replacement string in a variable, not directly in a processor's property.
I want to be able to only change the variables in the resulting template, so I can share it with other users via the Nifi registry.
I've tried using ExtractText to search for something, store that in an attribute, and then use the value of that attribute, but Nifi Expression Language scope is listed as unsupported for ExtractText user added properties. ReplaceText works fine if you want to put the regex in the ReplaceText property, but I don't, since I'm sharing the template via the registry and any time someone changes that property for their own search text, there will be a new version in the registry.
Update Attribute configuration
It is hard to tell from the screenshots provided, but it looks like the fields you want to remove are part of the content of the flow file, which is different then the attributes of the flow file. UpdateAttribute can only remove attributes, not anything in the content.
In order to modify the content you would need to use a processor specific to the type of content being processed. In your case it looks like JSON, so you could use a ConvertRecord processor with a JsonTreeReader and JsonRecordSetWriter, and configure the writer to have a different schema then the reader. Basically read in all the fields, but only write out the fields you want.
There is an UpdateRecord processor too, but it doesn't currently have the ability to remove fields.
To delete a FlowFile's attribute, you can use UpdateAttribute and a property named Delete Attributes Expression. You just need to fill it with a regular expression that matches the attributes you want to remove.
But as #Bryan Bende said, it doesn't look like you're trying to remove FlowFile's attributes, but content..
If you are willing to remove JSON attributes from your content, you can use JoltTransformJSON and Jolt Transformation DSL of Remove. Then just use specification of the attributes you are willing to remove. For example, I want to delete from this JSON the attribute t1:
{
"t1": "test",
"t2": "test2",
"t3": "test3"
}
So, my specification would be:
{
"t1": ""
}
You can read more about it here.
I have JSON objects coming into Nifi via MQTT from two different inputs - for instance, let's say one is from a top sensor, and one is from a bottom sensor. Each of the sensors has its own MQTT topic, so I am using two different ConsumeMQTT Processors to ingest this data into my Nifi Flow.
JSON Object for top sensor is {"Top_Data": "value"}
JSON Object for bottom sensor is {"Bottom_Data": "value"}
I am currently using two separate EvaluateJsonPath Processors to store either the value of Top_Data or Bottom_Data in an attribute called sensorData.
How can I use some kind of if/or statement to only use one processor to EvaluateJsonPath for both of the JSON objects I could get from MQTT? Basically, I want to have an expression that says "If my JSON object has a property called Top_Data, use its value for the attribute sensorData, otherwise, use the value from the property Bottom_Data."
Example of my EvaluateJsonPath Processor
maybe try JSONPath expression
$[Top_Data,Bottom_Data]
in the single EvaluateJSONPathProcessor.
According to https://goessner.net/articles/JsonPath/ there is a possibility to use alternate operator [,]:
[,] Union operator in XPath results in a combination of node sets. JSONPath allows alternate names or array indices as a set.
I have tested the expression using http://jsonpath.com/ and it should work.
Let us know if that helps.
You could try extracting them both using EvaluateJsonPath(property 1: top: $['top'], property 2: bottom: $['bottom']) and of course don't forget to set Destination to flowfile-attribute.
Then, transfer to UpdateAttribute and set property finalData as ${top:isEmpty():ifElse(${bottom}, ${top})}.
If EvaluateJsonPath won't find a full element, then it will set it as empty string, so all you need to do is check if either of them is empty and if it is, set the final data as the other one.
I am trying to insert the whole content of a row of an XML file into a JSON attribute (I am a newbie).
I am doing it this way (tell me if there is an easier way, it's good to now):
I have configured Extract text this way:
And to finish, I configure the Replace Text, giving a JSON format:
But he result appears to be wrong (doesn't work like a normal JSON file, for example if I a try to do a httpPost):
How can I fix this problem?
cheers
If you are concern regards to new lines and json key/values then use NiFi expression language functions on the extracted attribute(data).
ReplaceText Configs:
Replacement value:
{"name" : "user1","time" : "${now()}","data" : "${data:replaceAll('\s',''):escapeJson()}"}
Use escapeJson and replaceAll function to replace all spaces,newlines with ''
Replacement Strategy as Always Replace
(or)
Another way of preparing json message is by using AttributesToJson processor.
if we are using this processor then we need to prepare attributes/values before AttributesToJson processor by using UpdateAttribute processor
Flow:
1.SplitXml
2.ExtractText //add data property to extract content to flowfile attribute
3.UpdateAttribute //add name property -> user1
add time property -> ${now()}
add data property -> ${data:replaceAll('\s',''):escapeJson()}}
4.AttributeToJson //Attributes List -> name,time,data
Destination -> flowfile content
include core attributes -> false