Hi I am fairly new to apache nifi and I need to extract a suburb from a json string
flowfile looks like this
\/
{"UserName:"John Doe", "Address":"22 smith st, Smithville, NSW","IP":"10.10.10.1}
The suburb is always the 2nd last value in the commar delimited list of the "address" attribute. Sometime it wont be in the 2nd position from left as there might be something like
{"UserName:"John Doe", "Address":"Level 10, 22 smith st, Smithville, NSW","IP":"10.10.10.1}
I have tried to use extract text with regex [^,]+(?=,[^,]*$)
but was not able to make it extract the attribute correctly.
I think you haven't use extractText for extract the Json values and it is not proper way to do it.
You can EvaluateJsonPath processor for extract "Address" of the Json attribute with help of following configurations.
Just configure those attributes for destination to be "flowfile-content" and Return Type to be "Json".
Now you have to add new property named "Address":$.Address.
Here you can receive address of json will be stored in attribute named "Address" then you can extract 2nd column of the suburb present in Address like ${Address:substringBeforeLast(','):substringAfterLast(',')}.
Look at this expression guide which may useful for you.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#substringafterlast
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#substringbeforelast
Related
This is my flow file content:
{
"a":"b",
"c":"y",
"d":"z",
"e":"w",
"f":"u",
"g":"v",
"h":"o",
"x":"t"
}
The final result should look like that in Postgres :
| test |
|----------------------------------------------------------------|
|{"a":"b,"c":"y","d":"z","e":"w","f":"u","g":"v","h":"o","x":"t"}|
the table is: json_test
the column name is test
Those steps shows how i tried to solve the problem:
My method was to store the json record in a variable as string with "ExtractText":
the attribute data take only some key-values from the json not the entire record:
data = {"a":"b",
"c":"y",
"d":"z",
"e":"w",
"f":
so i have a problem in the regex expression.
next i used PutSQL with the following SQL statement:
Unfortunately the result isn't the wanted one.
I need to know the exact expression that i should set in ExtractText to get the entire json record in a variable as string.
The sql statement should be:
insert into schema.table_name(column_name) values(the_variable_where the flowfile data was stored)
I am working on a NiFi workflow. JSON is coming into the workflow and I am using EvaluateJsonPath and RouteOnAttribute processes. In the EvaluateJsonPath, I have the Null Value Representation set to empty string.
[
{
'a':null,
'b':null
},
{
'a':'1',
'b':'2'
}
]
I use a JsonSplitter to split the array so each element in the JSON array is handled separately.
In the first EvaluateJsonPath I do the following.
a=$.a
b=$.b
In the RouteToAttribute (I am using the same property, if I use a different or new property, I get the same issue)
a=${a:isEmpty():ifElse('NOT PROTECTED', 'PROTECTED')}
b=${b:isEmpty():ifElse('', 'ACTIVE')}
The RouteToAttribute processor is providing the following error for both properties:
'a' validated against '${a:isEmpty():ifElse('NOT PROTECTED', 'PROTECTED')}' is invalid because Expected Attribute Query to return type BOOLEAN but query returns type STRING
'b' validated against '${b:isEmpty():ifElse('', 'ACTIVE')}' is invalid because Expected Attribute Query to return type BOOLEAN but query returns type STRING
It's really not clear what you are trying to do.
You are using RouteOnAttribute, but it looks like you are trying to replace attribute values. This is not what RouteOnAttribute does.
RouteOnAttribute is designed to evaluate a query, and then route matching FlowFiles to a specified relationship.
With RouteOnAttribute you have 3 options for the Routing Strategy which are
Route to Property name
Route to 'matched' if all match
Route to 'matched' if any matches
When using option 1, that is, Route to Property name, the NAME of the property is the RELATIONSHIP that the FlowFile will be routed to. The VALUE of the property is a QUERY that must return a BOOLEAN (true or false). If the result of the query is True, the FlowFile is routed to this relationship. If you have multiple properties configured, it will evaluate them in order.
When using option 2 or 3, you do not specify the name of the relationship, it is always called matched. You can specify multiple properties, and the NAME of the property is irrelevent. The VALUE of the property is a QUERY that must return a BOOLEAN. With option 2, if ALL of the queries return True, the FlowFile is routed to matched. With option 3, if ANY of the queries return True, the FlowFile is routed to matched.
Read the documentation.
Your error is caused because ${a:isEmpty():ifElse('NOT PROTECTED', 'PROTECTED')} does not return a BOOLEAN. It returns a STRING. There is nothing RouteOnAttribute can do with this result.
Either way, you would be better off looking at Records.
Specifically, if you want to update the value of fields inside the data, look at UpdateRecord.
If you want to do routing logic on fields inside the data, look at QueryRecord and PartitionRecord.
Splitting each JSON element out using SplitJson is inefficient and avoidable when using Records.
I am trying to fetch xml data of a Sales Order record but I can't find a way to do it.
The data appears when we add '&xml=t' to the url of a record.
I want to fetch all the data in XML format into a variable.
Found the answer.
Just use record.load and store it in a variable.
var objRecord = record.load({
type: record.Type.SALES_ORDER,
id: 157,
isDynamic: true,
});
Logging this varialbe won't help, as there is character limit of 3999 characters. You can store it in File Cabinet.
Ex:
"ActivityId":7177,"ReferenceId":0,"Key":0,"FirstName":"Kerthana","LastName":"thanneru","CId":"00050068953"
"ActivityId":6216,"ReferenceId":42234,"Key":0,"FirstName":"Krishna","LastName":"thanneru","CId":null,"Specialty":null,"InviteeType":
I want to use ActivityId":6216 which I correlated. But I want use it only if CId has data. If CId is null, I need to skip it.
This is ForEach Controller configuration:
Could someone please help me to figure out how to skip rows which have null in CId column?
If your Regular Expression Extractor won't allow null values, it will ignore activity Ids with null CId.
This work for me with your given text for ignoring null in CId column:
ActivityId":(\d+)([^\t\n]+)CId":"
Use the IF controller and in that put the condition as ${__jexl3("${CId}"!= null,)}
And then if your CId is not null then that row should be picked up.!!
I am considering that your data is coming from an external source like csv file or so..!!
For every line you can do this:
if line =~ /.*?CId["]:"?[0-9]+"?.*/; puts line end
What's the best practice with NIFI to extract an attribute in a flowfile and transform it in a Text Format Example :
{ "data" : "ex" } ===> My data is ex
How can I do this with NIFI wihtout using a executeScript Processor
You could use ExtractText to extract the values into attributes. If you added a property in ExtractText like foo = {"(.+)" : "(.+)"} then your flow file would get two attributes for each of the capture groups in the regex:
foo.1 = data
foo.2 = ex
Then you can use ReplaceText with a Replacement Value of:
My ${foo.1} is ${foo.2}