I have created a RouteOnAttribute processor in nifi dataflow, i want it to select only certain files to parse on.
in the properties i have created a property "filetofetch"
with following expression
${filename:contains('INCOMING.D151221')
I need to fetch the file name INCOMING.D< YYYYMMDD>
so today 21 MARTS 2017 the filename would be
INCOMING.D20170321
I have tried with something like this to extraxt file name
${filename:contains('INCOMING.D'+ ${now():format('yyyymmdd')} )}
But i cannot concat with the date prefix
any suggestions ?
Havmaage,
You can concat by use append in Expression language like below.,
You have to use the updateAttribute to store the 'Incoming.D' in one attribute name like below.
fileStartsWith:Incoming.D
Date:${now():format('yyyyMMdd')
Then finally use routeonattribute to check like below.
${filename:contains(${fileStartsWith:append(${date})})}
You cann't be concat with '+' in Nifi.
EDIT-1:
Date:${now():format('yyyyMMdd')
Related
I am following nifi guide to parse a delimited file content.
Instead of coding the search text and replacement value i want to use the content of two attributes.
processor config
When executed the processor is not using the attribute content as a regexp even if it is a valid regex expression
I have some pipe delimited files. Each field is bounded by quotes like this.
"Created_Date__c"|"CreatedById"|"CreatedDate"|"Guid_c"
"2020-03-02 00:00:00"|"0053i000002XCpAAG"|"2020-03-02 16:01:34"|"94bf83ccf9daf610VgnVCM100000307882a2RCRD"
"2020-03-03 00:00:00"|"0053i000002XCpAAG"|"2020-03-03 09:15:56"|"1a4bb238cdedd610VgnVCM100000307882a2RCRD"
"2020-03-03 00:00:00"|"0053i000002XCpAAG"|"2020-03-03 09:52:33"|"22408baca6fee610VgnVCM100000307882a2RCRD"
I need to cleanse this data and the needs to look like this.
Created_Date__c|CreatedById|CreatedDate|Guid_c
2020-03-02 00:00:00|0053i000002XCpAAG|2020-03-02 16:01:34|94bf83ccf9daf610VgnVCM100000307882a2RCRD
2020-03-03 00:00:00|0053i000002XCpAAG|2020-03-03 09:15:56|1a4bb238cdedd610VgnVCM100000307882a2RCRD
2020-03-03 00:00:00|0053i000002XCpAAG|2020-03-03 09:52:33|22408baca6fee610VgnVCM100000307882a2RCRD
I tried using ReplaceText with these configurations.
search value - ^"(.*)"$ and Replacement Value - $1. But these configurations is not working and the file is routing to failure. not sure what might be the issue.
open to other suggestions. Thanks in advance.
I think you should only use "(.*?)" regex instead of ^"(.*)"$.
Some online services such as https://www.freeformatter.com/java-regex-tester.html can be useful for testing the regex replacement.
I think your best option here is a ConvertRecord processor, have CSVReader with infer schema + changing the csv sep to your own |, and a CSVRecordSetWritter with Option Quote Mode set to Do Not Quote Values and also set your sep as per your need.
I want to insert a comma between braces in a FlowFile.
I receive json objects and merge them together with a process. After the merge I get this:
{"tag":"a","bag":"b"}{"tag":"c","bag":"d"}
I want it to end up like this after the ReplaceText processor:
{"tag":"a","bag":"b"},{"tag":"c","bag":"d"}
However my ReplaceText process doesn't work. I have it set up like this:
With this process, nothing gets replaced.
Am I doing something wrong?
I suggest to add demarcator in MergeContent if you use it in you process
But you variant still must work. I try this on 1.10 and content has been changed.
In nifi, I need to transfer a bunch of json files to HDFS. The json files have a field called "creationDate" which has the date in UNIX format. I need to use the date in there to funnel the file to HDFS directories that are named after dates, like "2019-01-19" "2019-01-20" "2019-01-21" etc.
At first I used an "EvaluateJsonPath" processor going to a "PutHDFS" processor. The "Evaluate..." processor had "creationDate" as the property and "${creationDate} as the value. In the PutHDFS processor, for directory I put "/${creationDate}"
But then I realized that the date in the json file has the full timestamp, like "2019-01-19T04:34:28.527722+00:00
Obviously I don't need all that, just the first eight digits. So how can I turn this big string into a neat 8-digit directory name? Will I need to use a regex, and if so, how can this be implemented? Thanks in advance for any help.
You can use UpdateAttribute and use the date expression language functions to format it.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
Example (not specific to your format):
${creationDate:toDate('MM-dd-yyyy'):format('yyyy/MM/dd')}
In UpdateAttribute you would add a new property name creationDate and set the value to an expression like above.
I am trying to modify the name of some files with NiFi getting a value from a JSON an adding to the original filename (for example filename.csv (original name) + january (name that provides the incoming JSON)). To do so, I am redirecting the CSV files to updateAttribute to change de Filename.
From the other hand, I am receiving a JSON that has an attribute that will be part of the name of the file.
On the evaluateJsonPath the configuration is the next (I am receiving it correctly the value):
And finally I am trying to merge the values in the UpdateAttribute processor (here is where it doesn't work properly):
The response I am getting is _filename.csv
You have quotes around name inside your Expression Language expression, try
${name}_${filename} or
${name:append('_'):append(${filename})}