I have this flow in NiFi:
GetFile -> ConvertExcelToCSVProcessor -> ReplaceText ->
PutDatabaseRecord.
It's working ok but I want to set "table name" property of
PutDatabaseRecord
based on csv file name (and if it's posible to customize it). I can't find anything in docs or web.
Thanks!
According to documentation, the parameter table name of the PutDatabaseRecord processor supports nifi expression language.
So, if the attribute filename of your flow file contains value MyTableName.csv you could use expression with regular expression to convert file name to table name in PutDatabaseRecord processor like this:
Table Name = ${filename:replaceAll('\\..*','')}
Related
How can I get ParameterContext value with dynamic name in Apache NiFi?
For example, for each flowfile I need to get parameter according to ${type} attribute of this flowfile.
Like: #{${type}_parameter}.
Is it possible to do this in NiFi 1.19.1?
I tried ${literal('{${type}_parameter}'):prepend('#'):evaluateELString()}, but it always returns empty string.
I am new to apache NiFi.
I am trying to put data into elasticsearch using nifi.
I want to specify an index name by combining a specific string and the value converted from a timestamp field into date format.
I was able to create the desired shape with the expression below, but failed to create the index name with the value of the timestamp field of the content.
${now():format('yyyy-MM-dd')}
example json data
{
"timestamp" :1625579799000,
"data1": "abcd",
"date2": 12345
}
I would like to get the following result:
index : "myindex-2021.07.06"
What should I do? please tell me how
I know that if you use the PutElasticSearch Processor, you can provide it with a specific index name to use. And as long as the index name meets the proper ElasticSearch format for naming a new index, if the enable auto index creation in ElasticSearch is turned on, then when sent, Elastic will create the new index with the desired name. This has worked for me. Double check the Elastic Naming Rules that can be found here or https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-indexing.html
I have .csv file with several fields. One of them (for example 3th) contains email. How can I add additional filed that will contain only serverName from email field?
Example:
input:
01;city;name#servername.com;age;
result:
01;city;name#servername.com;age;servername;
I guess it possible through ReplaceText processor, but I can't choose correct value in "Search Value" and "Replacement Value" fields.
You can convert your flowfile, to a record with the help of ConvertRecord.
It allows you to pass to a JSON (or something else) format to whatever you prefer.
Then you can add a new field, like the following, with an UpdateRecordProcessor:
I recommend you the following reading:
Update Record tutorial
Suppose I have one JSON log that outputs the following information:
{"timestamp":"someText","alert":"someMoreText","Level":someInt}
And I have another JSON log that outputs the same kind of information but with different label's:
{"ts":"someText","alert":"someMoreText","Level":someInt}
The difference being "timestamp" and "ts" have different names yet same quality information. How would I reference, with one JSON path call, either one of the alternate names, if such a technique is possible?
So for example, if I wanted it to reference the timestamp of both logs, I would want to use something like $.[timestamp|ts]
Using the new record processors, you might be able to do something like...
Define a schema that has both 'timestamp' and 'ts'
Send all the records with 'ts' to an UpdateRecord processor
Set the UpdateRecord processor to make /timestamp = /ts
Define another version of the schema that doesn't have 'ts'
Use a ConvertRecord processor with a writer that uses the second schema
That last step would rewrite the records without the 'ts' field.
Alternatively, you could try defining a schema with a 'timestamp' field and an alias of 'ts' which should let any of the record processors access both fields by using 'timestamp'. It would depend what you are doing in your flow to see if it can be achieved with the record processors.
$.id:append('-') & ${id:append('-')} in Evaluate JsonPathProcessor gives me no results, what would we be the correct way to append a text at the end of an incoming attribute in Apache Nifi.
If your Apache NiFi had a sample input JSON document like:
{
"id": "foo"
}
Then you can extract and format attributes in two steps:
EvaluateJsonPath to extract a value from the input JSON document to a NiFi flowfile attribute. For the sample, you might add a custom property json.id with the JsonPath of $.id. A flowfile attribute called json.id will be added to the flowfile with the value foo.
UpdateAttribute to use NiFi Expression Language to format the id. You can assign any attribute, like formatted.id using an expression that references the attribute extracted earlier, json.id: