This is my flow file content:
{
"a":"b",
"c":"y",
"d":"z",
"e":"w",
"f":"u",
"g":"v",
"h":"o",
"x":"t"
}
The final result should look like that in Postgres :
| test |
|----------------------------------------------------------------|
|{"a":"b,"c":"y","d":"z","e":"w","f":"u","g":"v","h":"o","x":"t"}|
the table is: json_test
the column name is test
Those steps shows how i tried to solve the problem:
My method was to store the json record in a variable as string with "ExtractText":
the attribute data take only some key-values from the json not the entire record:
data = {"a":"b",
"c":"y",
"d":"z",
"e":"w",
"f":
so i have a problem in the regex expression.
next i used PutSQL with the following SQL statement:
Unfortunately the result isn't the wanted one.
I need to know the exact expression that i should set in ExtractText to get the entire json record in a variable as string.
The sql statement should be:
insert into schema.table_name(column_name) values(the_variable_where the flowfile data was stored)
Related
The following flowfile is the response of an "InvokeHttp":
[
{"data1":"[{....},{...},{....}]","info":"data-from_site"},
{"data2":"[{....},{...},{....}]","info":"data-from_site"},
{"data3":"[{....},{...},{....}]","info":"data-from_site"}
]
I did a "SplitJson", i got each json record as a single flowfile
flowfile 1:
{"data1":"[{....},{...},{....}]","info":"data-from_site"}
flowfile 2:
{"data2":"[{....},{...},{....}]","info":"data-from_site"}
flowfile 3:
{"data3":"[{....},{...},{....}]","info":"data-from_site"}
I want to store each json record in each flowfile in a variable like that:
variable1 = "{"data1":"[{....},{...},{....}]","info":"data-from_site"}"
variable2 = "{"data2":"[{....},{...},{....}]","info":"data-from_site"}"
variable3 = "{"data3":"[{....},{...},{....}]","info":"data-from_site"}"
can someone show me how to store the json record in a variable !
If I understand correctly what you want to do (by "variable", do you mean what is called "attribute" in NiFi?), you can use the EvaluateJsonPath processor configured with:
flowfile-attribute as Destination
json as Return type
I have a flow-files with the below structure
{
"PN" : "U0-WH",
"INPUT_DATE" : "44252.699895833335",
"LABEL" : "Marker",
"STATUS" : "Approved",
}
and I need to execute an update statement using some fields
update table1 set column1 = 'value' where pn=${PN}
I found convertJsonToSQL but am not sure how to use it in this case
You can use a processor namely ConvertjSONToSQL. Using this you can convert your json into an update query.
ConvertjSONToSQL Description
It takes the following parameters :
1. JDBC Connection Pool : Create a JDBC pool which takes DB connection information as input.
2. Statement Type : Here you need to provide type of statement you want to create. In your case its 'UPDATE'.
3. Table Name : Name of the table for which update query needed to be created
4. Schema Name : Name of the schema of your database.
5. Translate Field Names : If true, the Processor will attempt to translate JSON field names into the appropriate column names for the table specified. If false, the JSON field names must match the column names exactly, or the column will not be updated
6. Unmatched Field Behaviour : if an incoming JSON element has a field that does not map to any of the database table's columns, this property specifies how to handle the situation
7. Unmatched Column Behaviour : If an incoming JSON element does not have a field mapping for all of the database table's columns, this property specifies how to handle the situation
8. Update Keys : A comma-separated list of column names that uniquely identifies a row in the database for UPDATE statements. If the Statement Type is UPDATE and this property is not set, the table's Primary Keys are used. In this case, if no Primary Key exists, the conversion to SQL will fail if Unmatched Column Behaviour is set to FAIL. This property is ignored if the Statement Type is INSERT
Supports Expression Language: true (will be evaluated using flow file attributes and variable registry)
Read the description above and try to use the properties given. Detailed description of the processor is given in the link.
ConvertjSONToSQL Description
I have HTTP request whose response is below
{
"DATA": {
"G_1": {
"OR_ID": "100400",
"LEGAL_ENTITY": "TEST",
"BUSINESS_UNIT": "TEST BU"
},
"G_2": {
"OR_ID": "100500",
"LEGAL_ENTITY": "Test1 ",
"BUSINESS_UNIT": "Test1 "
},
"G_2": {
"OR_ID": "100100",
"LEGAL_ENTITY": "TEST3 ",
"BUSINESS_UNIT": "Test3"
}
}
I need to get OR_ID from the above response, which I am able to do it using Regular exp extractor.
There is Input CSV file which has multiple rows. For the CSV file, i need to check the OR_ID exists or not in column 2, if exists then I have to take columns 5 and 7 and pass it to my next post request in the body. In CSV the same OR_ID repeated, so i need repeat post request for all the repeated values of OR_ID in csv. CSV file has no header.
441919244,100010,QUTRN,TEST Inc.,100100,TEST,VCG and A, INC,USD,3409.0900,O,ICO-VCG-0140,2019-10-31,52 945,USD,USD,359409.0900,359409.0900,359409.0900,Processed,93901372,File,2019111NG52.csv,
441919028,100400,QUQED,TEST MEDICAL EDUCATION INC.,100020,QUINC,TEST INC.,USD,12.340,O,ICO-INC-8718,2019-10-31,52 729,USD,USD,12.3400,12.3400,12.3400,Processed,93901372,,File,20191113NG52.csv,
Can you please help.
Assuming that you can extract the OR_ID from the JSON response following solution could be useful.
In the CSV Data Set Config Element or Random CSV Data Set Config plugin read the CSV file assign the variable names to the respective columns
variable names = C1,OR_ID,C3,C4,C5,C6,C7,C8,C9
Add a While Controller as a parent to the CSV Data Set config elements and the HTTP Request where you want to send data from the CSV file.
${__jexl3("${OR_ID}"!="EOF")}
This will check EOF in the column 2 of the CSV file. Hence please add
,EOF,,
as the last line of the CSV file.
Add IF controller to the HTTP Request with following condition
${__jexl3("${OR_ID}"=="${OR_ID_J}")}
OR_ID_J is the OR_ID picked from the JASON response.
Use ${C5} and ${C7} in the places where you want to insert the data from the CSV file.
Reset the OR_ID to "" using a JSSR223 Sampler with following
vars.put("OR_ID", "");
Sample Test Plan is available in GitHub
Hi I am fairly new to apache nifi and I need to extract a suburb from a json string
flowfile looks like this
\/
{"UserName:"John Doe", "Address":"22 smith st, Smithville, NSW","IP":"10.10.10.1}
The suburb is always the 2nd last value in the commar delimited list of the "address" attribute. Sometime it wont be in the 2nd position from left as there might be something like
{"UserName:"John Doe", "Address":"Level 10, 22 smith st, Smithville, NSW","IP":"10.10.10.1}
I have tried to use extract text with regex [^,]+(?=,[^,]*$)
but was not able to make it extract the attribute correctly.
I think you haven't use extractText for extract the Json values and it is not proper way to do it.
You can EvaluateJsonPath processor for extract "Address" of the Json attribute with help of following configurations.
Just configure those attributes for destination to be "flowfile-content" and Return Type to be "Json".
Now you have to add new property named "Address":$.Address.
Here you can receive address of json will be stored in attribute named "Address" then you can extract 2nd column of the suburb present in Address like ${Address:substringBeforeLast(','):substringAfterLast(',')}.
Look at this expression guide which may useful for you.
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#substringafterlast
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#substringbeforelast
I have a table with a column that has urls. I want to query out a particular url param value from each record. the url param can occur in any position in the url data and the url can contain hashbangs and this param can contain special chars like -, _ and |.
data table column:
url
http://www.url.com?like=hobby&name=tom-_green
http://www.url.com?name=bob|ghost&like=hobby
and I want the query results to be
name
srini
tom-_green
bob|ghost
I tried a query like
Select regexp_extract(url, '(?<=name=)[^&?]*(?:|$&)',2) as name
From table_name
I see java exceptions when I run this query. the exceptions are pretty vague and checking if someone can help.
I found another Hive implementation for handling URLs specifically..
Select parse_url(url, 'QUERY', 'name') as name From table_name and this worked :)
ref: parse_url(string urlString, string partToExtract [, string keyToExtract])
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF