I want to send a dataflow to a webservice which will then respond based on the content of the dataflow. So it might respond with a JSON that says {'Errors':'None'} or {'Errors':'5'}. I would like to then use this information to continue processing the original dataflow if no errors are found, orlog the information if any errors are found. My question is, how can I route based on the values in the JSON response?
You can use EvaluateJsonPath to get the value of Errors into an attribute, then RouteOnAttribute to send the different values down different paths.
Alternatively you could use QueryRecord with queries like SELECT * FROM FLOWFILE WHERE Errors = 'None'. Each query gets its own downstream relationship, so it has the effect of routing in this case.
Related
I want to report to an API with some information with attributes of each flowfile. Sometimes the API return some important information in JSON. My goal is update the attributes of the original flowfile with the new data that return the API.
My sketch-strategy to update the FlowFile -> AttributeToJSON (But the entire content of the FlowFile is replaced by the JSON, 1°problem) -> HttpInvoke to send the information to the API -> The API return a JSON with some data -> Extract some data from the JSON with some Process and update the attributes of the Flowfile
1° problem: I can separate the flowfile, the original and another (to modify it with AttributeToJSON). But how can I merge them in the future? Which process I need to combine the original flowfile and the "new" attributes that I build with the response of the API?
Perhaps I can save the orginal file in a directory with PutFile, and by another way process the info, and some point use the FetchFile (with attributes know where is save the file), and then take the data and Attributes together.
extra Can I send with HttpInvoke POST Request with only the attributes(one of them written in JSON)?.
You may want to take a look at the lookup processors -- LookupAttribute and LookupRecord. These processors allow you to enrich the existing flowfile with additional information.
It looks like right now, the RestLookupService is available for record enrichment but not attribute enrichment. You may want to file a Jira requesting this, and in the meantime you can use SimpleScriptedLookupService to make an HTTP invocation from that processor.
I have a endpoint would return response as follow.
{
"result": [
{},
....
{}]
}
I am trying to use invokeHTTP and enable “Put Response Body In Attribute” to keep origin flowfile and response from api.
but it seems add a attribute named $.result as follow
Is there any way to set a proper name for result attribute ? Thanks.
You try to extract results using a JSON path. However this is not possible from within InvokeHttp. You may want to use EvaluateJsonPath processor.
Documentation for Put Response Body In Attribute:
If set, the response body received back will be put into an attribute
of the original FlowFile instead of a separate FlowFile. The attribute
key to put to is determined by evaluating value of this property.
To keep the original flow file after calling InvokeHttp, you can move it to an attribute (if the content is small enough) just before calling InvokeHttp or you can use MergeContent with original flow file and response flow file from InvokeHttp.
Before posting this question about Apache NiFi InvokeHTTP I have gone through all other questions and their answersbut I am still unsure the best flow I should have. My situation is as below:
1) From Apache Kakfa, I get raw metadata.
2) Using EvaluateJSONPath I get attribute I want.
3) Using RouteOnAttribute I created 3 routes based on the attribute value I got from step-2 above.
4) Now based on the attribute value I want to decide whether I should go for GET or for POST or for Delete.
5) My question is where/how to set POST message? GET message? Delete Message body?
6) I am able to set the URL in configuration part provided by InvokeHTTP. But message body I don't know which is that property? or its in flow file using ReplaceText?
I read somewhere that before you divert your Restful POST HTTP request to InvokeHTTP you must have another processor before which changes the content of flow file.
Ref: Configuring HTTP POST request from Nifi
Please help. Thanks.
regards,
Yeshwant
Adding on to what Bryan had explained, POST will use the FlowFile content as the message body so if you have some other data which you want to wipe/transform into something and then sent as the message body, you can leverage the following processors :
ExtractText to read data from the existing FlowFile content
ReplaceText to erase the existing content of the FlowFile and replace it with different one
To set the headers for the REST calls, InvokeHTTP has the property Attributes to Send property which takes a regex which will scanned against the incoming FlowFiles' attributes and whichever attributes are matched are taken and sent as HTTP header.
To add new attribute to your existing FlowFile, you can use UpdateAttribute
For a POST, the body will be whatever is in the flow file content.
a GET and DELETE typically wouldn't have a body since the information would typically be provided in the URL or query params.
I need to perform an HTTP Post from NiFi, but I don't want/need the request to carry all of the FlowFile's content.
Is there some way to pass attributes of a FlowFile but not the full content?
If the request body of your Http Post is JSON, you can use the AttributesToJSON Processor which allows you to pick which attributes you want to include in the resulting JSON. You can then configure the processor so the resulting JSON overwrites the existing flowfile content.
Keep in mind that the resulting JSON will be flat so you may need to transform it to the expected format. For that, you can use the JoltTransformJSON Processor.
Below is an example of what your dataflow might look like. I hope this helps!
I would like to read the exact value of a variable I use to pass through an HTTP Request. I first read in many values of variables using the CSV Data Set Config. For the username, it is in the form of an email address. So, I have a variable called "email" in the Data Set Config. In the actual HTTP Request, for "name", I call it "username". For the "Value" field for this same "username", I added a time() function to it like this so I would end up creating unique users in my tests:
${email}${__time()
When I view the "Request" in a View Results Tree, I can see my parameter is listed correctly:
username=email1%40email.com1390854377360
I do not care if this is correct in real world terms. I already know that is not a valid email. That is ok for now.
What I want to know is how can I log that email that I just created on the fly? I would like to not have to pull in the whole request every time also and then use some type of Regular Expression extractor. It just seems like there should be an easy way to do this.
I think there are 2 ways,
Beanshell Pre/Post processors : you can write custom code in which you can log all your variables in some custom log file
Simple data writer : you can configure it and check save url,save field names,save response data field checkboxes that will give you complete data but in that later postprocessing on result file is required to get all usernames (email in your case).
first approach is easier and allows you create your own logging format(easy to retrieve and use somewhere else).
second approach is somewhat tedious and requires post processing.