I'm working on a NiFi 1.15 flow where I have to send a request to a service that requires 2 pieces of form-data sent in a POST request as a multipart/form-data. The first part is a simple JSON object with a few parameters, while the second part is a CSV file. Here is an example of what I am trying to achieve.
Content-Type: multipart/form-data; boundary=1cf28ed799fe4325b8cd0637a67dc612
--1cf28ed799fe4325b8cd0637a67dc612
Content-Disposition: form-data; name="json"; filename="json"
{"Param1": "Value1","Param2": "Value2","Param3": true}
--1cf28ed799fe4325b8cd0637a67dc612
Content-Disposition: form-data; name="file"; filename="body.csv"
Field1,Field2,Field3,Field4,Field5
VALUE_FIELD_1,VALUE_FIELD_2,VALUE_FIELD_3,"Some value for field 4",VALUE_FIELD_5
--1cf28ed799fe4325b8cd0637a67dc612--
Another acceptable output would have the Content-Disposition lines empty.
Due to a few restrictions in the environment I am working on, I am not allowed to use scripting processors, such as ExecuteGroovyScript as suggested in another SO question.
Instead, I started creating a GenerateFlowFile -> InvokeHTTP flow. The GenerateFlowFile would output to a flow file a text similar to the one mentioned above. Here is the screenshot of the GenerateFlowFile.
The connected InvokeHTTP processor is configured to use the POST Http Method and to send headers (the Authorization header in my case) and Send Message Body is set to true. It also extracts the Content-Type from the flow file previsously generated attribute through a ${mime.type} function. You can see the details in the following screenshots.
Sadly, this does not work. The server responds with an "Unexpected end of MIME multipart stream. MIME multipart message is not complete." error.
After searching for a while in SO, I found another question describing what looks like a similar problem, but there they are getting a different error and is also posting parameters through a different method.
I am also aware about the blog post from Otto Fowler where he shows how InvokeHTTP supports POSTs with multipart/form-data. I did try this approach too, but did not manage to get it working. The service throws an error stating that NiFi does not send one of my post:form:parts.
Right now I am stuck and am not able to send that data. I did manage to write a simple Python script to test if the server is working properly and it is. For reference, here is the script:
import requests
server = 'https://targetserver.com'
#Authentication
result = requests.post(server + '/authentication',
{'grant_type': 'password',
'username': 'username',
'password': 'password'})
token = result.json()['access_token']
#Build the request
headers = {'Authorization': 'bearer ' + token}
json_data = '{"Param1": "Value1","Param2": "Value2","Param3": true}'
# First the JSON then the csv file.
files = {'json': json_data,
'file': open('body.csv', 'rb')}
result = requests.post(server + '/endpoint', headers = headers, files = files)
print(result.text)
Does anyone have a suggestion on how to get around this situation?
Related
When connecting to a website using Net::HTTP you can parse the URL and output each of the URL headers by using #.each_header. I understand what the encoding and the user agent and such means, but not what the "accept"=>["*/*"] part is. Is this the accepted payload? Or is it something else?
require 'net/http'
uri = URI('http://www.bible-history.com/subcat.php?id=2')
http://www.bible-history.com/subcat.php?id=2>
http_request = Net::HTTP::Get.new(uri)
http_request.each_header { |header| puts header }
# => {"accept-encoding"=>["gzip;q=1.0,deflate;q=0.6,identity;q=0.3"], "accept"=>["*/*"], "user-agent"=>["Ruby"], "host"=>["www.bible-history.com"]}
From https://www.w3.org/Protocols/HTTP/HTRQ_Headers.html#z3
This field contains a semicolon-separated list of representation schemes ( Content-Type metainformation values) which will be accepted in the response to this request.
Basically, it specifies what kinds of content you can read back. If you write an api client, you may only be interested in application/json, for example (and you couldn't care less about text/html).
In this case, your header would look like this:
Accept: application/json
And the app will know not to send any html your way.
Using the Accept header, the client can specify MIME types they are willing to accept for the requested URL. If the requested resource is e.g. available in multiple representations (e.g an image as PNG, JPG or SVG), the user agent can specify that they want the PNG version only. It is up to the server to honor this request.
In your example, the request header specifies that you are willing to accept any content type.
The header is defined in RFC 2616.
I'm trying to receive a json payload on a hapijs server, that json may not be valid, since is some custom format that i need to manually proxy the request to an elasticsearch cluster, cannot use the proxy option for hapijs because i need to do multiple requests to different clusters, for that i use nodejs http library.
Elasticsearch doesn't receive a valid json when doing bulk actions, it receives new lines instead of commas, to separate json objects:
{"index":[".kibana-devnull"],"ignore_unavailable":true}
{"size":500,"sort":[]}
Hapijs tries to validate the json payload when it gets application/json header in the request and it responds "Invalid request payload JSON format", as i cannot remove that header i need to look for another method to allow that invalid json in the route, even if the header is present.
I would look at the docs, in particular http://hapijs.com/api#route-configuration. If you set payload.output.parse to false you will receive the raw buffer inside handler which can then be parsed by yourself as opposed to by the framework.
I am trying to construct a HTTP Request through JMeter that uses a multipart data body.
I have a HTTP Header Manager that has COntent-Type set to multipart/form-data; boundary=AaB03x. I selected 'Use multipart/form-data for POST'.
I then have a data body created as,
`-----------------------------AaB03x
Content-Type: application/json
Content-Disposition: form-data; name="part1"
{"jsonKey1": "JsonValue1" }
-----------------------------AaB03x
Content-Type: application/json
Content-Disposition: form-data; name="part2"
{
"jsonKey2": "JsonValue2"
}
-----------------------------AaB03x
Content-Type: application/octet-stream
Content-Disposition: form-data; name="part3"
File Content here!!!!
-----------------------------AaB03x`
When I run this, I see that the request doesnt send the body correctly, instead it just sends some random data as,
POST data:
--vKOghkU7riRQbaANmmGpMs7j9TxCTUp3S2T0vE--
And gives an error response of,
`{"errorMsg":"Unable read headers from MultipartStream.","messageCode":"UnableReadHeadersFromMultipartStream","httpStat us":"BAD_REQUEST","requestName":"RequestName"}`
My second question is:
the part3 of the request sends a file to upload. Can I pass the file path somehow?
Given you set your own boundary and build your request manually I believe you need to uncheck Use multipart/form-data for POST in the HTTP Request Sampler
If your file encoding isn't very "exotic" you can try using __FileToString() function just instead of File Content here!!!!.
Looking into RFC 7578, it seems you also need a trailing -- at the end of the last line
You should try sending your JSON data as parameters. Also put your file path in the section for that... And even some servers don't actually need MIME type explicitly declared, you can check yours with some online tool like this one.
Your HTTP Request could look smethnig like:
I am using Sinatra with Thin.
When trying to get access the contents of a POST request with multipart data, I don't see any convenient way to access this, though it seems I can get to the raw body.
Is there something I am missing? Is there a convenient way to do this, or must I parse the data myself?
request.env['rack.request.form_hash'] gives an empty hash.
When dumping the body (request.body.read()) to stdout, it looks like this:
--KoZIhvcNAQcB
Content-Disposition: form-data; filename="1"
X-compId: 1
Content-Length: 1024
QeQL4k8RPBQVrP2u8Zf6Ie82labdA_
d7s FgvwSQvkP6e0TwaWWCfoHWbaP6
... (some lines omitted)
--KoZIhvcNAQcB--
I also see that it is not a chunked transfer encoding issue (I read that Thin has an issue with that); the Transfer-Encoding header is not set.
But as for my question, does Sinatra or Rack supply something in the API for accessing multipart details?
UPDATE: After doing some reading, I see the multipart data is supposed to be visible in the request parameters. As a result, I have logged an issue on GitHub. If I get it resolved, I will post an answer here. https://github.com/rack/rack/issues/695
Is there a way to overwrite the http response headers returned in Jmeter? I'm testing a web service that returns JSON and when an invalid request is sent, the JSON response returned doesn't contain application/json (or any for that matter) in the response header. If I save the response to a file, I see the actual JSON returned, but looking at the response in a Results tree doesn't show a response. Unless there is a way to load the response from file and parse the error message from the file, I'm hoping to somehow overwrite the HTTP response header and force jmeter to treat the response as JSON.
Any suggestions are welcome!
Using a beanshell post processor, you can write some script that would force the value for the header, or write out to a file.
You can also add a listener that would write the results to file for you. Granted - this is less convenient for debugging then Tree View.
As it turns out JMeter does not support response header overloading. While the response isn't displayed in the Results tree, it is actually available to other assertions. I was able to still provide assertions to validate responses even though the response was missing from the GUI.