Ruby within Python virtualenv HTTP Post Request very slow on the client side - ruby

I'm currently stuck with a response time problem with the Faraday HTTP library. I'm using the following code to initialize a POST request to a server running on my local machine at 127.0.0.1. The server returns a valid JSON string. The code to initialize the Faraday object on the client side is as follows:
url = 'http://127.0.0.1'
conn = Faraday.new(:url => url) do |faraday|
faraday.request :url_encoded
faraday.response :json, :content_type => 'application/json'
faraday.adapter Faraday.default_adapter
end
Then I send a JSON string via a POST request to the server. The code for sending the request looks like this (text size up to 5000 char):
payload = {:language => 'en', :text => 'some text'}.to_json
response = conn.post do |req|
req.url '/api'
req.headers['Content-Type'] = 'application/json'
req.body = payload
end
The expected result is a JSON string of the following structure:
{
"level1.1" : [
{
"level2.1" : [
{
"value1" : "Text",
"value2" : "Text",
"value(...)" : "Text",
"value(n)" : "Text"
},
{...
}
],
<and so on... - of course in a valid JSON structure ending>
When I run the code without doing anything with the result it performs pretty well and finishes in a reasonable time (< 0.5s). But as soon as I try to access the response object the script becomes terrible slow.
Just by adding the line:
p response.body
The processing time goes up to > 8s.
I have checked the server response with Postman and it works absolutely fine without any visible problems. Response time in Postman is >0.5 s as well. The slow down appears only on the client side when I try to access the response object. Just inspecting the response object doesn't effect the processing time either. But as soon as I start "to do" something with the response it becomes terribly slow.
I'm running Ruby 2.5.3 and Faraday 0.15.4 / Middleware 0.12.2
Any idea what might cause this slowdown is greatly appreciated.
Krid
EDIT
Despite my comment that HTTParty solved this issue this was just due to a smaller payload. HTTParty and Faraday both perform badly on my POST request while a POST request from Postman runs very fast even with large payloads. I have no idea what causes this different runtime behavior when querying exactly the same server app with exactly the same payload.
As said any ideas that might point me in the right direction greatly appreciated.
Krid

...after searching for a solution for days I finally found the issue. When you have Python and Ruby code combined within a folder that is connected with a virtualenv Python (and activated Python env of course) Ruby apparently has problems resolving the local address space. It works but it takes some strange detours that make the whole thing terribly slow.
My solution: I moved the Ruby code that has to play together with the Python code via HTTP connections in a folder outside of the Python code folder.

Related

Faraday gem not ungzipping response body

I'm experimenting with the UK NHS 'GP Connect' REST API and trying to wrap some of the connection complexity in a more simple Ruby layer.
I've been successfully playing with the API via Postman and the responses are all correctly unzipped to JSON/FHIR. Taking the simplest of the requests from my Postman session and converting it to Ruby using the Faraday HTTP gem, I can make the request/response cycle work fine, but the response is still gzipped.
(I'm using Ruby 2.6.1 and Faraday 0.15.4)
(The API requires 'accept-encoding': 'gzip' to work at all.)
Googling the problem I'm having turns up only Faraday middleware which is intended to be used when not using Net::HTTP (the default adapter, and what I am using)
The whole thing is a single script at this stage so it's easy enough to show. (There are additionally some client certs and a locally installed CA cert which are all working fine and no issues)
***** = I've blanked it to protect the innocent, although the healthcare endpoint I'm using here is a dummy-test with fictional data anyway.
require 'faraday'
identifier = "identifier=https://fhir.nhs.uk/Id/nhs-number|*****"
baseurl = "https://systmoneukdemo1.tpp-uk.com"
headers = {
"authorization": '*****',
"content-type": 'application/json+fhir',
"accept-encoding": 'gzip',
"accept": 'application/json+fhir',
"ssp-from": '*****',
"ssp-interactionid": 'urn:nhs:names:services:gpconnect:fhir:rest:search:patient-1',
"ssp-to": '123456',
"ssp-traceid": '*****',
"cache-control": 'no-cache',
}
connection = Faraday.new(
url: baseurl,
ssl: {
ca_file: './certs/cacert.crt',
client_cert: OpenSSL::X509::Certificate.new(File.read("certs/summer-school-cert.crt")),
client_key: OpenSSL::PKey::RSA.new(File.read("certs/summer-school-cert.key")),
}
)
response = connection.get do |request|
request.url "/SystmOneMHS/NHSConnect/Z12345/STU3/1/Patient?#{identifier}"
request.headers = headers
end
puts response.body
The response when I run ruby script.rb is:
�U�o�0������- |Z��U�nӪjrV�A�T����;;dM�n�4-������{wϽ'
������)9f�W�Һ�b�-+�e��7���2pZ���dPr�5o���,m��\;
`*�:�Ά������r��1Aju�H��셸T�Ϙ��ލ|�?�
߶ih>jy����D����5&=��m��/G��Ñ�'�Ϣ#?��'Q6�~���y��^[.�u��-��#�ȡSǛ^�~�b?|A�#צ�%eu��:
�� ���h�M�4dm��j�WB�l
8�%��f��b��sf����C�>iS[
��C�q$�ɛ���ǣ��h�oF�?�jٙR
�7�gh�h�����w�`��d�:iq�P� ������Ya�z<����k��d5X1����J�WfV��}�y�#�w%��0���
��;à)�!g����(��L�WMm�<xg��L����p�I�������9��]#
��i�N3�nz��z�'�Mf�țz�ڪ�����]"�y�����r'
j����gru��_��u��C,��U�� /
ý"���H����V��0�kT�ĸ��.f��pUNՊI�ݎvs��Z���잹˂�}��g��6Q��v���p<
�-.6z
R�Ğ��6�V�� ��-Pzs�L��Ի�4�_��L��Z�b��g:��;�OF=g{�y�?Ly�+����kk���H�y���
������a��s}��O/�W�=��r��F��%a
however if I pipe that output into gzip:
ruby script.rb | gzip -d
{"resourceType":"Bundle","id":"4D2EAEC0-A8DB-11E9-8B9A-1BA7090BDAD3","meta":{"profile":["https://fhir.nhs.uk/STU3/StructureDefinition/GPConnect-Searchset-Bundle-1"]},"type":"searchset","entry":[{"fullUrl":"Patient/9222500000000000","resource":{"resourceType":"Patient","id":"9222500000000000","meta":{"profile":["https://fhir.nhs.uk/STU3/StructureDefinition/CareConnect-GPC-Patient-1"],"versionId":"220262D942C746E9D8023C82ABC8F500","lastUpdated":"2019-07-17T22:39:05+01:00"},"identifier":[{"system":"https://fhir.nhs.uk/Id/nhs-number","value":"6656336566","extension":[{"url":"https://fhir.nhs.uk/STU3/StructureDefinition/Extension-CareConnect-GPC-NHSNumberVerificationStatus-1","valueCodeableConcept":{
"coding":
[
{
"system":"https://fhir.nhs.uk/STU3/CodeSystem/CareConnect-NHSNumberVerificationStatus-1",
"code":"01",
"display":"Number present and verified"
}
]
}}]}],"name":[{"prefix":["Mr"],"given":["Michael"],"family":"Smith","use":"official"}],"address":[{"use":"home","type":"both","line":["6 Wren Street"],"city":"London","postalCode":"WC1X 0HD"}],"telecom":[{"system":"phone","use":"home","value":"01132252263"},{"system":"email","use":"home","value":"liang.lin#tpp-uk.com"}],"birthDate":"1960-01-19","gender":"male","active":true,"managingOrganization":{"reference":"Organization/bd00000000000000"},"generalPractitioner":[{"reference":"Practitioner/e576600000000000"}],"extension":[{"url":"https://fhir.nhs.uk/STU3/StructureDefinition/Extension-CareConnect-GPC-RegistrationDetails-1","extension":[{"url":"registrationPeriod","valuePeriod":{"start":"2017-01-01T00:00:00Z"}},{"url":"registrationType","valueCodeableConcept":{"coding":[{"system":"https://fhir.nhs.uk/STU3/CodeSystem/CareConnect-RegistrationType-1","code":"O","display":"Other"}]}},{"url":"preferredBranchSurgery","valueReference":{"reference":"Location/bd00000000000000_0100000000000000"}}]}]}}]}
gzip: stdin: unexpected end of file
I get un-gzipped FHIR JSON content, which is what I'm expecting. So everything is working right up until the ungzipping stage, which is not being carried out, and the raw ugzipped content is returning to me as the response.
Somehow it seems that Faraday isn't automatically un-gzipping the response body. I've searched in Faraday's documentation for a flag or other control that I perhaps should be setting, but I can't find a clear explanation of how this should be handled.
Clearly I could un-gzip this output directly, but I would like to know if there is a proper internal 'Faraday' way of doing it. If it's using the Faraday Gzip middleware, then great, but how?
If it helps with working out what's going on, I've reimplemented this in plain Net::HTTP and the results are exactly the same, the reponse body is not being un-gzipped. Both Faraday and Net::HTTP say they do the ungzipping automatically, but neither of them are.
Any thoughts or advice welcome.
Given you are using the faraday-middleware gem, you can just use the :gzip middleware in connection setup.
request.use :gzip
Using your code as example:
response = connection.get do |request|
request.url "/SystmOneMHS/NHSConnect/Z12345/STU3/1/Patient?#{identifier}"
request.use :gzip
request.headers = headers
end

How to read POST data in rack request

When I run the curl command
curl -v -H "Content-type: application/json" -X POST -d '{"name":"abc", "id":"12", "subject":"my subject"}' http://localhost:9292
to send a POST request with data to my Rack application, my code prints out {}. That is coming from puts req.POST() in the code below.
Why does it print out {} instead of the POST data? And how do I correctly access the POST data in my Rack application?
require 'json'
class Greeter
def call(env)
req = Rack::Request.new(env)
if req.post?
puts req.POST()
end
[200, {"Content-Type" => "application/json"}, [{x:"Hello World!"}.to_json]]
end
end
run Greeter.new
From reading the docs for POST, looks like it is giving you parsed data based on other content types. If you want to process "application/json", you probably need to
JSON.parse( req.body.read )
instead. To check this, try
puts req.body.read
where you currently have puts req.POST.
req.body is an I/O object, not a string. See the body documentation and view the source. You can see that this is in fact the same as mudasobwa's answer.
Note that other code in a Rack application may expect to read the same I/O, such as the param parsers in Sinatra or Rails. To ensure that they see the same data and not get an error, you may want to call req.body.rewind, possibly both before and after reading the request body in your code. However, if you are in such a situation, you might instead consider whether your framework has options to process JSON directly via some option on the controller or request content-type handler declaration etc - most likely there will be an option to handle this kind of request within the framework.
Try:
env['rack.input'].read
I found it in "How to receive a JSON object with Rack" and, though it still sounds weird to me, it likely works.
You can try:
req.params
Hope this can help you.

How to send DELETE request with body using ruby gem?

I am communicating with API that requires DELETE request with JSON body. This works on console:
curl -XDELETE http://api.com/endpoint_path/rest_resource
-d '{"items":[{"type":"type1","item_id":"item1"}]}'
It seems that most gems for making HTTP requests don't support DELETE request with body (I tried RestClient and Curb). Is there a way to do it using some Ruby gem (preferably Curb) or Net::HTTP?
Here's one way using HTTParty:
HTTParty.delete("http://api.com/endpoint_path/rest_resource", {
:body => '{"items":[{"type":"type1","item_id":"item1"}]}'
})
it could be used her. This is ORM for api. https://github.com/remiprev/her
Example of usage:
RestResource.destroy_existing(id, body_params)
I also spent some time on this issue and #Casper's answer shed the light.
Seems to me that the key is to pass the body value as JSON string, which is not written in most of the documentations I found.
Here's another example using httpclient
require 'json'
body = { 'items': [{ 'type': 'type1', 'item_id': 'item1' }]}
HTTPClient.new.delete('http://api.com/endpoint_path/rest_resource', body.to_json)

RestClient multipart upload from IO

I am trying to upload data as multipart using RestClient like so:
response = RestClient.post(url, io, {
:cookies => {
'JSESSIONID' => #sessionid
},
:multipart => true,
:content_type => 'multipart/form-data'
})
The io argument is a StringIO that contains my file, so it's from memory instead of from the disk.
The server (Tomcat servlet) is unable to read the multipart data, giving an error:
org.apache.commons.fileupload.FileUploadException: the request was rejected because no multipart boundary was found
So I believe that RestClient is not sending it in multipart format? Anyone see the problem? I am assuming the problem is on the Ruby (client) side, but I can post my servlet (Spring) code if anyone thinks it might be a server-side problem.
I also wonder what RestClient would use for the uploaded filename, since there isn't an actual file... Can you have a multipart request without a filename?
You can do this, it simply requires subclassing StringIO and adding a non-nil path method to it:
class MailIO < StringIO
def path
'message'
end
end
I've just checked this, and the Mailgun api is pretty down with this.
After consulting with the author of the rest-client library (Archiloque), it seems that if this is possible, the API is not set up to handle it easily. Using the :multipart => true parameter will cause the IO to be treated like a file, and it looks for a non-nil #path on the IO, which for a StringIO is always nil.
If anyone needs this in the future, you'll need to consult with the library's mailing list (code#archiloque.net), as the author seems to think it is possible but perhaps not straightforward.
It CAN easily do streaming uploads from an IO as long as it's not multipart format, which is what I ended up settling for.

Parse body of POST reqest in self-made server in Ruby

I am writing mock Ruby servers to test components of API. I send a POST request with a body, and I'd like my mock server to return a body of that POST request. Currently i have this code:
require 'socket'
webserver = TCPServer.new('127.0.0.1', 7125)
loop do
session = webserver.accept
session.print "HTTP/1.1 200/OK\r\nContent-type:text/html\r\n\r\n"
request = session.gets
session.puts request
session.close
end
A POST request with body FOO returns a response with body that contains only POST / HTTP/1.1 How to fix it?
Writing your own HTTP server is really going to get you into trouble because the specification, while superficially simple, has a number of subtle nuances that can trip you up. In this case, you're reading one line with gets and ignoring the bulk of the submission. You're going to have to address that by reading in and properly decoding the posted data.
For something with a familiar interface you might start with Net::HTTP::Server instead.

Resources