I am trying to debug a request that I am making to a server, but I can't figure out what is wrong with it, as it seems to be just like the request that I am putting through on RESTClient.
I am initializing it like so:
request = Net::HTTP::Post.new(url.to_s)
request.add_field "HeaderKey", "HeaderValue"
request.body = requestBody
and then I am executing it like so:
Net::HTTP.start(url.host, url.port) do |http|
response = http.request(request)
end
The requestBody is a string that is encoded with Base64.encode64.
Is there a way to output the request to see exactly where it's going and with what contents? I've used Paros for checking my iOS connections and I can also output a description of requests from most platforms I've worked with, but I can't figure it out for Ruby.
I've found that HTTP Scoop works pretty well for grabbing network traffic (disclaimer - it's not free!)
Related
When doing web scraping with Nokogiri I occasionally get the following error message
undefined method `at_css' for nil:NilClass (NoMethodError)
I know that the selected element is present at some time, but the site is sometimes a bit slow to respond, and I guess this is the reason why I'm getting the error.
Is there some way to wait until a certain selector is present before proceeding with the script?
My current http request block looks like this
url = URL
body = BODY
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.read_timeout = 200 # default 60 seconds
http.open_timeout = 200 # default nil
http.use_ssl = true
request = Net::HTTP::Post.new(uri.request_uri)
request.body = body
request["Content-Type"] = "application/x-www-form-urlencoded"
begin
response = http.request(request)
doc = Nokogiri::HTML(response.body)
rescue
sleep 100
retry
end
While you can use a streaming Net::HTTP like #Stefan says in his comment, and an associated handler that includes Nokogiri, you can't parse a partial HTTP document using a DOM model, which is Nokogiri's default, because it expects the full document also.
You could use Nokogiri's SAX parser, but that's an entirely different programming style.
If you're retrieving an entire page, then use OpenURI instead of the lower-level Net::HTTP. It automatically handles a number of things that Net::HTTP will not do by default, such as redirection, which makes it a lot easier to retrieve pages and will greatly simplify your code.
I suspect the problem is either that the site is timing out, or the tag you're trying to find is dynamically loaded after the real page loads.
If it's timing out you'll need to increase your wait time.
If it's dynamically loading that markup, you can request the main page, locate the appropriate URL for the dynamic content and load it separately. Once you have it, you can either insert it into the first page if you need everything, or just parse it separately.
So when I list the images using the Ruby SDK, I get all of the publicly available ones, but the ones that I have created myself are not included. They do show up in the web console though... I've even tried using the REST API and constructed a Net:HTTP object as illustrated here. I get a 5xx error after setting the content-length (even though it isn't listed as a required header) to anything, including 0... I have had success using the same code on other azure RESTful urls, so I am unsure as to why this specific one is giving me an error....
Does anyone have any clue as to why my images aren't listed? Any experience with the endpoint linked above? Just fyi, heres my ruby request code:
# HTTP request code
def get(uri)
uri = URI.parse(uri)
pem = File.read('/path/to/management_cert')
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.cert = OpenSSL::X509::Certificate.new(pem)
http.key = OpenSSL::PKey::RSA.new(pem)
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
request = Net::HTTP::Get.new(uri.request_uri)
request['x-ms-version'] = '2014-06-01'
request['Content-Length'] = 0
http.request(request)
end
Here is the calling code:
# The invoking line
get 'https://management.core.windows.net/<subscription-id>/services/vmimages'
???
You must go through the API first. Here is link of rest API http://msdn.microsoft.com/en-us/library/azure/dn499770.aspx
You are making POST request instead of GET. The Method should be 'GET'
request = Net::HTTP::Get.new(uri.request_uri)
You have to set Content-Length
I found the answer (kinda).... I guess the servers were having an issue that day as I re-ran the code and got the data I needed... The above code (now fixed) works!!!
I am trying to make fetch a PDF file that gets generated on-demand behind an auth wall. Based on my testing, the flow is as follows:
I make a GET request with several parameters (including auth credentials) to the appropriate page. That page validates my credentials and then processes my request. When the request is finished processing (nearly instantly), I am sent a 302 response that redirects me to the location of the generated PDF. This PDF can then only be accessed by that session.
Using a browser, there's really nothing strange that happens. I attempted to do the same via curl and wget without any optional parameters, but those both failed. I was able to get curl working by adding -L -b /tmp/cookie.txt as options, though (to follow redirects and store cookies).
According to the ruby-doc, using Net::HTTP.start should get me close to what I want. After playing around with it, I was indeed fairly close. I believe the only issue, however, was that my Set-Cookie values were different between requests, even though they were using the same http object in the same start block.
I tried keeping it as simple as possible and then expanding once I got the results I was looking for:
url = URI.parse("http://dev.example.com:8888/path/to/page.jsp?option1=test1&option2=test2&username=user1&password=password1")
Net::HTTP.start(url.host, url.port) do |http|
# Request the first URL
first_req = Net::HTTP::Get.new url
first_res = http.request first_req
# Grab the 302 redirect location (it will always be relative like "../servlet/sendfile/result/543675843657843965743895642865273847328.pdf")
redirect_loc = URI.parse(first_res['Location']
# Request the PDF
second_req = Net::HTTP::Get.new redirect_loc
second_res = http.request first_req
end
I also attempted to use http.get instead of creating a new request each time, but still no luck.
The problem is with cookie: it should be passed within the second request. Smth like:
second_req = Net::HTTP::Get.new(uri.path, {'Cookie' => first_req['Set-Cookie']})
I want to access the Kippt API through Ruby without the usage of any external libraries whatsoever, i.e. everything that comes packed with Ruby is fine, but nothing else (except for the standard library).
How should I go about doing this? Please detail the process.
This is very basic access, showing it is possible:
require "net/https"
require "uri"
uri = URI.parse( 'https://kippt.com/api/users/1/' )
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
http.verify_mode = OpenSSL::SSL::VERIFY_NONE
request = Net::HTTP::Get.new(uri.request_uri)
response = http.request(request)
data = JSON.parse( response.body )
=> {
"username"=>"jorilallo",
"bio"=>"Co-founder of Kippt. I love building products.",
"app_url"=>"/jorilallo",
"avatar_url"=>"https://d19weqihs4yh5u.cloudfront.net/avatars/147d86b9-0830-49d8-a449-0421a6a4bf05/160x160",
"twitter"=>"jorilallo",
"id"=>1, "github"=>"jorde",
"website_url"=>"http://about.me/jorilallo",
"full_name"=>"Jori Lallo",
"dribbble"=>"jorilallo",
"counts"=>{"follows"=>1192, "followed_by"=>23628},
"is_pro"=>true, "resource_uri"=>"/api/users/1/"
}
There is a fair amount of work to take this demonstration and put it into some re-usable code that copes with authentication, posting params, request failure and other standard issues for HTTP-based APIs.
I'd suggest reading http://www.rubyinside.com/nethttp-cheat-sheet-2940.html for some examples of how to build and process the requests in more detail. That's how I did the above (until writing the answer, I'd never used Ruby's net/http directly before, and I just grabbed a likely looking block of code from that site).
I am writing mock Ruby servers to test components of API. I send a POST request with a body, and I'd like my mock server to return a body of that POST request. Currently i have this code:
require 'socket'
webserver = TCPServer.new('127.0.0.1', 7125)
loop do
session = webserver.accept
session.print "HTTP/1.1 200/OK\r\nContent-type:text/html\r\n\r\n"
request = session.gets
session.puts request
session.close
end
A POST request with body FOO returns a response with body that contains only POST / HTTP/1.1 How to fix it?
Writing your own HTTP server is really going to get you into trouble because the specification, while superficially simple, has a number of subtle nuances that can trip you up. In this case, you're reading one line with gets and ignoring the bulk of the submission. You're going to have to address that by reading in and properly decoding the posted data.
For something with a familiar interface you might start with Net::HTTP::Server instead.