I am using Typhoeus with Hydra in order to make parallel requests . my end goal is to parse the typhoeus response into mechanize object.
url = "http://example.com/"
hydra = Typhoeus::Hydra.new
agent = Mechanize.new
request = Typhoeus::Request.new(url, :method => :get, :proxy => "#{proxy_host}:#{proxy_port}")
request.on_complete do |response| #Typhoeus::response object
body = response.body
uri = request.parsed_uri
page = agent.parse(uri, response, body)
end
hydra.queue(request)
hydra.run
the agent.parse method is giving me error because it cannot parse the typhoeus response object
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5.1/lib/mechanize.rb:1165:in `parse': undefined method `[]' for #<Typhoeus::Response:0x00000012cd9da0> (NoMethodError)
Is there anyway i can convert Typhoeus response into Net::HTTPResponse object ?
Or is there any other way I can club Mechanize and Typhoeus together? So that, I can make parallel requests with typhoeus and scrape the data with Mechanize library.
I tried to create a Net::HTTPResponse(https://github.com/ruby/ruby/blob/trunk/lib/net/http/response.rb) from a Typhoeus::Response, but it didn't work out. Calling the initializer is easy, but setting the response body or headers not.
I looked into mechanize to see if it can be changed to use Typhoeus for making requests but I don't think thats possible right now. Net/http is really hard-wired into mechanize. I thought of a mechanize-typhoeus adapter, which would be nice.
Related
I'm make a little program in sinatra and I'm wanted to perfom some dynamic call of post, with diynamic uri, so I make a Connexion class like this:
class Connexion
def initialize(path)
#path = path
end
def sinatraPost
post "/#{#path}" do
# some code
end
end
end
But when I'm launch sinatraPost, I've got this error:
undefined method `post' for #<Connexion:0x000000026206b8> (NoMethodError)
How can I call the sinatra post method in my class ?
EDIT: Okay ! So, I change my strategy, I have this following code:
class Webhook < Sinatra::Base
get '/:name' do
# compare with names array
end
end
Webhook.run!
Thank's to everyone !
It looks like you're going about this the wrong way. If you want to set up your app to receive a POST request, you'll need routing logic in your controller. Sinatra controllers normally look like this:
require 'sinatra'
get '/route1' do
# do stuff
end
post '/route2' do
# do stuff
end
If you're using a modular app, you'll want to have your app inherit from Sinatra::Base. See the Sinatra docs for more.
Making a post request is different, and doesn't rely on Sinatra methods.
require 'net/http'
uri = URI("http://google.com")
headers = {}
http = Net::HTTP.new(uri.host, uri.port)
request = Net::HTTP::Post.new(uri.request_uri, headers)
response = http.request(request)
Or something like that. Good luck!
Im using Net::HTTP in my ruby code to make http requests. For example to make a post request i do
require 'net/http'
Net::HTTP.post_form(url,{'email' => email,'password' => password})
This works. But im unable to make a delete request, i.e.
require 'net/http'
Net::HTTP::Delete(url)
gives the following error
NoMethodError: undefined method `Delete' for Net::HTTP:Class
The documentation at http://ruby-doc.org/stdlib-1.9.3/libdoc/net/http/rdoc/Net/HTTP.html shows Delete is available. So why is it not working in my case ?
Thank You
The documentation tells you that Net::HTTP::Delete is a class, not a method.
Try Net::HTTP.new('www.server.com').delete('/path') instead.
uri = URI('http://localhost:8080/customer/johndoe')
http = Net::HTTP.new(uri.host, uri.port)
req = Net::HTTP::Delete.new(uri.path)
res = http.request(req)
puts "deleted #{res}"
Simple post and delete requests, see docs for more:
puts Net::HTTP.new("httpbin.org").post("/post", "a=1").body
puts Net::HTTP.new("httpbin.org").delete("/delete").body
This works for me:
uri = URI(YOUR_URL)
req = Net::HTTP::Delete.new(uri, {}) # params on second place
response = Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
http.request req
end
I'm using the Savon gem to make a SOAP request using code similar to what's below. It's working, but I would like to view/capture the request XML without actually making a call to their server. I can view it now after a request is made by sticking a debugger line after the request and inspecting the client variable.
Does anyone know of a way to view the request XML without actually making a request? I want to be able to validate the XML against a schema using Cucumber or Rspec.
client = Savon::Client.new do |wsdl, http|
wsdl.document = "http://fakesite.org/fake.asmx?wsdl"
end
client.request(:testpostdata, :xmlns => "http://fakesite.org/") do
soap.header = { :cAuthentication => {"UserName" => "MyName", "Password" => "MyPassword" } }
soap.body = { :xml_data => to_xml }
end
Using Savon 2 I do it this way, write a method that return the request body from the client.
client = Savon::Client.new(....)
this is not mentioned in the documentation
def get_request
# list of operations can be found using client.operations
ops = client.operation(:action_name_here)
# build the body of the xml inside the message here
ops.build(message: { id: 42, name: "Test User", age: 20 }).to_s
end
You can directly via the Savon::Client#build_request method.
Example:
request = client.build_request(:some_operation, some_payload)
request.body # Get the request body
request.headers # Get the request headers
Take a peak # https://github.com/savonrb/savon/blob/master/lib/savon/request.rb for the full doc.
I am using Savon 2.11 and I can accomplish it with globals in the client:
def client
#client ||= Savon.client(soap_version: 2,
wsdl: config.wsdl,
logger: Rails.logger,
log: true)
end
More info on the globals here.
Then the logger spits out the host, the http verb and the complete xml ("headers" and body) for both request and response.
While I'm sure there's a better way to do this, I just overrode response.
class Savon::SOAP::Request
def response
pp self.request.headers
puts
puts self.request.body
exit
end
end
They've updated the API since the last post. Set this setting in Savon.client: :pretty_print_xml => true. After your call, search the logs for SOAP request:. The output is put to stdout. Check the console console history if you're testing your connection from the console.
Savon uses HTTPI to execute SOAP requests. HTTPI is a common interface on top of various Ruby HTTP clients. You could probably mock/stub the HTTP request executed by Savon via:
HTTPI.expects(:post).with do |http|
SchemaValidation.validate(:get_user, http.body)
end
Please note that I used Mocha for mocking the SOAP request, getting the HTTP body and validating it against some validation method (pseudo-code).
Currently, Savon does not support building up requests without executing them. So the only way to validate the request would be to intercept it.
If you would need Savon to support this feature, please let me know and open a ticket over at GitHub.
EDIT: There's also savon_spec, which is a little helper for basic fixture-based testing with Savon.
I had the same issue and patched Savon as follows:
module Savon
class Client
def get_request_xml operation_name, locals
Savon::Builder.new(operation_name, #wsdl, #globals, locals).pretty
end
end
end
This builds the XML and returns it as a string without sending it to the API endpoint. It doesn't accept a block argument in the same way client.call does, so it won't be able to reproduce every type of request you're making, but it meets my needs for now.
Could someone tell me how I can fetch (GET) a URL (with params) using Ruby? I found a bunch of examples online but I couldn't find one that explained how I can also pass the parameters.
require 'net/http'
require 'uri'
uri = URI.parse("http://www.example.com/?test=1")
response = Net::HTTP.get_response uri
p response.body
There are also some other good HTTP clients or wrappers, such as HTTParty.
require 'rubygems'
require 'httparty'
response = HTTParty.get("http://www.example.com/?test=1")
p response.body
I use something like the following, it's pretty simple and doesn't make you build your own query string:
require 'net/http'
response = nil
Net::HTTP.start "example.com", 80 do |http|
request = Net::HTTP::Get.new "/endpoint"
request.form_data = {:q => "123"}
response = http.request(request)
end
I missed this one. The solutions are here.
Parametrized get request in Ruby?
I would like to take information from another website. Therefore (maybe) I should make a request to that website (in my case a HTTP GET request) and receive the response.
How can I make this in Ruby on Rails?
If it is possible, is it a correct approach to use in my controllers?
You can use Ruby's Net::HTTP class:
require 'net/http'
url = URI.parse('http://www.example.com/index.html')
req = Net::HTTP::Get.new(url.to_s)
res = Net::HTTP.start(url.host, url.port) {|http|
http.request(req)
}
puts res.body
Net::HTTP is built into Ruby, but let's face it, often it's easier not to use its cumbersome 1980s style and try a higher level alternative:
HTTP Gem
HTTParty
RestClient
Excon
Feedjira (RSS only)
OpenURI is the best; it's as simple as
require 'open-uri'
response = open('http://example.com').read
require 'net/http'
result = Net::HTTP.get(URI.parse('http://www.example.com/about.html'))
# or
result = Net::HTTP.get(URI.parse('http://www.example.com'), '/about.html')
I prefer httpclient over Net::HTTP.
client = HTTPClient.new
puts client.get_content('http://www.example.com/index.html')
HTTParty is a good choice if you're making a class that's a client for a service. It's a convenient mixin that gives you 90% of what you need. See how short the Google and Twitter clients are in the examples.
And to answer your second question: no, I wouldn't put this functionality in a controller--I'd use a model instead if possible to encapsulate the particulars (perhaps using HTTParty) and simply call it from the controller.
Here is the code that works if you are making a REST api call behind a proxy:
require "uri"
require 'net/http'
proxy_host = '<proxy addr>'
proxy_port = '<proxy_port>'
proxy_user = '<username>'
proxy_pass = '<password>'
uri = URI.parse("https://saucelabs.com:80/rest/v1/users/<username>")
proxy = Net::HTTP::Proxy(proxy_host, proxy_port, proxy_user, proxy_pass)
req = Net::HTTP::Get.new(uri.path)
req.basic_auth(<sauce_username>,<sauce_password>)
result = proxy.start(uri.host,uri.port) do |http|
http.request(req)
end
puts result.body
My favorite two ways to grab the contents of URLs are either OpenURI or Typhoeus.
OpenURI because it's everywhere, and Typhoeus because it's very flexible and powerful.