Profiling Sinatra with rack-perftools_profiler (using net/http) - ruby

One of my Sinatra actions performs another request using net/http and computes something based on the response:
api_response = Net::HTTP.get_response(uri)
It works like a charm, but when I try to profile this:
configure do
use ::Rack::PerftoolsProfiler, default_printer: 'pdf', frequency: 4000, mode: :walltime
end
(and put ?profile=true into the params of the request I get this:
Errno::EADDRINUSE - Address already in use - connect(2):
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:762:in `initialize'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:762:in `open'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:762:in `block in connect'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:762:in `connect'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:744:in `start'
/Users/tomek/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/net/http.rb:454:in `get_response'
Any idea what's happening?

I try to setup these the same way as you.
require 'sinatra'
require 'rack/perftools_profiler'
require 'net/http'
configure do
use ::Rack::PerftoolsProfiler, default_printer: 'pdf', frequency: 4000, mode: :walltime
end
get "/" do
uri = URI('http://google.com')
api_response = Net::HTTP.get_response(uri)
end
And it run with out a problem on ruby 1.9.3-p286 and with rack-perftools_profiler (0.6.1).
Two recommendations:
update your ruby version
update your gems
If these don't help check which program has bound your port with sudo netstat -nlp
If these all don't help please tell us more about your setup.

Related

Ruby and Https: A socket operation was attempted to an unreachable network

I'm trying to download all of my class notes from coursera. I figured that since I'm learning ruby this would be a good practice exercise, downloading all the PDFs they have for future use. Unfortunately though, I'm getting an exception saying ruby can't connect for some reason. Here is my code:
require 'net/http'
module Coursera
class Downloader
attr_accessor :page_url
attr_accessor :destination_directory
attr_accessor :cookie
def initialize(page_url,dest,cookie)
#page_url=page_url
#destination_directory = dest
#cookie=cookie
end
def download
puts #page_url
request = Net::HTTP::Get.new(#page_url)
puts #cookie.encoding
request['Cookie']=#cookie
# the line below is where the exception is thrown
res = Net::HTTP.start(#page_url.hostname, use_ssl=true,#page_url.port) {|http|
http.request(request)
}
html_page = res.body
pattern = /http[^\"]+\.pdf/
i=0
while (match = pattern.match(html_page,i)) != nil do
# 0 is the entire string.
url_string = match[0]
# make sure that 'i' is updated
i = match.begin(0)+1
# we want just the name of the file.
j = url_string.rindex("/")
filename = url_string[j+1..url_string.length]
destination = #destination_directory+"\\"+filename
# I want to download that resource to that file.
uri = URI(url_string)
res = Net::HTTP.get_response(uri)
# write that body to the file
f=File.new(destination,mode="w")
f.print(res.body)
end
end
end
end
page_url_string = 'https://class.coursera.org/datasci-002/lecture'
puts page_url_string.encoding
dest='C:\\Users\\michael\\training material\\data_science'
page_url=URI(page_url_string)
# I copied this from my browsers developer tools, I'm omitting it since
# it's long and has my session key in it
cookie="..."
downloader = Coursera::Downloader.new(page_url,dest,cookie)
downloader.download
At runtime the following is written to console:
Fast Debugger (ruby-debug-ide 0.4.22, debase 0.0.9) listens on 127.0.0.1:65485
UTF-8
https://class.coursera.org/datasci-002/lecture
UTF-8
Uncaught exception: A socket operation was attempted to an unreachable network. - connect(2)
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `initialize'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `open'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `block in connect'
C:/Ruby200-x64/lib/ruby/2.0.0/timeout.rb:52:in `timeout'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:877:in `connect'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:862:in `do_start'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:851:in `start'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:582:in `start'
C:/Users/michael/Documents/Aptana Studio 3 Workspace/practice/CourseraDownloader.rb:20:in `download'
C:/Users/michael/Documents/Aptana Studio 3 Workspace/practice/CourseraDownloader.rb:52:in `<top (required)>'
C:/Ruby200-x64/bin/rdebug-ide:23:in `load'
C:/Ruby200-x64/bin/rdebug-ide:23:in `<main>'
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `initialize': A socket operation was attempted to an unreachable network. - connect(2) (Errno::ENETUNREACH)
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `open'
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `block in connect'
from C:/Ruby200-x64/lib/ruby/2.0.0/timeout.rb:52:in `timeout'
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:877:in `connect'
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:862:in `do_start'
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:851:in `start'
from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:582:in `start'
from C:/Users/michael/Documents/Aptana Studio 3 Workspace/practice/CourseraDownloader.rb:20:in `download'
from C:/Users/michael/Documents/Aptana Studio 3 Workspace/practice/CourseraDownloader.rb:52:in `<top (required)>'
from C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ruby-debug-ide-0.4.22/lib/ruby-debug-ide.rb:86:in `debug_load'
from C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ruby-debug-ide-0.4.22/lib/ruby-debug-ide.rb:86:in `debug_program'
from C:/Ruby200-x64/lib/ruby/gems/2.0.0/gems/ruby-debug-ide-0.4.22/bin/rdebug-ide:110:in `<top (required)>'
from C:/Ruby200-x64/bin/rdebug-ide:23:in `load'
from C:/Ruby200-x64/bin/rdebug-ide:23:in `<main>'
I was following instructions here to write all the HTTP code. As far as I can see I'm following them ver-batim.
I'm using Windows 7, ruby 2.0.0p481, and Aptana Studio 3. When I copy the url into my browser it goes straight to the page without a problem. When I look at the request headers in my browser for that url, I don't see anything else I think I'm missing. I also tried setting the Host and Referer request headers, it made no difference.
I am out of ideas, and have already searched Stack Overflow for similar questions but that didn't help. Please let me know what I'm missing.
So, I had this same error message with a different project and the problem was that my machine literally couldn't connect to the IP / Port. Have you tried connecting with curl? If it works in your browser, it could be using a proxy or something to actually get there. Testing the URL with curl solved the problem for me.

100% unending CPU usage fetching SSL page without cert

Using Mechanize 2.6.0 on Ruby 1.9.3 I'm trying to fetch a web page over HTTPS from Windows 7x64. When I attempt to get() the URL the CPU usage goes to 100% and the method never returns:
require 'mechanize'
uri = "https://my.com/wiki/api.php?action=query&titles=US4&prop=info&format=xml"
agent = Mechanize.new
u,p = %w[myusername mypassword]
agent.add_auth( uri, u, p )
agent.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE
info = agent.get( uri )
When I interrupt it, I get these stack traces (three different runs):
>> info = agent.get( page_api )
IRB::Abort: abort then interrupt!
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:27:in `call'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:27:in `parse'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:716:in `response_authenticate'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:306:in `fetch'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize.rb:431:in `get'
from (irb):10
from C:/Ruby193/bin/irb:12:in `<main>'
>> info = agent.get( page_api )
IRB::Abort: abort then interrupt!
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:29:in `call'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:29:in `new'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:29:in `parse'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:716:in `response_authenticate'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:306:in `fetch'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize.rb:431:in `get'
from (irb):11
from C:/Ruby193/bin/irb:12:in `<main>'
>> info = agent.get( page_api )
IRB::Abort: abort then interrupt!
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:114:in `call'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:114:in `token'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/www_authenticate_parser.rb:31:in `parse'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:716:in `response_authenticate'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize/http/agent.rb:306:in `fetch'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/mechanize-2.6.0/lib/mechanize.rb:431:in `get'
from (irb):12
from C:/Ruby193/bin/irb:12:in `<main>'
How can I work around this problem and properly fetch an HTTPS URL via Ruby on Windows? (If there's a better solution than Mechanize for this—since I only need the source of the page to feed to Nokogiri anyhow—I'm open to not using Mechanize at all.)
Another datapoint: trying the same code on OS X produces the same result.
Here's the actual content of the page, using the alternative fetching method described in my workaround answer below:
p fetch_https_without_ssl_verification(uri, u, p)
#=> "\t\t <?xml version=\"1.0\"?><api><query><normalized><n from=\"Devtools/UI_Composer/DesignSpec/US7294\" to=\"Devtools/UI Composer/DesignSpec/US7294\" /></normalized><pages><page ns=\"0\" title=\"Devtools/UI Composer/DesignSpec/US7294\" missing=\"\" /></pages></query></api>"
If you simply need the contents of the URL (as I do) then using curl instead of Mechanize is far easier, and it works:
def fetch_https_without_ssl_verification( uri, user=nil, pass=nil )
`curl -s -k #{%Q{-u "#{user}#{":"<<pass if pass}"} if user} "#{uri}"`
end

Need to implement Watirgrid,

I want to implement watirgrid, but I'm not able to do that, every time I'm getting errors related with controller and provider starting process, Also all the example over internet, none of them are working.
Could any one please help me to implement this, a full running example with steps will be a great help.
I'm trying this code:
require 'rubygems'
require 'watirgrid'
require 'watir'
require 'watir-webdriver'
# setup a controller on port 12351 for your new grid
controller = Controller.new(:ring_server_port => 12351, :loglevel => Logger::ERROR)
controller.start
# add a provider to your grid
# :browser_type => 'webdriver' if using webdriver or
# :browser_type => 'ie' if using watir...
provider = Provider.new(:controller_uri => 'druby://127.0.0.1:11235',
:ring_server_port => 12351,
:loglevel => Logger::ERROR,
:browser_type => 'webdriver')
provider.start
# connect to the grid and take all providers from it (this time only one)
grid = Watir::Grid.new(:ring_server_port => 12351, :ring_server_host => '127.0.0.1')
grid.start(:take_all => true)
# for each provider on the grid, launch a new thread to start multiple browsers
threads = []
grid.browsers.each do |browser|
threads << Thread.new do
p browser[:hostname]
p browser[:architecture]
p browser[:browser_type]
# in this case we are starting a new IE browser
b = browser[:object].new_browser(:ie)
b.goto("http://www.google.com")
b.text_field(:name, 'q').set("watirgrid")
b.button(:name, "btnI").click
end
end
threads.each {|thread| thread.join}
And Errors I'm getting is
DRb::DRbConnError: druby://127.0.0.1:11235 - #<Errno::ECONNREFUSED: No connection could
be made because the target machine actively refused it. - connect(2)>
from C:/Ruby192/lib/ruby/1.9.1/drb/drb.rb:736:in `rescue in block in open'
from C:/Ruby192/lib/ruby/1.9.1/drb/drb.rb:730:in `block in open'
from C:/Ruby192/lib/ruby/1.9.1/drb/drb.rb:729:in `each'
from C:/Ruby192/lib/ruby/1.9.1/drb/drb.rb:729:in `open'
from C:/Ruby192/lib/ruby/1.9.1/drb/drb.rb:1191:in `initialize'
from C:/Ruby192/lib/ruby/1.9.1/drb/drb.rb:1171:in `new'
from C:/Ruby192/lib/ruby/1.9.1/drb/drb.rb:1171:in `open'
from C:/Ruby192/lib/ruby/1.9.1/drb/drb.rb:1087:in `block in method_missing'
from C:/Ruby192/lib/ruby/1.9.1/drb/drb.rb:1105:in `with_friend'
from C:/Ruby192/lib/ruby/1.9.1/drb/drb.rb:1086:in `method_missing'
from C:/Ruby192/lib/ruby/gems/1.9.1/gems/watirgrid-1.1.5/lib/provider.rb:141:in `start'
from (irb):44
from C:/Ruby192/bin/irb:12:in `<main>'
irb(main):045:0>
I got the same problem, I got the provider started successfully by changing the
controller_uri
to
controller_uri => 'druby://machineIPAddress:11235'

Sinatra, Mongoid, Heroku, MongoHQ: connecting to Mongodb

Trying to get Mongoid up and running with Sinatra on Heroku (MongoHQ). Previous experience with Rails but first time with the stack and Sinatra.
Started with one of the simple examples on the web (app.rb):
require 'rubygems'
require 'sinatra'
require 'mongo'
require 'mongoid'
configure do
Mongoid.load!('mongoid.yml')
Mongoid.configure do |config|
if ENV['MONGOHQ_URL']
conn = Mongo::Connection.from_uri(ENV['MONGOHQ_URL'])
uri = URI.parse(ENV['MONGOHQ_URL'])
# problem happens here
config.master = conn.db(uri.path.gsub(/^\//, ''))
else
config.master = Mongo::Connection.from_uri("mongodb://localhost:27017").db('test')
end
end
end
# Models
class Counter
include Mongoid::Document
field :count, :type => Integer
def self.increment
c = first || new({:count => 0})
c.inc(:count, 1)
c.save
c.count
end
end
# Controllers
get '/' do
"Hello visitor n" + Counter.increment.to_s
end
For reference, mongoid.yml looks like:
development:
sessions:
default:
database: localhost
production:
sessions:
default:
uri: <%= ENV['MONGOHQ_URL'] %>
As per app.rb (# problem happens here), my logs say:
/app/app.rb:15:in `block (2 levels) in <top (required)>': undefined method `master=' for Mongoid::Config:Module (NoMethodError)
from /app/vendor/bundle/ruby/1.9.1/gems/mongoid-3.0.3/lib/mongoid.rb:112:in `configure'
from /app/app.rb:11:in `block in <top (required)>'
from /app/vendor/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1273:in `configure'
from /app/app.rb:8:in `<top (required)>'
I have also tried variants, including:
config.master = Mongo::Connection.from_uri(ENV['MONGOHQ_URL']).db('appXXXXX')
Mongoid.database = Mongo::Connection.from_uri(ENV['MONGOHQ_URL']).db('appXXXXXXX')
But get the same error:
undefined method `master` for Mongoid::Config:Module (NoMethodError)
or:
undefined method `database=` for Mongoid::Config:Module (NoMethodError)
What am I missing?
Shouldn't be
configure do
Mongoid.load!('mongoid.yml')
end
enough?
That's what the mongid docs are saying. The MONGOHQ_URL environment variable already contains every information to initialize the connection to the db.
So was using Mongoid 3.x ... which:
Doesn't use 10gen driver: uses Moped
Doesn't use config.master
The canonical sample code above which is all over the web works out of the box with Mongoid 2.x so have dropped back to that for the time being.
Thanks!

Get request to twitter API using Ruby, REST

I try to learn REST in Ruby using Twitter API.
According https://dev.twitter.com/docs/api/1/get/trends I have to write GET request to http://api.twitter.com/1/trends.json.
My Ruby code is:
require 'rubygems'
require 'rest-client'
require 'json'
url = 'http://api.twitter.com/1/trends.json'
response = RestClient.get(url)
puts response.body
But i'm getting next errors:
/home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient /abstract_response.rb:48:in `return!': 404 Resource Not Found (RestClient::ResourceNotFound)
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit'
from /home/danik/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:745:in `start'
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit'
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
from /home/danik/.rvm/gems/ruby-1.9.3-p194/gems/rest-client-1.6.7/lib/restclient.rb:68:in `get'
from TwitterTrends.rb:5:in `<main>'
What is wrong?
You are getting that error because the resource you are trying to fetch with http://api.twitter.com/1/trends.json does not exist, as is explained in this doc trends docs
This method is deprecated and has been replaced by GET trends/:woeid.
Please update your applications with the new endpoint.
You want to fetch a URL like this https://api.twitter.com/1/trends/1.json. So, in your code, try doing this:
require 'rubygems'
require 'rest-client'
require 'json'
url = 'https://api.twitter.com/1/trends/1.json'
response = RestClient.get(url)
puts response.body
And you should get a response.

Resources