Patching classes in Ruby's core lib - ruby

I'm trying to set the timeout for subsequent http calls to a very unreliable API. I tried multiple attempts at using Ruby's built-in Timeout.timeout() method but had had no such luck getting it to extend to sub calls. For example, Timeout.timeout(300) will set the first timeout to 300 but sub calls go back to 60. I added a print of the seconds_delay and here is what I saw:
[16:55:16 miker#laughwhat-lm ~/optisol/src/rails/tools_app/trunk/adhoc/ticket] $ bundle exec ruby buck.rb
300
nil
warning: peer certificate won't be verified in this SSL session
60
60
nil
warning: peer certificate won't be verified in this SSL session
60
Here is the error I receive with full stack trace:
[16:49:50 miker#laughwhat-lm ~/optisol/src/rails/tools_app/trunk/adhoc/ticket] $ bundle exec ruby buck.rb
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
/Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill': execution expired (Timeout::Error)
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/net/protocol.rb:126:in `readline'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:2017:in `read_new'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:1051:in `request_without_fakeweb'
from /Users/miker/.rvm/gems/ree-1.8.7-2011.03/gems/fakeweb-1.3.0/lib/fake_web/ext/net_http.rb:50:in `request'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:845:in `post'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/soap/netHttpClient.rb:93:in `post'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/soap/netHttpClient.rb:116:in `start'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/net/http.rb:543:in `start'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/soap/netHttpClient.rb:115:in `start'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/soap/netHttpClient.rb:92:in `post'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/soap/streamHandler.rb:170:in `send_post'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/soap/streamHandler.rb:109:in `send'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/soap/rpc/proxy.rb:170:in `route'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/soap/rpc/proxy.rb:141:in `call'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/soap/rpc/driver.rb:178:in `call'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/soap/rpc/driver.rb:232:in `getByBuyer'
from buck.rb:9
from /Users/miker/.rvm/gems/ree-1.8.7-2011.03/gems/yieldmanager-0.8.2/lib/yieldmanager/client.rb:131:in `session'
from buck.rb:8
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/timeout.rb:67:in `timeout'
from /Users/miker/.rvm/rubies/ree-1.8.7-2011.03/lib/ruby/1.8/timeout.rb:101:in `timeout'
from buck.rb:6
So I guess my question would be how can I go about patching the protocol.rb BufferedIO's method to look like this:
class BufferedIO
private
def rbuf_fill
puts "working"
timeout(300) { # forced 300 second timeout
#rbuf << #io.sysread(BUFSIZE)
}
end
end
Adding that to my ruby file before or after I do my requires/includes does not have an affect (i.e. no "working" is ever printed out). Hope someone has a solution. Thanks!

Of all the various Ruby library's, the clumsiest to work with and ugliest, in my opinion, would have to be net::http. Have you considered switching to something like this:
https://github.com/dbalatero/typhoeus
The requests in typhoeus occur via lib-curl and so aren't bound by Ruby's not-so-reliable when nested or threaded timeout method.
In terms of the rbuf_fill issue, I'm not sure if that will get you there. If I remember correctly, when a timeout exception fires, it always shows the location that the code is currently at in the stack. That location is only incidental. Take the example below I just ran in irb. Notice how it tells you the timeout occured in "sleep"? Where the timeout is reported is just what it happened to be doing at that moment, not where the timeout code is necessarily implemented, nor do you know which timeout it is that is tripping if there are multiple. I'd have to chase down the rbuf_fill to confirm this for you though, and I have to run at the moment...
irb> timeout(2){sleep 5}
Timeout::Error: execution expired
from (irb):3:in sleep'
from (irb):3:inblock in irb_binding'
from (irb):3
from /home/ebelan

Related

Cannot send emails via AWS-SES and Sidekiq

We are migrating from DelayedJob to Sidekiq. And things seem fine, except, we use Amazon's SES to send emails. I believe the issue is one of threadsafe problems.
Whenever we queue up more than 1 email we get tons of errors like these
WARN EmailJob::perform IOError closed stream #<SendMailQueueItemWorker:0x00007f7fb00b7ae8 #jid="242cafc31a55b6121959103f"> /usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/protocol.rb:176:in `wait_readable'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/protocol.rb:176:in `rbuf_fill'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/protocol.rb:124:in `read'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http/response.rb:291:in `block in read_body_0'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http/response.rb:276:in `inflater'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http/response.rb:281:in `read_body_0'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http/response.rb:202:in `read_body'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:1408:in `block in send_entity'
/mnt/xxx-production/shared/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/net_http/patches.rb:40:in `block in new_transport_request'
Or
EmailJob::perform OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=unknown state: unexpected record #<SendMailQueueItemWorker:0x000055c08033f748 #jid="47a8cdc65878dcaa9e6ad817"> /usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/protocol.rb:44:in `connect_nonblock'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/protocol.rb:44:in `ssl_socket_connect'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:928:in `connect'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:852:in `start'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:1384:in `request'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:1407:in `send_entity'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:1195:in `post'
/mnt/xxx-production/shared/bundle/ruby/2.3.0/bundler/gems/aws-ses-feac6b6b069d/lib/aws/ses/base.rb:165:in `request'
/mnt/xxx-production/shared/bundle/ruby/2.3.0/bundler/gems/aws-ses-feac6b6b069d/lib/aws/ses/send_email.rb:103:in `send_raw_email'
The code is literally identical to that called from inside of DelayedJob for years without issue.
Is there anything we can do here or do we go back to DJ for Amazon-related calls? We make lots of calls to different Amazon services.
Sounds like you are trying to use one global AWS connection in multiple threads. That's the biggest difference between Sidekiq and DJ: your code must be threadsafe. Switch to a connection pool.
https://github.com/mperham/sidekiq/wiki/Advanced-Options#connection-pooling

SSL problems in ruby but not curl

I have the following working curl command:
curl -k -E some_cert.pem https://someurl.com/__dirlist__
Trying to implement this in Ruby I have:
uri = URI.parse('https://someurl.com/__dirlist__')
http_session = Net::HTTP.new(uri.host, uri.port)
http_session.ca_file = "some_cert.pem"
http_session.use_ssl = true
http_session.verify_mode = OpenSSL::SSL::VERIFY_NONE
res = http_session.get(uri.request_uri)
I've played around with using all the different versions of SSL (using http_session.ssl_version = :SSLv2_client etc), all which failed (some with different messages), I matched up versions using wireshark to see what curl was using so don't think that's the problem (although ruby was sending a bunch of extra settings none seemed pertinent).
From reading other bug reports I've seen people have a lot of problems related to not having the appropriate certificates in their cert store however with SSL::VERIFY_NONE I don't see how that could matter.
I can't rule out that it could be the openssl baked into my Ruby but it seems unlikely to me given I've also run this code on another machine and gotten the same error and I would assume curl is linking against the same openssl (I don't know how to check this).
I've looked through the rdocs like I've exhausted all the settings available in Net:HTTP.
This is the nondescript error that I'm seeing (anonymised slightly):
OpenSSL::SSL::SSLError: SSL_read:: ssl handshake failure
from /Users/a_user/.rvm/gems/ruby-1.9.3-p0#project/gems/right_http_connection-1.3.0/lib/net_fix.rb:52:in `sysread'
from /Users/a_user/.rvm/gems/ruby-1.9.3-p0#project/gems/right_http_connection-1.3.0/lib/net_fix.rb:52:in `block in rbuf_fill'
from /Users/a_user/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
from /Users/a_user/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
from /Users/a_user/.rvm/gems/ruby-1.9.3-p0#project/gems/right_http_connection-1.3.0/lib/net_fix.rb:51:in `rbuf_fill'
from /Users/a_user/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
from /Users/a_user/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
from /Users/a_user/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
from /Users/a_user/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
from /Users/a_user/.rvm/gems/ruby-1.9.3-p0#project/gems/right_http_connection-1.3.0/lib/net_fix.rb:146:in `request'
from /Users/a_user/.rvm/gems/ruby-1.9.3-p0#project/gems/right_http_connection-1.3.0/lib/net_fix.rb:131:in `block in request'
from /Users/a_user/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:745:in `start'
from /Users/a_user/.rvm/gems/ruby-1.9.3-p0#project/gems/right_http_connection-1.3.0/lib/net_fix.rb:129:in `request'
from /Users/a_user/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/net/http.rb:1026:in `get'
From the curl manual:
-k, --insecure
(SSL) This option explicitly allows curl to perform "insecure"
SSL connections and transfers. All SSL connections are attempted
to be made secure by using the CA certificate bundle installed
by default. This makes all connections considered "insecure"
fail unless -k, --insecure is used.
So running without -k might reveal the problem on curl as well.

Why ruby process starts getting: "getaddrinfo: Temporary failure in name resolution" on all remote calls before restart

I have a couple of dedicated servers where I have background workers running for days sometimes before restart. When this happens (that they run for more than 24 hours it seems) then I start to get errors like this:
getaddrinfo: Temporary failure in name resolution
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/smtp.rb:540:in `open'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/smtp.rb:540:in `tcp_socket'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/smtp.rb:549:in `block in do_start'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/smtp.rb:549:in `do_start'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/smtp.rb:519:in `start'
The error will occur for ALL remote calls from that specific process, until it has been restarted (NOT hardware/server restart, just the ruby process). I have double-checked/confirmed that restarting the process works as a fix, since I have tried to SSH into the server, and run a new process along side the failing one, and this one would work flawlessly, while the faulty one still failed.
So my question is, why is this happening, and how can I resolve the issue in an other way than restarting the process every day?
I am on ruby 1.9.3 p194 and the background jobs are processed with the sidekiq gem (however I don't suspect that one to be responsible for the mess).
The servers are situated at http://www.hetzner.de (however I don't think they are to blame, since it's happening in-process)
UPDATE
Along side the above smtp call errors, I'm also getting errors like this, when doing simple http requests. This also resolves after a process restart:
# (Errno::EHOSTUNREACH) "No route to host - connect(2)"
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `initialize'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `open'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `block in connect'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `connect'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/http.rb:750:in `start'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:628:in `start'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:888:in `reset'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:567:in `connection_for'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:926:in `request'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:258:in `fetch'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/mechanize-2.5.1/lib/mechanize.rb:407:in `get'

watir-webdriver with firefox 6.0 see following error Errno::ECONNREFUSED

Currently run 150 plus scenarios nightly approximately 5000 steps. I see the following error occur around 10 times in the 5000 steps. Not a lot, nor on the same step, however don't know what to do to fix. Currently wrapping in a rescue block and retrying to work around error.
Any suggestions would be great.
Thanks,
Jim
Environment:
Windows 2003 Server 32 bit
FireFox 6.0.2
Ruby 1.8.7
watir-webdriver 0.3.4
selenium-webdriver 2.7.0
watir-page-helper 0.3.0
Errno::ECONNREFUSED: No connection could be made because the target machine actively refused it. - connect(2)
Stack trace:
G:/Ruby187/lib/ruby/1.8/net/http.rb:560:in `initialize'
G:/Ruby187/lib/ruby/1.8/net/http.rb:560:in `open'
G:/Ruby187/lib/ruby/1.8/net/http.rb:560:in `connect'
G:/Ruby187/lib/ruby/1.8/timeout.rb:53:in `timeout'
G:/Ruby187/lib/ruby/1.8/timeout.rb:101:in `timeout'
G:/Ruby187/lib/ruby/1.8/net/http.rb:560:in `connect'
G:/Ruby187/lib/ruby/1.8/net/http.rb:553:in `do_start'
G:/Ruby187/lib/ruby/1.8/net/http.rb:542:in `start'
G:/Ruby187/lib/ruby/1.8/net/http.rb:1035:in `request'
./features/support/../../lib/pages/base_page_class.rb:37:in `initialize'
./features/support/env.rb:147:in `new'
./features/support/env.rb:147:in `on'
./features/support/env.rb:143:in `visit'
./features/step_definitions/login_steps.rb:32:in `/^A user logs into Connect using (new|existing) rid using correct environment dictated by environment variable$/'
features\ReservationDailyView.feature:6:in `And A user logs into Connect using existing rid using correct environment dictated by environment variable'
One thing to note, I am closing the browser after each scenario and opening it up again at start of the next scenario.
If I leave the browser open instead I get this error and my firefox instance is totally running out of memory 600,000+ K VM Size 700,000+ K
Timeout::Error: execution expired
Stack trace:
G:/Ruby187/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill'
G:/Ruby187/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
G:/Ruby187/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
G:/Ruby187/lib/ruby/1.8/net/protocol.rb:126:in `readline'
G:/Ruby187/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'
G:/Ruby187/lib/ruby/1.8/net/http.rb:2017:in `read_new'
G:/Ruby187/lib/ruby/1.8/net/http.rb:1051:in `request'
G:/Ruby187/lib/ruby/1.8/net/http.rb:1037:in `request'
G:/Ruby187/lib/ruby/1.8/net/http.rb:543:in `start'
G:/Ruby187/lib/ruby/1.8/net/http.rb:1035:in `request'
./features/support/env.rb:148:in `call'
./features/support/env.rb:148:in `on'
Looks like you are running out of ephemeral ports. You might want to change settings in the registry to use more ports. Refer below
http://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx

Ruby networking problem on windows

I am running windows XP with ruby 1.8.6 patchlevel 111. I am using HTTP to connect to a remote server and it has been running fine. All of a sudden it started to through the exception listed below (I did not change any code since the last time I ran it successfully). Does anybody know what is going on?
c:/ruby/lib/ruby/1.8/timeout.rb:54:in `rbuf_fill': execution expired (Timeout::E
rror)
from c:/ruby/lib/ruby/1.8/timeout.rb:56:in `timeout'
from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
from c:/ruby/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
from c:/ruby/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
from c:/ruby/lib/ruby/1.8/net/protocol.rb:126:in `readline'
from c:/ruby/lib/ruby/1.8/net/http.rb:2029:in `read_status_line'
from c:/ruby/lib/ruby/1.8/net/http.rb:2018:in `read_new'
from c:/ruby/lib/ruby/1.8/net/http.rb:1059:in `request'
... 19 levels...
from c:/ruby/lib/ruby/1.8/test/unit/autorunner.rb:216:in `run'
from c:/ruby/lib/ruby/1.8/test/unit/autorunner.rb:12:in `run'
from c:/ruby/lib/ruby/1.8/test/unit.rb:278
from c:/ruby/lib/ruby/gems/1.8/gems/rake-0.8.3/lib/rake/rake_test_loader
.rb:5
rake aborted!
Command failed with status (3): [c:/ruby/bin/ruby -Ilib;test "c:/ruby/lib/r...]
Maybe the remote host is down? Or a new firewall has been put between your machine and the remote host?
"Timeout::Error" usually points to that direction.
besides the obvious (firewall, you got blacklisted for bad user-agent or ignoring robots.txt), you can try curl
http://curl.haxx.se/libcurl/ruby/
OR increase net/http timeout to say, 30+ seconds
http://groups.google.com/group/rubyonrails-talk/msg/cc89e8ae6703d6fb
It could be related to this known Ruby bug where Timeout::Error does not subclass Exception. (fixed in 1.9.2 I believe)
http://lindsaar.net/2007/12/9/rbuf_filltimeout-error
It can be fixed by rescuing from Timeout::Error like rescue Timeout::Error => e

Resources