We are migrating from DelayedJob to Sidekiq. And things seem fine, except, we use Amazon's SES to send emails. I believe the issue is one of threadsafe problems.
Whenever we queue up more than 1 email we get tons of errors like these
WARN EmailJob::perform IOError closed stream #<SendMailQueueItemWorker:0x00007f7fb00b7ae8 #jid="242cafc31a55b6121959103f"> /usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/protocol.rb:176:in `wait_readable'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/protocol.rb:176:in `rbuf_fill'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/protocol.rb:124:in `read'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http/response.rb:291:in `block in read_body_0'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http/response.rb:276:in `inflater'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http/response.rb:281:in `read_body_0'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http/response.rb:202:in `read_body'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:1408:in `block in send_entity'
/mnt/xxx-production/shared/bundle/ruby/2.3.0/gems/aws-sdk-core-3.22.1/lib/seahorse/client/net_http/patches.rb:40:in `block in new_transport_request'
Or
EmailJob::perform OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=unknown state: unexpected record #<SendMailQueueItemWorker:0x000055c08033f748 #jid="47a8cdc65878dcaa9e6ad817"> /usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/protocol.rb:44:in `connect_nonblock'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/protocol.rb:44:in `ssl_socket_connect'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:928:in `connect'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:852:in `start'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:1384:in `request'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:1407:in `send_entity'
/usr/local/rubies/2.3.7/lib/ruby/2.3.0/net/http.rb:1195:in `post'
/mnt/xxx-production/shared/bundle/ruby/2.3.0/bundler/gems/aws-ses-feac6b6b069d/lib/aws/ses/base.rb:165:in `request'
/mnt/xxx-production/shared/bundle/ruby/2.3.0/bundler/gems/aws-ses-feac6b6b069d/lib/aws/ses/send_email.rb:103:in `send_raw_email'
The code is literally identical to that called from inside of DelayedJob for years without issue.
Is there anything we can do here or do we go back to DJ for Amazon-related calls? We make lots of calls to different Amazon services.
Sounds like you are trying to use one global AWS connection in multiple threads. That's the biggest difference between Sidekiq and DJ: your code must be threadsafe. Switch to a connection pool.
https://github.com/mperham/sidekiq/wiki/Advanced-Options#connection-pooling
Related
I am trying to parse an rss feed with Ruby. This is my (working) code, with the actual feed url:
include 'rss'
myfeed = RSS::Parser.parse('http://allmusicnews.altervista.org/blog/feed/', false)
If I run it from my development computer (Mac OS 10.9), all works fine. If I run it from my server (Debian 7), all i get is:
/home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:879:in `initialize': Connection refused - connect(2) for "allmusicnews.altervista.org" port 80 (Errno::ECONNREFUSED)
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:879:in `open'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:879:in `block in connect'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/timeout.rb:74:in `timeout'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:878:in `connect'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:863:in `do_start'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:852:in `start'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:318:in `open_http'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:736:in `buffer_open'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:211:in `block in open_loop'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:209:in `catch'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:209:in `open_loop'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:150:in `open_uri'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:716:in `open'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/open-uri.rb:724:in `read'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rss/parser.rb:103:in `normalize_rss'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rss/parser.rb:89:in `initialize'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rss/parser.rb:76:in `new'
from /home/daniele/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rss/parser.rb:76:in `parse'
This does not happen with other rss feeds. example: feeds.blogo.it/soundsblog/it seems working well on both platforms.
I am using the same Ruby version (2.2.0), installed with RVM on both the computers. Same gems too.
Any clue, at least about what to look for?
ps: it may seem something related to connection timeout, but as I run my script suddenly I see the error message above, there's no idle waiting time...
Please ignore my question. I was shooting to the wrong target.
Actually I'm afraid to have some blacklistish problem with my production machine. I cannot connect to altervista.org server in any way (lynx, telnet), I keep getting "connection refused error", so there's nothing to do with my Ruby code...
I'm following the Jumpstart Labs MicroBlogger tutorial and I'm running into an issue where I'm unable to connect to twitter. I am getting the following errors:
C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:799:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (Twitter::Error::ClientError)
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:799:in `block in connect'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:799:in `connect'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:744:in `start'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:1284:in `request'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/faraday-0.8.8/lib/faraday/adapter/net_http.rb:75:in `perform_request'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/faraday-0.8.8/lib/faraday/adapter/net_http.rb:38:in `call'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/faraday-0.8.8/lib/faraday/response.rb:8:in `call'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/faraday-0.8.8/lib/faraday/response.rb:8:in `call'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/faraday-0.8.8/lib/faraday/response.rb:8:in `call'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/faraday-0.8.8/lib/faraday/request/url_encoded.rb:14:in `call'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/faraday-0.8.8/lib/faraday/request/multipart.rb:13:in `call'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/twitter-4.8.1/lib/twitter/request/multipart_with_file.rb:14:in `call'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/faraday-0.8.8/lib/faraday/connection.rb:253:in `run_request'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/faraday-0.8.8/lib/faraday/connection.rb:118:in `post'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/twitter-4.8.1/lib/twitter/client.rb:108:in `request'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/twitter-4.8.1/lib/twitter/client.rb:72:in `post'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/twitter-4.8.1/lib/twitter/api/utils.rb:82:in `object_from_response'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/twitter-4.8.1/lib/twitter/api/tweets.rb:129:in `update'
from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/jumpstart_auth-0.3.0/lib/jumpstart_auth.rb:32:in `update'
from C:/Users/Steve/RubymineProjects/MicroBlogger/micro_blogger.rb:12:in `tweet'
from C:/Users/Steve/RubymineProjects/MicroBlogger/micro_blogger.rb:18:in `<top (required)>'
from -e:1:in `load'
from -e:1:in `<main>'
This is my code so far:
require 'jumpstart_auth'
class MicroBlogger
attr_reader :client
def initialize
puts "Initializing"
#client = JumpstartAuth.twitter
end
def tweet(message)
#client.update(message)
end
end
blogger = MicroBlogger.new
blogger.tweet("hey")
As indicated in the error message, the SSL library couldn't verify the SSL certificate returned from the Twitter server. In this case, it is most likely because no certificate authorities were installed for you.
The details and solution are available here:
https://gist.github.com/fnichol/867550#file-readme-md
I suggest using the "boring" method, since it does not involve running code copied off the internet (instructions in case the link disappears, but better to read info at link):
Download http://curl.haxx.se/ca/cacert.pem
Save it to somewhere like C:\RailsInstaller\cacert.pem
Run set SSL_CERT_FILE=C:\RailsInstaller\cacert.pem to set the location for the current shell
Run your script in the same shell
If you have installed ruby directly to windows you can copy the cacert.pem file to the Ruby200dir(eg. c:\ruby200) and run the set SSL_CERT_FILE=[path to the cert file] command from the command prompt.
Keep in mind that this will require you to add the SSL cert for every boot of your computer
I have a couple of dedicated servers where I have background workers running for days sometimes before restart. When this happens (that they run for more than 24 hours it seems) then I start to get errors like this:
getaddrinfo: Temporary failure in name resolution
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/smtp.rb:540:in `open'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/smtp.rb:540:in `tcp_socket'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/smtp.rb:549:in `block in do_start'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/smtp.rb:549:in `do_start'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/smtp.rb:519:in `start'
The error will occur for ALL remote calls from that specific process, until it has been restarted (NOT hardware/server restart, just the ruby process). I have double-checked/confirmed that restarting the process works as a fix, since I have tried to SSH into the server, and run a new process along side the failing one, and this one would work flawlessly, while the faulty one still failed.
So my question is, why is this happening, and how can I resolve the issue in an other way than restarting the process every day?
I am on ruby 1.9.3 p194 and the background jobs are processed with the sidekiq gem (however I don't suspect that one to be responsible for the mess).
The servers are situated at http://www.hetzner.de (however I don't think they are to blame, since it's happening in-process)
UPDATE
Along side the above smtp call errors, I'm also getting errors like this, when doing simple http requests. This also resolves after a process restart:
# (Errno::EHOSTUNREACH) "No route to host - connect(2)"
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `initialize'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `open'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `block in connect'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `connect'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
/home/deployer/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/net/http.rb:750:in `start'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:628:in `start'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:888:in `reset'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:567:in `connection_for'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:926:in `request'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:258:in `fetch'
/home/deployer/apps/au/shared/bundle/ruby/1.9.1/gems/mechanize-2.5.1/lib/mechanize.rb:407:in `get'
Currently run 150 plus scenarios nightly approximately 5000 steps. I see the following error occur around 10 times in the 5000 steps. Not a lot, nor on the same step, however don't know what to do to fix. Currently wrapping in a rescue block and retrying to work around error.
Any suggestions would be great.
Thanks,
Jim
Environment:
Windows 2003 Server 32 bit
FireFox 6.0.2
Ruby 1.8.7
watir-webdriver 0.3.4
selenium-webdriver 2.7.0
watir-page-helper 0.3.0
Errno::ECONNREFUSED: No connection could be made because the target machine actively refused it. - connect(2)
Stack trace:
G:/Ruby187/lib/ruby/1.8/net/http.rb:560:in `initialize'
G:/Ruby187/lib/ruby/1.8/net/http.rb:560:in `open'
G:/Ruby187/lib/ruby/1.8/net/http.rb:560:in `connect'
G:/Ruby187/lib/ruby/1.8/timeout.rb:53:in `timeout'
G:/Ruby187/lib/ruby/1.8/timeout.rb:101:in `timeout'
G:/Ruby187/lib/ruby/1.8/net/http.rb:560:in `connect'
G:/Ruby187/lib/ruby/1.8/net/http.rb:553:in `do_start'
G:/Ruby187/lib/ruby/1.8/net/http.rb:542:in `start'
G:/Ruby187/lib/ruby/1.8/net/http.rb:1035:in `request'
./features/support/../../lib/pages/base_page_class.rb:37:in `initialize'
./features/support/env.rb:147:in `new'
./features/support/env.rb:147:in `on'
./features/support/env.rb:143:in `visit'
./features/step_definitions/login_steps.rb:32:in `/^A user logs into Connect using (new|existing) rid using correct environment dictated by environment variable$/'
features\ReservationDailyView.feature:6:in `And A user logs into Connect using existing rid using correct environment dictated by environment variable'
One thing to note, I am closing the browser after each scenario and opening it up again at start of the next scenario.
If I leave the browser open instead I get this error and my firefox instance is totally running out of memory 600,000+ K VM Size 700,000+ K
Timeout::Error: execution expired
Stack trace:
G:/Ruby187/lib/ruby/1.8/timeout.rb:64:in `rbuf_fill'
G:/Ruby187/lib/ruby/1.8/net/protocol.rb:134:in `rbuf_fill'
G:/Ruby187/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
G:/Ruby187/lib/ruby/1.8/net/protocol.rb:126:in `readline'
G:/Ruby187/lib/ruby/1.8/net/http.rb:2028:in `read_status_line'
G:/Ruby187/lib/ruby/1.8/net/http.rb:2017:in `read_new'
G:/Ruby187/lib/ruby/1.8/net/http.rb:1051:in `request'
G:/Ruby187/lib/ruby/1.8/net/http.rb:1037:in `request'
G:/Ruby187/lib/ruby/1.8/net/http.rb:543:in `start'
G:/Ruby187/lib/ruby/1.8/net/http.rb:1035:in `request'
./features/support/env.rb:148:in `call'
./features/support/env.rb:148:in `on'
Looks like you are running out of ephemeral ports. You might want to change settings in the registry to use more ports. Refer below
http://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx
I was wondering if somebody has seen this error before?
C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:675:in `initialize': The requested address is not valid in its context. - connect(2) (://:0) (Errno::EADDRNOTAVAIL)
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:675:in `new'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:675:in `create_socket'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:632:in `connect'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/timeout.rb:128:in `timeout'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:631:in `connect'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:522:in `query'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient/session.rb:147:in `query'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:953:in `do_get_block'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:765:in `do_request'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:848:in `protect_keep_alive_disconnected'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:764:in `do_request'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:666:in `request'
from C:/Ruby/lib/ruby/gems/1.8/gems/httpclient-2.1.5.2/lib/httpclient.rb:596:in `post'
from C:/Ruby/lib/ruby/gems/1.8/gems/soap4r-1.5.8/lib/soap/streamHandler.rb:238:in `send_post'
from C:/Ruby/lib/ruby/gems/1.8/gems/soap4r-1.5.8/lib/soap/streamHandler.rb:172:in `send'
from C:/Ruby/lib/ruby/gems/1.8/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb:179:in `route'
from C:/Ruby/lib/ruby/gems/1.8/gems/soap4r-1.5.8/lib/soap/rpc/proxy.rb:143:in `call'
from C:/Ruby/lib/ruby/gems/1.8/gems/soap4r-1.5.8/lib/soap/rpc/driver.rb:181:in `call'
from (eval):6:in `preRepairAuthorizationQA'
from C:/documents and settings/ngorbikoff/Desktop/GMW/WSDL/ProcessMessageClient.rb:21
I'm trying to connect to a service, I just generated this soap client from wsdl2ruby. Everything went fine. So I have no idea why this error is happening. This is a fresh install of ruby 1.8.7 on Windows, but I tested this on another Windows machine with Ruby 1.8.6 and on a Debian server with REE 1.8.7 - same error. My gut feeling is that it has to do with the httpclient lib - but I can't find anything on google - regarding this err, other than some references to Pythong and tcl - which seem to be unrelated. Also I'm trying to connect to wsdl service that is on httpS - but I didn't have this problem before and there were no changes on the server.
Does anyone have any insight?
OK nevermind people. For anyone else looking into this cryptic message if you are using wsdl2ruby - make sure that you define your endpoint_url in the WhateverServiceClient.rb file generated by the wsdl3ruby.