I have a ruby program which listens to a redis channel:
module Listener
class << self
def listen
redis.subscribe "messaging" do |on|
on.message do |_, msg|
Notify.about(msg)
end
end
end
def redis
#redis ||= Redis.new(driver: :hiredis)
end
end
end
Every time I deploy the app I restart the process with
kill -15 listener-pid
But Airbrake notifies me about the SignalException: SIGTERM with the following backtrace
/gems/hiredis-0.6.1/lib/hiredis/ext/connection.rb:19 in read
/gems/hiredis-0.6.1/lib/hiredis/ext/connection.rb:19 in read
/gems/redis-3.3.3/lib/redis/connection/hiredis.rb:54 in read
/gems/redis-3.3.3/lib/redis/client.rb:262 in block in read
/gems/redis-3.3.3/lib/redis/client.rb:250 in io
/gems/redis-3.3.3/lib/redis/client.rb:261 in read
/gems/redis-3.3.3/lib/redis/client.rb:136 in block (3 levels) in call_loop
/gems/redis-3.3.3/lib/redis/client.rb:135 in loop
/gems/redis-3.3.3/lib/redis/client.rb:135 in block (2 levels) in call_loop
/gems/redis-3.3.3/lib/redis/client.rb:231 in block (2 levels) in process
/gems/redis-3.3.3/lib/redis/client.rb:367 in ensure_connected
/gems/redis-3.3.3/lib/redis/client.rb:221 in block in process
/gems/redis-3.3.3/lib/redis/client.rb:306 in logging
/gems/redis-3.3.3/lib/redis/client.rb:220 in process
/gems/redis-3.3.3/lib/redis/client.rb:134 in block in call_loop
/gems/redis-3.3.3/lib/redis/client.rb:280 in with_socket_timeout
/gems/redis-3.3.3/lib/redis/client.rb:133 in call_loop
/gems/redis-3.3.3/lib/redis/subscribe.rb:43 in subscription
/gems/redis-3.3.3/lib/redis/subscribe.rb:12 in subscribe
/gems/redis-3.3.3/lib/redis.rb:2765 in _subscription
/gems/redis-3.3.3/lib/redis.rb:2143 in block in subscribe
/gems/redis-3.3.3/lib/redis.rb:58 in block in synchronize
/usr/lib/ruby/2.4.0/monitor.rb:214 in mon_synchronize
/gems/redis-3.3.3/lib/redis.rb:58 in synchronize
/gems/redis-3.3.3/lib/redis.rb:2142 in subscribe
Is it possible to restart the listener process gracefully so I wont receive SIGTERM errors?
I found a pubsub example in redis-rb
After I added trap('SIGTERM') { exit } the problem was fixed
Now my listener class looks like this:
module Listener
class << self
def listen
trap('SIGTERM') { exit }
redis.subscribe "messaging" do |on|
on.message do |_, msg|
Notify.about(msg)
end
end
end
def redis
#redis ||= Redis.new(driver: :hiredis)
end
end
end
Related
I have an activemq topic subscriber in ruby which uses stomp protocol with failover to connect to the broker and if somehow the activemq gets restarted then sometimes i get an exception :
undefined method `command' for nil:NilClass
/usr/lib64/ruby/gems/1.8/gems/stomp-1.1.8/lib/stomp/client.rb:295:in `start_listeners'
/usr/lib64/ruby/gems/1.8/gems/stomp-1.1.8/lib/stomp/client.rb:108:in `join'
/usr/lib64/ruby/gems/1.8/gems/stomp-1.1.8/lib/stomp/client.rb:108:in `join'
./lib/active_mq_topic_reader.rb:31:in `active_mq_topic_reader'
main.rb:164
main.rb:163:in `initialize'
main.rb:163:in `new'
but i get this exception only when i use join() method on the broker thread, otherwise no exception appears and the subscriber get unsubscribed from the topic.
The problem which i am facing is i have a different mechanism of shutting down the process by sending shutdown signal, and till then the process waits, but if we use join() then the process will get stuck on this line and i will not be able to close the method by shutdown signal.So what should i do to catch the exception and restart the listener thread?
active_mq_topic_reader.rb :
require 'rubygems'
require 'ffi-rzmq'
require 'msgpack'
require 'zmq_helper'
require 'stomp'
include ZmqHelper
def active_mq_topic_reader(context, notice_agg_fe_url, signal_pub_url, monitor_url, active_mq_broker, topic)
begin
sender = create_connect_socket(context, ZMQ::PUSH, notice_agg_fe_url)
monitor = create_connect_socket(context, ZMQ::PUSH, monitor_url)
active_mq_broker.subscribe(topic, {}) do |msg|
notice = {}
["entity_id","entity_type","system_name","change_type"].each do |key|
notice[key] = msg.headers[key].to_s
end
monitor.send_string("qreader-#{topic.slice(7..-1)}")
sender.send_string(notice.to_msgpack)
end
active_mq_broker.join() #cannot use this
signal_subscriber.recv_string() #here the code waits for shutdown signal in case of process shutdown
sender.close()
monitor.close()
signal_subscriber.close()
active_mq_broker.unsubscribe(topic)
return
rescue Exception => e
puts "#{topic}: #{e}"
puts e.backtrace
$stdout.flush
end
end
main.rb :
context = ZMQ::Context.new(1)
active_mq_broker_audit = Stomp::Client.new("failover:(stomp://localhost:61613,stomp://localhost:61613)")
new_thread = Thread.new do
active_mq_topic_reader(context,
"inproc://notice_agg_fe",
"inproc://signal_pub",
"tcp://localhost:xxxx",
active_mq_broker_audit,
"/topic/myTopic")
end
new_thread.join()
I have been searching the whole day for socket accept non blocking. I found recv non blocking but that wouldn't benefit me in anyway. My script first starts a new socket class. It binds to the client with ip 127.0.0.1 and port 6112. Then it starts multi threading. Multi threading takes #sock.accept. << That is blocking. I have then used accept_nonblock. Though, that would throw me the following error:
IO::EWOULDBLOCKWaitReadable : A non-blocking socket operation could not be completed immediately. - accept(2) would block
I am using Ruby 2.2.
NOTE: I do not intend to use Rails to solve my problem, or give me a shortcut. I am sticking with pure Ruby 2.2.
Here is my script:
require 'socket'
include Socket::Constants
#sock = Socket.new(AF_INET, SOCK_STREAM, 0)
#sockaddr = Socket.sockaddr_in(6112, '127.0.0.1')
#sock.bind(#sockaddr)
#sock.listen(5)
Thread.new(#sock.accept_nonblock) do |connection|
#client = Client.new(ip, connection, self)
#clients.push(#client)
begin
while connection
packet = connection.recv(55555)
if packet == nil
DeleteClient(connection)
else
#toput = "[RECV]: #{packet}"
puts #toput
end
end
rescue Exception => e
if e.class != IOError
line1 = e.backtrace[0].split(".rb").last
line = line1.split(":")[1]
#Log.Error(e, e.class, e.backtrace[0].split(".rb").first + ".rb",line)
puts "#{ e } (#{ e.class })"
end
end
def DeleteClient(connection)
#clients.delete(#client)
connection.close
end
http://docs.ruby-lang.org/en/2.2.0/Socket.html#method-i-accept_nonblock
accept_nonblock raises an exception when it can't immediately accept a connection. You are expected to rescue this exception and then IO.select the socket.
begin # emulate blocking accept
client_socket, client_addrinfo = socket.accept_nonblock
rescue IO::WaitReadable, Errno::EINTR
IO.select([socket])
retry
end
A patch has recently been accepted which will add an exception: false option to accept_nonblock, which will allow you to use it without using exceptions for flow control. I don't know that it's shipped yet, though.
I'm going on a limb here, and posting a large chunk of code.
I hope it will answer both your question and the any related questions others reading this answer might raise.
I'm sorry if I went overboard, I just thought it was almost all relevant.
Issues like looping through an event stack, using IO.select to push events in a non-block manner and other performance issues are all related (in my opinion) to the nonblocking concept of socket programming.
So i'm posting a ruby module which acts as a server with a reactor, using a limited number of threads, rather than thousands of threads, each per connection (12 threads will give you better performance than a hundred). The reactor utilizes the IO.select method with a timeout once all it's active events are handled.
The module can set up multiple listening sockets which use #accept_nonblock, and they all currently act as an echo server.
It's basically the same code I used for the Plezi framework's core... with some stripped down functionality.
The following is a thread-pool with 12 working threads + the main thread (which will sleep and wait for the "TERM" signal)...
...And it's an example of an accept_nonblock with exception handling and a thread pool.
It's a simple socket echo server, test it as a remote client using telnet:
> telnet localhost 3000
Hi!
# => Hi!
bye
#=> will disconnect
here's the code - Good Luck!!!
require 'socket'
module SmallServer
module_function
####
# Replace this method with your actual server logic.
#
# this code will be called when a socket recieves data.
#
# For now, we will just echo.
def got_data io, io_params
begin
got = io.recv_nonblock( 1048576 ) # with maximum number of bytes to read at a time...
puts "echoing: #{got}"
if got.match /^(exit|bye|q)\R/
puts 'closing connection.'
io.puts "bye bye!"
remove_connection io
else
io.puts "echoing: #{got}"
end
rescue => e
# should also log error
remove_connection io
end
end
#########
# main loop and activation code
#
# This will create a thread pool and set them running.
def start
# prepare threads
exit_flag = false
max_threads = 12
threads = []
thread_cycle = Proc.new do
io_review rescue false
true while fire_event
end
(max_threads).times { Thread.new { thread_cycle.call until exit_flag } }
# set signal tarps
trap('INT'){ exit_flag = true; raise "close!" }
trap('TERM'){ exit_flag = true; raise "close!" }
puts "Services running. Press ^C to stop"
# sleep until trap raises exception (cycling might cause the main thread to loose signals that might be caught inside rescue clauses)
(sleep unless SERVICES.empty?) rescue true
# start shutdown.
exit_flag = true
# set fallback tarps
trap('INT'){ puts 'Forced exit.'; Kernel.exit }
trap('TERM'){ puts 'Forced exit.'; Kernel.exit }
puts 'Started shutdown process. Press ^C to force quit.'
# shut down listening sockets
stop_services
# disconnect active connections
stop_connections
# cycle down threads
puts "Waiting for workers to cycle down"
threads.each {|t| t.join if t.alive?}
# rundown any active events
thread_cycle.call
end
#######################
## Events (Callbacks) / Multi-tasking Platform
EVENTS = []
E_LOCKER = Mutex.new
# returns true if there are any unhandled events
def events?
E_LOCKER.synchronize {!EVENTS.empty?}
end
# pushes an event to the event's stack
# if a block is passed along, it will be used as a callback: the block will be called with the values returned by the handler's `call` method.
def push_event handler, *args, &block
if block
E_LOCKER.synchronize {EVENTS << [(Proc.new {|a| push_event block, handler.call(*a)} ), args]}
else
E_LOCKER.synchronize {EVENTS << [handler, args]}
end
end
# Runs the block asynchronously by pushing it as an event to the event's stack
#
def run_async *args, &block
E_LOCKER.synchronize {EVENTS << [ block, args ]} if block
!block.nil?
end
# creates an asynchronous call to a method, with an optional callback (shortcut)
def callback object, method, *args, &block
push_event object.method(method), *args, &block
end
# event handling FIFO
def fire_event
event = E_LOCKER.synchronize {EVENTS.shift}
return false unless event
begin
event[0].call(*event[1])
rescue OpenSSL::SSL::SSLError => e
puts "SSL Bump - SSL Certificate refused?"
rescue Exception => e
raise if e.is_a?(SignalException) || e.is_a?(SystemExit)
error e
end
true
end
#####
# Reactor
#
# IO review code will review the connections and sockets
# it will accept new connections and react to socket input
IO_LOCKER = Mutex.new
def io_review
IO_LOCKER.synchronize do
return false unless EVENTS.empty?
united = SERVICES.keys + IO_CONNECTION_DIC.keys
return false if united.empty?
io_r = (IO.select(united, nil, united, 0.1) )
if io_r
io_r[0].each do |io|
if SERVICES[io]
begin
callback self, :add_connection, io.accept_nonblock, SERVICES[io]
rescue Errno::EWOULDBLOCK => e
rescue => e
# log
end
elsif IO_CONNECTION_DIC[io]
callback(self, :got_data, io, IO_CONNECTION_DIC[io] )
else
puts "what?!"
remove_connection(io)
SERVICES.delete(io)
end
end
io_r[2].each { |io| (remove_connection(io) || SERVICES.delete(io)).close rescue true }
end
end
callback self, :clear_connections
true
end
#######################
# IO - listening sockets (services)
SERVICES = {}
S_LOCKER = Mutex.new
def add_service port = 3000, parameters = {}
parameters[:port] ||= port
parameters.update port if port.is_a?(Hash)
service = TCPServer.new(parameters[:port])
S_LOCKER.synchronize {SERVICES[service] = parameters}
callback Kernel, :puts, "Started listening on port #{port}."
true
end
def stop_services
puts 'Stopping services'
S_LOCKER.synchronize {SERVICES.each {|s, p| (s.close rescue true); puts "Stoped listening on port #{p[:port]}"}; SERVICES.clear }
end
#####################
# IO - Active connections handling
IO_CONNECTION_DIC = {}
C_LOCKER = Mutex.new
def stop_connections
C_LOCKER.synchronize {IO_CONNECTION_DIC.each {|io, params| io.close rescue true} ; IO_CONNECTION_DIC.clear}
end
def add_connection io, more_data
C_LOCKER.synchronize {IO_CONNECTION_DIC[io] = more_data} if io
end
def remove_connection io
C_LOCKER.synchronize { IO_CONNECTION_DIC.delete io; io.close rescue true }
end
# clears closed connections from the stack
def clear_connections
C_LOCKER.synchronize { IO_CONNECTION_DIC.delete_if {|c| c.closed? } }
end
end
start the echo server in irb with:
SmallServer.add_service(3000) ; SmallServer.start
I am using AirBnb Nerve service. It's service code looks like this:
require 'logger'
require 'json'
require 'timeout'
require 'nerve/version'
require 'nerve/utils'
require 'nerve/log'
require 'nerve/ring_buffer'
require 'nerve/reporter'
require 'nerve/service_watcher'
module Nerve
class Nerve
include Logging
def initialize(opts={})
log.info 'nerve: starting up!'
# set global variable for exit signal
$EXIT = false
...some code...
# Any exceptions in the watcher threads should wake the main thread so
# that we can fail fast.
Thread.abort_on_exception = true
log.debug 'nerve: completed init'
end
def run
log.info 'nerve: starting run'
#services.each do |name, config|
launch_watcher(name, config)
end
begin
sleep
rescue StandardError => e
log.error "nerve: encountered unexpected exception #{e.inspect} in main thread"
raise e
ensure
$EXIT = true
log.warn 'nerve: reaping all watchers'
#watchers.each do |name, watcher_thread|
reap_watcher(name)
end
end
log.info 'nerve: exiting'
ensure
$EXIT = true
end
def launch_watcher(name, config)
... some code ...
end
def reap_watcher(name)
... some code ...
end
end
end
I do not see any stop method. What is the right way of stopping such a service? I am using JRuby and intend to write a JSVC adapter for this service.
There is no way to do this via the current API, short of sending it a signal.
If sending a signal isn't going to work and you want to handle stop explicitly, it looks like you will need to change the following things:
Add a #stop method to Nerve that sets $EXIT = true.
Modify #run so that rather than sleeping forever (sleep) it wakes up and checks $EXIT.
My script gets elements from redis database end doing some work with it. I need to be sure that if script finishes with ^C or other Signal, the element will be returned back in the database.
I'm trying to do it
require "redis"
class Test
attr_reader :quit
def initialize
#redis = Redis.new
end
def trap_signal
trap("INT") {
puts "get ready to exit"
#redis.rpush "TestQueue", #elem # I need to be sure that #emelent always puts back in the database
#quit = true}
end
def run!
trap_signal
#elem = "test string"
#redis.rpush "TestQueue", #elem
while !quit
#redis.blpop "TestQueue", #elem
# Do some work whith #element
sleep 1
# And put it back in the database
#redis.rpush "TestQueue", #elem
end
end
end
Test.new.run!
but get this error
^Cget ready to exit
/usr/lib/ruby/2.1.0/monitor.rb:185:in `lock': can't be called from trap context (ThreadError)
from /usr/lib/ruby/2.1.0/monitor.rb:185:in `mon_enter'
from /usr/lib/ruby/2.1.0/monitor.rb:209:in `mon_synchronize'
from /home/kusayu/.gem/ruby/2.1.0/gems/redis-3.2.0/lib/redis.rb:37:in `synchronize'
from /home/kusayu/.gem/ruby/2.1.0/gems/redis-3.2.0/lib/redis.rb:991:in `rpush'
from test.rb:13:in `block in trap_signal'
from test.rb:24:in `call'
from test.rb:24:in `sleep'
from test.rb:24:in `run!'
from test.rb:32:in `<main>'
Your code already works properly, just remove the #redis.rpush from the signal handler.
You shouldn't run "heavy" operations in the signal handler (and you get an exception because of it anyway). It's much better to use a variable like #quit = true to signal the main loop that it's time to finish, and then let the main loop handle the proper cleanup.
So if you just remove the #redis.rpush from your INT signal handler, then you will ensure that the element is returned back into the database because the main loop will only finish once #quit is true.
I've got a Ruby TCPSocket client that works great except when I'm trying to close it. When I call the disconnect method in my code below, I get this error:
./smartlinc.rb:70:in `start_listen': stream closed (IOError)
from ./smartlinc.rb:132:in `initialize'
from ./smartlinc.rb:132:in `new'
from ./smartlinc.rb:132:in `start_listen'
from bot.rb:45:in `initialize'
from bot.rb:223:in `new'
from bot.rb:223
Here's the (simplified) code:
class Smartlinc
def initialize
#socket = TCPSocket.new(HOST, PORT)
end
def disconnect
#socket.close
end
def start_listen
# Listen on a background thread
th = Thread.new do
Thread.current.abort_on_exception = true
# Listen for Ctrl-C and disconnect socket gracefully.
Kernel.trap('INT') do
self.disconnect
exit
end
while true
ready = IO.select([#socket])
readable = ready[0]
readable.each do |soc|
if soc == #socket
buf = #socket.recv_nonblock(1024)
if buf.length == 0
puts "The socket connection is dead. Exiting."
exit
else
puts "Received Message"
end
end
end # end each
end # end while
end # end thread
end # end message callback
end
Is there a way I can prevent or catch this error? I'm no expert in socket programming (obviously!), so all help is appreciated.
Your thread is sitting in IO.select() while the trap code happily slams the door in its face with #socket.close, hence you get some complaining.
Don't set abort_on_exception to true, or then handle the exception properly in your code:
Something along these lines...
Kernel.trap('INT') do
#interrupted = true
disconnect
exit
end
...
ready = nil
begin
ready = IO.select(...)
rescue IOError
if #interrupted
puts "Interrupted, we're outta here..."
exit
end
# Else it was a genuine IOError caused by something else, so propagate it up..
raise
end
...