I have multiple clients connecting on UDP. Their addresses are stored for future communications. If one dies and message is sent to it then recvfrom(2) gives me econnreset.
Is there a way to know which peer died at this point?
the scenario looks like this
def broadcast message
# one of the clients is dead
clients.each { |x| #socket.send message, 0, x.ip, x.port }
end
def fill_inbox
#inbox << #socket.recvfrom(1000) while IO.select([#socket],nil,nil,0)
rescue Errno::ECONNRESET => e
# who died?
end
Related
I'm new both to IMAP and multi-thread programming, and I'd like to write a script to fetch incoming mails and process them in parallel.
Thread, Queue, Mutex and Monitor are new concepts to me, I may miss-use them in the following question and example.
The script's goal is to:
receive IDLE notifications from the IMAP server when a new mail arrives
fetch the corresponding mail
parse its body
generate a PDF from parsed data
send the report through SMTP
Another aproach could be to fetch and identify mails based on their UID, thought learning IMAP & multi-thread programing is another goal.
Here's what I've got so far (IMAP part):
def self.idle(imap)
#imap = Net::IMAP.new 'mail.company.org', 143, false # Not using TLS for test purposes
#imap.login 'username', 'password'
#idler_listener = Thread.new do
loop do
begin
imap.examine 'INBOX'
imap.idle do |res|
if res.kind_of?(Net::IMAP::UntaggedResponse) and res.name == 'EXISTS'
imap.idle_done
Thread.new { process_email(imap) }
end
end
rescue => e
puts e.inspect
end
end
end.join
end
def self.process_email(imap)
begin
uid = imap.uid_search(['SUBJECT', 'MyFavoriteSubject']).last
mail = imap.uid_fetch(uid, 'BODY[TEXT]')[0].attr['BODY[TEXT]']
puts mail
rescue => e
puts e.inspect
end
end
This example successfully prints out the body of an incoming mail. However, if several mail arrives at the same time, only one will be treated.
Q:
Is this behavior due to the fact that the loop is executed in a single thread ?
Does it illustrate the need for a queue, a thread pool, or a producer / consumer pattern ?
Chances that this script will attempt to access the same resource at the same time are low, however, would Mutex.new.synchronize protect me against race conditions & deadlocks ?
(Eventhough I've not fully understand it, I think it's worth sharing this great resource from Masatoshi Seki: https://www.druby.org/sidruby/)
I am working on a project where I have implemented a TCP client server for a device communication. In order to send a command from the server to the client, I am building a command that the device understands and sending to it but the response is not what should be returned
while 1
Thread.start(#otd.accept) do |client|
loop do
command_to_send ="<R-2,3,4>"
client.puts command_to_send
puts "Command #{command_to_send}sent"
#sleep 2
response = **client.gets** # here it halts and never puts the the next statement.
puts "Reponse #{response}"
end # end of nested loop
client.close
end #END OF THREAD.
end #end of while loop
Can someone tell me what I am missing?
Do not use gets as it expects '\n' to be a delimiter of the message.
Instead use: recv here is a method that could help you:
def read(timeout = 2, buffer = 1024)
message = ''
begin
Timeout::timeout(timeout) do
buffer = client.recv(buffer)
message += buffer
end
rescue Timeout::Error
puts "Received nothing from client: #{client.__id__}"
message = ''
rescue Exception => e
raise "Client failed to read for reason - #{e.message}"
end
message
end
You do not have to use sleep anymore as recv like gets is blocking. But the timeout makes sure you are not stuck waiting for a response not existing.
I have a 2 part question on reading from sockets and how is it managed on Ruby servers like Unicorn or Mongrel
I've learnt that to read from a socket is different from reading a file and that there are no distinct EOF message sent and the data is an endless stream. So how do you know when to stop reading? My TCPServer for example in this case when I hit my server by accessing http://localhost:9799 from a browser, it hangs after there is no more data to read and it won't throw the EOFError either.
require 'socket'
READ_CHUNK = 1024
socket = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM)
addr = Socket.pack_sockaddr_in(9799, '127.0.0.1')
socket.bind(addr)
socket.listen(Socket::SOMAXCONN)
socket.setsockopt(:SOCKET, :REUSEADDR, true)
puts "Server is listening on port = 9799"
loop do
connection, addr_info = socket.accept
data_buffer = ""
loop do
begin
connection.read_nonblock(READ_CHUNK, data_buffer)
puts "Buffer = #{data_buffer}"
rescue Errno::EAGAIN => e
IO.select([connection])
retry
rescue EOFError
break
end
end
connection.write("HTTP/1.1 200 \r\n")
connection.write("Content-Type: text/html\r\n")
connection.write("Status 200 \r\n")
connection.write("Connection: close \r\n")
connection.write("Hello World \r\n")
connection.close
end
I'd like to know whats the best practice/standard approach used by Ruby servers. I see the Unicorn uses read_nonblock from kgio library and mongrel uses readpartial (I'm not sure about these but going through the code this is what I feel is the approach adopted.) Even with checks for \r\n how does the server know the input is complete.
Could explain how this should be done (and I think gets is not the approach - its with read, readpartial, read_nonblock).
2). I would really appreciate a few lines on how this is achieved in servers like unicorn or passenger
Thank you.
It's done in unicorn here
https://github.com/defunkt/unicorn/blob/master/lib/unicorn/http_request.rb#L69-L71
There is add_parse method(read the comments above methods)
https://github.com/defunkt/unicorn/blob/master/ext/unicorn_http/unicorn_http.rl#L760-L778
Also take a look at some explanations here http://www.ruby-forum.com/topic/2267632#1014288
Here is your working code using http_parser.rb https://gist.github.com/4136962
gem install http_parser.rb
require 'socket'
require "http/parser"
READ_CHUNK = 1024 * 4
socket = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM)
addr = Socket.pack_sockaddr_in(9799, '127.0.0.1')
socket.bind(addr)
socket.listen(Socket::SOMAXCONN)
socket.setsockopt(:SOCKET, :REUSEADDR, true)
puts "Server is listening on port = 9799"
loop do
connection, addr_info = socket.accept
parser = Http::Parser.new
begin
data = connection.readpartial(READ_CHUNK)
puts "Buffer = #{data}"
parser << data
end until parser.headers
connection.write("HTTP/1.1 200 \r\n")
connection.write("Content-Type: text/html\r\n")
connection.write("Status 200 \r\n")
connection.write("Connection: close \r\n")
connection.write("\r\n\r\n")
connection.write("Hello World \r\n")
connection.close
end
I have a circumstance where my server may close TCPServer and restart, saving all the users to a file, and immediately reloading them; their connections do not sever.
The problem is I can't seem to reinitialize their streams.
When we restart (and attempt to maintain connections), I reinitialize TCPServer, and load my array of connected users – Since these each have an existing socket address, stored as <TCPSocket:0x00000000000000>, can I reinitialize these addresses with TCPServer?
Normally, each user connects and is accepted:
$nCS = TCPServer.new(HOST, PORT)
begin
while socket = $nCS.accept
Thread.new( socket ) do |sock|
begin
d = User.new(sock)
while sock.gets
szIn = $_.chomp
DBG( "Received '" + szIn + "' from Client " + sock.to_s )
d.parseInput( szIn )
end
rescue => e
$stdout.puts "ERROR: Caught error in Client Thread: #{e} \r\n #{e.backtrace.to_s.gsub(",", ",\r\n")}"
sock.write("Sorry, an error has occurred, and you have been disconnected."+EOL+"Please try again later."+EOL)
d.closeConnection
end
end
end
rescue => e
$stdout.puts "ERROR: Caught error in Server Thread: #{e} \r\n #{e.backtrace.to_s.gsub(",", ",\r\n")}"
exit
end
To give it a command to hot reboot, we use exec('./main --copyover') to flag that a copy over is occurring.
If $connected holds an array of all users, and each user has a socket, how do I reinitialize the socket that was open before the restart (assuming the other end is still connected)?
I suspect that using exec("./main", "--copyover", *$nCS, *$connected) is getting me closer, since this simply replaces the process, and should maintain the files (not close them).
You can't. The socket is only valid for the lifetime of the process: it is closed by the OS when the process exits. That in turn invalidates the connection, so the other end is not still connected.
How to Hot-Reboot a TCPServer in Ruby
Hot-Rebooting (aka Copyover) is a process by which an administrator can reload the application (along with any new changes made since last boot) without losing the client connections. This is useful in managing customer expectations as the application does not need to suffer severe downtime and disruption if in use.
What I propose below may not be the best practice, but it's functioning and perhaps will guide others to a similar solution.
The Command
I use a particular style of coding that makes use of command tables to find functions and their accessibility. All command functions are prefixed with cmd. I'll clean up the miscellany to improve readability:
def cmdCopyover
#$nCS is the TCPServer object
#$connected holds an array of all users sockets
#--copyover flags that this is a hot reboot.
connected_args = $connected.map do |sock|
sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
sock.fileno.to_s
end.join(",")
exec('./main.rb', '--copyover', $nCS.fileno.to_s, connected_args)
end
What we're passing are strings; $nCS.fileno.to_s provides us the file descriptor of the main TCPServer object, while connected_args is a comma-delineated list of file descriptors for each user connected. When we restart, ARGV will be an array holding each argument:
ARGV[0] == "--copyover"
ARGV[1] == "5" (Or whatever the file descriptor for TCPServer was)
ARGV[2] == "6,7,8,9" (Example, assuming 4 connected users)
What To Expect When You're Expecting (a Copyover)
Under normal circumstances, we may have a basic server (in main.rb that looks something like this:
puts "Starting Server"
$connected = Array.new
$nCS = TCPServer.new("127.0.0.1",9999)
begin
while socket = $nCS.accept
# NB: Move this loop to its own function, threadLoop()
Thread.new( socket ) do |sock|
begin
while sock.gets
szIn = $_.chomp
#do something with input.
end
rescue => e
puts "ERROR: Caught error in Client Thread: #{e}"
puts #{e.backtrace.to_s.gsub(",", ",\r\n")}"
sock.write("Sorry, an error has occurred, and you have been disconnected."+EOL+"Please try again later."+EOL)
sock.close
end
end
end
rescue => e
puts "Error: Caught Error in Server Thread: #{e}"
puts "#{e.backtrace.to_s.gsub(",", ",\r\n")}"
exit
end
We want to move that main loop to its own function to make it accessible -- our reconnecting users will need to be reinserted in the loop.
So let's get main.rb ready for accepting a hot reboot:
def threadLoop( socket )
Thread.new( socket ) do |sock|
begin
while sock.gets
szIn = $_.chomp
#do something with input.
end
rescue => e
puts "ERROR: Caught error in Client Thread: #{e}"
puts #{e.backtrace.to_s.gsub(",", ",\r\n")}"
sock.write("Sorry, an error has occurred, and you have been disconnected."+EOL+"Please try again later."+EOL)
sock.close
end
end
end
puts "Starting Server"
$connected = Array.new
if ARGV[0] == '--copyover'
$nCS = TCPServer.for_fd( ARGV[1].to_i )
$nCS.close_on_exec = false if $nCS.respond_to?(:close_on_exec=)
connected_args = ARGV[2]
connected_args.split(/,/).map do |sockfd|
$connected << sockfd
$connected.each {|c| threadLoop( c ) }
else
$nCS = TCPServer.new("127.0.0.1",9999)
$nCS.close_on_exec = false if $nCS.respond_to?(:close_on_exec=)
end
begin
while socket = $nCS.accept
threadLoop( socket )
end
rescue => e
puts "Error: Caught Error in Server Thread: #{e}"
puts "#{e.backtrace.to_s.gsub(",", ",\r\n")}"
exit
end
Caveat
My actual usage was a lot more ridiculously complicated, so I did my best to strip out all the garbage; however, I was realizing when I got the end here that you could probably do without $connected (it's a part of a larger system for me). There may be some errors, so please comment if you find them and I'll correct.
Hope this helps anyone who finds it.
I am using ruby-smpp and redis to achive a queue based background worker to send SMPP messages.
And i am wondering if I am using eventmachine in the right way. It works but it doesnt feel right.
#!/usr/bin/env ruby
# Sample SMS gateway that can receive MOs (mobile originated messages) and
# DRs (delivery reports), and send MTs (mobile terminated messages).
# MTs are, in the name of simplicity, entered on the command line in the format
# <sender> <receiver> <message body>
# MOs and DRs will be dumped to standard out.
require 'smpp'
require 'redis/connection/hiredis'
require 'redis'
require 'yajl'
require 'time'
LOGFILE = File.dirname(__FILE__) + "/sms_gateway.log"
PIDFILE = File.dirname(__FILE__) + '/worker_test.pid'
Smpp::Base.logger = Logger.new(LOGFILE)
#Smpp::Base.logger.level = Logger::WARN
REDIS = Redis.new
class MbloxGateway
# MT id counter.
##mt_id = 0
# expose SMPP transceiver's send_mt method
def self.send_mt(sender, receiver, body)
if sender =~ /[a-z]+/i
source_addr_ton = 5
else
source_addr_ton = 2
end
##mt_id += 1
##tx.send_mt(('smpp' + ##mt_id.to_s), sender, receiver, body, {
:source_addr_ton => source_addr_ton
# :service_type => 1,
# :source_addr_ton => 5,
# :source_addr_npi => 0 ,
# :dest_addr_ton => 2,
# :dest_addr_npi => 1,
# :esm_class => 3 ,
# :protocol_id => 0,
# :priority_flag => 0,
# :schedule_delivery_time => nil,
# :validity_period => nil,
# :registered_delivery=> 1,
# :replace_if_present_flag => 0,
# :data_coding => 0,
# :sm_default_msg_id => 0
#
})
end
def logger
Smpp::Base.logger
end
def start(config)
# Write this workers pid to a file
File.open(PIDFILE, 'w') { |f| f << Process.pid }
# The transceiver sends MT messages to the SMSC. It needs a storage with Hash-like
# semantics to map SMSC message IDs to your own message IDs.
pdr_storage = {}
# Run EventMachine in loop so we can reconnect when the SMSC drops our connection.
loop do
EventMachine::run do
##tx = EventMachine::connect(
config[:host],
config[:port],
Smpp::Transceiver,
config,
self # delegate that will receive callbacks on MOs and DRs and other events
)
# Let the connection start before we check for messages
EM.add_timer(3) do
# Maybe there is some better way to do this. IDK, But it works!
EM.defer do
loop do
# Pop a message
message = REDIS.lpop 'messages:send:queue'
if message # If there is a message. Process it and check the queue again
message = Yajl::Parser.parse(message, :check_utf8 => false) # Parse the message from Json to Ruby hash
if !message['send_after'] or (message['send_after'] and Time.parse(message['send_after']) < Time.now)
self.class.send_mt(message['sender'], message['receiver'], message['body']) # Send the message
REDIS.publish 'log:messages', "#{message['sender']} -> #{message['receiver']}: #{message['body']}" # Push the message to the redis queue so we can listen to the channel
else
REDIS.lpush 'messages:queue', Yajl::Encoder.encode(message)
end
else # If there is no message. Sleep for a second
sleep 1
end
end
end
end
end
sleep 2
end
end
# ruby-smpp delegate methods
def mo_received(transceiver, pdu)
logger.info "Delegate: mo_received: from #{pdu.source_addr} to #{pdu.destination_addr}: #{pdu.short_message}"
end
def delivery_report_received(transceiver, pdu)
logger.info "Delegate: delivery_report_received: ref #{pdu.msg_reference} stat #{pdu.stat}"
end
def message_accepted(transceiver, mt_message_id, pdu)
logger.info "Delegate: message_accepted: id #{mt_message_id} smsc ref id: #{pdu.message_id}"
end
def message_rejected(transceiver, mt_message_id, pdu)
logger.info "Delegate: message_rejected: id #{mt_message_id} smsc ref id: #{pdu.message_id}"
end
def bound(transceiver)
logger.info "Delegate: transceiver bound"
end
def unbound(transceiver)
logger.info "Delegate: transceiver unbound"
EventMachine::stop_event_loop
end
end
# Start the Gateway
begin
puts "Starting SMS Gateway. Please check the log at #{LOGFILE}"
# SMPP properties. These parameters work well with the Logica SMPP simulator.
# Consult the SMPP spec or your mobile operator for the correct settings of
# the other properties.
config = {
:host => 'server.com',
:port => 3217,
:system_id => 'user',
:password => 'password',
:system_type => 'type', # default given according to SMPP 3.4 Spec
:interface_version => 52,
:source_ton => 0,
:source_npi => 1,
:destination_ton => 1,
:destination_npi => 1,
:source_address_range => '',
:destination_address_range => '',
:enquire_link_delay_secs => 10
}
gw = MbloxGateway.new
gw.start(config)
rescue Exception => ex
puts "Exception in SMS Gateway: #{ex} at #{ex.backtrace.join("\n")}"
end
Some easy steps to make this code more EventMachine-ish:
Get rid of the blocking Redis driver, use em-hiredis
Stop using defer. Pushing work out to threads with the Redis driver will make things even worse as it relies on locks around the socket it's using.
Get rid of the add_timer(3)
Get rid of the inner loop, replace it by rescheduling a block for the next event loop using EM.next_tick. The outer one is somewhat unnecessary. You shouldn't loop around EM.run as well, it's cleaner to properly handle a disconnect by doing a reconnect in your unbound method instead of stopping and restarting the event loop, by calling the ##tx.reconnect.
Don't sleep, just wait. EventMachine will tell you when new things come in on a network socket.
Here's how the core code around EventMachine would look like with some of the improvements:
def start(config)
File.open(PIDFILE, 'w') { |f| f << Process.pid }
pdr_storage = {}
EventMachine::run do
##tx = EventMachine::connect(
config[:host],
config[:port],
Smpp::Transceiver,
config,
self
)
REDIS = EM::Hiredis.connect
pop_message = lambda do
REDIS.lpop 'messages:send:queue' do |message|
if message # If there is a message. Process it and check the queue again
message = Yajl::Parser.parse(message, :check_utf8 => false) # Parse the message from Json to Ruby hash
if !message['send_after'] or (message['send_after'] and Time.parse(message['send_after']) < Time.now)
self.class.send_mt(message['sender'], message['receiver'], message['body'])
REDIS.publish 'log:messages', "#{message['sender']} -> #{message['receiver']}: #{message['body']}"
else
REDIS.lpush 'messages:queue', Yajl::Encoder.encode(message)
end
end
EM.next_tick &pop_message
end
end
end
end
Not perfect and could use some cleaning up too, but this is more what it should be like in an EventMachine manner. No sleeps, avoid using defer if possible, and don't use network drivers that potentially block, implement traditional loop by rescheduling things on the next reactor loop. In terms of Redis, the difference is not that big, but it's more EventMachine-y this way imho.
Hope this helps. Happy to explain further if you still have questions.
You're doing blocking Redis calls in EM's reactor loop. It works, but isn't the way to go. You could take a look at em-hiredis to properly integrate Redis calls with EM.