I'm new both to IMAP and multi-thread programming, and I'd like to write a script to fetch incoming mails and process them in parallel.
Thread, Queue, Mutex and Monitor are new concepts to me, I may miss-use them in the following question and example.
The script's goal is to:
receive IDLE notifications from the IMAP server when a new mail arrives
fetch the corresponding mail
parse its body
generate a PDF from parsed data
send the report through SMTP
Another aproach could be to fetch and identify mails based on their UID, thought learning IMAP & multi-thread programing is another goal.
Here's what I've got so far (IMAP part):
def self.idle(imap)
#imap = Net::IMAP.new 'mail.company.org', 143, false # Not using TLS for test purposes
#imap.login 'username', 'password'
#idler_listener = Thread.new do
loop do
begin
imap.examine 'INBOX'
imap.idle do |res|
if res.kind_of?(Net::IMAP::UntaggedResponse) and res.name == 'EXISTS'
imap.idle_done
Thread.new { process_email(imap) }
end
end
rescue => e
puts e.inspect
end
end
end.join
end
def self.process_email(imap)
begin
uid = imap.uid_search(['SUBJECT', 'MyFavoriteSubject']).last
mail = imap.uid_fetch(uid, 'BODY[TEXT]')[0].attr['BODY[TEXT]']
puts mail
rescue => e
puts e.inspect
end
end
This example successfully prints out the body of an incoming mail. However, if several mail arrives at the same time, only one will be treated.
Q:
Is this behavior due to the fact that the loop is executed in a single thread ?
Does it illustrate the need for a queue, a thread pool, or a producer / consumer pattern ?
Chances that this script will attempt to access the same resource at the same time are low, however, would Mutex.new.synchronize protect me against race conditions & deadlocks ?
(Eventhough I've not fully understand it, I think it's worth sharing this great resource from Masatoshi Seki: https://www.druby.org/sidruby/)
The application I'm working on allows users to add additional connections to other databases through a UI. I'm simply trying to create a validation that ensures that a connection can be made.
I created a separate class to test the DB connections:
class StoredProcConnection < ActiveRecord::Base
def self.abstract_class?
true # So it gets its own connection
end
end
I then create the connection:
def connect
adapter = sql_server? ? 'mssql' : 'mysql2'
default_port = sql_server? ? '1443' : '3306'
#connection_pool = StoredProcConnection.establish_connection(
adapter: adapter,
username: username,
password: password,
host: host,
database: database_name,
port: port || default_port,
timeout: 300)
end
def connection_pool
connect unless #connection_pool
#connection_pool
end
Then I validate it with this method:
def connection_test
if connection_pool.connection
#remove the connection from the StoredProcConnection pool
connection_pool.remove(connection_pool.connection)
return true
else
return false
end
rescue Exception => error
logger.info "unable to create connection with connection.id = #{id} - #{error}"
return false
end
Unfortunately, when it gets to this line with a bad host address like 127.0.0.abcdefg or 666.666.666.666
if connection_pool.connect
The app gets stuck, no errors raised or anything. It just freezes and I have to shut down the server manually.
I have a workaround but it feels quite sloppy. I am just inserting my own timeout in there, but I feel like Active Record should be throwing some kind of error.
def connection_test
Timeout::timeout(3) {
if connection_pool.connection
#remove the connection from the StoredProcConnection pool
connection_pool.remove(connection_pool.connection)
return true
else
return false
end
}
rescue Exception => error
logger.info "unable to create connection with connection.id = #{id} - #{error}"
return false
end
Does anyone see anything that might be causing the freeze? It seems pretty straight forward to me. I'm not sure why the connection pool is even created in the first place with a bad host passed in.
It just freezes...
Odds are good it's not frozen, it's just waiting to see if that connection can be made. TCP/IP has a long timeout value, which fools people into thinking things are frozen, when actually it's being patient.
Few people really understand how the internet works, and how it's really a house built of straw that we try to keep running no matter what. Long IP timeouts are one of the ways that we try to make it self-healing. Software doesn't care about how long something takes, only people care.
Since it appears you're concerned about malformed IP addresses, why aren't you pre-testing them to make sure they're at least in a valid format?
Use Ruby's built-in IPAddr class and try to parse them:
require 'ipaddr'
%w[
127.0.0.abcdefg
666.666.666.666
127.0.0.1
192.168.0.1
255.255.255.255
].each do |ip|
begin
IPAddr.new ip
puts "Good: #{ ip }"
rescue IPAddr::InvalidAddressError => e
puts "#{ e }: #{ ip }"
end
end
# >> invalid address: 127.0.0.abcdefg
# >> invalid address: 666.666.666.666
# >> Good: 127.0.0.1
# >> Good: 192.168.0.1
# >> Good: 255.255.255.255
For "fun reading" there's "TCP Timeout and Retransmission" and "TCP Socket no connection timeout".
I have a circumstance where my server may close TCPServer and restart, saving all the users to a file, and immediately reloading them; their connections do not sever.
The problem is I can't seem to reinitialize their streams.
When we restart (and attempt to maintain connections), I reinitialize TCPServer, and load my array of connected users – Since these each have an existing socket address, stored as <TCPSocket:0x00000000000000>, can I reinitialize these addresses with TCPServer?
Normally, each user connects and is accepted:
$nCS = TCPServer.new(HOST, PORT)
begin
while socket = $nCS.accept
Thread.new( socket ) do |sock|
begin
d = User.new(sock)
while sock.gets
szIn = $_.chomp
DBG( "Received '" + szIn + "' from Client " + sock.to_s )
d.parseInput( szIn )
end
rescue => e
$stdout.puts "ERROR: Caught error in Client Thread: #{e} \r\n #{e.backtrace.to_s.gsub(",", ",\r\n")}"
sock.write("Sorry, an error has occurred, and you have been disconnected."+EOL+"Please try again later."+EOL)
d.closeConnection
end
end
end
rescue => e
$stdout.puts "ERROR: Caught error in Server Thread: #{e} \r\n #{e.backtrace.to_s.gsub(",", ",\r\n")}"
exit
end
To give it a command to hot reboot, we use exec('./main --copyover') to flag that a copy over is occurring.
If $connected holds an array of all users, and each user has a socket, how do I reinitialize the socket that was open before the restart (assuming the other end is still connected)?
I suspect that using exec("./main", "--copyover", *$nCS, *$connected) is getting me closer, since this simply replaces the process, and should maintain the files (not close them).
You can't. The socket is only valid for the lifetime of the process: it is closed by the OS when the process exits. That in turn invalidates the connection, so the other end is not still connected.
How to Hot-Reboot a TCPServer in Ruby
Hot-Rebooting (aka Copyover) is a process by which an administrator can reload the application (along with any new changes made since last boot) without losing the client connections. This is useful in managing customer expectations as the application does not need to suffer severe downtime and disruption if in use.
What I propose below may not be the best practice, but it's functioning and perhaps will guide others to a similar solution.
The Command
I use a particular style of coding that makes use of command tables to find functions and their accessibility. All command functions are prefixed with cmd. I'll clean up the miscellany to improve readability:
def cmdCopyover
#$nCS is the TCPServer object
#$connected holds an array of all users sockets
#--copyover flags that this is a hot reboot.
connected_args = $connected.map do |sock|
sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
sock.fileno.to_s
end.join(",")
exec('./main.rb', '--copyover', $nCS.fileno.to_s, connected_args)
end
What we're passing are strings; $nCS.fileno.to_s provides us the file descriptor of the main TCPServer object, while connected_args is a comma-delineated list of file descriptors for each user connected. When we restart, ARGV will be an array holding each argument:
ARGV[0] == "--copyover"
ARGV[1] == "5" (Or whatever the file descriptor for TCPServer was)
ARGV[2] == "6,7,8,9" (Example, assuming 4 connected users)
What To Expect When You're Expecting (a Copyover)
Under normal circumstances, we may have a basic server (in main.rb that looks something like this:
puts "Starting Server"
$connected = Array.new
$nCS = TCPServer.new("127.0.0.1",9999)
begin
while socket = $nCS.accept
# NB: Move this loop to its own function, threadLoop()
Thread.new( socket ) do |sock|
begin
while sock.gets
szIn = $_.chomp
#do something with input.
end
rescue => e
puts "ERROR: Caught error in Client Thread: #{e}"
puts #{e.backtrace.to_s.gsub(",", ",\r\n")}"
sock.write("Sorry, an error has occurred, and you have been disconnected."+EOL+"Please try again later."+EOL)
sock.close
end
end
end
rescue => e
puts "Error: Caught Error in Server Thread: #{e}"
puts "#{e.backtrace.to_s.gsub(",", ",\r\n")}"
exit
end
We want to move that main loop to its own function to make it accessible -- our reconnecting users will need to be reinserted in the loop.
So let's get main.rb ready for accepting a hot reboot:
def threadLoop( socket )
Thread.new( socket ) do |sock|
begin
while sock.gets
szIn = $_.chomp
#do something with input.
end
rescue => e
puts "ERROR: Caught error in Client Thread: #{e}"
puts #{e.backtrace.to_s.gsub(",", ",\r\n")}"
sock.write("Sorry, an error has occurred, and you have been disconnected."+EOL+"Please try again later."+EOL)
sock.close
end
end
end
puts "Starting Server"
$connected = Array.new
if ARGV[0] == '--copyover'
$nCS = TCPServer.for_fd( ARGV[1].to_i )
$nCS.close_on_exec = false if $nCS.respond_to?(:close_on_exec=)
connected_args = ARGV[2]
connected_args.split(/,/).map do |sockfd|
$connected << sockfd
$connected.each {|c| threadLoop( c ) }
else
$nCS = TCPServer.new("127.0.0.1",9999)
$nCS.close_on_exec = false if $nCS.respond_to?(:close_on_exec=)
end
begin
while socket = $nCS.accept
threadLoop( socket )
end
rescue => e
puts "Error: Caught Error in Server Thread: #{e}"
puts "#{e.backtrace.to_s.gsub(",", ",\r\n")}"
exit
end
Caveat
My actual usage was a lot more ridiculously complicated, so I did my best to strip out all the garbage; however, I was realizing when I got the end here that you could probably do without $connected (it's a part of a larger system for me). There may be some errors, so please comment if you find them and I'll correct.
Hope this helps anyone who finds it.
I'm using the following code to connect to a network service i'm writing (thats backed by EventMachine) and I'm having a bit of trouble getting into a situation allowing me to use one socket connection to execute multiple commands.
#!/usr/bin/env ruby
require 'socket'
opts = {
:address => "0.0.0.0",
:port => 2478
}
connection = TCPSocket.open opts[:address], opts[:port]
# Get ID
connection.print "ID something"
puts connection.read
# Status
connection.print "STATUS"
puts connection.read
# Close the connection
connection.close
Here's what my EventMachine server hander looks like...
module ConnectionHandler
def receive_data data
send_data "Some output #{data}"
end
end
However, my first ruby script hangs when it executes connection.read as I presume its waiting for the connection to close so it knows its got all of the data? This is not what I want to happen.
My socket server will just take one command (on one line) and return one line of output.
Any ideas how I can do this? Thanks.
It turns out the connection.gets method will return a line of data received if the server sends a response ending in a \n character. So I just added \n to the end of my send_data call and switch to using puts connection.gets and it worked great!
I use the following code to check the server status of a certain game server to see if the game server is online.
begin
sock = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM, 0)
sockaddr = Socket.sockaddr_in(game_server.gameserver_port, game_server.gameserver_hostname)
sock.connect(sockaddr)
server_status.status = 1
rescue
server_status.status = 0
end
However it seems that the code blindly hangs up on the line without proceeding anywhere sock.connect(sockaddr) and does not throw an error when there's no services listening on that port. Is there a better way to do this in Ruby?
Could timeout be a good solution?
require 'timeout'
begin
timeout(5) do
# socket stuff...
end
rescue Timeout::Error
puts "Timed out!"
end