Ruby DB-connection (SQLITE) with concurrent gem - ruby

I have a question and already the workaround for it, but I am not sure if this is "right" solution.
The followning example does not work. It gets stuck by the SQL-Update.
def mysynccmd(username,log)
exit_code = system("Doing stuff")
if exit_code != true
log.error "Error: #{username} not done."
else
log.info "Successfully done user: #{username}"
end
return exit_code
end
# Create thread pool
pool = Concurrent::FixedThreadPool.new(threadlimit)
# Running loop over the list of users with status 0
begin
#SQLite3::Database.new(dbfile) do |db|
db = SQLite3::Database.open dbfile
db.execute( "SELECT user FROM T_users WHERE status = 0" ) do |user|
# Puts work into the thread pool
pool.post do
# Check if the sync was correct and update the DB
if mysynccmd(user[0],log) == true
db.execute("UPDATE T_users SET status=1 WHERE user = '#{username}';")
end
end
end
#end
rescue SQLite3::Exception => e
log.error "Exception occurred: #{e}"
ensure
db.close if db
end
And I was thinking during the multithreading the DB-connection gets lost, because of the new process/thread.
So I did it this way and opend a new connection in every thread.
def mysynccmd(username,log,dbfile)
exit_code = system("Doing stuf")
if exit_code != true
log.error "Error: #{username} is not done."
else
log.info "Successfully done user: #{username}"
begin
db = SQLite3::Database.open dbfile
db.execute("UPDATE T_users SET status=1 WHERE user = '#{username}';")
rescue SQLite3::Exception => e
log.error "mysynccmd: Exception occurred: #{e}"
ensure
db.close if db
end
end
end
# Create thread pool
pool = Concurrent::FixedThreadPool.new(threadlimit)
# Running loop over the list of not synced users
begin
#SQLite3::Database.new(dbfile) do |db|
db = SQLite3::Database.open dbfile
db.execute( "SELECT user FROM T_users WHERE status = 0" ) do |user|
# Puts work into the thread pool
pool.post do
# Check if the sync was correct and update the DB
mysynccmd(user[0],log,dbfile)
end
end
#end
rescue SQLite3::Exception => e
log.error "Exception occurred: #{e}"
ensure
db.close if db
end
Was my thinking right or wrong?
Thanks for your answers and/or alternative ideas.

Related

Failed to connect to OpenTok

I created a Room chat, used Opentok like this:
in room.rb model
require 'opentok'
def config_opentok
#opentok ||= OpenTok::OpenTok.new Rails.application.secrets.open_tok_api_key, Rails.application.secrets.open_tok_secret
end
def set_opentok_params
session = config_opentok.create_session
self.session_id = session.session_id
self.opentok_token = session.generate_token({
expire_time: Time.now.to_i + 30.days
})
end
It worked well but sometimes I got this error:
OpenTok::OpenTokError: Failed to connect to OpenTok. Response code: execution expired
How can I fix it?
Thanks.
def config_opentok
tries = 3
begin
#opentok = OpenTok::OpenTok.new api_key, api_secret
logger.debug "opentok connected."
rescue Errno::ETIMEDOUT => e
log.error e
tries -= 1
if tries > 0
logger.debug "retrying opentok.new..."
retry
else
logger.debug "opentok.new timed out..."
puts "ERROR: #{e.message}"
end
end
end
This worked for me. See also: Cleanly Retrying Blocks of Code After an Exception in Ruby

Unable to make socket Accept Non Blocking ruby 2.2

I have been searching the whole day for socket accept non blocking. I found recv non blocking but that wouldn't benefit me in anyway. My script first starts a new socket class. It binds to the client with ip 127.0.0.1 and port 6112. Then it starts multi threading. Multi threading takes #sock.accept. << That is blocking. I have then used accept_nonblock. Though, that would throw me the following error:
IO::EWOULDBLOCKWaitReadable : A non-blocking socket operation could not be completed immediately. - accept(2) would block
I am using Ruby 2.2.
NOTE: I do not intend to use Rails to solve my problem, or give me a shortcut. I am sticking with pure Ruby 2.2.
Here is my script:
require 'socket'
include Socket::Constants
#sock = Socket.new(AF_INET, SOCK_STREAM, 0)
#sockaddr = Socket.sockaddr_in(6112, '127.0.0.1')
#sock.bind(#sockaddr)
#sock.listen(5)
Thread.new(#sock.accept_nonblock) do |connection|
#client = Client.new(ip, connection, self)
#clients.push(#client)
begin
while connection
packet = connection.recv(55555)
if packet == nil
DeleteClient(connection)
else
#toput = "[RECV]: #{packet}"
puts #toput
end
end
rescue Exception => e
if e.class != IOError
line1 = e.backtrace[0].split(".rb").last
line = line1.split(":")[1]
#Log.Error(e, e.class, e.backtrace[0].split(".rb").first + ".rb",line)
puts "#{ e } (#{ e.class })"
end
end
def DeleteClient(connection)
#clients.delete(#client)
connection.close
end
http://docs.ruby-lang.org/en/2.2.0/Socket.html#method-i-accept_nonblock
accept_nonblock raises an exception when it can't immediately accept a connection. You are expected to rescue this exception and then IO.select the socket.
begin # emulate blocking accept
client_socket, client_addrinfo = socket.accept_nonblock
rescue IO::WaitReadable, Errno::EINTR
IO.select([socket])
retry
end
A patch has recently been accepted which will add an exception: false option to accept_nonblock, which will allow you to use it without using exceptions for flow control. I don't know that it's shipped yet, though.
I'm going on a limb here, and posting a large chunk of code.
I hope it will answer both your question and the any related questions others reading this answer might raise.
I'm sorry if I went overboard, I just thought it was almost all relevant.
Issues like looping through an event stack, using IO.select to push events in a non-block manner and other performance issues are all related (in my opinion) to the nonblocking concept of socket programming.
So i'm posting a ruby module which acts as a server with a reactor, using a limited number of threads, rather than thousands of threads, each per connection (12 threads will give you better performance than a hundred). The reactor utilizes the IO.select method with a timeout once all it's active events are handled.
The module can set up multiple listening sockets which use #accept_nonblock, and they all currently act as an echo server.
It's basically the same code I used for the Plezi framework's core... with some stripped down functionality.
The following is a thread-pool with 12 working threads + the main thread (which will sleep and wait for the "TERM" signal)...
...And it's an example of an accept_nonblock with exception handling and a thread pool.
It's a simple socket echo server, test it as a remote client using telnet:
> telnet localhost 3000
Hi!
# => Hi!
bye
#=> will disconnect
here's the code - Good Luck!!!
require 'socket'
module SmallServer
module_function
####
# Replace this method with your actual server logic.
#
# this code will be called when a socket recieves data.
#
# For now, we will just echo.
def got_data io, io_params
begin
got = io.recv_nonblock( 1048576 ) # with maximum number of bytes to read at a time...
puts "echoing: #{got}"
if got.match /^(exit|bye|q)\R/
puts 'closing connection.'
io.puts "bye bye!"
remove_connection io
else
io.puts "echoing: #{got}"
end
rescue => e
# should also log error
remove_connection io
end
end
#########
# main loop and activation code
#
# This will create a thread pool and set them running.
def start
# prepare threads
exit_flag = false
max_threads = 12
threads = []
thread_cycle = Proc.new do
io_review rescue false
true while fire_event
end
(max_threads).times { Thread.new { thread_cycle.call until exit_flag } }
# set signal tarps
trap('INT'){ exit_flag = true; raise "close!" }
trap('TERM'){ exit_flag = true; raise "close!" }
puts "Services running. Press ^C to stop"
# sleep until trap raises exception (cycling might cause the main thread to loose signals that might be caught inside rescue clauses)
(sleep unless SERVICES.empty?) rescue true
# start shutdown.
exit_flag = true
# set fallback tarps
trap('INT'){ puts 'Forced exit.'; Kernel.exit }
trap('TERM'){ puts 'Forced exit.'; Kernel.exit }
puts 'Started shutdown process. Press ^C to force quit.'
# shut down listening sockets
stop_services
# disconnect active connections
stop_connections
# cycle down threads
puts "Waiting for workers to cycle down"
threads.each {|t| t.join if t.alive?}
# rundown any active events
thread_cycle.call
end
#######################
## Events (Callbacks) / Multi-tasking Platform
EVENTS = []
E_LOCKER = Mutex.new
# returns true if there are any unhandled events
def events?
E_LOCKER.synchronize {!EVENTS.empty?}
end
# pushes an event to the event's stack
# if a block is passed along, it will be used as a callback: the block will be called with the values returned by the handler's `call` method.
def push_event handler, *args, &block
if block
E_LOCKER.synchronize {EVENTS << [(Proc.new {|a| push_event block, handler.call(*a)} ), args]}
else
E_LOCKER.synchronize {EVENTS << [handler, args]}
end
end
# Runs the block asynchronously by pushing it as an event to the event's stack
#
def run_async *args, &block
E_LOCKER.synchronize {EVENTS << [ block, args ]} if block
!block.nil?
end
# creates an asynchronous call to a method, with an optional callback (shortcut)
def callback object, method, *args, &block
push_event object.method(method), *args, &block
end
# event handling FIFO
def fire_event
event = E_LOCKER.synchronize {EVENTS.shift}
return false unless event
begin
event[0].call(*event[1])
rescue OpenSSL::SSL::SSLError => e
puts "SSL Bump - SSL Certificate refused?"
rescue Exception => e
raise if e.is_a?(SignalException) || e.is_a?(SystemExit)
error e
end
true
end
#####
# Reactor
#
# IO review code will review the connections and sockets
# it will accept new connections and react to socket input
IO_LOCKER = Mutex.new
def io_review
IO_LOCKER.synchronize do
return false unless EVENTS.empty?
united = SERVICES.keys + IO_CONNECTION_DIC.keys
return false if united.empty?
io_r = (IO.select(united, nil, united, 0.1) )
if io_r
io_r[0].each do |io|
if SERVICES[io]
begin
callback self, :add_connection, io.accept_nonblock, SERVICES[io]
rescue Errno::EWOULDBLOCK => e
rescue => e
# log
end
elsif IO_CONNECTION_DIC[io]
callback(self, :got_data, io, IO_CONNECTION_DIC[io] )
else
puts "what?!"
remove_connection(io)
SERVICES.delete(io)
end
end
io_r[2].each { |io| (remove_connection(io) || SERVICES.delete(io)).close rescue true }
end
end
callback self, :clear_connections
true
end
#######################
# IO - listening sockets (services)
SERVICES = {}
S_LOCKER = Mutex.new
def add_service port = 3000, parameters = {}
parameters[:port] ||= port
parameters.update port if port.is_a?(Hash)
service = TCPServer.new(parameters[:port])
S_LOCKER.synchronize {SERVICES[service] = parameters}
callback Kernel, :puts, "Started listening on port #{port}."
true
end
def stop_services
puts 'Stopping services'
S_LOCKER.synchronize {SERVICES.each {|s, p| (s.close rescue true); puts "Stoped listening on port #{p[:port]}"}; SERVICES.clear }
end
#####################
# IO - Active connections handling
IO_CONNECTION_DIC = {}
C_LOCKER = Mutex.new
def stop_connections
C_LOCKER.synchronize {IO_CONNECTION_DIC.each {|io, params| io.close rescue true} ; IO_CONNECTION_DIC.clear}
end
def add_connection io, more_data
C_LOCKER.synchronize {IO_CONNECTION_DIC[io] = more_data} if io
end
def remove_connection io
C_LOCKER.synchronize { IO_CONNECTION_DIC.delete io; io.close rescue true }
end
# clears closed connections from the stack
def clear_connections
C_LOCKER.synchronize { IO_CONNECTION_DIC.delete_if {|c| c.closed? } }
end
end
start the echo server in irb with:
SmallServer.add_service(3000) ; SmallServer.start

Skip to next host in array if an error is thrown

Say I have an array which is populated with IP addresses. Now say I have a method that tries to connect to each host in the array. In this particular instance I don't need to know any details about the failed connections, I just want to skip to the next host if one fails to connect or if ANY errors are thrown. How do I do this?
The problem I'm having is that, occasionally, when connecting to one of the hosts the program will throw an ENETUNREACH error and then kill the program. I tried solving it by just rescuing the error, but then what happens is the program will just stop executing without throwing any errors. How do I get it to skip the host in the array, and just move on to the next host?
def popen(host)
addr = Socket.getaddrinfo(host, nil)
sock = Socket.new(Socket.const_get(addr[0][0]), Socket::SOCK_STREAM, 0)
begin
sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
rescue Errno::EINPROGRESS
resp = IO.select(nil, [sock], nil, #timeout.to_i)
if resp.nil?
false
end
begin
if sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
sshlog(host, #user, #pass)
end
rescue Errno::ECONNREFUSED
false
end
end
sock
end
That is the connect method. And this:
def randIP
begin
threads = []
if #arg1 =~ /^([1-9][0-9]{0,2}|1000)$/
t1 = Time.now
s = 1
while s <= #arg1.to_i do
#host_arr << Array.new(4){rand(254)}.join('.')
s += 1
end
#host_arr.each do |ip|
threads << Thread.new do
begin
popen(ip)
rescue Errno::ENETUNREACH
end
end
end
threads.each do |thread|
thread.join
end
t2 = Time.now
time = t2 - t1
STDOUT.puts "done"
proxylst(#res_arr, #user, #pass)
else
puts "Host range too large! Must be a number between 1 and 1000."
end
rescue
false
end
end
Is a method that generates random IP addresses, puts them into an array, and uses the popen method above to attempt a connection on each host. Considering the nature of random IP address generation, chances are at least one out of 1000 hosts is going to be either invalid, or unreachable in some way. So what I need is a way to try every host in the array skipping any that throw errors.
typically how I handle this is like so:
exception_array = []
Connections.each do |con|
begin
#connection code
rescue Exception => e
exception_array << e
end
end
unless exception_array.empty?
msg = ""
exception_array.each do |e|
msg << e.to_s
end
raise msg
end
This way you keep track of all exceptions that were thrown without breaking the code on a failure.

DB call terminates the program

I have a program written in Ruby with Sql Server 2008 on the back end. Right now I am confronting a problem which I am going to discuss under the following.
Consider I have functionA() and functionB(). fuctionA() and functionB() are being called in a loop. Of course it DB, connections are open and closed after each call inside these function.
I have a Rufus Scheduler scheduled after 1m which calls fuctionA() in its own thread by default. When functionA() in Rufus triggers the program terminates keeping in mind that functionA() and functionB() are being run constantly in a loop. When I comment out functionA() in Rufus the program executes smoothly. What could be the reason? Please guide.
class TestRufus
def initialize
#dps = nil
#mu = Mutex.new
end
def get_data()
victim_device = VictimDevice.new()
app_config = YAML.load_file("#{File.dirname(FILE)}/../../config/sa.gw.c.victim.yml")
scheduler = Rufus::Scheduler.new
scheduler.every "10s" do
handle_rufus
end
loop do
begin
#mu.synchronize {
#dps = victim_device.get_device_proximitysett(8)
}
puts "DeviceID : #{#dps.device_id} AlertDistance: #{#dps.victim_prox_alert_dist} VoilationDistance : #{#dps.victim_prox_viol_dist}"
rescue => e
time_of_error = Time.now.gmtime.to_s()
puts("Error occurred: #{time_of_error} - #{e.message}")
end
end
end #end of function
def handle_rufus()
puts 'In Rufus.'
begin
#mu.synchronize {
#dps = victim_device.get_device_proximitysett(8)
}
rescue => e
puts 'Error #{e.message}'
end
puts "Rufus got! #{#dps.device_id} "
puts 'Out Rufus.'
end
end # end of class
obj= TestRufus.new
obj.get_data()
Thank you in advance.

How to get exit status with Ruby's Net::SSH library?

I have a snippet of code, simply trying to execute a script on a remote server, in the event that it fails, I'd like to make a follow-up call, imagine this:
require 'rubygems'
require 'net/ssh'
require 'etc'
server = 'localhost'
Net::SSH.start(server, Etc.getlogin) do |ssh|
puts (ssh.exec("true") ? 'Exit Success' : "Exit Failure")
puts (ssh.exec("false") ? 'Exit Success' : "Exit Failure")
end
I would expect (ignoring that stdout and stderr are printed in my contrived example) - but first line should exit with 0 which I would expect Ruby would interperate as false and display "Exit Failure" (sure, so the logic is wrong, the ternary needs to be flipped) - but the second line should exit with the opposite status, and it doesn't.
I can't even find anything in the documentation about how to do this, and I'm a little worried that I might be doing it wrong?!
I find the following way of running processes with Net::SSH much more useful. It provides you with distinct stdout and stderr, exit code and exit signal.
require 'rubygems'
require 'net/ssh'
require 'etc'
server = 'localhost'
def ssh_exec!(ssh, command)
stdout_data = ""
stderr_data = ""
exit_code = nil
exit_signal = nil
ssh.open_channel do |channel|
channel.exec(command) do |ch, success|
unless success
abort "FAILED: couldn't execute command (ssh.channel.exec)"
end
channel.on_data do |ch,data|
stdout_data+=data
end
channel.on_extended_data do |ch,type,data|
stderr_data+=data
end
channel.on_request("exit-status") do |ch,data|
exit_code = data.read_long
end
channel.on_request("exit-signal") do |ch, data|
exit_signal = data.read_long
end
end
end
ssh.loop
[stdout_data, stderr_data, exit_code, exit_signal]
end
Net::SSH.start(server, Etc.getlogin) do |ssh|
puts ssh_exec!(ssh, "true").inspect
# => ["", "", 0, nil]
puts ssh_exec!(ssh, "false").inspect
# => ["", "", 1, nil]
end
Hope this helps.
Building on the answer by flitzwald - I've monkey patched my version of this into Net::SSH (Ruby 1.9+)
class Net::SSH::Connection::Session
class CommandFailed < StandardError
end
class CommandExecutionFailed < StandardError
end
def exec_sc!(command)
stdout_data,stderr_data = "",""
exit_code,exit_signal = nil,nil
self.open_channel do |channel|
channel.exec(command) do |_, success|
raise CommandExecutionFailed, "Command \"#{command}\" was unable to execute" unless success
channel.on_data do |_,data|
stdout_data += data
end
channel.on_extended_data do |_,_,data|
stderr_data += data
end
channel.on_request("exit-status") do |_,data|
exit_code = data.read_long
end
channel.on_request("exit-signal") do |_, data|
exit_signal = data.read_long
end
end
end
self.loop
raise CommandFailed, "Command \"#{command}\" returned exit code #{exit_code}" unless exit_code == 0
{
stdout:stdout_data,
stderr:stderr_data,
exit_code:exit_code,
exit_signal:exit_signal
}
end
end
For newer versions of Net::SSH, you can just pass a status hash to Net::SSH::Connection::Session#exec:
status = {}
Net::SSH.start(hostname, user, options) do |ssh|
channel = ssh.exec(command, status: status)
channel.wait # wait for the command to actually be executed
end
puts status.inspect
# {:exit_code=>0}
By default, exec streams its output to $stdout and $stderr. You can pass a block to exec to do something different, a la:
ssh.exec(command, status: status) do |ch, stream, data|
if stream == :stdout
do_something_with_stdout(data)
else
do_something_with_stderr(data)
end
end
This works on 6.1.0 - not sure about availability for older versions. See http://net-ssh.github.io/net-ssh/Net/SSH/Connection/Session.html#method-i-exec for more details.

Resources