I currently have a script written in Ruby that scans a range of IP addresses and tries to connect to them. It's extremely slow at the moment. It takes up to 300 seconds to scan 254 hosts on the network, and that's obviously not very practical. What I'm trying to do is give the script some concurrency in hopes of speeding up the script. So far this is what I have:
require 'socket'
require 'celluloid'
$res_arr = []
class Ranger
include Celluloid
def initialize(host)
#host = host
#timeout = 1
end
def ip_range(host)
host =~ /(?:\d{1,3}\.){3}[xX*]{1,3}/
end
def ctrl(host)
begin
if ip_range(host)
strIP = host.gsub(/[xX*]/, '')
(1..254).each do |oct|
$res_arr << strIP+oct.to_s
end
else
puts "Invalid host!"
end
rescue
puts "onnection terminated."
end
end
def connect
addr = Socket.getaddrinfo(#host, nil)
sock = Socket.new(Socket.const_get(addr[0][0]), Socket::SOCK_STREAM, 0)
begin
sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
rescue Errno::EINPROGRESS
resp = IO.select(nil, [sock], nil, #timeout.to_i)
if resp.nil?
$res_arr << "#{#host} Firewalled!"
end
begin
if sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
$res_arr << "#{#host}Connected!"
end
rescue Errno::ECONNREFUSED
$res_arr << "#{#host} Refused!"
rescue
false
end
end
sock
end
def output(contents)
puts contents.value
end
end # Ranger
main = Ranger.new(ARGV[0])
main.ctrl(ARGV[0])
$res_arr.each do |ip|
scan = Ranger.new(ip)
scnftr = scan.future :connect
scan.output(scnftr)
end
The script works, but it takes just as long as before I included Celluloid at all. Am I misunderstanding how Celluloid works and what it's supposed to do?
Your problem is that each iteration of your loop starts a future, then immediately waits for it to return a value. What you want instead is start all futures, then wait for all futures to finish in two separate steps:
futures = $res_arr.map do |ip|
scan = Ranger.new(ip)
scan.future :connect
end
# now that all futures are running, we can start
# waiting for the first one to finish
futures.each do |future|
puts future.value
end
Here's another example from the celluloid source: https://github.com/celluloid/celluloid/blob/master/examples/simple_pmap.rb
Related
I have a module that should send GPS location to my server. Below I describe how communication happens between module and server:
When the module connects to the server, module sends its IMEI.
If server accepts data, then should reply to module 01. Note that confirmation should be sent as binary packet. I.e. 1 byte 0x01.
I'm struggling at second step. Have tried various combination, but neither worked.
client.puts('\x01')
client.send([0x01].pack("C"), 0)
client.write["01"].pack('H*')
Below is a full code example:
require 'socket'
class ClientThread
def initialize(port)
#server = TCPServer.open(port)
end
def run
puts "Started TCP Server"
loop do
Thread.start(#server.accept) do |client|
2.times do |index|
data = client.recv(8192)
if index == 0
client.send('\x01'.encode('utf-8'), 0) # RESPONSE TO DEVISE THAT SERVER IS READY TO ACCEPT DATA
elsif index == 1
puts self.log("Done! Closing Connection")
client.close
else
client.send('\x00'.encode('utf-8'), 0) # RESPONSE TO DEVISE THAT SERVER IS NOT READY TO ACCEPT DATA
end
end # end of loop twice
end # end of Thread
end # end of infinite loop
end # run method
end # end of class
new_thread = ClientThread.new(65432)
p new_thread.run
Device does not understand that server is ready to accept data. Most likely, because binary packet is formed incorrectly. How do you form response, so device would understand to send GSP data?
Ask me please if any questions. Thanks.
I have managed to make it work.
Instead of client.send('\x01'.encode('utf-8'), 0) I have used client.puts [0x01].pack("C") and module started to send data.
Below is a full working example for Teltonika FMT100.
Module sends its IMEI to the server.
Server accepts data and replies as binary packet 0x01.
Module send GPS data to the server. At this point you will need to decode data.
require 'socket'
class ClientThread
def initialize(port)
#server = TCPServer.open(port)
#imei = "unknown"
end
def log(msg)
"#{Time.now.utc} #{msg}"
end
def run
puts self.log("Started TCP Server")
loop do
Thread.start(#server.accept) do |client|
if client
2.times do |index|
begin
data = client.recv(8192)
if index == 0
#imei = data
p self.log("Device Authenticated | IMEI: #{#imei}")
client.puts [0x01].pack("C")
elsif index == 1
p data.unpack('H*').first
p self.log("Done! Closing Connection")
client.close
else
client.puts [0x00].pack("C")
end
rescue IOError
p self.log("Stream is already closed")
end
end # end of loop twice
else
p self.log('Socket is null')
end # if conditional
end # end of Thread
end # end of infinite loop
end # run method
end # end of class
new_thread = ClientThread.new(65432)
p new_thread.run
You should get something similar to:
"2021-12-30 11:12:22 UTC Device Authenticated | IMEI: \u0000\u000F357544374597827"
"00000000000004d608130000017dfcc4f8f8000f1753862097342800000000000000f00c05ef00f0011505c800450205b50000b60000422fbb430fc944000002f100006019100009d77a000000017dfcc5f2f8000f1753862097342800000000000000f00c05ef00f0001504c800450205b50000b60000422fbe430fcb44000002f100006019100009d77a000000017dfcce8a30000f1753862097342800000000000000000c05ef00f0001505c800450205b50000b60000422f6a430fbd44000002f100006019100009d77a000000017dfcce8e18000f1753862097342800000000000000f00c05ef00f0011505c800450205b50000b60000422f78430fc244000002f100006019100009d77a000000017dfccf8430000f1753862097342800000000000000f00c05ef00f0001504c800450205b50000b60000422fbd430fcb44000002f100006019100009d77a000000017dfcdbd488000f1753862097342800000000000000000c05ef00f0001504c800450205b50000b60000422fc0430fcb44000002f100006019100009d77a000000017dfcdbd870000f1753862097342800000000000000f00c05ef00f0011504c800450205b50000b60000422fac430fc744000002f100006019100009d77a000000017dfcdcbee8000f1753862097342800000000000000f00c05ef00f0001504c800450205b50000b60000422fb8430fcb44000002f100006019100009d77a000000017dfcddb500000f1753862097342800000000000000f00c05ef00f0011504c800450205b50000b60000422fb5430fcb44000002f100006019100009d77a000000017dfcdebab8000f1753862097342800000000000000f00c05ef00f0001504c800450205b50000b60000422fbe430fca44000002f100006019100009d77a000000017dfced4948000f1753862097342800000000000000000c05ef00f0001504c800450205b50000b60000422fbe430fc644000002f100006019100009d77a000000017dfced4d30000f1753862097342800000000000000f00c05ef00f0011504c800450205b50000b60000422fbf430fc644000002f100006019100009d77a000000017dfcee3790000f1753862097342800000000000000f00c05ef00f0001504c800450205b50000b60000422fc2430fc644000002f100006019100009d77a000000017dfcefd5a0000f1753862097342800000000000000f00c05ef00f0011504c800450205b50000b60000422fbe430fc644000002f100006019100009d77a000000017dfcf13918000f1753862097342800000000000000f00c05ef00f0001504c800450205b50000b60000422fc1430fc644000002f100006019100009d77a000000017dfcf2b7e8000f1753862097342800000000000000f00c05ef00f0011504c800450205b50000b60000422fb7430fc644000002f100006019100009d77a000000017dfcf3a630000f1753862097342800000000000000f00c05ef00f0001504c800450205b50000b60000422fb8430fc644000002f100006019100009d77a000000017dfd2aac20000f1753862097342800000000000000000c05ef00f0001504c800450205b50000b60000422fa0430fcb44000002f100006019100009d77a000000017dfd61b210000f1753862097342800000000000000000c05ef00f0001504c800450205b50000b60000422f9b430fcb44000002f100006019100009d77a00130000b8e0"
"2021-12-30 11:12:24 UTC Done! Closing Connection"
im building a small game in ruby to practice programming, so far everything has went well but im trying to implement multiplayer support, i can connect to the server and i can send information but when I try to read form the server it just freezes and my screen goes completely black. and i cant find the cause, ive read the documentation for the gem im using for TCP and i dont know, maybe i missed something, but if any of you have some insight I would really appreciate it
heres the repo if this code isnt enough
https://github.com/jaypitti/ruby-2d-gosu-game
heres the client side code
class Client
include Celluloid::IO
def initialize(server, port)
begin
#socket = TCPSocket.new(server, port)
rescue
$error_message = "Cannot find game server."
end
end
def send_message(message)
#socket.write(message) if #socket
end
def read_message
#socket.readpartial(4096) if #socket
end
end
heres the gameserver
require 'celluloid/autostart'
require 'celluloid/io'
class Server
include Celluloid::IO
finalizer :shutdown
def initialize(host, port)
puts "Starting Server on #{host}:#{port}."
#server = TCPServer.new(host, port)
#objects = Hash.new
#players = Hash.new
async.run
end
def shutdown
#server.close if #server
end
def run
loop { async.handle_connection #server.accept }
end
def handle_connection(socket)
_, port, host = socket.peeraddr
user = "#{host}:#{port}"
puts "#{user} has joined the arena."
loop do
data = socket.readpartial(4096)
data_array = data.split("\n")
if data_array and !data_array.empty?
begin
data_array.each do |row|
message = row.split("|")
if message.size == 10
case message[0]
when 'obj'
#players[user] = message[1..9] unless #players[user]
#objects[message[1]] = message[1..9]
when 'del'
#objects.delete message[1]
end
end
response = String.new
#objects.each_value do |obj|
(response << obj.join("|") << "\n") if obj
end
socket.write response
end
rescue Exception => exception
puts exception.backtrace
end
end # end data
end # end loop
rescue EOFError => err
player = #players[user]
puts "#{player[3]} has left"
#objects.delete player[0]
#players.delete user
socket.close
end
end
server, port = ARGV[0] || "0.0.0.0", ARGV[1] || 1234
supervisor = Server.supervise(server, port.to_i)
trap("INT") do
supervisor.terminate
exit
end
sleep
it just freezes and my screen goes completely black. and i cant find the cause
A good trick you can look at is attaching to your process with either rbspy or rbtrace to see that is going on when it is stuck.
You can also try first reducing dependencies here a bit and doing this with a simple threadpool prior to going full async with celluloid or event machine.
First of all you should not be rescuing Exception all over the place. Wrapping long begin rescue blocks around nested iterators is begging for trouble.
It sounds like a threading issues, memory and/or CPU but that's just a guess. Try to monitor your resources or use some performance checking gems. But for the love of Satoshi Nakamoto, please write some test coverage and see your methods fail miserably, then fix them!
Some of these may help:
group :development do
gem 'bullet', require: false
gem 'flamegraph', require: false
gem 'memory_profiler', require: false
gem 'rack-mini-profiler', require: false
gem 'seed_dump'
gem 'stackprof', require: false
gem 'traceroute', require: false
end
I have been searching the whole day for socket accept non blocking. I found recv non blocking but that wouldn't benefit me in anyway. My script first starts a new socket class. It binds to the client with ip 127.0.0.1 and port 6112. Then it starts multi threading. Multi threading takes #sock.accept. << That is blocking. I have then used accept_nonblock. Though, that would throw me the following error:
IO::EWOULDBLOCKWaitReadable : A non-blocking socket operation could not be completed immediately. - accept(2) would block
I am using Ruby 2.2.
NOTE: I do not intend to use Rails to solve my problem, or give me a shortcut. I am sticking with pure Ruby 2.2.
Here is my script:
require 'socket'
include Socket::Constants
#sock = Socket.new(AF_INET, SOCK_STREAM, 0)
#sockaddr = Socket.sockaddr_in(6112, '127.0.0.1')
#sock.bind(#sockaddr)
#sock.listen(5)
Thread.new(#sock.accept_nonblock) do |connection|
#client = Client.new(ip, connection, self)
#clients.push(#client)
begin
while connection
packet = connection.recv(55555)
if packet == nil
DeleteClient(connection)
else
#toput = "[RECV]: #{packet}"
puts #toput
end
end
rescue Exception => e
if e.class != IOError
line1 = e.backtrace[0].split(".rb").last
line = line1.split(":")[1]
#Log.Error(e, e.class, e.backtrace[0].split(".rb").first + ".rb",line)
puts "#{ e } (#{ e.class })"
end
end
def DeleteClient(connection)
#clients.delete(#client)
connection.close
end
http://docs.ruby-lang.org/en/2.2.0/Socket.html#method-i-accept_nonblock
accept_nonblock raises an exception when it can't immediately accept a connection. You are expected to rescue this exception and then IO.select the socket.
begin # emulate blocking accept
client_socket, client_addrinfo = socket.accept_nonblock
rescue IO::WaitReadable, Errno::EINTR
IO.select([socket])
retry
end
A patch has recently been accepted which will add an exception: false option to accept_nonblock, which will allow you to use it without using exceptions for flow control. I don't know that it's shipped yet, though.
I'm going on a limb here, and posting a large chunk of code.
I hope it will answer both your question and the any related questions others reading this answer might raise.
I'm sorry if I went overboard, I just thought it was almost all relevant.
Issues like looping through an event stack, using IO.select to push events in a non-block manner and other performance issues are all related (in my opinion) to the nonblocking concept of socket programming.
So i'm posting a ruby module which acts as a server with a reactor, using a limited number of threads, rather than thousands of threads, each per connection (12 threads will give you better performance than a hundred). The reactor utilizes the IO.select method with a timeout once all it's active events are handled.
The module can set up multiple listening sockets which use #accept_nonblock, and they all currently act as an echo server.
It's basically the same code I used for the Plezi framework's core... with some stripped down functionality.
The following is a thread-pool with 12 working threads + the main thread (which will sleep and wait for the "TERM" signal)...
...And it's an example of an accept_nonblock with exception handling and a thread pool.
It's a simple socket echo server, test it as a remote client using telnet:
> telnet localhost 3000
Hi!
# => Hi!
bye
#=> will disconnect
here's the code - Good Luck!!!
require 'socket'
module SmallServer
module_function
####
# Replace this method with your actual server logic.
#
# this code will be called when a socket recieves data.
#
# For now, we will just echo.
def got_data io, io_params
begin
got = io.recv_nonblock( 1048576 ) # with maximum number of bytes to read at a time...
puts "echoing: #{got}"
if got.match /^(exit|bye|q)\R/
puts 'closing connection.'
io.puts "bye bye!"
remove_connection io
else
io.puts "echoing: #{got}"
end
rescue => e
# should also log error
remove_connection io
end
end
#########
# main loop and activation code
#
# This will create a thread pool and set them running.
def start
# prepare threads
exit_flag = false
max_threads = 12
threads = []
thread_cycle = Proc.new do
io_review rescue false
true while fire_event
end
(max_threads).times { Thread.new { thread_cycle.call until exit_flag } }
# set signal tarps
trap('INT'){ exit_flag = true; raise "close!" }
trap('TERM'){ exit_flag = true; raise "close!" }
puts "Services running. Press ^C to stop"
# sleep until trap raises exception (cycling might cause the main thread to loose signals that might be caught inside rescue clauses)
(sleep unless SERVICES.empty?) rescue true
# start shutdown.
exit_flag = true
# set fallback tarps
trap('INT'){ puts 'Forced exit.'; Kernel.exit }
trap('TERM'){ puts 'Forced exit.'; Kernel.exit }
puts 'Started shutdown process. Press ^C to force quit.'
# shut down listening sockets
stop_services
# disconnect active connections
stop_connections
# cycle down threads
puts "Waiting for workers to cycle down"
threads.each {|t| t.join if t.alive?}
# rundown any active events
thread_cycle.call
end
#######################
## Events (Callbacks) / Multi-tasking Platform
EVENTS = []
E_LOCKER = Mutex.new
# returns true if there are any unhandled events
def events?
E_LOCKER.synchronize {!EVENTS.empty?}
end
# pushes an event to the event's stack
# if a block is passed along, it will be used as a callback: the block will be called with the values returned by the handler's `call` method.
def push_event handler, *args, &block
if block
E_LOCKER.synchronize {EVENTS << [(Proc.new {|a| push_event block, handler.call(*a)} ), args]}
else
E_LOCKER.synchronize {EVENTS << [handler, args]}
end
end
# Runs the block asynchronously by pushing it as an event to the event's stack
#
def run_async *args, &block
E_LOCKER.synchronize {EVENTS << [ block, args ]} if block
!block.nil?
end
# creates an asynchronous call to a method, with an optional callback (shortcut)
def callback object, method, *args, &block
push_event object.method(method), *args, &block
end
# event handling FIFO
def fire_event
event = E_LOCKER.synchronize {EVENTS.shift}
return false unless event
begin
event[0].call(*event[1])
rescue OpenSSL::SSL::SSLError => e
puts "SSL Bump - SSL Certificate refused?"
rescue Exception => e
raise if e.is_a?(SignalException) || e.is_a?(SystemExit)
error e
end
true
end
#####
# Reactor
#
# IO review code will review the connections and sockets
# it will accept new connections and react to socket input
IO_LOCKER = Mutex.new
def io_review
IO_LOCKER.synchronize do
return false unless EVENTS.empty?
united = SERVICES.keys + IO_CONNECTION_DIC.keys
return false if united.empty?
io_r = (IO.select(united, nil, united, 0.1) )
if io_r
io_r[0].each do |io|
if SERVICES[io]
begin
callback self, :add_connection, io.accept_nonblock, SERVICES[io]
rescue Errno::EWOULDBLOCK => e
rescue => e
# log
end
elsif IO_CONNECTION_DIC[io]
callback(self, :got_data, io, IO_CONNECTION_DIC[io] )
else
puts "what?!"
remove_connection(io)
SERVICES.delete(io)
end
end
io_r[2].each { |io| (remove_connection(io) || SERVICES.delete(io)).close rescue true }
end
end
callback self, :clear_connections
true
end
#######################
# IO - listening sockets (services)
SERVICES = {}
S_LOCKER = Mutex.new
def add_service port = 3000, parameters = {}
parameters[:port] ||= port
parameters.update port if port.is_a?(Hash)
service = TCPServer.new(parameters[:port])
S_LOCKER.synchronize {SERVICES[service] = parameters}
callback Kernel, :puts, "Started listening on port #{port}."
true
end
def stop_services
puts 'Stopping services'
S_LOCKER.synchronize {SERVICES.each {|s, p| (s.close rescue true); puts "Stoped listening on port #{p[:port]}"}; SERVICES.clear }
end
#####################
# IO - Active connections handling
IO_CONNECTION_DIC = {}
C_LOCKER = Mutex.new
def stop_connections
C_LOCKER.synchronize {IO_CONNECTION_DIC.each {|io, params| io.close rescue true} ; IO_CONNECTION_DIC.clear}
end
def add_connection io, more_data
C_LOCKER.synchronize {IO_CONNECTION_DIC[io] = more_data} if io
end
def remove_connection io
C_LOCKER.synchronize { IO_CONNECTION_DIC.delete io; io.close rescue true }
end
# clears closed connections from the stack
def clear_connections
C_LOCKER.synchronize { IO_CONNECTION_DIC.delete_if {|c| c.closed? } }
end
end
start the echo server in irb with:
SmallServer.add_service(3000) ; SmallServer.start
Say I have an array which is populated with IP addresses. Now say I have a method that tries to connect to each host in the array. In this particular instance I don't need to know any details about the failed connections, I just want to skip to the next host if one fails to connect or if ANY errors are thrown. How do I do this?
The problem I'm having is that, occasionally, when connecting to one of the hosts the program will throw an ENETUNREACH error and then kill the program. I tried solving it by just rescuing the error, but then what happens is the program will just stop executing without throwing any errors. How do I get it to skip the host in the array, and just move on to the next host?
def popen(host)
addr = Socket.getaddrinfo(host, nil)
sock = Socket.new(Socket.const_get(addr[0][0]), Socket::SOCK_STREAM, 0)
begin
sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
rescue Errno::EINPROGRESS
resp = IO.select(nil, [sock], nil, #timeout.to_i)
if resp.nil?
false
end
begin
if sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
sshlog(host, #user, #pass)
end
rescue Errno::ECONNREFUSED
false
end
end
sock
end
That is the connect method. And this:
def randIP
begin
threads = []
if #arg1 =~ /^([1-9][0-9]{0,2}|1000)$/
t1 = Time.now
s = 1
while s <= #arg1.to_i do
#host_arr << Array.new(4){rand(254)}.join('.')
s += 1
end
#host_arr.each do |ip|
threads << Thread.new do
begin
popen(ip)
rescue Errno::ENETUNREACH
end
end
end
threads.each do |thread|
thread.join
end
t2 = Time.now
time = t2 - t1
STDOUT.puts "done"
proxylst(#res_arr, #user, #pass)
else
puts "Host range too large! Must be a number between 1 and 1000."
end
rescue
false
end
end
Is a method that generates random IP addresses, puts them into an array, and uses the popen method above to attempt a connection on each host. Considering the nature of random IP address generation, chances are at least one out of 1000 hosts is going to be either invalid, or unreachable in some way. So what I need is a way to try every host in the array skipping any that throw errors.
typically how I handle this is like so:
exception_array = []
Connections.each do |con|
begin
#connection code
rescue Exception => e
exception_array << e
end
end
unless exception_array.empty?
msg = ""
exception_array.each do |e|
msg << e.to_s
end
raise msg
end
This way you keep track of all exceptions that were thrown without breaking the code on a failure.
I have a consumer which pulls messages off of a queue via an evented subscription. It takes those messages and then connects with a rather slow http interface. I have a worker pool of 8 and once those are all filled up I need to stop pulling requests from the queue and have the fibers that are working on the http jobs keep working. Here is an example I've thrown together.
def send_request(callback)
EM.synchrony do
while $available <= 0
sleep 2
puts "sleeping"
end
url = 'http://example.com/api/Restaurant/11111/images/?image%5Bremote_url%5D=https%3A%2F%2Firs2.4sqi.net%2Fimg%2Fgeneral%2Foriginal%2F8NMM4yhwsLfxF-wgW0GA8IJRJO8pY4qbmCXuOPEsUTU.jpg&image%5Bsource_type_enum%5D=3'
result = EM::Synchrony.sync EventMachine::HttpRequest.new(url, :inactivity_timeout => 0).send("apost", :head => {:Accept => 'services.v1'})
callback.call(result.response)
end
end
def display(value)
$available += 1
puts value.inspect
end
$available = 8
EM.run do
EM.add_periodic_timer(0.001) do
$available -= 1
puts "Available: #{$available}"
puts "Tick ..."
puts send_request(method(:display))
end
end
I have found that if I call sleep within a while loop in the synchrony block, the reactor loop gets stuck. If I call sleep within an if statement(sleeping just once) then most times it is enough time for the requests to finish but it is unreliable at best. If I use EM::Synchrony.sleep, then the main reactor loop will keep creating new requests.
Is there a way to pause the main loop but have the fibers finish their execution?
sleep 2
...
add_periodic_timer(0.001)
Are you serious?
Have you ever though how many send_request's are sleeping in the loop? And it's adding 1000 every second.
What about this:
require 'eventmachine'
require 'em-http'
require 'fiber'
class Worker
URL = 'http://example.com/api/whatever'
def initialize callback
#callback = callback
end
def work
f = Fiber.current
loop do
http = EventMachine::HttpRequest.new(URL).get :timeout => 20
http.callback do
#callback.call http.response
f.resume
end
http.errback do
f.resume
end
Fiber.yield
end
end
end
def display(value)
puts "Done: #{value.size}"
end
EventMachine.run do
8.times do
Fiber.new do
Worker.new(method(:display)).work
end.resume
end
end