Put contents of array all at once - ruby

I don't understand why this won't do what the title states.
#!/usr/bin/env ruby
require 'socket'
require 'timeout'
class Scanner
def initialize(host, port)
#host = host
#port = port
end
def popen
begin
array = []
sock = Socket.new(:INET, :STREAM)
sockaddr = Socket.sockaddr_in(#port, #host)
Timeout::timeout(5) do
array.push("Port #{#port}: Open") if sock.connect(sockaddr)
end
puts array
rescue Timeout::Error
puts "Port #{#port}: Filtered"
rescue Errno::ECONNREFUSED
end
end
end # end Scanner
def main
begin
p = 1
case ARGV[0]
when '-p'
eport = ARGV[1]
host = ARGV[2]
else
eport = 65535
host = ARGV[0]
end
t1 = Time.now
puts "\n"
puts "-" * 70
puts "Scanning #{host}..."
puts "-" * 70
while p <= eport.to_i do
scan = Scanner.new(host, p)
scan.popen
p += 1
end
t2 = Time.now
time = t2 - t1
puts "\nScan completed: #{host} scanned in #{time} seconds."
rescue Errno::EHOSTUNREACH
puts "This host appears to be unreachable"
rescue Interrupt
puts "onnection terminated."
end
end
main
What I'm trying to achieve is an output similar to nmap, in the way that it scans everything, and then shows all open or closed ports at the end. Instead what happens is that it prints them out as it discovers them. I figured pushing the output into an array then printing the array would achieve such an output, yet it still prints out the ports one at a time. Why is this happening?
Also, I apologize for the formatting, the code tags are a little weird.

Your loop calls popen once per iteration. Your popen method sets array = [] each time it is called, then populates it with one item, then you print it with puts. On the next loop iteration, you reset array to [] and do it all again.
You only asked "why," but – you could solve this by setting array just once in the body of main and then passing it to popen (or any number of ways).

Related

Message size varies TCPServer Ruby

I'm working with an AVL (Skypatrol TT8750+) and the messages that it sends (using TCP) are supposed to be 59bytes long but it always sends a first message (the message has some information about the AVL, so the user can identify it) of 33bytes.
So the question is, How can I handle those different size messages on ruby?
require 'socket'
portnumber = 12050
socketServer = TCPServer.open(portnumber)
while true
Thread.new(socketServer.accept) do |connection|
puts "Accepting connection from: #{connection.peeraddr[2]}"
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
out_file = File.new(file_name, "w+")
begin
while connection
incomingData = connection.gets()
if incomingData != nil
incomingData = incomingData
end
hex_line = incomingData.unpack('H*')[0]
out_file.puts(hex_line)
puts "Incoming: #{hex_line}"
end
rescue Exception => e
# Displays Error Message
puts "#{ e } (#{ e.class })"
ensure
connection.close
puts "ensure: Closing"
end
end
end
This is the experimental code that I'm using.
I'm posting this answer to explain a comment I made to Anderson's answer. Most of the code isn't mine.
moving the if out of the loop
When the if statement is within a loop, it will be evaluated each and every time the loop runs, increasing the number of CPU instructions and the complexity of each loop.
You could improve performance by moving the conditional statement out of the loop like so:
require 'socket'
require 'celluloid/io'
portnumber = 12050
socketServer = TCPServer.open(portnumber)
incomingData = nil
while true
Thread.new(socketServer.accept) do |connection|
puts "Accepting connection from: #{connection.peeraddr[2]}"
# this should probably be changed,
# it ignores the possibility of two connections arriving at the same timestamp.
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
out_file = File.new(file_name, "w+")
begin
if connection
incomingData = conection.recv(33)
if incomingData != nil
incomingData = incomingData.unpack('H*')[0]
out_file.puts(incomingData)
puts "Incoming: #{incomingData}"
end
end
while connection
incomingData = connection.recv(59)
if incomingData != nil
incomingData = incomingData.unpack('H*')[0]
out_file.puts(incomingData)
puts "Incoming: #{incomingData}"
end
end
rescue Exception => e
# Displays Error Message
puts "#{ e } (#{ e.class })"
ensure
connection.close
out_file.close
puts "ensure: Closing"
end
end
end
Optimizing the recv method
Another optimization I should probably mention (but won't implement here) would be the recv method call.
This is both an optimization and a possible source for errors that should be addressed.
recv is a system call and as network messages might be combined (or fragmented) across TCP/IP packets, it might become more expensive to call recv than to handle an internal buffer of data that resolved fragmentation and overflow states.
Reconsidering the thread-per-client design
I would also recommend avoiding the thread-per client design.
In general, for a small number of clients it probably doesn't matter much.
However, as clients multiply and threads become busier, you might find the system spends more resources on context switches than actual tasks.
Another concern might be the allocated stack each thread requires (1Mb or 2Mb for Ruby threads, if I remember correctly)... In a best case scenario, 1,000 clients will require more than a GigaByte of memory allocation just for the stack (I'm ignoring kernel structure data table and other resources).
I would consider using EventMachine or Iodine (I'm iodine's author, so I'm biased).
An evented design could save you many resources.
For example (untested):
require 'iodine'
# define the protocol for our service
class ExampleProtocol
#timeout = 10
def on_open
puts "New Connection Accepted."
# this should probably be changed,
# it ignores the possibility of two connections arriving at the same timestamp.
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
#out_file = File.new(file_name, "w+")
# a rolling buffer for fragmented messages
#expecting = 33
#msg = ""
end
def on_message buffer
length = buffer.length
pos = 0
while length >= #expecting
#msg << (buffer[pos, #expecting])
out_file.puts(msg.unpack('H*')[0])
length -= #expecting
pos += #expecting
#expecting = 59
#msg.clear
end
if(length > 0)
#msg << (buffer[pos, length])
#expecting = 59-length
end
end
def on_close
#out_file.close
end
end
# create the service instance
Iodine.listen 12050, ExampleProtocol
# start the service
Iodine.start
The solution was quite simple
require 'socket'
require 'celluloid/io'
portnumber = 12050
socketServer = TCPServer.open(portnumber)
while true
Thread.new(socketServer.accept) do |connection|
puts "Accepting connection from: #{connection.peeraddr[2]}"
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
out_file = File.new(file_name, "w+")
messagecounter = 1
begin
while connection
if messagecounter == 1
incomingData = conection.recv(33)
messagecounter += 1
else
incomingData = connection.recv(59)
end
if incomingData != nil
incomingData = incomingData.unpack('H*')[0]
end
out_file.puts(incomingData)
puts "Incoming: #{incomingData}"
end
rescue Exception => e
# Displays Error Message
puts "#{ e } (#{ e.class })"
ensure
connection.close
puts "ensure: Closing"
end
end
end
I just needed an extra variable and an if to auto increment the variable, and that's it.

Add multithreads/concurency in script

I created a script which checks healthcheck and ports status from a .json file populated with microservices.
So for every microservice from the .json file the script will output the HTTP status and healthcheck body and other small details, and I want to add multithreading here in order to return all the output at once.Please see the script below:
#!/usr/bin/env ruby
... get the environment argument part...
file = File.read('./services.json')
data_hash = JSON.parse(file)
threads = []
service = data_hash.keys
service.each do |microservice|
threads << Thread.new do
begin
puts "Microservice: #{microservice}"
port = data_hash["#{microservice}"]['port']
puts "Port: #{port}"
nodes = "knife search 'chef_environment:#{env} AND recipe:#{microservice}' -i"
node = %x[ #{nodes} ].split
node.each do |n|
puts "Node: #{n}"
uri = URI("http://#{n}:#{port}/healthcheck?count=10")
res = Net::HTTP.get_response(uri)
status = Net::HTTP.get(uri)
puts res.code
puts status
puts res.message
end
rescue Net::ReadTimeout
puts "ReadTimeout Error"
next
end
end
end
threads.each do |thread|
thread.join
end
Anyway in this way the script return first the puts "Microservice: #{microservice}" and puts "Port: #{port}" and after this it will return the nodes and only after the STATUS.
How can I return all the data for each loop together?
Instead of puts write output to a variable (hash).
If you wand to wait for all threads to finish their job before showing the output, use ThreadsWait class.
require 'thwait'
file = File.read('./services.json')
data_hash = JSON.parse(file)
h = {}
threads = []
service = data_hash.keys
service.each do |microservice|
threads << Thread.new do
thread_id = Thread.current.object_id.to_s(36)
begin
h[thread_id] = "Microservice: #{microservice}"
port = data_hash["#{microservice}"]['port']
h[thread_id] << "Port: #{port}"
nodes = "knife search 'chef_environment:#{env} AND recipe:#{microservice}' -i"
node = %x[ #{nodes} ].split
node.each do |n|
h[thread_id]<< "Node: #{n}"
uri = URI("http://#{n}:#{port}/healthcheck?count=10")
res = Net::HTTP.get_response(uri)
status = Net::HTTP.get(uri)
h[thread_id] << res.code
h[thread_id] << status
h[thread_id] << res.message
end
rescue Net::ReadTimeout
h[thread_id] << "ReadTimeout Error"
next
end
end
end
threads.each do |thread|
thread.join
end
# wait untill all threads finish their job
ThreadsWait.all_waits(*threads)
p h
[edit]
ThreadsWait.all_waits(*threads) is redundant in above code and can be omitted, since line treads.each do |thread| thread.join end does exactely the same thing.
Instead of outputting the data as you get it using puts, you can collect it all in a string and then puts it once at the end. Strings can take the << operator (implemented as a method in Ruby), so you can just initialize the string, add to it, and then output it at the end, like this:
report = ''
report << 'first thing'
report << 'second thing'
puts report
You could even save them all up together and print them all after all were finished if you want.

Celluloid output is out of order and formatted erratically

I have a working script that utilizes celluloid for network parallelism. What it does is scan a range of IP addresses and tries to connect to them. It will output either ip_addr: Filtered, Refused, or Connected. The only problem with the script is the way the results are printed. Instead of being in order, like so:
192.168.0.20: Filtered
192.168.0.21: Connected
It outputs like this:
192.168.0.65 Firewalled!
192.168.0.11 Firewalled!192.168.0.183 Firewalled!192.168.0.28 Firewalled!192.168.0.171 Firewalled!192.168.0.228 Firewalled!
192.168.0.238 Firewalled!192.168.0.85 Firewalled!192.168.0.148 Firewalled!192.168.0.154 Firewalled!192.168.0.76 Firewalled!192.168.0.115 Firewalled!
192.168.0.215 Firewalled!
In the terminal. As you can see it's completely erratic. Here's the relevant code:
def connect
addr = Socket.getaddrinfo(#host, nil)
sock = Socket.new(Socket.const_get(addr[0][0]), Socket::SOCK_STREAM, 0)
begin
sock.connect_nonblock(Socket.pack_sockaddr_in(#port, addr[0][3]))
rescue Errno::EINPROGRESS
resp = IO.select(nil, [sock], nil, #timeout.to_i)
if resp.nil?
puts "#{#host} Firewalled!"
end
begin
if sock.connect_nonblock(Socket.pack_sockaddr_in(#port, addr[0][3]))
puts "#{#host} Connected!"
end
rescue Errno::ECONNREFUSED
puts "#{#host} Refused!"
rescue
false
end
end
sock
end
range = []
main = Ranger.new(ARGV[0], ARGV[1])
(1..254).each do |oct|
range << main.strplace(ARGV[0]+oct.to_s)
end
threads = []
range.each do |ip|
threads << Thread.new do
scan = Ranger.new(ip, ARGV[1])
scan.future :connect
end
end
threads.each do |thread|
thread.join
end
I think I know what the problem is. You see, puts is not thread-safe. When you call puts, it does 2 things: a) It prints whatever you want to the screen and b) It inserts a newline \n at the end. So one thread (thread A) could do a) but then stop and another thread (thread B) could also do a), then the operating system might go again to thread A which will do b) etc., thus producing the input you're seeing.
So the solution would be to replace all instances of puts with "print whatever-you-want \n". For example, this:
puts "#{#host} Firewalled!"
could be converted into:
print "#{#host} Firewalled!\n"
Unlike puts, print is thread-safe and cannot be interrupted before it's complete.

Nasty race conditions with Celluloid

I have a script that generates a user-specified number of IP addresses and tries to connect to them all on some port. I'm using Celluloid with this script to allow for reasonable speeds, since scanning 2000 hosts synchronously could take a long time. However, say I tell the script to scan 2000 random hosts. What I find is that it actually only ends up scanning about half that number. If I tell it to scan 3000, I get the same basic results. It seems to work much better if I do 1000 or less, but even if I just scan 1000 hosts it usually only ends up doing about 920 with relative consistency. I realize that generating random IP addresses will obviously fail with some of them, but I find it hard to believe that there are around 70 improperly generated IP addresses, every single time. So here's the code:
class Scan
include Celluloid
def initialize(arg1)
#arg1 = arg1
#host_arr = []
#timeout = 1
end
def popen(host)
addr = Socket.getaddrinfo(host, nil)
sock = Socket.new(Socket.const_get(addr[0][0]), Socket::SOCK_STREAM, 0)
begin
sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
rescue Errno::EINPROGRESS
resp = IO.select(nil, [sock], nil, #timeout.to_i)
if resp.nil?
puts "#{host}:Firewalled"
end
begin
if sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
puts "#{host}:Connected"
end
rescue Errno::ECONNREFUSED
puts "#{host}:Refused"
rescue
false
end
end
sock
end
def asynchronous
s = 1
threads = []
while s <= #arg1.to_i do
#host_arr << Array.new(4){rand(254)}.join('.')
s += 1
end
#host_arr.each do |ip|
threads << Thread.new do
begin
popen(ip)
rescue
end
end
end
threads.each do |thread|
thread.join
end
end
end
scan = Scan.pool(size: 100, args: [ARGV[0]])
(0..20).to_a.map { scan.future.asynchronous }
Around half the time I get this:
D, [2014-09-30T17:06:12.810856 #30077] DEBUG -- : Terminating 11 actors...
W, [2014-09-30T17:06:12.812151 #30077] WARN -- : Terminating task: type=:finalizer, meta={:method_name=>:shutdown}, status=:receiving
Celluloid::TaskFiber backtrace unavailable. Please try Celluloid.task_class = Celluloid::TaskThread if you need backtraces here.
and the script does nothing at all. The rest of the time (only if I specify more then 1000) I get this: http://pastebin.com/wTmtPmc8
So, my question is this. How do I avoid race conditions and deadlocking, while still achieving what I want in this particular script?
Starting low-level Threads by yourself interferes with Celluloid's functionality. Instead create a Pool of Scan objects and feed them the IP's all at once. They will queue up for the available
class Scan
def popen
…
end
end
scanner_pool = Scan.pool(50)
resulsts = #host_arr.map { |host| scanner_pool.scan(host) }

Ruby output is not displayed on the sinatra browser

I want to bulid a multi threaded application. If i do not use threads, everything works fine. When i try to use threads, then nothing is displayed on the browser. when i use the syntax 'puts "%s" %io.read' then it displays on the command prompt and not on the browser. Any help would be appreciated.
require 'sinatra'
require 'thread'
set :environment, :production
get '/price/:upc/:rtype' do
Webupc = "#{params[:upc]}"
Webformat = "#{params[:rtype]}"
MThread = Thread.new do
puts "inside thread"
puts "a = %s" %Webupc
puts "b = %s" %Webformat
#call the price
Maxupclen = 16
padstr = ""
padupc = ""
padlen = (Maxupclen - Webupc.length)
puts "format type: #{params[:rtype]}"
puts "UPC: #{params[:upc]}"
puts "padlen: %s" %padlen
if (Webformat == 'F')
puts "inside format"
if (padlen == 0 ) then
IO.popen("tstprcpd.exe #{Webupc}")
{ |io|
"%s" %io.read
}
elsif (padlen > 0 ) then
for i in 1 .. padlen
padstr = padstr + "0"
end
padupc = padstr + Webupc
puts "padupc %s" %padupc
IO.popen("tstprcpd.exe #{padupc}") { |io|
"%s" %io.read
}
elsif (padlen < 0 ) then
IO.popen("date /T") { |io|
"UPC length must be 16 digits or less." %io.read
}
end
end
end
end
Your code has several problems:
It is not formatted properly
You are using Uppercase names for variables; that makes them constants!
puts will not output to the browser, but to the console. The browser will recieve the return value of the block, i.e. the return value of the last statement in the block. Therefore, you need to build your output differently (see below).
You are never joining the thread
Here's a minimal sinatra app that uses a thread. However, the thread makes no sense in this case because you must wait for its termination anyway before you can output the result to the browser. In order to build the output I have used StringIO, which you can use with puts to build a multiline string conveniently. However, you could also simply initialize res with an empty string with res = "" and then append your lines to this string with res << "new line\n".
require 'sinatra'
require 'thread'
require 'stringio'
get '/' do
res = StringIO.new
th = Thread.new do
res.puts 'Hello, world!'
end
th.join
res.string
end

Resources