Right now I'm developing a some sort of parser for the messages of the Skypatrol TT8750+ and my threaded TCP server is working. The problem is that it isn't a good approach if there are to many devices connected at the same time. I'm using iodine but I can't make work some code that was given to me. My goal is to receive first a 33bytes message to identify the device and then start to receive 86bytes messages with information of the vehicle.
require 'iodine'
# define the protocol for our service
class TT8750plus
#timeout = 10
def on_open
puts "New Connection Accepted."
# this file is just for testing purposes.
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
#out_file = File.new(file_name, "w+")
# a rolling buffer for fragmented messages
#expecting = 33
#msg = ""
end
def on_message buffer
length = buffer.length
pos = 0
while length >= #expecting
#msg << (buffer[pos, #expecting])
#out_file.puts(#msg.unpack('H*')[0])
length -= #expecting
po += #expecting
#expecting = 86
#msg.clear
end
if(length > 0)
#msg << (buffer[pos, length])
#expecting = 86 - length
end
puts #msg
end
def on_close
#out_file.close
end
end
# create the service instance
Iodine.listen 12050, TT8750plus
# start the service
Iodine.start
And this error appears on every message
New Connection Accepted.
Iodine caught an unprotected exception - NoMethodError: undefined method `+' for nil:NilClass
iodineServer.rb:26:in `on_message'
iodineServer.rb:1:in `on_data'Iodine caught an unprotected exception - NoMethodError: undefined method `+' for nil:NilClass
Also this implementation doesn't get the messages I need
These are the first two lines that I got from this implementation:
0021000a0800000000000120202020202038363332383630323034333433373020
0021000a08000000000001202020202020383633323836303230343334333730200056000a08100020202020202038363332383630323034333433373020014b0000
And these are the first two lines from the threaded implementation
0021000a0800000000000120202020202038363332383630323034333433373020
0056000a08100020202020202038363332383630323034333433373020000b00000013090044709bfb8109e400000000001100000000000067eb11090c1512012e970020000000000005000000000005000000000007
0056000a08100020202020202038363332383630323034333433373020010b00000013090044709bfb8109e400000000001200000000000067eb11090c1512042e970020000000000005000000000005000000000008
This looks like a variation of the untested code I posted earlier.
It seems the issue is with a spelling mistake in the code (probably mine?).
The po += ... should have been pos += ...
require 'iodine'
# define the protocol for our service
class TT8750plus
#timeout = 10
def on_open
puts "New Connection Accepted."
# this file is just for testing purposes.
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
#out_file = File.new(file_name, "w+")
# a rolling buffer for fragmented messages
#expecting = 33
#msg = ""
end
def on_message buffer
length = buffer.length
pos = 0
while length >= #expecting
#msg << (buffer[pos, #expecting])
#out_file.puts(#msg.unpack('H*')[0])
length -= #expecting
pos += #expecting # the spelling mistake was here
#expecting = 86
puts "wrote:", #msg
#msg.clear
end
if(length > 0)
#msg << (buffer[pos, length])
#expecting = 86 - length
end
puts("Waiting for more data:", #msg) unless #msg.empty?
end
def on_close
#out_file.close
end
end
# create the service instance
Iodine.listen 12050, TT8750plus
# start the service
Iodine.start
Again, lacking an emulation for the Skypatrol TT8750+, I can't test the code. But it should be possible to follow error messages to slowly track down these types of issues.
P.S.
To protect from exceptions, consider using Ruby's:
begin
# code
rescue => e
# oops something happened. i.e.
puts e.message, e.backtrace
end
i.e., the on_message method might look like this:
def on_message buffer
begin
length = buffer.length
pos = 0
while length >= #expecting
#msg << (buffer[pos, #expecting])
#out_file.puts(#msg.unpack('H*')[0])
length -= #expecting
pos += #expecting # the spelling mistake was here
#expecting = 86
#msg.clear
end
if(length > 0)
#msg << (buffer[pos, length])
#expecting = 86 - length
end
puts #msg unless #msg.empty? # print leftovers for testing...?
rescue => e
# oops something happened. React
puts e.message, e.backtrace
end
end
Also, FYI, iodine allows you to control the number of processes (workers) and threads you'r using. See the documentation for details.
Related
I'm pretty new to Ruby, having decided to try my hand with another programming language. I have been trying to get to grips with sockets, which is an area I'm not too familiar with in general. I have created a basic 'game' that allows a player to move his icon around the screen, however, I am trying to make this into something that could be multiplayer using TCP Sockets.
At the moment, when a player launches the game as a host, it creates a server. If the player starts a game as a client, it connects to the server. Currently, it only works on the same machine, but the connection has been established successfully, and when connecting as a client, the server creates a username, sends this back to the client, which then uses it to create a player.
The problem comes when i try to communicate between the server and the client, the messages appear to be sending from server to client, but only partially, and it appears they are largely truncated at either the beginning or end.
If anyone could advice on what is causing this, it would be greatly appreciated. I am using Ruby with Gosu and Celluloid-IO.
SERVER CLASS
require 'src/Game.rb'
require 'celluloid/io'
class Server
include Celluloid::IO
finalizer:shutdown
def initialize
puts ("server init")
#super
#playersConnected = Array.new
#actors = Array.new
#server = TCPServer.new("0.0.0.0", 28888)
##server.open(8088)
puts #server
async.run
end
def run
loop {
async.handle_connection #server.accept
}
end
def readMessage(socket, player)
msg = socket.recv(30)
data = msg.split"|"
#playersConnected.each do |p|
if p.getUser() == player
p.setX(data[1])
p.setY(data[2])
end
#puts "END"
end
end
def handle_connection(socket)
_, port, host = socket.peeraddr
user = "#{host}:#{port}"
puts "#{user} has joined the game"
#playersConnected.push(Player.new(user))
socket.send "#{user}", 0
#socket.send "#{Player}|#{user}", 0
#socket.send "#{port}", 0
puts "PLAYER LIST"
#playersConnected.each do |player|
puts player
end
Thread.new{
loop{
readMessage(socket, user)
#divide message
#find array index
#update character position in array
}
}
Thread.new{
loop{
#playersConnected.each do |p|
msg = p.getUser() + "|" + "#{p.getX}" + "|" + "#{p.getY}"
socket.send(msg, 0)
end
}
}
end
Server.new
end
CLIENT CLASS
require 'src/Game.rb'
class Client < Game
#finalizer :shutdown
def initialize
super
#socket = TCPSocket.new("localhost", 28888)
#188.222.55.241
while #player.nil?
#player = Player.new(#socket.recv(1024))
puts #player
puts #player.getUser()
#player.warp(0, 0)
pulse
end
end
def pulse
Thread.new{
loop{
msg = #player.getUser() + "|" + "#{#player.getX()}" + "|" + "#{#player.getY()}"
#socket.write(msg)
}
}
Thread.new{
loop{
msg = #socket.recv(1024)
data = msg.split"|"
puts data[0]
match = false
#players.each do |player|
if player.getUser() == data[0]
puts "MATCHX"
player.setX(data[1])
player.setY(data[2])
match = true
end
if match == false
p = Player.new(data[0])
#p.warp(data[1],data[2])
#players.push(p)
end
end
puts "end"
}
}
end
Client.new.show
end
Side note; There is also a Host class, which mimics the client class, only it calls the server. I am aware this is a terrible way to do things, i intend to fix it once i overcome the current issue.
Many thanks in advance
I have a few commands from a Twitch chat, it works and it's awesome! People can type commands like !about and it's going to send a response. There's just one problem, if you go down to the thread and look under # // COMMANDS // COMMANDS // COMMANDS. There's a problem, there´s a few admin commands (look under # ADMIN COMMANDS // ADMIN COMMANDS). I have this: if msg.include?("!project") then msg.slice!("!project "), but it's not working like I want it to. I want to be able to type !project Then some substring here. But when a substring is attached it doesn't work, even though I can print msg and it clearly includes !project. Both the !disconnect and !project command work when it's only those commands by themselves. Meaning that something like !disconnect Hello World wouldn't work even though it clearly includes !disconnect.
# Message formatting in console
class String
def red; "\e[31m#{self}\e[0m" end
def yellow; "\e[33m#{self}\e[0m" end
def green; "\e[32m#{self}\e[0m" end
def cyan; "\e[36m#{self}\e[0m" end
def bold; "\e[1m#{self}\e[22m" end
end
# Requied packages / modules
require 'socket'
require 'logger'
require 'open-uri'
# Create logger
File.delete("log.txt") # Clear previous log
log = Logger.new("log.txt", formatter: proc {|severity, datetime, progname, msg|
"#{datetime}: #{msg}\n"})
# Required Info
load "credentials.txt"
log.info("Loading \"credentials.txt\"")
# -------- IGNORE -------- #
OAUTH.downcase!
BOTNAME.downcase!
CHANNEL.downcase!.gsub!("#", "")
# //- AGE -// #
time = Time.new
age = time.year - 2000
if time.month == 10
if time.day < 28
age -= 1
end
elsif time.month < 10
age -= 1
end
# -------- IGNORE -------- #
# Save "Preparing to connect" to "log.txt"
log.info("Preparing to connect")
# Variables
socket = TCPSocket.new('irc.chat.twitch.tv', 6667)
send = "PRIVMSG ##{CHANNEL} :" # shortcut for sending messages
running = true
content = nil
message_count = 0
message_limit = Time.now.to_i
# Commands
commands = ["!about","!uptime","!commands","!cortexio","!followed"]
api_commands = ["!followed","!uptime"]
admin_commands = ["!disconnect","!project"]
# Authorization Login
socket.puts("PASS #{OAUTH}") # Send the password(oauth) to Twitch
socket.puts("NICK #{BOTNAME}") # Send the botname to Twitch
socket.puts("JOIN ##{CHANNEL}") # Send the channel to Twitch
# Save "Connected!" to "log.txt
log.info("Joining #{CHANNEL.capitalize} as #{BOTNAME.capitalize} using OAUTH Token: #{OAUTH[6,OAUTH.length-12]}" + "*"*12)
# Thread.abort_on_exception = true
# Loop (Background Thread) for recieving Twitch chat data
Thread.start do
socket.puts(send + "Connected!") # Send "Connected!" to the Twitch channel
puts "#{BOTNAME} Joined ##{CHANNEL}" # Connection Status
puts "You should be fully connected now" # Connection Status
puts ""
puts "Type \"clear\" to clear terminal"
puts ""
while (running) do
ready = IO.select([socket])
ready[0].each do |s|
line = s.gets
# Respond to Twitch IRC "PING" Message
if line.index("PING") == 0
line.strip!
socket.puts("PONG :tmi.twitch.tv\r\n")
log.info("[IRC Message]: " + line)
log.info("[IRC Response]: PONG :tmi.twitch.tv")
puts("-".bold.red*line.length)
puts "[Twitch] ".bold.cyan + "IRC: ".bold.yellow + line.bold.green
puts "[Response] ".bold.cyan + "IRC: ".bold.yellow + "PONG :tmi.twitch.tv".bold.green
puts("-".bold.red*line.length)
end
match = line.match(/^:(.+)!(.+)PRIVMSG ##{CHANNEL} :(.+)$/)
message = match && match[3]
if message =~ /^/
message.strip!
user = match[1] # Get username
# Twitch message limit - (Max 100 messages in 30 secs - Applies to mods and above)
# Avoid global ban
if Time.now.to_i - message_limit > 30 # If more than 30 seconds has passed
message_count = 0 # Reset "message_count"
end
if message_count == 0 # If "message_count" is 0
message_limit = Time.now.to_i # Start counting to 30 again
end
message_count = message_count + 1
end
# // COMMANDS // COMMANDS // COMMANDS
if message != nil
msg = message.downcase
# ADMIN COMMANDS // ADMIN COMMANDS
if admin_commands.include?(msg) and user == CHANNEL
if msg.include?("!disconnect")
socket.puts(send + "Disconnecting") # Disconnect from the channel
socket.puts("PART ##{CHANNEL}") # Disconnect from the channel
Disconnect()
log.info("[Command] #{user}: #{message}")
elsif msg.include?("!project")
msg.slice!("!project ")
File.write("Responses/project.txt", msg)
end
user = user.capitalize # Capitalize first letter (Cuz I'm that kind of person)
elsif commands.include?(msg) and message_count < 80
puts "[Command] ".bold.cyan + "#{user}: ".bold + "#{message}".bold.cyan
# API COMMANDS // API COMMANDS
if api_commands.include?(msg)
if msg.include?("!uptime")
file = open("https://decapi.me/twitch/uptime?channel=#{CHANNEL}")
content = "#{CHANNEL} has been streaming for: " + file.read
elsif msg.include?("!followed")
file = open("https://decapi.me/twitch/followage/#{CHANNEL}/#{user}")
content = file.read
if content == "Follow not found"
content = "#{user} is not following #{CHANNEL}"
else
content = "#{user} has been following #{CHANNEL} for " + content
end
end
puts "[Response] ".bold.red + "Cortexio: ".bold + "API: ".bold.yellow + "\"#{content}\"".bold.red
else
file = open "Responses/" + msg.gsub!("!","") + ".txt" # open matching file
content = file.read
file.close
puts "[Response] ".bold.red + "Cortexio: ".bold + "File: ".bold.yellow + "\"#{msg}.txt\"".bold.red
end
file.close
log.info("[Command] #{user}: #{message}")
else
puts "[Message] ".bold.green + "#{user}: ".bold + "#{message}".bold.green
log.info("[Message] #{user}: #{message}")
if message[0] == "!" # Unrecognized command
content = "Unrecognized command: \"#{message}\" - Type !commands to see a list of available commands."
end
end
# Response handling
if content != nil
content.gsub!("USER", "##{user}")
content.gsub!("AGE", "#{age}")
content.gsub!("CHANNEL", "#{CHANNEL}")
if content.include?("COMMANDS")
content.gsub!("COMMANDS", "#{commands}")
content.gsub!("\"", "")
content.gsub!("[","")
content.gsub!("]","")
end
socket.puts(send + content) # Send response if any
content = nil # Too avoid multiple messages with the same response
end
end
end
end
end
def Disconnect() # End script
running = false
exit
end
# Loop to keep bot going
while (running) do
input = gets.chomp
if input == "clear"
system "clear" or system "cls"
end
end
The problem isn't with
if msg.include?("!project")
it's with the line before it:
admin_commands.include?(msg)
admin_commands is an array which contains the 2 strings ["!disconnect","!project"]. It does not, however, contain the string "!project Then some substring here", so you are never checking if msg.include?("!project"). You should be seeing your log messages about an unrecognized command, which means you aren't making it into your first if statement. What you'll want to be doing is something like:
if admin_commands.any? { |command| msg.include?(command) }
if msg.include?("!project")
# ...
I'm working with an AVL (Skypatrol TT8750+) and the messages that it sends (using TCP) are supposed to be 59bytes long but it always sends a first message (the message has some information about the AVL, so the user can identify it) of 33bytes.
So the question is, How can I handle those different size messages on ruby?
require 'socket'
portnumber = 12050
socketServer = TCPServer.open(portnumber)
while true
Thread.new(socketServer.accept) do |connection|
puts "Accepting connection from: #{connection.peeraddr[2]}"
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
out_file = File.new(file_name, "w+")
begin
while connection
incomingData = connection.gets()
if incomingData != nil
incomingData = incomingData
end
hex_line = incomingData.unpack('H*')[0]
out_file.puts(hex_line)
puts "Incoming: #{hex_line}"
end
rescue Exception => e
# Displays Error Message
puts "#{ e } (#{ e.class })"
ensure
connection.close
puts "ensure: Closing"
end
end
end
This is the experimental code that I'm using.
I'm posting this answer to explain a comment I made to Anderson's answer. Most of the code isn't mine.
moving the if out of the loop
When the if statement is within a loop, it will be evaluated each and every time the loop runs, increasing the number of CPU instructions and the complexity of each loop.
You could improve performance by moving the conditional statement out of the loop like so:
require 'socket'
require 'celluloid/io'
portnumber = 12050
socketServer = TCPServer.open(portnumber)
incomingData = nil
while true
Thread.new(socketServer.accept) do |connection|
puts "Accepting connection from: #{connection.peeraddr[2]}"
# this should probably be changed,
# it ignores the possibility of two connections arriving at the same timestamp.
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
out_file = File.new(file_name, "w+")
begin
if connection
incomingData = conection.recv(33)
if incomingData != nil
incomingData = incomingData.unpack('H*')[0]
out_file.puts(incomingData)
puts "Incoming: #{incomingData}"
end
end
while connection
incomingData = connection.recv(59)
if incomingData != nil
incomingData = incomingData.unpack('H*')[0]
out_file.puts(incomingData)
puts "Incoming: #{incomingData}"
end
end
rescue Exception => e
# Displays Error Message
puts "#{ e } (#{ e.class })"
ensure
connection.close
out_file.close
puts "ensure: Closing"
end
end
end
Optimizing the recv method
Another optimization I should probably mention (but won't implement here) would be the recv method call.
This is both an optimization and a possible source for errors that should be addressed.
recv is a system call and as network messages might be combined (or fragmented) across TCP/IP packets, it might become more expensive to call recv than to handle an internal buffer of data that resolved fragmentation and overflow states.
Reconsidering the thread-per-client design
I would also recommend avoiding the thread-per client design.
In general, for a small number of clients it probably doesn't matter much.
However, as clients multiply and threads become busier, you might find the system spends more resources on context switches than actual tasks.
Another concern might be the allocated stack each thread requires (1Mb or 2Mb for Ruby threads, if I remember correctly)... In a best case scenario, 1,000 clients will require more than a GigaByte of memory allocation just for the stack (I'm ignoring kernel structure data table and other resources).
I would consider using EventMachine or Iodine (I'm iodine's author, so I'm biased).
An evented design could save you many resources.
For example (untested):
require 'iodine'
# define the protocol for our service
class ExampleProtocol
#timeout = 10
def on_open
puts "New Connection Accepted."
# this should probably be changed,
# it ignores the possibility of two connections arriving at the same timestamp.
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
#out_file = File.new(file_name, "w+")
# a rolling buffer for fragmented messages
#expecting = 33
#msg = ""
end
def on_message buffer
length = buffer.length
pos = 0
while length >= #expecting
#msg << (buffer[pos, #expecting])
out_file.puts(msg.unpack('H*')[0])
length -= #expecting
pos += #expecting
#expecting = 59
#msg.clear
end
if(length > 0)
#msg << (buffer[pos, length])
#expecting = 59-length
end
end
def on_close
#out_file.close
end
end
# create the service instance
Iodine.listen 12050, ExampleProtocol
# start the service
Iodine.start
The solution was quite simple
require 'socket'
require 'celluloid/io'
portnumber = 12050
socketServer = TCPServer.open(portnumber)
while true
Thread.new(socketServer.accept) do |connection|
puts "Accepting connection from: #{connection.peeraddr[2]}"
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
out_file = File.new(file_name, "w+")
messagecounter = 1
begin
while connection
if messagecounter == 1
incomingData = conection.recv(33)
messagecounter += 1
else
incomingData = connection.recv(59)
end
if incomingData != nil
incomingData = incomingData.unpack('H*')[0]
end
out_file.puts(incomingData)
puts "Incoming: #{incomingData}"
end
rescue Exception => e
# Displays Error Message
puts "#{ e } (#{ e.class })"
ensure
connection.close
puts "ensure: Closing"
end
end
end
I just needed an extra variable and an if to auto increment the variable, and that's it.
I'm trying to define methods to parse through an apache log file and pull ip addresses, URLs, requests per hour, and error codes. I've got everything working outside of methods, but when attempting to put that code into the methods I keep getting the error message "Stack level too deep." Here is the code in question.
class CommonLog
def initialize(logfile)
#logfile = logfile
end
def readfile
#readfile = File.readlines(#logfile).map { |line|
line.split()
}
#readfile = #readfile.to_s.split(" ")
end
def ip_histogram
#ip_count = 0
#readfile.each_index { |index|
if (#readfile[index] =~ /\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/ )
puts #readfile[index]
puts #ip_count += 1
end
}
end
def url_histogram
url_count = 0
cleaned_file.each_index { |index|
if (cleaned_file[index] =~ /\/{1}(([a-z]{4,})|(\~{1}))\:{0}\S+/ )
puts cleaned_file[index]
puts url_count += 1
end
}
end
def requests_per_hour
end
def sorted_list
end
end
my_file = CommonLog.new("test_log")
cleaned_file = my_file.readfile
puts cleaned_file.ip_histogram
It looks like the problem lies on you CommonLog#readfile method:
def readfile
#readfile = File.readlines(#logfile).map { |line|
line.split()
}
#readfile = readfile.to_s.split(" ")
end
Notice that inside the implementation of readfile your calling readfile recursively? When it executes it reads the lines from the file, maps them and assign the result the #readfile; then it calls readfile and the method starts to execute again; this goes on forever, until you stack blows up because of too many recursive method calls.
I assume what you actually meant is:
#readfile = #readfile.to_s.split(" ")
I don't understand why this won't do what the title states.
#!/usr/bin/env ruby
require 'socket'
require 'timeout'
class Scanner
def initialize(host, port)
#host = host
#port = port
end
def popen
begin
array = []
sock = Socket.new(:INET, :STREAM)
sockaddr = Socket.sockaddr_in(#port, #host)
Timeout::timeout(5) do
array.push("Port #{#port}: Open") if sock.connect(sockaddr)
end
puts array
rescue Timeout::Error
puts "Port #{#port}: Filtered"
rescue Errno::ECONNREFUSED
end
end
end # end Scanner
def main
begin
p = 1
case ARGV[0]
when '-p'
eport = ARGV[1]
host = ARGV[2]
else
eport = 65535
host = ARGV[0]
end
t1 = Time.now
puts "\n"
puts "-" * 70
puts "Scanning #{host}..."
puts "-" * 70
while p <= eport.to_i do
scan = Scanner.new(host, p)
scan.popen
p += 1
end
t2 = Time.now
time = t2 - t1
puts "\nScan completed: #{host} scanned in #{time} seconds."
rescue Errno::EHOSTUNREACH
puts "This host appears to be unreachable"
rescue Interrupt
puts "onnection terminated."
end
end
main
What I'm trying to achieve is an output similar to nmap, in the way that it scans everything, and then shows all open or closed ports at the end. Instead what happens is that it prints them out as it discovers them. I figured pushing the output into an array then printing the array would achieve such an output, yet it still prints out the ports one at a time. Why is this happening?
Also, I apologize for the formatting, the code tags are a little weird.
Your loop calls popen once per iteration. Your popen method sets array = [] each time it is called, then populates it with one item, then you print it with puts. On the next loop iteration, you reset array to [] and do it all again.
You only asked "why," but – you could solve this by setting array just once in the body of main and then passing it to popen (or any number of ways).