I'm using Celluloid IO to read from sockets. The incoming message has the following syntax
sometextsometextsometext
where
SOH = Hex 1
FS = Hex 1C
STX = Hex 2
ETX = Hex 3
EOT = Hex 4
My read code is something like this -
message = ""
begin
data = socket.readpartial(4096)
message << data
end until message =~ /not sure what goes here/
I'm looking for a reliable way to read from the socket until EOT. Once the message is read, i'll regex out the relevant sections.
Some guidance on detecting the above mentioned hex characters in socket read stream and in regex would be very helpful. Guidance?
And this does the trick for me thanks
def parse(message)
if message =~ /\001(.*)\01C(.*)\002(.*)\003\004/
return ($1,$2,$3)
end
end
def read_until_eot(socket)
eot_found = false
message = ''
begin
data = socket.read()
eot_found = !!data['\004']
message << data
end until eot_found
message.chomp!
end
def handle_connection(socket)
# read from socket until EOT
message = read_until_eot(socket) # <-- need help with
if (origin,target,payload) = parse(message) #message can be parsed
# process message
output_message = process(payload)
end
# write to socket
socket.write output_message
# close socket
socket.close
end
Related
I'm new to ruby and I'm trying to make a client to connect to a TCPServer, and it seems that in order to do so I have to call the method close_write every time I finish sending data one way, to let the client/server know that the other end is finished sending data. Whenever I do that then Im not able to write info to the server or client again because the socket is not opened for writing anymore.
This is my code:
client.rb
require "socket"
socket = TCPSocket.open("localhost", 6666)
loop do
input = gets.chomp
socket.puts input # Send data to server
socket.close_write
while(line = socket.gets)
puts line
end # Print sever response
break if input=="EXIT"
end
socket.close
server.rb
require "socket"
server = TCPServer.new 6666
data = Hash.new { |hash, key| hash[key] = {} }
WAITING_SET_VALUE = "1"
WAITING_NEW_COMMAND = "0"
loop do
Thread.start(server.accept) do |session|
thread_status ||= WAITING_NEW_COMMAND
....
puts "Entering If..."
if(thread_status == WAITING_NEW_COMMAND) #Check thread status
puts "thread_status == WAITING_NEW_COMMAND"
puts "CHECKING COMMAND.."
case line.strip
when /^set \w* \w* \d{1,7} \d{1,7}$/
puts "COMMAND SET"
thread_status = WAITING_SET_VALUE
lineArr = line.strip.split(" ")
varName = lineArr[1]
flag = lineArr[2]
ttl = lineArr[3]
size = lineArr[4]
puts "END SET EXECUTION"
session.write "Executed"
session.close_write
...
Is there a way to open the socket for writing again, or a better way to do this back and forth connection between server and client without losing context? Thanks!
When designing a client-server protocol, you have to decide:
How a client knows when a response has more lines/data.
How a client knows when a response is complete.
How a client knows when a response is invalid/valid.
How a client knows when there was some type of server error.
A simple approach is for the server to return a response with the number of lines (as in the code below). However, instead, you could use END or something so that the client knows when there is no more data to read. I would strongly suggest looking into tutorials about Protocols.
Save the below into a file called client_server.rb. First, run the server with ruby ./client_server.rb s and then in a separate terminal run the client with ruby ./client_server.rb c. Next, type in list over and over to see the different responses. I added list so that you don't have to type in set w w 1 1 over and over for testing purposes. Check it out and let me know if you have any questions.
# frozen_string_literal: true
require 'socket'
# Use:
# First, run the server: ruby ./client_server.rb s
# Then, run the client: ruby ./client_server.rb c
# Use "netcat localhost 6666" on the command line to test
# without implementing a client.
# Returns false to close this client socket.
# Returns true to continue reading from this client socket.
def handle_client(client_id, client_socket, command)
# TODO: Define some type of client-server Protocol here.
case command
when /^set \w* \w* \d{1,7} \d{1,7}$/
puts "Running command for client #{client_id}: #{command}"
# This is just for testing purposes.
case rand(3)
when 0
client_socket.puts 'lines 0'
when 1
client_socket.puts 'lines 3'
client_socket.puts 'This is line 1.'
client_socket.puts 'This is line 2.'
client_socket.puts 'This is line 3.'
when 2
client_socket.puts 'The set command returned an error.'
end
when 'list'
puts "Responding to client #{client_id} with list of messages."
# This is just for testing purposes.
case rand(3)
when 0
client_socket.puts 'messages 0'
when 1
client_socket.puts 'messages 2'
client_socket.puts 'This is message 1.'
client_socket.puts 'This is message 2.'
when 2
client_socket.puts 'Unable to retrieve messages due to error.'
end
when 'bye'
puts "Killing client #{client_id}."
return false # Disconnect and kill the thread.
else
client_socket.puts "[ERROR] Invalid command: #{command}"
end
client_socket.flush # Flush all output just in case.
true # Continue connection.
end
case ARGV[0].to_s.downcase
when 's' # server
TCPServer.open(6666) do |server_socket|
global_client_id = 1
loop do
Thread.start(global_client_id, server_socket.accept) do |client_id, client_socket|
puts "Accepting new client #{client_id}."
loop do
command = client_socket.gets
if command.nil?
puts "Client #{client_id} disconnected manually."
break
end
command = command.strip
keep_alive = handle_client(client_id, client_socket, command)
break unless keep_alive
end
client_socket.close
end
global_client_id += 1
end
end
when 'c' # client
TCPSocket.open('localhost', 6666) do |socket|
puts '[Commands]'
puts 'set <word> <word> <num> <num> Run set command.'
puts 'list Get list of messages.'
puts 'exit, bye Exit the client.'
puts
loop do
print '> '
input = $stdin.gets.strip
# TODO: Define some type of client-server Protocol here.
case input
when /EXIT|BYE/i
socket.puts 'bye'
socket.flush
break
when /\Aset .+\z/
socket.puts input
socket.flush
response = socket.gets
if response.nil?
puts 'Server is not running anymore! Disconnecting.'
break
end
response = response.strip
match_data = response.match(/\Alines (?<lines>\d+)\z/)
if match_data
line_count = match_data[:lines].to_i
puts "Received #{line_count} lines from server."
1.upto(line_count) do |i|
line = socket.gets
puts ">> Resp[#{i}] #{line}"
end
else
# Can check for "response == 'ERROR'" or something.
puts "Server error or invalid response from server: #{response}"
end
when 'list'
socket.puts input
socket.flush
response = socket.gets
if response.nil?
puts 'Server is not running anymore! Disconnecting.'
break
end
response = response.strip
match_data = response.match(/\Amessages (?<messages>\d+)\z/)
if match_data
message_count = match_data[:messages].to_i
puts "Received #{message_count} messages from server."
1.upto(message_count) do |i|
line = socket.gets
puts ">> Resp[#{i}] #{line}"
end
else
# Can check for "response == 'ERROR'" or something.
puts "Server error or invalid response from server: #{response}"
end
else
puts "Invalid command: #{input}"
end
end
end
else
puts "Pass in 'c' for client or 's' for server."
end
I have a few commands from a Twitch chat, it works and it's awesome! People can type commands like !about and it's going to send a response. There's just one problem, if you go down to the thread and look under # // COMMANDS // COMMANDS // COMMANDS. There's a problem, there´s a few admin commands (look under # ADMIN COMMANDS // ADMIN COMMANDS). I have this: if msg.include?("!project") then msg.slice!("!project "), but it's not working like I want it to. I want to be able to type !project Then some substring here. But when a substring is attached it doesn't work, even though I can print msg and it clearly includes !project. Both the !disconnect and !project command work when it's only those commands by themselves. Meaning that something like !disconnect Hello World wouldn't work even though it clearly includes !disconnect.
# Message formatting in console
class String
def red; "\e[31m#{self}\e[0m" end
def yellow; "\e[33m#{self}\e[0m" end
def green; "\e[32m#{self}\e[0m" end
def cyan; "\e[36m#{self}\e[0m" end
def bold; "\e[1m#{self}\e[22m" end
end
# Requied packages / modules
require 'socket'
require 'logger'
require 'open-uri'
# Create logger
File.delete("log.txt") # Clear previous log
log = Logger.new("log.txt", formatter: proc {|severity, datetime, progname, msg|
"#{datetime}: #{msg}\n"})
# Required Info
load "credentials.txt"
log.info("Loading \"credentials.txt\"")
# -------- IGNORE -------- #
OAUTH.downcase!
BOTNAME.downcase!
CHANNEL.downcase!.gsub!("#", "")
# //- AGE -// #
time = Time.new
age = time.year - 2000
if time.month == 10
if time.day < 28
age -= 1
end
elsif time.month < 10
age -= 1
end
# -------- IGNORE -------- #
# Save "Preparing to connect" to "log.txt"
log.info("Preparing to connect")
# Variables
socket = TCPSocket.new('irc.chat.twitch.tv', 6667)
send = "PRIVMSG ##{CHANNEL} :" # shortcut for sending messages
running = true
content = nil
message_count = 0
message_limit = Time.now.to_i
# Commands
commands = ["!about","!uptime","!commands","!cortexio","!followed"]
api_commands = ["!followed","!uptime"]
admin_commands = ["!disconnect","!project"]
# Authorization Login
socket.puts("PASS #{OAUTH}") # Send the password(oauth) to Twitch
socket.puts("NICK #{BOTNAME}") # Send the botname to Twitch
socket.puts("JOIN ##{CHANNEL}") # Send the channel to Twitch
# Save "Connected!" to "log.txt
log.info("Joining #{CHANNEL.capitalize} as #{BOTNAME.capitalize} using OAUTH Token: #{OAUTH[6,OAUTH.length-12]}" + "*"*12)
# Thread.abort_on_exception = true
# Loop (Background Thread) for recieving Twitch chat data
Thread.start do
socket.puts(send + "Connected!") # Send "Connected!" to the Twitch channel
puts "#{BOTNAME} Joined ##{CHANNEL}" # Connection Status
puts "You should be fully connected now" # Connection Status
puts ""
puts "Type \"clear\" to clear terminal"
puts ""
while (running) do
ready = IO.select([socket])
ready[0].each do |s|
line = s.gets
# Respond to Twitch IRC "PING" Message
if line.index("PING") == 0
line.strip!
socket.puts("PONG :tmi.twitch.tv\r\n")
log.info("[IRC Message]: " + line)
log.info("[IRC Response]: PONG :tmi.twitch.tv")
puts("-".bold.red*line.length)
puts "[Twitch] ".bold.cyan + "IRC: ".bold.yellow + line.bold.green
puts "[Response] ".bold.cyan + "IRC: ".bold.yellow + "PONG :tmi.twitch.tv".bold.green
puts("-".bold.red*line.length)
end
match = line.match(/^:(.+)!(.+)PRIVMSG ##{CHANNEL} :(.+)$/)
message = match && match[3]
if message =~ /^/
message.strip!
user = match[1] # Get username
# Twitch message limit - (Max 100 messages in 30 secs - Applies to mods and above)
# Avoid global ban
if Time.now.to_i - message_limit > 30 # If more than 30 seconds has passed
message_count = 0 # Reset "message_count"
end
if message_count == 0 # If "message_count" is 0
message_limit = Time.now.to_i # Start counting to 30 again
end
message_count = message_count + 1
end
# // COMMANDS // COMMANDS // COMMANDS
if message != nil
msg = message.downcase
# ADMIN COMMANDS // ADMIN COMMANDS
if admin_commands.include?(msg) and user == CHANNEL
if msg.include?("!disconnect")
socket.puts(send + "Disconnecting") # Disconnect from the channel
socket.puts("PART ##{CHANNEL}") # Disconnect from the channel
Disconnect()
log.info("[Command] #{user}: #{message}")
elsif msg.include?("!project")
msg.slice!("!project ")
File.write("Responses/project.txt", msg)
end
user = user.capitalize # Capitalize first letter (Cuz I'm that kind of person)
elsif commands.include?(msg) and message_count < 80
puts "[Command] ".bold.cyan + "#{user}: ".bold + "#{message}".bold.cyan
# API COMMANDS // API COMMANDS
if api_commands.include?(msg)
if msg.include?("!uptime")
file = open("https://decapi.me/twitch/uptime?channel=#{CHANNEL}")
content = "#{CHANNEL} has been streaming for: " + file.read
elsif msg.include?("!followed")
file = open("https://decapi.me/twitch/followage/#{CHANNEL}/#{user}")
content = file.read
if content == "Follow not found"
content = "#{user} is not following #{CHANNEL}"
else
content = "#{user} has been following #{CHANNEL} for " + content
end
end
puts "[Response] ".bold.red + "Cortexio: ".bold + "API: ".bold.yellow + "\"#{content}\"".bold.red
else
file = open "Responses/" + msg.gsub!("!","") + ".txt" # open matching file
content = file.read
file.close
puts "[Response] ".bold.red + "Cortexio: ".bold + "File: ".bold.yellow + "\"#{msg}.txt\"".bold.red
end
file.close
log.info("[Command] #{user}: #{message}")
else
puts "[Message] ".bold.green + "#{user}: ".bold + "#{message}".bold.green
log.info("[Message] #{user}: #{message}")
if message[0] == "!" # Unrecognized command
content = "Unrecognized command: \"#{message}\" - Type !commands to see a list of available commands."
end
end
# Response handling
if content != nil
content.gsub!("USER", "##{user}")
content.gsub!("AGE", "#{age}")
content.gsub!("CHANNEL", "#{CHANNEL}")
if content.include?("COMMANDS")
content.gsub!("COMMANDS", "#{commands}")
content.gsub!("\"", "")
content.gsub!("[","")
content.gsub!("]","")
end
socket.puts(send + content) # Send response if any
content = nil # Too avoid multiple messages with the same response
end
end
end
end
end
def Disconnect() # End script
running = false
exit
end
# Loop to keep bot going
while (running) do
input = gets.chomp
if input == "clear"
system "clear" or system "cls"
end
end
The problem isn't with
if msg.include?("!project")
it's with the line before it:
admin_commands.include?(msg)
admin_commands is an array which contains the 2 strings ["!disconnect","!project"]. It does not, however, contain the string "!project Then some substring here", so you are never checking if msg.include?("!project"). You should be seeing your log messages about an unrecognized command, which means you aren't making it into your first if statement. What you'll want to be doing is something like:
if admin_commands.any? { |command| msg.include?(command) }
if msg.include?("!project")
# ...
or is there anyway to get this function working in ruby socket programming:
conn = pwn.remote('localhost', 4567)
conn.recvuntil("> ")
With native Ruby socket module, socket.read(bytes) will wait until it receives exact number of bytes, which i cannot decide apriori. And, socket.recv(bytes), will print as soon as it gets some data, regardless of message being complete.
Since, I do know the message format and its ending, it would be easy, if there was a gem as mentioned above or any way to get the functionality in Ruby.
For eg, if i know the message from the server ends with colon,
socket = Socket.new( AF_INET, SOCK_STREAM, 0 )
sockaddr = Socket.pack_sockaddr_in(port, hostname)
socket.connect( sockaddr )
socket.recvuntil ": "
# do something
socket.send "y"
require 'socket'
def read_until(socket, re)
accu = ''
re = Regexp.escape(re) if String === re
re = /#{re}\Z/
socket.each_char do |c|
accu << c
return accu if re === accu
end
return accu
end
socket = Socket.new(Socket::AF_INET, Socket::SOCK_STREAM, 0 )
sockaddr = Socket.pack_sockaddr_in(80, 'www.google.com')
socket.connect(sockaddr)
socket.write "GET / HTTP/1.0\n\n"
headers = read_until(socket, /\r\n\r\n/)
body = socket.read
puts "HEADERS:\n#{headers}BODY:\n#{body}"
Here is my code to listen client TCP socket:
def initialize
#msg = ''
#messages = Queue.new
#socket = TCPSocket.open('127.0.0.1', 2000)
Thread.new do
loop do
ch = #socket.recv(1)
if ch == "\n"
puts #msg unless #msg.blank?
#msg = ''
else
#msg += ch
end
end
end
end
What I don't like is byte-by-byte string concatenation. It should be not memory-efficient.
The read method of socket reads until newline. Could the socket read until some custom terminator character, for example 0x00 ?
If not, then which memory-efficient string appenging do you know?
You could use IO#gets with a custom separator:
# tcp_test.rb
require 'socket'
TCPSocket.open('127.0.0.1', 2000) do |socket|
puts socket.gets("\0").chomp("\0") # chomp removes the separator
end
Test server using Netcat:
$ echo -ne "foo\0bar" | nc -l 2000
Output:
$ ruby tcp_test.rb
foo
You could even set the input record separator to "\0":
$/ = "\0"
puts socket.gets.chomp
I need to read a file in MB chunks, is there a cleaner way to do this in Ruby:
FILENAME="d:\\tmp\\file.bin"
MEGABYTE = 1024*1024
size = File.size(FILENAME)
open(FILENAME, "rb") do |io|
read = 0
while read < size
left = (size - read)
cur = left < MEGABYTE ? left : MEGABYTE
data = io.read(cur)
read += data.size
puts "READ #{cur} bytes" #yield data
end
end
Adapted from the Ruby Cookbook page 204:
FILENAME = "d:\\tmp\\file.bin"
MEGABYTE = 1024 * 1024
class File
def each_chunk(chunk_size = MEGABYTE)
yield read(chunk_size) until eof?
end
end
open(FILENAME, "rb") do |f|
f.each_chunk { |chunk| puts chunk }
end
Disclaimer: I'm a ruby newbie and haven't tested this.
Alternatively, if you don't want to monkeypatch File:
until my_file.eof?
do_something_with( my_file.read( bytes ) )
end
For example, streaming an uploaded tempfile into a new file:
# tempfile is a File instance
File.open( new_file, 'wb' ) do |f|
# Read in small 65k chunks to limit memory usage
f.write(tempfile.read(2**16)) until tempfile.eof?
end
You can use IO#each(sep, limit), and set sep to nil or empty string, for example:
chunk_size = 1024
File.open('/path/to/file.txt').each(nil, chunk_size) do |chunk|
puts chunk
end
If you check out the ruby docs:
http://ruby-doc.org/core-2.2.2/IO.html
there's a line that goes like this:
IO.foreach("testfile") {|x| print "GOT ", x }
The only caveat is. Since, this process can read the temp file faster than the
generated stream, IMO, a latency should be thrown in.
IO.foreach("/tmp/streamfile") {|line|
ParseLine.parse(line)
sleep 0.3 #pause as this process will discontine if it doesn't allow some buffering
}
https://ruby-doc.org/core-3.0.2/IO.html#method-i-read gives an example of iterating over fixed length records with read(length):
# iterate over fixed length records
open("fixed-record-file") do |f|
while record = f.read(256)
# ...
end
end
If length is a positive integer, read tries to read length bytes without any conversion (binary mode). It returns nil if an EOF is encountered before anything can be read. Fewer than length bytes are returned if an EOF is encountered during the read. In the case of an integer length, the resulting string is always in ASCII-8BIT encoding.
FILENAME="d:/tmp/file.bin"
class File
MEGABYTE = 1024*1024
def each_chunk(chunk_size=MEGABYTE)
yield self.read(chunk_size) until self.eof?
end
end
open(FILENAME, "rb") do |f|
f.each_chunk {|chunk| puts chunk }
end
It works, mbarkhau. I just moved the constant definition to the File class and added a couple of "self"s for clarity's sake.