Has anyone tried streaming html/text/content from webrick? I've tried assigning an IO to the response body, but webrick is waiting for the stream to be closed first.
found this link by accident (http://redmine.ruby-lang.org/attachments/download/161) which contains webrick patch
# Copyright (C) 2008 Brian Candler, released under Ruby Licence.
#
# A collection of small monkey-patches to webrick.
require 'webrick'
module WEBrick
class HTTPRequest
# Generate HTTP/1.1 100 continue response. See
# http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/18459
def continue
if self['expect'] == '100-continue' && #config[:HTTPVersion] >= "1.1"
#socket.write "HTTP/#{#config[:HTTPVersion]} 100 continue\r\n\r\n"
#header.delete('expect')
end
end
end
class HTTPResponse
alias :orig_setup_header :setup_header
# Correct termination of streamed HTTP/1.1 responses. See
# http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/18454 and
# http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/18565
def setup_header
orig_setup_header
unless chunked? || #header['content-length']
#header['connection'] = "close"
#keep_alive = false
end
end
# Allow streaming of zipfile entry. See
# http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/18460
def send_body(socket)
if #body.respond_to?(:read) then send_body_io(socket)
elsif #body.respond_to?(:call) then send_body_proc(socket)
else send_body_string(socket)
end
end
# If the response body is a proc, then we invoke it and pass in
# an object which supports "write" and "<<" methods. This allows
# arbitary output streaming.
def send_body_proc(socket)
if #request_method == "HEAD"
# do nothing
elsif chunked?
#body.call(ChunkedWrapper.new(socket, self))
_write_data(socket, "0#{CRLF}#{CRLF}")
else
size = #header['content-length'].to_i
#body.call(socket) # TODO: StreamWrapper which supports offset, size
#sent_size = size
end
end
class ChunkedWrapper
def initialize(socket, resp)
#socket = socket
#resp = resp
end
def write(buf)
return if buf.empty?
data = ""
data << format("%x", buf.size) << CRLF
data << buf << CRLF
socket = #socket
#resp.instance_eval {
_write_data(socket, data)
#sent_size += buf.size
}
end
alias :<< :write
end
end
end
if RUBY_VERSION < "1.9"
old_verbose, $VERBOSE = $VERBOSE, nil
# Increase from default of 4K for efficiency, similar to
# http://svn.ruby-lang.org/cgi-bin/viewvc.cgi/branches/ruby_1_8/lib/net/protocol.rb?r1=11708&r2=12092
# In trunk the default is 64K and can be adjusted using :InputBufferSize,
# :OutputBufferSize
WEBrick::HTTPRequest::BUFSIZE = 16384
WEBrick::HTTPResponse::BUFSIZE = 16384
$VERBOSE = old_verbose
end
to use simply pass a proc to as the response body, like so
res.body = proc { |w|
10.times do
w << Time.now.to_s
sleep(1)
end
}
woot!
I would suggest against using WEBrick for anything really, it's junk. I would say try Mongrel.
I know that wasn't your question, it's just some friendly advice.
Related
I'm pretty new to Ruby, having decided to try my hand with another programming language. I have been trying to get to grips with sockets, which is an area I'm not too familiar with in general. I have created a basic 'game' that allows a player to move his icon around the screen, however, I am trying to make this into something that could be multiplayer using TCP Sockets.
At the moment, when a player launches the game as a host, it creates a server. If the player starts a game as a client, it connects to the server. Currently, it only works on the same machine, but the connection has been established successfully, and when connecting as a client, the server creates a username, sends this back to the client, which then uses it to create a player.
The problem comes when i try to communicate between the server and the client, the messages appear to be sending from server to client, but only partially, and it appears they are largely truncated at either the beginning or end.
If anyone could advice on what is causing this, it would be greatly appreciated. I am using Ruby with Gosu and Celluloid-IO.
SERVER CLASS
require 'src/Game.rb'
require 'celluloid/io'
class Server
include Celluloid::IO
finalizer:shutdown
def initialize
puts ("server init")
#super
#playersConnected = Array.new
#actors = Array.new
#server = TCPServer.new("0.0.0.0", 28888)
##server.open(8088)
puts #server
async.run
end
def run
loop {
async.handle_connection #server.accept
}
end
def readMessage(socket, player)
msg = socket.recv(30)
data = msg.split"|"
#playersConnected.each do |p|
if p.getUser() == player
p.setX(data[1])
p.setY(data[2])
end
#puts "END"
end
end
def handle_connection(socket)
_, port, host = socket.peeraddr
user = "#{host}:#{port}"
puts "#{user} has joined the game"
#playersConnected.push(Player.new(user))
socket.send "#{user}", 0
#socket.send "#{Player}|#{user}", 0
#socket.send "#{port}", 0
puts "PLAYER LIST"
#playersConnected.each do |player|
puts player
end
Thread.new{
loop{
readMessage(socket, user)
#divide message
#find array index
#update character position in array
}
}
Thread.new{
loop{
#playersConnected.each do |p|
msg = p.getUser() + "|" + "#{p.getX}" + "|" + "#{p.getY}"
socket.send(msg, 0)
end
}
}
end
Server.new
end
CLIENT CLASS
require 'src/Game.rb'
class Client < Game
#finalizer :shutdown
def initialize
super
#socket = TCPSocket.new("localhost", 28888)
#188.222.55.241
while #player.nil?
#player = Player.new(#socket.recv(1024))
puts #player
puts #player.getUser()
#player.warp(0, 0)
pulse
end
end
def pulse
Thread.new{
loop{
msg = #player.getUser() + "|" + "#{#player.getX()}" + "|" + "#{#player.getY()}"
#socket.write(msg)
}
}
Thread.new{
loop{
msg = #socket.recv(1024)
data = msg.split"|"
puts data[0]
match = false
#players.each do |player|
if player.getUser() == data[0]
puts "MATCHX"
player.setX(data[1])
player.setY(data[2])
match = true
end
if match == false
p = Player.new(data[0])
#p.warp(data[1],data[2])
#players.push(p)
end
end
puts "end"
}
}
end
Client.new.show
end
Side note; There is also a Host class, which mimics the client class, only it calls the server. I am aware this is a terrible way to do things, i intend to fix it once i overcome the current issue.
Many thanks in advance
I'm working with an AVL (Skypatrol TT8750+) and the messages that it sends (using TCP) are supposed to be 59bytes long but it always sends a first message (the message has some information about the AVL, so the user can identify it) of 33bytes.
So the question is, How can I handle those different size messages on ruby?
require 'socket'
portnumber = 12050
socketServer = TCPServer.open(portnumber)
while true
Thread.new(socketServer.accept) do |connection|
puts "Accepting connection from: #{connection.peeraddr[2]}"
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
out_file = File.new(file_name, "w+")
begin
while connection
incomingData = connection.gets()
if incomingData != nil
incomingData = incomingData
end
hex_line = incomingData.unpack('H*')[0]
out_file.puts(hex_line)
puts "Incoming: #{hex_line}"
end
rescue Exception => e
# Displays Error Message
puts "#{ e } (#{ e.class })"
ensure
connection.close
puts "ensure: Closing"
end
end
end
This is the experimental code that I'm using.
I'm posting this answer to explain a comment I made to Anderson's answer. Most of the code isn't mine.
moving the if out of the loop
When the if statement is within a loop, it will be evaluated each and every time the loop runs, increasing the number of CPU instructions and the complexity of each loop.
You could improve performance by moving the conditional statement out of the loop like so:
require 'socket'
require 'celluloid/io'
portnumber = 12050
socketServer = TCPServer.open(portnumber)
incomingData = nil
while true
Thread.new(socketServer.accept) do |connection|
puts "Accepting connection from: #{connection.peeraddr[2]}"
# this should probably be changed,
# it ignores the possibility of two connections arriving at the same timestamp.
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
out_file = File.new(file_name, "w+")
begin
if connection
incomingData = conection.recv(33)
if incomingData != nil
incomingData = incomingData.unpack('H*')[0]
out_file.puts(incomingData)
puts "Incoming: #{incomingData}"
end
end
while connection
incomingData = connection.recv(59)
if incomingData != nil
incomingData = incomingData.unpack('H*')[0]
out_file.puts(incomingData)
puts "Incoming: #{incomingData}"
end
end
rescue Exception => e
# Displays Error Message
puts "#{ e } (#{ e.class })"
ensure
connection.close
out_file.close
puts "ensure: Closing"
end
end
end
Optimizing the recv method
Another optimization I should probably mention (but won't implement here) would be the recv method call.
This is both an optimization and a possible source for errors that should be addressed.
recv is a system call and as network messages might be combined (or fragmented) across TCP/IP packets, it might become more expensive to call recv than to handle an internal buffer of data that resolved fragmentation and overflow states.
Reconsidering the thread-per-client design
I would also recommend avoiding the thread-per client design.
In general, for a small number of clients it probably doesn't matter much.
However, as clients multiply and threads become busier, you might find the system spends more resources on context switches than actual tasks.
Another concern might be the allocated stack each thread requires (1Mb or 2Mb for Ruby threads, if I remember correctly)... In a best case scenario, 1,000 clients will require more than a GigaByte of memory allocation just for the stack (I'm ignoring kernel structure data table and other resources).
I would consider using EventMachine or Iodine (I'm iodine's author, so I'm biased).
An evented design could save you many resources.
For example (untested):
require 'iodine'
# define the protocol for our service
class ExampleProtocol
#timeout = 10
def on_open
puts "New Connection Accepted."
# this should probably be changed,
# it ignores the possibility of two connections arriving at the same timestamp.
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
#out_file = File.new(file_name, "w+")
# a rolling buffer for fragmented messages
#expecting = 33
#msg = ""
end
def on_message buffer
length = buffer.length
pos = 0
while length >= #expecting
#msg << (buffer[pos, #expecting])
out_file.puts(msg.unpack('H*')[0])
length -= #expecting
pos += #expecting
#expecting = 59
#msg.clear
end
if(length > 0)
#msg << (buffer[pos, length])
#expecting = 59-length
end
end
def on_close
#out_file.close
end
end
# create the service instance
Iodine.listen 12050, ExampleProtocol
# start the service
Iodine.start
The solution was quite simple
require 'socket'
require 'celluloid/io'
portnumber = 12050
socketServer = TCPServer.open(portnumber)
while true
Thread.new(socketServer.accept) do |connection|
puts "Accepting connection from: #{connection.peeraddr[2]}"
t = Time.now.strftime("%d-%m-%Y %H%M")
file_name = t + '.txt'
out_file = File.new(file_name, "w+")
messagecounter = 1
begin
while connection
if messagecounter == 1
incomingData = conection.recv(33)
messagecounter += 1
else
incomingData = connection.recv(59)
end
if incomingData != nil
incomingData = incomingData.unpack('H*')[0]
end
out_file.puts(incomingData)
puts "Incoming: #{incomingData}"
end
rescue Exception => e
# Displays Error Message
puts "#{ e } (#{ e.class })"
ensure
connection.close
puts "ensure: Closing"
end
end
end
I just needed an extra variable and an if to auto increment the variable, and that's it.
I was trying to make a script that downloads all images or videos from a thread in my favourite imageboard: 2ch.hk
I was successful until I wanted to download these files asynchronously (for example, to improve performance)
Here is the code http://ideone.com/k2l4Hm
file = http.get(source).body
require 'net/http'
multithreading = false
Net::HTTP.start("2ch.hk", :use_ssl => true) do |http|
thread = http.get("/b/res/133467978.html").body
sources = []
thread.scan(/<a class="desktop" target="_blank" href=".+">.+<\/a>/).each do |a|
source = "/b#{/<a class="desktop" target="_blank" href="\.\.(.+)">.+<\/a>/.match(a).to_a[1]}"
sources << source
end
i = 0
start = Time.now
if multithreading
threads = []
sources.each do |source|
threads << Thread.new(i) do |j|
file = http.get(source).body #breaks everything
# type = /.+\.(.+)/.match(source)[1]
# open("#{j}.#{type}","wb") { |new_file|
# new_file.write(file)
# }
end
i += 1
end
threads.each do |thr|
thr.join
end
# until downloade=sources.size
#
# end
else
sources.each do |source|
file = http.get(source).body
type = /.+\.(.+)/.match(source)[1]
open("#{i}.#{type}","wb") { |new_file|
new_file.write(file)
}
i += 1
print "#{(((i).to_f / sources.size) * 100).round(2)}% "
end
puts
end
puts "Done. #{i} files were downloaded. It took #{Time.now - start} seconds"
end
I suppose that this line crashes everything.
file = http.get(source).body
Or maybe that's the problem.
threads.each do |thr|
thr.join
end
Error messages are always different, from Bad File Descriptor and IO errors to "You may have encountered a bug in the Ruby interpreter or extension libraries."
If you want to try and run my code, please substitute a link to thread in 4th line with a new thread (from 2ch.hk/b), because the one in my code may be deleted by the time you run my code
Version of ruby: 2.3.1, OS Xubuntu 16.10
You'll probably have much better performance using a ruby http lib that supports parallel requests:
https://github.com/typhoeus/typhoeus
e.g.
hydra = Typhoeus::Hydra.new
10.times.map{ hydra.queue(Typhoeus::Request.new("www.example.com", followlocation: true)) }
hydra.run
The problem with my code is that I can't make multiple requests on a Net::HTTP instance at the same time.
The solution is to open an HTTP connection for each thread.
I created a script which checks healthcheck and ports status from a .json file populated with microservices.
So for every microservice from the .json file the script will output the HTTP status and healthcheck body and other small details, and I want to add multithreading here in order to return all the output at once.Please see the script below:
#!/usr/bin/env ruby
... get the environment argument part...
file = File.read('./services.json')
data_hash = JSON.parse(file)
threads = []
service = data_hash.keys
service.each do |microservice|
threads << Thread.new do
begin
puts "Microservice: #{microservice}"
port = data_hash["#{microservice}"]['port']
puts "Port: #{port}"
nodes = "knife search 'chef_environment:#{env} AND recipe:#{microservice}' -i"
node = %x[ #{nodes} ].split
node.each do |n|
puts "Node: #{n}"
uri = URI("http://#{n}:#{port}/healthcheck?count=10")
res = Net::HTTP.get_response(uri)
status = Net::HTTP.get(uri)
puts res.code
puts status
puts res.message
end
rescue Net::ReadTimeout
puts "ReadTimeout Error"
next
end
end
end
threads.each do |thread|
thread.join
end
Anyway in this way the script return first the puts "Microservice: #{microservice}" and puts "Port: #{port}" and after this it will return the nodes and only after the STATUS.
How can I return all the data for each loop together?
Instead of puts write output to a variable (hash).
If you wand to wait for all threads to finish their job before showing the output, use ThreadsWait class.
require 'thwait'
file = File.read('./services.json')
data_hash = JSON.parse(file)
h = {}
threads = []
service = data_hash.keys
service.each do |microservice|
threads << Thread.new do
thread_id = Thread.current.object_id.to_s(36)
begin
h[thread_id] = "Microservice: #{microservice}"
port = data_hash["#{microservice}"]['port']
h[thread_id] << "Port: #{port}"
nodes = "knife search 'chef_environment:#{env} AND recipe:#{microservice}' -i"
node = %x[ #{nodes} ].split
node.each do |n|
h[thread_id]<< "Node: #{n}"
uri = URI("http://#{n}:#{port}/healthcheck?count=10")
res = Net::HTTP.get_response(uri)
status = Net::HTTP.get(uri)
h[thread_id] << res.code
h[thread_id] << status
h[thread_id] << res.message
end
rescue Net::ReadTimeout
h[thread_id] << "ReadTimeout Error"
next
end
end
end
threads.each do |thread|
thread.join
end
# wait untill all threads finish their job
ThreadsWait.all_waits(*threads)
p h
[edit]
ThreadsWait.all_waits(*threads) is redundant in above code and can be omitted, since line treads.each do |thread| thread.join end does exactely the same thing.
Instead of outputting the data as you get it using puts, you can collect it all in a string and then puts it once at the end. Strings can take the << operator (implemented as a method in Ruby), so you can just initialize the string, add to it, and then output it at the end, like this:
report = ''
report << 'first thing'
report << 'second thing'
puts report
You could even save them all up together and print them all after all were finished if you want.
I'm trying to define methods to parse through an apache log file and pull ip addresses, URLs, requests per hour, and error codes. I've got everything working outside of methods, but when attempting to put that code into the methods I keep getting the error message "Stack level too deep." Here is the code in question.
class CommonLog
def initialize(logfile)
#logfile = logfile
end
def readfile
#readfile = File.readlines(#logfile).map { |line|
line.split()
}
#readfile = #readfile.to_s.split(" ")
end
def ip_histogram
#ip_count = 0
#readfile.each_index { |index|
if (#readfile[index] =~ /\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/ )
puts #readfile[index]
puts #ip_count += 1
end
}
end
def url_histogram
url_count = 0
cleaned_file.each_index { |index|
if (cleaned_file[index] =~ /\/{1}(([a-z]{4,})|(\~{1}))\:{0}\S+/ )
puts cleaned_file[index]
puts url_count += 1
end
}
end
def requests_per_hour
end
def sorted_list
end
end
my_file = CommonLog.new("test_log")
cleaned_file = my_file.readfile
puts cleaned_file.ip_histogram
It looks like the problem lies on you CommonLog#readfile method:
def readfile
#readfile = File.readlines(#logfile).map { |line|
line.split()
}
#readfile = readfile.to_s.split(" ")
end
Notice that inside the implementation of readfile your calling readfile recursively? When it executes it reads the lines from the file, maps them and assign the result the #readfile; then it calls readfile and the method starts to execute again; this goes on forever, until you stack blows up because of too many recursive method calls.
I assume what you actually meant is:
#readfile = #readfile.to_s.split(" ")