I'm developing API, based on EventMachine echo server. It listens requests on specific port and returns html-page with qr-code on it, which is generated depending on params from request query. The problem is that evaluating method, which packs string into qr-code takes from 8 to 11 seconds, it is unacceptable. I have no ideas why it happens, except it could be associated with event-machine.
P.S. In irb the same code RQRCode::QRCode.new(my_string, :size => 10,:level => :l) takes less than 1 second.
I have tried two different gems: rqrcode and barby+rqrcode. Both show the same results.
Code example:
require 'eventmachine'
require 'rqrcode'
class Handler < EventMachine::Connection
def receive_data(data)
puts Time.now
qrcode = RQRCode::QRCode.new('some_string', :size => 10,:level => :l)
puts Time.now
return qrcode
end
end
EventMachine::run {
EventMachine::start_server("0.0.0.0", 8081, Handler)
puts "Listening..."
}
Output:
2015-05-12 18:03:38 +0300
2015-05-12 18:03:48 +0300
It looks like you need to call close_connection. Here's EventMachine's echo server example
require 'eventmachine'
module EchoServer
def post_init
puts "-- someone connected to the echo server!"
end
def receive_data data
send_data ">>>you sent: #{data}"
close_connection if data =~ /quit/i # <---- This is what you're missing -----
end
def unbind
puts "-- someone disconnected from the echo server!"
end
end
# Note that this will block current thread.
EventMachine.run {
EventMachine.start_server "127.0.0.1", 8081, EchoServer
}
If your timestamps are always (nearly) exactly 10 seconds apart, I'd wager the request is ending because of a timeout rather than because of a response. After you respond with the QR data, call close_connection to let the browser know you're finished properly.
Edit It also looks like you're not calling send_data (and referencing Tume instead of Time). It looks like you need to use send_data, not return. Is your code snippet what's actually running on your server?
Update Running in debug mode was significantly slowing down QR generation.
The problem was that I've run server in debug-mode from RubyMine. Starting server from console solves the problem, qr-code generation takes about 1 sec.
Related
I am encountering an interesting issue with Ruby TCPServer, where once a client connects, it continually uses more and more CPU processing power until it hits 100% and then the entire system starts to bog down and can't process incoming data.
The processing class that is having an issue is designed to be a TCP Client that receives data from an embedded system, processes it, then returns the processed data to be further used (either by other similar data processors, or output to a user).
In this particular case, there is an external piece of code that would like this processed data, but cannot access it from the main parent code (the thing that the original process class is returning it's data to). This external piece may or may not be connected at any point while it is running.
To solve this, I set up a Thread with a TCPServer, and the processing class continually adds to a queue, and the Thread pulls from the queue and sends it to the client.
It works great, except for the performance issues. I am curious if I have something funky going on in my code, or if it's just the nature of this methodology and it will never be performant enough to work.
Thanks in advance for any insight/suggestions with this problem!
Here is my code/setup, with some test helpers:
process_data.rb
require 'socket'
class ProcessData
def initialize
super
#queue = Queue.new
#client_active = false
Thread.new do
# Waiting for connection
#server = TCPServer.open('localhost', 5000)
loop do
Thread.start(#server.accept) do |client|
puts 'Client connected'
# Connection established
#client_active = true
begin
# Continually attempt to send data to client
loop do
unless #queue.empty?
# If data exists, send it to client
begin
until #queue.empty?
client.puts(#queue.pop)
end
rescue Errno::EPIPE => error
# Client disconnected
client.close
end
end
sleep(1)
end
rescue IOError => error
# Client disconnected
#client_active = false
end
end # Thread.start(#server.accept)
end # loop do
end # Thread.new do
end
def read(data)
# Data comes in from embedded system on this method
# Do some processing
processed_data = data.to_i + 5678
# Ready to send data to external client
if #client_active
#queue << processed_data
end
return processed_data
end
end
test_embedded_system.rb (source of the original data)
require 'socket'
#data = '1234'*100000 # Simulate lots of data coming ing
embedded_system = TCPServer.open('localhost', 5555)
client_connection = embedded_system.accept
loop do
client_connection.puts(#data)
sleep(0.1)
end
parent.rb (this is what will create/call the ProcessData class)
require_relative 'process_data'
processor = ProcessData.new
loop do
begin
s = TCPSocket.new('localhost', 5555)
while data = s.gets
processor.read(data)
end
rescue => e
sleep(1)
end
end
random_client.rb (wants data from ProcessData)
require 'socket'
loop do
begin
s = TCPSocket.new('localhost', 5000)
while processed_data = s.gets
puts processed_data
end
rescue => e
sleep(1)
end
end
To run the test in linux, open 3 terminal windows:
Window 1: ./test_embedded_system.rb
Window 2: ./parent.rb
\CPU usage is stable
Window 3: ./random_client.rb
\CPU usage continually grows
I ended up figuring out what the issue was, and unfortunately I lead folks astray with my example.
It turns out my example didn't quite have the issue I was having, and the main difference was the sleep(1) was not in my version of process_data.rb.
That sleep is actually incredibly important, because it is inside of a loop do, and without the sleep, the Thread won't yield the GVL, and will continually eat up CPU resources.
Essentially, it was unrelated to TCP stuff, and more related to Threads and loops.
If you stumble on this question later on, you can put a sleep(0) in your loops if you don't want it to wait, but you want it to yield the GVL.
Check out these answers as well for more info:
Ruby infinite loop causes 100% cpu load
sleep 0 has special meaning?
I have a faye server (nodejs) running on localhost, and I am trying to setup a server side ruby client which needs to publish on the server on a regular basis. This is the code I am trying to use.
(Please ignore the commented code to start with).
I make a class variable ##client and initialize it as soon as the class loads. I define a class method pub whose task is to publish something on the faye server.
In the end, I just call the pub method twice. The first publication callback is received successfully, but the second publication doesn't make either of callback or the errback. And since the control has not been given back to the app, the app just hangs there.
If I make the gobal variable $client (currently commented), the behaviour is the same. But if I make the client everytime pub is called, then the publish goes on smoothly. I initiate it in EM.run loop or outside, the behavior is same. (as expected)
I don't want to make a new connection everytime I want to publish something since that defeats the purpose. Also, if I create a new client in EM.run everytime I call the method, the client connections don't close by themselves. I can see them open in lsof command as open files, and soon I'll start getting too many open files error I think.
I don't really understand Event Machine correctly, maybe I am missing something there.
require 'faye'
require 'eventmachine'
# $client = Faye::Client.new('http://localhost:5050/faye')
class Fayeclient
puts "#{__LINE__}: Reactor running: " + EM.reactor_running?.to_s
# if !defined? ##client or ##client.nil?
##client = Faye::Client.new('http://localhost:5050/faye')
puts "Created client: " + ##client.inspect
# end
def self.pub
puts "#{__LINE__}: Reactor running: " + EM.reactor_running?.to_s
# client = Faye::Client.new('http://localhost:5050/faye') #$client
# client = ##client
EM.run {
#client = Faye::Client.new('http://localhost:5050/faye') #$client
puts "#{__LINE__}: Reactor running: " + EM.reactor_running?.to_s
puts ##client.inspect
publication = ##client.publish('/foo', 'text' =>'Hello world')
puts "Publishing: #{publication.inspect}"
# puts "Publication methods: #{publication.methods}"
publication.callback do
puts "Did it #{publication.inspect}"
EM.stop_event_loop
puts "#{__LINE__}: Reactor running: " + EM.reactor_running?.to_s
# puts "#{client.methods}"
# puts client.inspect
# client.remove_all_listeners
# puts client.inspect
end
publication.errback do |error |
puts error.inspect
EM.stop_event_loop
end
}
puts "Outside event loop"
puts "#{__LINE__}: Reactor running: " + EM.reactor_running?.to_s
end
end
Fayeclient.pub
Fayeclient.pub
EM.run call is blocking, you have to run it on a separate thread, and eventually join it when all is over. In the example I'm using Singleton but it's up to you.
This does correctly the 2 faye calls.
#!/usr/bin/env ruby
#
require 'faye'
require 'singleton'
require 'eventmachine'
class Fayeclient
include Singleton
attr_accessor :em_thread, :client
def initialize
self.em_thread = Thread.new do
EM.run
end
self.client = Faye::Client.new('http://localhost:8890/faye')
end
def pub
puts "#{__LINE__}: Reactor running: " + EM.reactor_running?.to_s
puts client.inspect
publication = client.publish('/foo', 'text' =>'Hello world')
puts "Publishing: #{publication.inspect}"
publication.callback do
puts "Did it #{publication.inspect}"
EM.stop_event_loop
puts "#{__LINE__}: Reactor running: " + EM.reactor_running?.to_s
end
publication.errback do |error |
puts error.inspect
EM.stop_event_loop
end
end
end
Fayeclient.instance.pub
Fayeclient.instance.pub
Fayeclient.instance.em_thread.join
In my personal experience, anyway, having to deal with EventMachine inside Rails application can be a mess, some webserver uses EM, other does not so, and when you want to test from console it may not work as expected.
My solution is to fallback to http calls:
RestClient.post "http://localhost:#{Rails.configuration.faye_port}/faye", message: {foo: 'bar'}.to_json
I found this solution simpler and easy to customize, if you don't need to receive message from this piece of code.
My first question concerning Ruby.
I'm trying to test EventMachine interaction inside the Reactor loop - I guess it could be classified as "functional" testing.
Say I have two classes - a server and a client. And I want to test both sides - I need to be sure about their interaction.
Server:
require 'singleton'
class EchoServer < EM::Connection
include EM::Protocols::LineProtocol
def post_init
puts "-- someone connected to the echo server!"
end
def receive_data data
send_data ">>>you sent: #{data}"
close_connection if data =~ /quit/i
end
def unbind
puts "-- someone disconnected from the echo server!"
end
end
Client:
class EchoClient < EM::Connection
include EM::Protocols::LineProtocol
def post_init
send_data "Hello"
end
def receive_data(data)
#message = data
p data
end
def unbind
puts "-- someone disconnected from the echo server!"
end
end
So, I've tried different approaches and came up with nothing.
The fundamental question is - could I somehow test my code with RSpec, using should_recive?
EventMachine parameter should be a class or a module, so I can't send instantiated/mocked code inside. Right?
Something like this?
describe 'simple rspec test' do
it 'should pass the test' do
EventMachine.run {
EventMachine::start_server "127.0.0.1", 8081, EchoServer
puts 'running echo server on 8081'
EchoServer.should_receive(:receive_data)
EventMachine.connect '127.0.0.1', 8081, EchoClient
EventMachine.add_timer 1 do
puts 'Second passed. Stop loop.'
EventMachine.stop_event_loop
end
}
end
end
And, if not, how would you do it with EM::SpecHelper? I have this code using it, and can't figure out what I'm doing wrong.
describe 'when server is run and client sends data' do
include EM::SpecHelper
default_timeout 2
def start_server
EM.start_server('0.0.0.0', 12345) { |ws|
yield ws if block_given?
}
end
def start_client
client = EM.connect('0.0.0.0', 12345, FakeWebSocketClient)
yield client if block_given?
return client
end
describe "examples from the spec" do
it "should accept a single-frame text message" do
em {
start_server
start_client { |client|
client.onopen {
client.send_data("\x04\x05Hello")
}
}
}
end
end
end
Tried a lot of variations of these tests and I just can't figure it out. I'm sure I'm missing something here...
Thanks for your help.
The simplest solution that I can think of is to change this:
EchoServer.should_receive(:receive_data)
To this:
EchoServer.any_instance.should_receive(:receive_data)
Since EM is expecting a class to start a server, the above any_instance trick will expect any instance of that class to receive that method.
The EMSpecHelper example (while being official/standard) is quite convoluted, I'd rather stick with the first rspec and use any_instance, just for simplicity's sake.
I'm studying how to use ZeroMQ together with EventMachine.
To test things out, I wrote a small program in ruby (echo client server) where i used XREQ and XREP sockets. The client application is sending messages to server (consecutive numbers) and getting them back in responce. The interval between sendings is 0.1s.
Everything works... until a certain moment. When current number reaches about 400, server just freezes and doesn't respond to client anymore. I tested this on several computers, and still got that strange issue.
The code is pretty straightforward:
server.rb
require 'rubygems'
require 'bundler/setup'
require 'em-zeromq'
Thread.abort_on_exception = true
ADDRESS = 'tcp://127.0.0.1:2091'
class EMServerHandler
attr_reader :received
def on_readable(socket, messages)
client_identity = messages.shift.copy_out_string #getting client identity from the 1st part of the message
messages.shift #skip the delimeter
messages.each do |m|
msg = m.copy_out_string
puts "server received from #{client_identity}: " + msg
socket.send_msg("#{client_identity}",'',"#{msg}") #echo message back to the client
end
end
end
trap('INT') do
EM::stop()
end
puts "Program started (with zmq #{ZMQ::Util.version.join('.')})."
EM.run do
EventMachine.epoll
ctx = EM::ZeroMQ::Context.new(1)
server = ctx.bind(ZMQ::XREP, ADDRESS, EMServerHandler.new, {:identity => "server"})
end
client.rb
require 'rubygems'
require 'bundler/setup'
require 'em-zeromq'
Thread.abort_on_exception = true
ADDRESS = 'tcp://127.0.0.1:2091'
class EMClientHandler
attr_reader :received
def on_readable(socket, messages)
messages.shift #skip the delimeter
messages.each do |m|
puts "client recieved: " + m.copy_out_string
end
end
end
trap('INT') do
EM::stop()
end
puts "Program started (with zmq #{ZMQ::Util.version.join('.')})."
EM.run do
EventMachine.epoll
ctx = EM::ZeroMQ::Context.new(1)
puts "client"
puts "enter client name >> "
identity = gets.strip
client = ctx.connect(ZMQ::XREQ, ADDRESS, EMClientHandler.new, {:identity => identity})
client.send_msg('', "hello from client #{identity}")
count = 0
EM::PeriodicTimer.new(0.1) do
client.send_msg('', "#{count += 1}")
end
end
Please help me figure out the reason for this.
Your ZeroMQ context is being reaped by the garbage collector.
You need to move your call to EM::ZeroMQ::Context#new outside of the EM loop.
See the README
At last I figured out that this issue only appeared when using ruby 1.9.3p0, so it feels like this is a bug of that version of ruby.
With ruby 1.9.2 everything works like a charm.
I have an eventmachine app where one script is reading from a file, sending data to another script line by line, and the "server" script is acting upon that data. Unfortunately, the "server" script fails to execute receive_data as it should. I know that a connection is being made because it eecutes post_init, and I know the sender script is sending data. Here is some of my code along with how I start the server.
module BT_Server
def post_init
puts "-- someone connected to the echo server!"
end
def receive_data(data)
puts "hi"
int, time, *int_macs = data.split("-")
# more stuff that isn't needed here
end
def bt_left(dev)
dev.save
if t = Device.macs.index(dev.mac)
Device.all[t].add_int(dev.int, dev.t_0, dev.t_l)
else
Device.new(dev.mac, dev.int, dev.t_0, dev.t_l)
end
return false
end
def unbind
puts "disconnection"
end
end
EventMachine::run {
EventMachine::start_server 'localhost', 8081, BT_Server
puts t_0 = Time.new
puts 'listening...'
}
Note: I have the Module definition in a separate file, along with my classes, which I require into the server script, if that makes any difference.
i tested your code and it outputs 'hi' every time i send something via telnet.
from my point of view, the code is correct.
are you sure the sending script is working? try with a manual telnet on port 8081.
regards.