Redis Pub/Sub causes web socket connection to hang - ruby

Question

I'm building a web application that connects to a server via web sockets. The server component is a small Ruby application based on sinatra, redis, and faye-websocket. The server is running on Phusion Passenger. A separate updater daemon is constantly pulling updates from various sources and publishes them to redis (using the redis gem and Redis::publish).
In order to push the updates to the clients I tried the following in my Sinatra app:
get '/' do
if Faye::WebSocket.websocket?(request.env)
store = Redis.new
ws = Faye::WebSocket.new(request.env)
ws.on(:open) do |event|
store.incr('connection_count')
puts 'Client connected (connection count: %s)' % store.get('connection_count')
end
ws.on(:close) do |event|
store.decr('connection_count')
puts 'Client disconnected (connection count: %s)' % store.get('connection_count')
end
ws.rack_response
store.subscribe(:updates) do |on|
on.message do |ch, payload|
puts "Got update"
ws.send(payload) if payload
end
end
end
end
This works only partially. A client can connect successfully and also receives updates but the store.incr and store.decr calls don't work. Also, the connections don't seem to be closed correctly—when I fire up multiple clients, I noticed that the connections pile up and the Passenger server stops working eventually.
Log output:
devserver_1 | App 614 stdout: Got update
devserver_1 | App 614 stdout: Got update
devserver_1 | App 614 stdout: Got update
When I comment out the following block, keeping track of the connections suddenly works:
store.subscribe(:updates) do |on|
on.message do |ch, payload|
puts "Got update"
ws.send(payload) if payload
end
end
Log output:
devserver_1 | App 1028 stdout: Client connected (connection count: 1)
devserver_1 | App 1039 stdout: Client connected (connection count: 2)
devserver_1 | App 1039 stdout: Client disconnected (connection count: 1)
devserver_1 | App 1028 stdout: Client disconnected (connection count: 0)
So using Redis::subscribe seems to somehow interfere with the web socket connection.
How can I solve this?
Phusion Passenger version 4.0.58
ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-linux-gnu]
sinatra (1.4.6)
faye-websocket (0.9.2)

Answer 1

I think the problem here is that Faye uses EventMachine which means there's a reactor on your thread that is handling events, and calling your callbacks ws.on(:open) and ws.on(:close).
Now when you hit
store.subscribe(:updates) do |on|
on.message do |ch, payload|
puts "Got update"
ws.send(payload) if payload
end
end
This is a blocking operation - it entirely blocks the current thread. If your current thread is blocked, the reactor can't listen for events and then call your callbacks.
One solution to this is to run your store.subscribe on a different thread so it doesn't matter if it blocks that thread.
But I think a better solution is to use a non-blocking version of the Redis library:
From the documentation:
redis = EM::Hiredis.connect
pubsub = redis.pubsub
pubsub.subscribe(:updates).callback do
puts "Got update"
ws.send(payload) if payload
end
Both of these (Redis + Faye) should register with the EventMachine reactor loop, so that it dispatches events to both.