this is my ruby code:
require 'redis'
redis = Redis.new
Thread.new do
redis.subscribe('foo_bar') do |on|
on.message do |key, id|
puts id
end
end
end
redis.publish('foo_bar', 1337)
Can anyone tell me, why the "publish" instruction in last line is blocking my process?
Thanks
It appears that redis.subscribe blocks redis access in the entire process. So when you try to publish it locks.
Solution: trying using polling instead of subscribe
Solution 2: use several REDIS connections, i.e. create REDIS connection pool
To start with, the publish isnt blocking the process.
require 'redis'
redis = Redis.new
Thread.new do
redis.subscribe('foo_bar') do |on|
on.message do |key, id|
puts id
end
end
end
redis.publish('foo_bar', 1337)
Should output:
1337
And also,
require 'redis'
redis = Redis.new
Thread.new do
redis.subscribe('foo_bar') do |on|
on.message do |key, id|
puts id
end
end
end
redis.publish('foo_bar', 1337)
redis.publish('foo_bar', 1338)
Would output:
1337
1338
In the pub/sub system, the publisher pushes the message, and the subscriber is an always blocking synchronous process. Publisher is always asynchronous.
Related
I was wondering if I could stop execution of an operation that has been deffered.
require 'rubygems'
require 'em-websocket'
EM.run do
EM::WebSocket.start(:host => '0.0.0.0', :port => 8080) do |ws|
ws.onmessage do |msg|
op = proc do
sleep 5 # Thread safe IO here that is safely killed
true
end
callback = proc do |result|
puts "Done!"
end
EM.defer(op, callback)
end
end
end
This is an example web socket server. Sometimes when I get a message I want to do some IO, later on another message might come in that needs to read the same thing, the next thing always has precedence over the previous thing. So I want to cancel the first op and do the second.
Here is my solution. It is similar to the EM.queue solution, but just uses a hash.
require 'rubygems'
require 'em-websocket'
require 'json'
EM.run do
EM::WebSocket.start(:host => '0.0.0.0', :port => 3333) do |ws|
mutex = Mutex.new # to make thread safe. See https://github.com/eventmachine/eventmachine/blob/master/lib/eventmachine.rb#L981
queue = EM::Queue.new
ws.onmessage do |msg|
message_type = JSON.parse(msg)["type"]
op = proc do
mutex.synchronize do
if message_type == "preferred"
puts "killing non preferred\n"
queue.size.times { queue.pop {|thread| thread.kill } }
end
queue << Thread.current
end
puts "doing the long running process"
sleep 15 # Thread safe IO here that is safely killed
true
end
callback = proc do |result|
puts "Finished #{message_type} #{msg}"
end
EM.defer(op, callback)
end
end
end
I'm using eventmachine to read from a HornetQ topic, push to a Channel which is subscribed to by EM websocket connections. I need to prevent the #topic.receive loop from blocking, so have created a proc and am calling EventMachine.defer with no callback. This will run indefinitely. This works fine. I could also have just used Thread.new.
My question is, is this the correct way to read from a stream/queue and pass the data to the channel and is there a better/any other way to do this?
require 'em-websocket'
require 'torquebox-messaging'
class WebsocketServer
def initialize
#channel = EM::Channel.new
#topic = TorqueBox::Messaging::Topic.new('/topics/mytopic')
end
def start
EventMachine.run do
topic_to_channel = proc do
while true
msg = #topic.receive
#channel.push msg
end
end
EventMachine.defer(topic_to_channel)
EventMachine::WebSocket.start(:host => "127.0.0.1", :port => 8081, :debug => false) do |connection|
connection.onopen do
sid = #channel.subscribe { |msg| connection.send msg }
connection.onclose do
#channel.unsubscribe(sid)
end
end
end
end
end
end
WebsocketServer.new.start
This is ok, but EM.defer will spawn 20 threads, so I would avoid it for your use case. In general I would avoid EM entirely, especially the Java reactor as we never finished it.
The Torquebox has a native stomp over websockets solution that would be a much better way to go in this context, and solves a bunch of other encapsulation challenges for you.
If you really want to stick with EM for this, then I'd use Thread.new instead of defer, so as to avoid having 19 idle threads taking up extra ram for no reason.
I have the following sinatra app:
require 'sinatra'
require 'redis'
require 'json'
class FeedStream < Sinatra::Application
helpers do
include SessionsHelper
def redis
#redis ||= Redis.connect
end
end
get '/feed', provides: 'text/event-stream' do
stream do |out|
redis.subscribe "feed" do |on|
on.message do |channel, message|
event_data = JSON.parse message
logger.info "received event = #{event_data}"
out << "event: #{event_data['event']}\n"
out << "data: #{{:data => event_data['data'],
:by => current_user}}.to_json\n\n"
end
end
end
end
end
basically, it receives events published by other users to a feed using redis pubsub, and then it sends those events with the sinatra streaming api.
The problem is that, when the browser reconnects to the feed, the redis client keeps connected, and it keeps receiving events, so the redis server gets full of useless connections.
how can i close all this connections once the broser closes the connection to the web server?
I know it's been a while.
Were you looking for quit?
After much research and experimentation, here's the code I'm using with sinatra + sinatra sse gem (which should be easily adapted to Rails 4):
class EventServer < Sinatra::Base
include Sinatra::SSE
set :connections, []
.
.
.
get '/channel/:channel' do
.
.
.
sse_stream do |out|
settings.connections << out
out.callback {
puts 'Client disconnected from sse';
settings.connections.delete(out);
}
redis.subscribe(channel) do |on|
on.subscribe do |channel, subscriptions|
puts "Subscribed to redis ##{channel}\n"
end
on.message do |channel, message|
puts "Message from redis ##{channel}: #{message}\n"
message = JSON.parse(message)
.
.
.
if settings.connections.include?(out)
out.push(message)
else
puts 'closing orphaned redis connection'
redis.unsubscribe
end
end
end
end
end
The redis connection blocks on.message and only accepts (p)subscribe/(p)unsubscribe commands. Once you unsubscribe, the redis connection is no longer blocked and can be released by the web server object which was instantiated by the initial sse request. It automatically clears when you receive a message on redis and sse connection to the browser no longer exists in the collection array.
I'm studying how to use ZeroMQ together with EventMachine.
To test things out, I wrote a small program in ruby (echo client server) where i used XREQ and XREP sockets. The client application is sending messages to server (consecutive numbers) and getting them back in responce. The interval between sendings is 0.1s.
Everything works... until a certain moment. When current number reaches about 400, server just freezes and doesn't respond to client anymore. I tested this on several computers, and still got that strange issue.
The code is pretty straightforward:
server.rb
require 'rubygems'
require 'bundler/setup'
require 'em-zeromq'
Thread.abort_on_exception = true
ADDRESS = 'tcp://127.0.0.1:2091'
class EMServerHandler
attr_reader :received
def on_readable(socket, messages)
client_identity = messages.shift.copy_out_string #getting client identity from the 1st part of the message
messages.shift #skip the delimeter
messages.each do |m|
msg = m.copy_out_string
puts "server received from #{client_identity}: " + msg
socket.send_msg("#{client_identity}",'',"#{msg}") #echo message back to the client
end
end
end
trap('INT') do
EM::stop()
end
puts "Program started (with zmq #{ZMQ::Util.version.join('.')})."
EM.run do
EventMachine.epoll
ctx = EM::ZeroMQ::Context.new(1)
server = ctx.bind(ZMQ::XREP, ADDRESS, EMServerHandler.new, {:identity => "server"})
end
client.rb
require 'rubygems'
require 'bundler/setup'
require 'em-zeromq'
Thread.abort_on_exception = true
ADDRESS = 'tcp://127.0.0.1:2091'
class EMClientHandler
attr_reader :received
def on_readable(socket, messages)
messages.shift #skip the delimeter
messages.each do |m|
puts "client recieved: " + m.copy_out_string
end
end
end
trap('INT') do
EM::stop()
end
puts "Program started (with zmq #{ZMQ::Util.version.join('.')})."
EM.run do
EventMachine.epoll
ctx = EM::ZeroMQ::Context.new(1)
puts "client"
puts "enter client name >> "
identity = gets.strip
client = ctx.connect(ZMQ::XREQ, ADDRESS, EMClientHandler.new, {:identity => identity})
client.send_msg('', "hello from client #{identity}")
count = 0
EM::PeriodicTimer.new(0.1) do
client.send_msg('', "#{count += 1}")
end
end
Please help me figure out the reason for this.
Your ZeroMQ context is being reaped by the garbage collector.
You need to move your call to EM::ZeroMQ::Context#new outside of the EM loop.
See the README
At last I figured out that this issue only appeared when using ruby 1.9.3p0, so it feels like this is a bug of that version of ruby.
With ruby 1.9.2 everything works like a charm.
I'm trying to build a websocket server where each client establish its own redis connections used for publish and subscribe.
When the redis server is running I can see the two new connections being established when a client connects to the websocket server and I can also publish data to the client, but when the client drops the connection to the websocket server I also want to disconnect from Redis . How can I do this?
Maybe I'm doing it wrong, but this is my code.
#require 'redis'
require 'em-websocket'
require 'em-hiredis'
require 'json'
CLIENTS = Hash.new
class PubSub
def initialize(client)
#socket = client.ws
# These clients can only be used for pub sub commands
#publisher = EM::Hiredis.connect #Later I will like to disconnect this
#subscriber = EM::Hiredis.connect #Later I will like to disconnect this
client.connections << #publisher << #subscriber
end
def subscribe(channel)
#channel = channel
#subscriber.subscribe(channel)
#subscriber.on(:message) { |chan, message|
#socket.send message
}
end
def publish(channel,msg)
#publisher.publish(channel, msg).errback { |e|
puts [:publisherror, e]
}
end
def unsubscribe()
#subscriber.unsubscribe(#channel)
end
end
class Client
attr_accessor :connections, :ws
def initialize(ws)
#connections = []
#ws = ws
end
end
EventMachine.run do
# Creates a websocket listener
EventMachine::WebSocket.start(:host => '0.0.0.0', :port => 8081) do |ws|
ws.onopen do
# I instantiated above
puts 'CLient connected. Creating socket'
#client = Client.new(ws)
CLIENTS[ws] = #client
end
ws.onclose do
# Upon the close of the connection I remove it from my list of running sockets
puts 'Client disconnected. Closing socket'
#client.connections.each do |con|
#do something to disconnect from redis
end
CLIENTS.delete ws
end
ws.onmessage { |msg|
puts "Received message: #{msg}"
result = JSON.parse(msg)
if result.has_key? 'channel'
ps = PubSub.new(#client)
ps.subscribe(result['channel'])
elsif result.has_key? 'publish'
ps = PubSub.new(ws)
ps.publish(result['publish']['channel'],result['publish']['msg']);
end
}
end
end
This version of em-hiredis supports close connection: https://github.com/whatupdave/em-hiredis
Here is how I would (and did many times) this:
instead of always opening and closing connections for each client you can keep 1 connection open per Thread/Fiber dependeing on what you are basing your concurrency on, that way if you are using a poll of Thread/Fibers once each one of them have its connections they will keep it and reuse them.
I did not worked much with websocket until now (I was waiting for a standard implementation) but I am sure you can apply that thinking to it too.
You can also do what rails/activerecord: keeo a pool of redis connection, each time you need to use a connection you request one, use it and realease it, it could look like this:
def handle_request(request)
#redis_pool.get_connection do |c|
# [...]
end
end
before yielding the block a connection is taken from the available ones and after it the connection is marked as free.
This was added to em-hiredis: https://github.com/mloughran/em-hiredis/pull/6