I'm working with redis and ruby and attempting to issue a blpop within a thread, so that I can wait for an incoming item on a list.
The problem is that the code within the block for blpop never seems to get called. Here's the sample code that I'm running (ruby 1.9.3):
require 'rubygems'
require 'redis'
def start_thread
#thread = Thread.new do
r = Redis.new
r.blpop("test", 0) do |key, message|
process_message(key, message)
end
end
redis = Redis.new
redis.rpush "test", "hello world"
end
def process_message(key, message)
#message = "#{key} was sent #{message}"
end
start_thread
#thread.join
p #message
Any help is greatly appreciated!
require 'rubygems'
require 'redis'
def start_thread
#thread = Thread.new do
r = Redis.new
key, message = r.blpop(:test, 0)
process_message(key, message)
end
redis = Redis.new
redis.rpush :test, "hello world"
end
def process_message(key, message)
#message = "#{key} was sent #{message}"
end
start_thread
#thread.join
p #message
Related
I'm playing with celluloid gem. The example works well, but when I press Ctrl-C I get the unexpected message:
^CD, [2015-10-07T09:53:19.784411 #16326] DEBUG -- : Terminating 8 actors...
and after few seconds, I get the error:
E, [2015-10-07T09:53:29.785162 #16326] ERROR -- : Couldn't cleanly terminate all actors in 10 seconds!
/usr/local/rvm/gems/ruby-2.0.0-p353/gems/eventmachine-1.0.7/lib/eventmachine.rb:187:in `run_machine': Interrupt
from /usr/local/rvm/gems/ruby-2.0.0-p353/gems/eventmachine-1.0.7/lib/eventmachine.rb:187:in `run'
Strange that I create only 4 actors, not 8, and my TERM, INT signals handler isn't be called.
#!/usr/bin/env ruby
require './config/environment'
opts = CommandlineOptions.new.to_h
iface = opts[:iface] || '0.0.0.0'
port = opts[:port] || 3000
App.logger.info('Starting communication server')
connections = Connections.new
local_inbox = LocalQueue.new
auth_server = AuthServer.new(connections, local_inbox)
inbox_service = InboxService.new('inbox', iface, port)
inbox_service.async.process_inbox(local_inbox) # <--------
remote_outbox_name = "outbox_#{iface}:#{port}"
outbox_service = OutboxService.new(connections)
outbox_service.async.subscribe(remote_outbox_name) # <--------
conn_server_opts = { host: iface, port: port }
conn_server_opts.merge!(auth_server.callbacks)
conn_server = ConnServer.new(conn_server_opts)
%W(INT TERM).each do |signal|
trap(signal) do
info("Shutting down...")
conn_server.stop
end
end
conn_server.start
Here InboxService is an actor which creates another actor - there are 2 actors, then OutboxService also creates one actor, so I got created 4 actors.
require 'redis'
require 'celluloid/current'
class InboxServiceActor
include Celluloid
def initialize(remote_inbox_name)
#remote_inbox_name = remote_inbox_name
create_redis_connection
end
def publish(full_msg)
#redis.publish(#remote_inbox_name, full_msg)
end
private
def create_redis_connection
#redis = Redis.new
end
end
require 'json'
require 'redis'
require 'celluloid/current'
class OutboxServiceActor
include Celluloid
include HasLoggerMethods
def initialize
create_redis_connection
end
def subscribe(remote_outbox_name, &block)
#redis.subscribe(remote_outbox_name) do |on|
on.message do |_channel, full_msg|
debug("Outbox message received: '#{full_msg}'")
hash = parse_msg(full_msg)
block.call(hash['signature'], hash['msg']) if message_valid?(hash)
end
end
end
private
def create_redis_connection
#redis = Redis.new
end
def parse_msg(full_msg)
JSON.parse(full_msg)
rescue JSON::ParserError
error('Outbox message JSON parse error')
nil
end
def message_valid?(msg)
msg.is_a?(Hash) && msg.key?('signature') && msg.key?('msg') ||
error('Invalid outbox message. Should '\
'contain "signature" and "msg" keys') && false
end
end
I currently have a script written in Ruby that scans a range of IP addresses and tries to connect to them. It's extremely slow at the moment. It takes up to 300 seconds to scan 254 hosts on the network, and that's obviously not very practical. What I'm trying to do is give the script some concurrency in hopes of speeding up the script. So far this is what I have:
require 'socket'
require 'celluloid'
$res_arr = []
class Ranger
include Celluloid
def initialize(host)
#host = host
#timeout = 1
end
def ip_range(host)
host =~ /(?:\d{1,3}\.){3}[xX*]{1,3}/
end
def ctrl(host)
begin
if ip_range(host)
strIP = host.gsub(/[xX*]/, '')
(1..254).each do |oct|
$res_arr << strIP+oct.to_s
end
else
puts "Invalid host!"
end
rescue
puts "onnection terminated."
end
end
def connect
addr = Socket.getaddrinfo(#host, nil)
sock = Socket.new(Socket.const_get(addr[0][0]), Socket::SOCK_STREAM, 0)
begin
sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
rescue Errno::EINPROGRESS
resp = IO.select(nil, [sock], nil, #timeout.to_i)
if resp.nil?
$res_arr << "#{#host} Firewalled!"
end
begin
if sock.connect_nonblock(Socket.pack_sockaddr_in(22, addr[0][3]))
$res_arr << "#{#host}Connected!"
end
rescue Errno::ECONNREFUSED
$res_arr << "#{#host} Refused!"
rescue
false
end
end
sock
end
def output(contents)
puts contents.value
end
end # Ranger
main = Ranger.new(ARGV[0])
main.ctrl(ARGV[0])
$res_arr.each do |ip|
scan = Ranger.new(ip)
scnftr = scan.future :connect
scan.output(scnftr)
end
The script works, but it takes just as long as before I included Celluloid at all. Am I misunderstanding how Celluloid works and what it's supposed to do?
Your problem is that each iteration of your loop starts a future, then immediately waits for it to return a value. What you want instead is start all futures, then wait for all futures to finish in two separate steps:
futures = $res_arr.map do |ip|
scan = Ranger.new(ip)
scan.future :connect
end
# now that all futures are running, we can start
# waiting for the first one to finish
futures.each do |future|
puts future.value
end
Here's another example from the celluloid source: https://github.com/celluloid/celluloid/blob/master/examples/simple_pmap.rb
In Java, there is a class called ArrayBlockingQueue as part of its concurrent package. It is a thread-safe class where you can add and remove items from the queue without worrying about thread-safety. This class has a put method which allows you to put items in queue. And a take method removes items from the queue. Two great things about put and take is there is no need of a synchronized keyword for synchronization against thread interleaving, and take patiently waits until something is added to the queue, rather than throwing an exception if nothing is in it.
I try to implement something similar in ruby, but the issue is queue.pop seems to block even when items are added to the queue (at least for one of the queues), as shown below:
require 'redis'
require 'date'
def log_debug(str)
debug_str = "#{DateTime.now} #{str}"
puts debug_str
end
class EmailsmsResponder
def initialize
#queue = Queue.new
end
# add to queue
def produce(channel, msg)
#queue << {channel: channel, msg: msg}
puts "queue size: #{#queue.size}"
end
# take from queue
def consume
loop do
log_debug "Whats going on??"
sleep(1)
if !#queue.empty?
item = #queue.pop
log_debug "removing channel #{item[:channel]} and msg #{item[:msg]} from email-sms thread from queue"
end
end
end
end
class SidekiqResponder
def initialize
#queue = Queue.new
end
def produce(channel, msg)
#queue << {channel: channel, msg: msg}
puts "queue size: #{#queue.size}"
end
def consume
loop do
log_debug "Whats going on??"
sleep(1)
if !#queue.empty?
value = #queue.pop
log_debug "removing channel #{item[:channel]} and msg #{item[:msg]} from sidekiq thread from queue"
end
end
end
end
class RedisResponder
def initialize(host,port)
#host = host
#port = port
#email_sms = EmailsmsResponder.new
#sidekiq = SidekiqResponder.new
# timeout so we wait for messages forever
#redis = Redis.new(:host => #host, :port => #port, :timeout => 0)
end
def start_producers
thread = Thread.new do
#redis.subscribe('juggernaut') do |on|
# message block fired for new messages
on.message do |channel, msg|
log_debug "New message"
#email_sms.produce(channel, msg)
#sidekiq.produce(channel, msg)
end
end
end
end
def start_consumers
thread = Thread.new do
#email_sms.consume
#sidekiq.consume
end
end
end
responder = RedisResponder.new('127.0.0.1', 6379)
responder.start_producers.join(responder.start_consumers.join)
While one queue seems to be working properly, the other queue never retrieves anything:
$ ruby redis-client4.rb
2014-07-22T14:53:24-04:00 Whats going on??
2014-07-22T14:53:25-04:00 Whats going on??
2014-07-22T14:53:25-04:00 New message
queue size: 1
queue size: 1
2014-07-22T14:53:26-04:00 removing channel juggernaut and msg {"channels":["/reports/6561/new"],"data":"New reports for unit 6561"} from email-sms thread from queue
2014-07-22T14:53:26-04:00 Whats going on??
2014-07-22T14:53:27-04:00 Whats going on??
2014-07-22T14:53:28-04:00 Whats going on??
2014-07-22T14:53:28-04:00 New message
queue size: 1
queue size: 2
2014-07-22T14:53:29-04:00 removing channel juggernaut and msg {"channels":["/reports/6561/new"],"data":"New reports for unit 6561"} from email-sms thread from queue
2014-07-22T14:53:29-04:00 Whats going on??
2014-07-22T14:53:30-04:00 Whats going on??
2014-07-22T14:53:31-04:00 Whats going on??
2014-07-22T14:53:31-04:00 New message
queue size: 1
queue size: 3
2014-07-22T14:53:32-04:00 removing channel juggernaut and msg {"channels":["/reports/6561/new"],"data":"New reports for unit 6561"} from email-sms thread from queue
2014-07-22T14:53:32-04:00 Whats going on??
2014-07-22T14:53:33-04:00 Whats going on??
2014-07-22T14:53:34-04:00 Whats going on??
2014-07-22T14:53:34-04:00 New message
queue size: 1
queue size: 4
2014-07-22T14:53:35-04:00 removing channel juggernaut and msg {"channels":["/reports/6561/new"],"data":"New reports for unit 6561"} from email-sms thread from queue
2014-07-22T14:53:35-04:00 Whats going on??
2014-07-22T14:53:36-04:00 Whats going on??
2014-07-22T14:53:37-04:00 Whats going on??
2014-07-22T14:53:37-04:00 New message
queue size: 1
queue size: 5
What might I be doing wrong?
I got it working with the code below. I didn't like the fact I had to use 4 threads just to get it working, so if anyone has a more nice solution I would be glad to recommend their solution. But this seems to be working for now:
require 'redis'
require 'date'
def log_debug(str)
debug_str = "#{DateTime.now} #{str}"
puts debug_str
end
class EmailsmsResponder
def initialize
#queue = Queue.new
end
# add to queue
def produce(channel, msg)
#queue << {channel: channel, msg: msg}
puts "queue size: #{#queue.size}"
end
# take from queue
def consume
loop do
log_debug "Whats going on??"
sleep(1)
if !#queue.empty?
item = #queue.pop
log_debug "removing channel #{item[:channel]} and msg #{item[:msg]} from email-sms thread from queue"
end
end
end
end
class SidekiqResponder
def initialize
#queue = Queue.new
end
def produce(channel, msg)
#queue << {channel: channel, msg: msg}
puts "queue size: #{#queue.size}"
end
def consume
loop do
log_debug "Whats going on??"
sleep(1)
if !#queue.empty?
item = #queue.pop
log_debug "removing channel #{item[:channel]} and msg #{item[:msg]} from sidekiq thread from queue"
end
end
end
end
class RedisResponder
def initialize(host,port)
#host = host
#port = port
#email_sms = EmailsmsResponder.new
#sidekiq = SidekiqResponder.new
# timeout so we wait for messages forever
#redis = Redis.new(:host => #host, :port => #port, :timeout => 0)
end
def start_producers
thread = Thread.new do
#redis.subscribe('juggernaut') do |on|
# message block fired for new messages
on.message do |channel, msg|
log_debug "New message"
#email_sms.produce(channel, msg)
#sidekiq.produce(channel, msg)
end
end
end
end
def start_consumers
thread = Thread.new do
t1 = Thread.new { #email_sms.consume }
t2 = Thread.new { #sidekiq.consume }
t1.join(t2.join)
end
end
end
responder = RedisResponder.new('127.0.0.1', 6379)
responder.start_producers.join(responder.start_consumers.join)
I try to implement something similar in ruby, but the issue is
queue.pop seems to block even when items are added to the queue
That's easy to disprove:
require 'thread'
q = Queue.new
q << 'hello'
x = q.pop
puts x
x = q.pop
--output:--
hello
deadlock detected (fatal)
What might I be doing wrong?
Start deleting code and simplifying things to pinpoint where the problem occurs. The fact that you have two classes that are exactly the same means you haven't even begun to simplify.
Then there is this:
def consume
loop do
log_debug "Whats going on??"
sleep(1)
value = queue.pop
log_debug "removing channel: #{channel} msg: #{msg} of sidekiq thread from queue"
end
end
***Error in `consume': undefined local variable or method `queue'
I was wondering if I could stop execution of an operation that has been deffered.
require 'rubygems'
require 'em-websocket'
EM.run do
EM::WebSocket.start(:host => '0.0.0.0', :port => 8080) do |ws|
ws.onmessage do |msg|
op = proc do
sleep 5 # Thread safe IO here that is safely killed
true
end
callback = proc do |result|
puts "Done!"
end
EM.defer(op, callback)
end
end
end
This is an example web socket server. Sometimes when I get a message I want to do some IO, later on another message might come in that needs to read the same thing, the next thing always has precedence over the previous thing. So I want to cancel the first op and do the second.
Here is my solution. It is similar to the EM.queue solution, but just uses a hash.
require 'rubygems'
require 'em-websocket'
require 'json'
EM.run do
EM::WebSocket.start(:host => '0.0.0.0', :port => 3333) do |ws|
mutex = Mutex.new # to make thread safe. See https://github.com/eventmachine/eventmachine/blob/master/lib/eventmachine.rb#L981
queue = EM::Queue.new
ws.onmessage do |msg|
message_type = JSON.parse(msg)["type"]
op = proc do
mutex.synchronize do
if message_type == "preferred"
puts "killing non preferred\n"
queue.size.times { queue.pop {|thread| thread.kill } }
end
queue << Thread.current
end
puts "doing the long running process"
sleep 15 # Thread safe IO here that is safely killed
true
end
callback = proc do |result|
puts "Finished #{message_type} #{msg}"
end
EM.defer(op, callback)
end
end
end
i'm having an issue configuring the sidekiq server, the process seems to be running in the foreground as soon as i refresh my page. /consumers/fetch i need to put it in the background permanently.
consumers_controller.rb
require 'kafka'
class ConsumersController < ApplicationController
def fetch
#consumer = Kafka::Consumer.new( { :host => ENV["host"],
:port => ENV["port"],
:topic => ENV["topic"]})
#consumer.loop do |message|
logger.info "-------------#{message.inspect}--------------"
logger.info "-------------#{message.first.payload.inspect}--------------"
unless message.blank?
ConsumerWorker.perform_async(message.first.payload)
end
end
end
end
consumer_worker.rb
class ConsumerWorker
include Sidekiq::Worker
def perform(message)
payload = message.first["payload"]
hash = JSON.parse(payload)
return #message = Message.new(hash) if hash["concern"] == 'order_create' or hash["concern"] == 'first_payment'
end
end
message.rb
class Message
attr_reader :bundle_id, :order_id, :order_number, :event
def initialize(message)
#payload = message["payload"]
#bundle_id = #payload["bundle_id"]
#order_id = #payload["order_id"]
#order_number = #payload["order_number"]
#event = message["concern"]
end
end
I think you need to move this block
#consumer.loop do |message|
end
inside your worker somehow, as I think the consumption is done after block execution.