Is there any method to temporarily pause a consumer and resume it at later time?
Here is an example of what i want to do:
require "bunny"
conn = Bunny.new
conn.start
ch1 = conn.create_channel
publisher = ch.direct('test', :auto_delete => false)
consumer1 = nil
Thread.new do
ch2 = conn.create_channel(nil, 8) #Using eight worker
queue1 = ch2.queue('', :exclusive => true)
queue1.bind(publisher, :routing_key => 'low_priority')
consumer1 = queue1.subscribe(:block => true) do |delivery_info, properties, payload|
#do some work
end
end
Thread.new do
ch3 = conn.create_channel
queue2 = ch3.queue('', :exclusive => true)
queue2.bind(publisher, :routing_key => 'high_priority')
consumer2 = queue2.subscribe(:block => true) do |delivery_info, properties, payload|
consumer1.pause #pause the other consumer
#do other things
consumer1.resume #resume the consumer
end
end
#rest of the code
I want to pause consumer1 when I'm doing the work at consumer2. Is there any valid way to do that?
If you want to create priority policies, bunny has implemented this:
http://rubybunny.info/articles/queues.html#consumer_priorities
q = ch.queue("a.queue")
q.subscribe(:manual_ack => true, :arguments => {"x-priority" => 5}) do |delivery_info, properties, payload|
# ...
end
q.subscribe(:manual_ack => true, :arguments => {"x-priority" => 2}) do |delivery_info, properties, payload|
# ...
end
And if you want to process it in parallel I suggest this gem:
https://github.com/grosser/parallel
Example:
results = Parallel.map(['a','b','c'], in_processes: 3) { |one_letter| ... }
Hope it helps.
Related
I have a program that uses a method for verification, if that verification failed I would like to return to the method it was called from, for example:
def obtain_pokemon_name
print 'Enter Pokemon: '
pokemon = gets.chomp.capitalize
obtain_basic_attack(pokemon)
end
def obtain_basic_attack(poke)
print 'Enter basic attack: '
basic_attack = gets.chomp.downcase
check_attacks(poke, basic_attack)
obtain_spec_attack(poke)
end
def obtain_spec_attack(poke)
print 'Enter special attack: '
spec_attack = gets.chomp.downcase
check_attacks(poke, spec_attack)
end
def check_attacks(pokemon, attack)
if POKEMON_ATTACKS[pokemon][attack] == nil
puts "#{attack} is not one of #{pokemon}'s attacks, try again.."
return # to where this function was called
else
attack
end
end
begin
obtain_pokemon_name
rescue => e
puts "Failed with error code: #{e}"
end
When this is run:
Enter Pokemon: arbok
Enter basic attack: eat
eat is not one of Arbok's attacks, try again..
Enter special attack: test
test is not one of Arbok's attacks, try again..
Attack list:
POKEMON_ATTACKS = {
'Bulbasaur' => {'tackle' => 10.9, 'vine whip' => 15.4, 'power whip' => 21.4, 'seed bomb' => 12.5, 'sludge bomb' => 19.2},
'Ivysaur' => {'razor leaf' => 10.3, 'vine whip' => 15.4, 'power whip' => 21.4, 'sludge bomb' => 19.2, 'solar beam' => 13.3},
'Kakuna' => {'bug bite' => 13.3, 'poison sting' => 10.3, 'struggle' => 8.8},
'Beedrill' => {'bug bite' => 13.3, 'poison jab' => 14.3, 'aerial ace' => 8.6, 'sludge bomb' => 19.2, 'x-scissor' => 14.3},
'Pidgey' => {'quick attack' => 7.5, 'tackle' => 10.9, 'aerial ace' => 8.6, 'air cutter' => 7.6, 'twister' => 5.6},
'Ekans' => {'acid' => 9.5, 'poison sting' => 10.3, 'gunk shot' => 20.0, 'sludge bomb' => 19.2, 'wrap' => 3.8},
'Arbok' => {'acid' => 9.5, 'bite' => 12.0, 'dark pulse' => 12.9, 'gunk shot' => 20.0, 'sludge wave' => 17.6},
}
So my question is, if the attack is not present in the data, how can I return back to the calling method? So for instance if I call arbok and his attack is tackle if it doesn't exist in the hash, how would I return to the obtain_basic_attack(poke) method?
RIght here:
puts "#{attack} is not one of #{pokemon}'s attacks, try again.."
return # to where this function was called
you should call the original method again. i.e.
if POKEMON_ATTACKS[pokemon][attack] == nil
puts "#{attack} is not one of #{pokemon}'s attacks, try again.."
return obtain_spec_attack(poke)
You could alternatively add this logic to obtain_spec_attack:
def obtain_spec_attack(poke)
loop do
print 'Enter special attack: '
spec_attack = gets.chomp.downcase
attack_found = check_attacks(poke, spec_attack)
if attack_found
break attack_found # this will return attack_found from the loop
else
puts "attack not found"
end
end
end
edit
looking at your question again, I realize you want to return to a method multiple levels up. You could use the approaches I've already outlined, or alternatively use rescue:
def obtain_basic_attack(poke)
begin
print 'Enter basic attack: '
basic_attack = gets.chomp.downcase
check_attacks(poke, basic_attack)
obtain_spec_attack(poke)
rescue AttackNotFoundError
retry # runs the 'begin' block again
end
end
def obtain_spec_attack(poke)
print 'Enter special attack: '
spec_attack = gets.chomp.downcase
check_attacks(poke, spec_attack)
end
def check_attacks(pokemon, attack)
if POKEMON_ATTACKS[pokemon][attack] == nil
puts "#{attack} is not one of #{pokemon}'s attacks, try again.."
raise AttackNotFoundError
else
attack
end
end
In order to use a custom error like AttackNotFoundError, you need to define the error class somewhere:
class AttackNotFoundError < StandardError; end
You could use any error really, such as raise StandardError, but it's better to restrict what errors you're rescuing so that you don't accidentally rescue an unrelated error.
I'm having a difficulty with processing resque tasks which were enqueued.
The whole enqueuing part goes well - I can see it in Redis and also Resque.info shows the pending tasks number incrementing as it should.
If I run the perform method of the Job class explicitly - everything works fine.
Once the worker come alive - all the tasks fail.
This is the Job class:
class TestJob
#queue = :test1_queue
def self.perform(str)
begin
f = open('/tmp/test.txt', 'a')
f.write("#{Time.now.to_s} #{str} in self.perform\n")
rescue Exception => e
f.write("#{Time.now.to_s} #{str} in self.perform\n#{e.message}\n#{e.backtrace}\n")
ensure
f.close
end
end
end
The resque.rb initializer:
require 'resque'
require 'redis'
Dir['../../jobs'].each { |file| require file }
Resque.redis = $resque_redis
Resque.logger.level = Logger::WARN
Resque.after_fork do |_|
$resque_redis.client.reconnect
end
Redis initializer:
require 'redis'
$resque_redis = Redis.new(:host => REDIS_BITMAP_HOST, :port => REDIS_PORT, :db => 0, :timeout => 30)
config file to start the worker with god:
require_relative './common.rb'
watch_resque_process('test1', 1)
God definitions:
$home_dir = ENV["HOME"]
$rack_env = ENV["ETL_ENV"] || "development"
def create_deafult_monitoring_scheme(watch)
# Restart if memory is above 150 Megabytes or CPU is above 50% for 5 consecutive intervals
watch.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.above = 150.megabytes
c.times = [3, 5] # 3 out of 5 intervals
end
restart.condition(:cpu_usage) do |c|
c.above = 50.percent
c.times = 5
end
end
# The :flapping condition guards against the edge case wherein god rapidly starts or restarts your application.
# If this watch is started or restarted five times withing 5 minutes, then unmonitor it for 10 minutes.
# After 10 minutes, monitor it again to see if it was just a temporary problem; if the process is seen to be flapping five times within two hours, then give up completely.
watch.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minute
c.transition = :unmonitored
c.retry_in = 10.minute
c.retry_times = 5
c.retry_within = 30.minute
end
end
end
def watch_resque_process(resque_process_name, worker_count=8)
God.watch do |w|
w.name = "resque_work-#{resque_process_name}"
w.start = "cd #{$home_dir}/rtb-etl && COUNT=#{worker_count} QUEUE='#{resque_process_name}_queue' RACK_ENV=#{$rack_env} rake resque:workers"
w.interval = 30.seconds
w.log = File.join($home_dir, 'logs', 'resque', "resque_#{resque_process_name}.log")
w.err_log = File.join($home_dir, 'logs', 'resque', "resque_#{resque_process_name}.log")
w.env = { 'PIDFILE' => "#{$home_dir}/pids/#{w.name}.pid" }
# Check if the process is still up every 5 seconds
w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
end
end
create_deafult_monitoring_scheme(w)
end
end
def watch_rake_task(rake_task_name, interval=30.seconds)
God.watch do |w|
w.name = "rake_#{rake_task_name}"
# w.start = "cd #{$home_dir}/rtb-etl && RACK_ENV=#{$rack_env} bundle exec rake #{rake_task_name}"
w.start = "cd #{$home_dir}/rtb-etl && RACK_ENV=#{$rack_env} rake #{rake_task_name}"
w.interval = interval
w.log = File.join($home_dir, 'logs', 'resque', "rake_#{rake_task_name}.log")
w.err_log = File.join($home_dir, 'logs', 'resque', "rake_#{rake_task_name}.log")
w.env = { 'PIDFILE' => "#{$home_dir}/pids/#{w.name}.pid" }
# Check if the process is still up every 30 seconds
w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = interval
c.running = false
end
end
create_deafult_monitoring_scheme(w)
end
end
when I run the follwoing:
irb(main):004:0> Resque.enqueue(TestJob, 'foo')
=> true
In order to check what went wrong, I run in irb the following:
Resque::Failure.all(0,20).each { |job|
puts "#{job["exception"]} #{job["backtrace"]}"
}
and get this result:
[{"failed_at"=>"2015/08/26 17:35:00 UTC",
"payload"=>{"class"=>"TestJob", "args"=>["foo"]},
"exception"=>"NoMethodError",
"error"=>"undefined method `client' for nil:NilClass",
"backtrace"=>[], "worker"=>"ip-172-31-11-211:5006:test1_queue",
"queue"=>"test1_queue"}]
Any ideas ?
I am trying to set up RabbitMQ rpc. I want one queue to listen, and when it receives a message I want it to reply to an anonymous queue that is specified via the reply_to header with multiple messages.
I have the following thor task creates a queue and then uses EM:Iterator to send a number of messages back to the queue specified with the replyt_to routing key:
desc "start_consumer", "start the test consumer"
def start_consumer
conf = {
:host => "localhost",
:user => "guest",
:password => "guest",
:vhost => "/",
:logging => true,
:port => 5672
}
# n = 1
AMQP.start(conf) do |connection|
channel = AMQP::Channel.new(connection)
requests_queue = channel.queue("one")
requests_queue.purge
Signal.trap("INT") do
connection.close do
EM.stop{exit}
end
end
channel.prefetch(1)
requests_queue.subscribe(:ack => true) do |header, body|
url_search = MultiJson.decode(body)
EM::Iterator.new(0..5).each do |n, iter|
lead = get_lead(n, (n == 5))
puts "about to publish #{n} message is_last = #{lead.is_last} at #{Time.now}"
AMQP::Exchange.default.publish(
MultiJson.encode(lead),
:immediate => true,
:routing_key => header.reply_to,
:correlation_id => header.correlation_id
)
iter.next
end
end
puts " [x] Awaiting RPC requests"
end
end
The code beloow sends a message to the queue specified above and also creates a queue that will be used to listen for the messages sent by the EM::Iterator code. This queue's name is the routing key for the first queues reply_to header.
def publish(urlSearch, routing_key)
EM.run do
corr_id = rand(10_000_000).to_s
requests ||= Hash.new
connection = AMQP.connect(:host => "localhost")
callback_queue = AMQP::Channel.new(connection).queue("", :exclusive => false)
callback_queue.subscribe do |header, body|
lead = safe_json_decode(body)
puts "company = #{lead["company"]} is_last = #{lead["is_last"]} received at #{Time.now}"
if lead["is_last"]
puts "in exit"
connection.close do
EM.stop{exit}
end
end
end
callback_queue.append_callback(:declare) do
AMQP::Exchange.default.publish(MultiJson.encode(urlSearch), :routing_key => routing_key, :reply_to => callback_queue.name, :correlation_id => corr_id)
end
puts "initial message sent"
end
end
The above code works as I want with one annoying exception. Something is blocking the EM::Iterator code from being executed asynchronously. It is only after the EM::Iterator code has completed that the messages are sent. I want the messages to be sent asynchronously and handled by the anonymous queue after each iteration. At the moment, it is only after the EM::Iterator code has completed its last iteration that all the messages are sent.
Can anyone see what I am doing wrong or suggest a different approach? I tried EM::defer and had the same behaviour.
Spinning up a new thread was the answer to my problem:
Thread.new do
5.times do
lead = get_lead(n, (n == 5))
puts "message #{n} is_last = #{lead.is_last} at #{Time.now}";
AMQP::Exchange.default.publish(
MultiJson.encode(lead),
:routing_key => header.reply_to,
:correlation_id => header.correlation_id
)
n += 1
sleep(2)
end
end
Creating a new thread stops the EventMachine reactor being blocked and the messages are sent async.
post '/payment/transactions/amount' do # => creates a new charge
content_type :json, :charset => 'utf-8'
class Transaction
def initialize(endUserId, code, referenceCode, callback=false)
#endUserId = endUserId
#code = code
#referenceCode = referenceCode
$callback = callback
$transId = rand(1000)
end
def transId
return $transId
end
def callback
return $callback
end
def postCallback
sleep(5)
RestClient.post"#{$callback}", :message => 'CALLBACK SUCCESS', :content_type => :json
end
def to_json(*a)
{"amountTransaction" => {
"endUserId" => #endUserId,
"paymentAmount" =>{
"chargingInformation" => {"code" => #code, "description" => #description},
"totalAmountCharged"=> "0"
},
"referenceCode" => #referenceCode,
"serverReferenceCode" =>"#{(0...6).map{65.+(rand(25)).chr}.join}", # just an example, can be any string
"resourceURL" => "http://localhost:4567/payment/#{#endUserId}/transactions/amount/#{Trans.transId}",
"transactionOperationStatus" => "Processing"
}}.to_json(*a)
end
end
Trans = Transaction.new(params[:endUserId],params[:code], params[:referenceCode], params[:callback])
jsonTrans = Trans.to_json
receipts.addReceipt(Trans.transId,jsonTrans)
jsonTrans
# fire callback
if Trans.callback then
sleep(10)
Trans.postCallback
end
end
Problem for me is the code following #fire callback. If I omit this if...then , the jsonTrans JSON object is returned as expected. When I include the if...then to fire the callback, then the desired Trans.postcallback occurs after 5 seconds - but the preceding jsonTrans is not returned, it is simply ignored. Any ideas on what I'm doing wrong? To confirm, the desired behaviour is to return the jsonTrans in the HTTP response and then call the postCallback method after 5 seconds (assuming the if evaluates to true). Cheers!
EDIT: Solved by spawning a new thread:
Thread.new{
if Trans.callback then
sleep(5)
Trans.postCallback
end}
jsonTrans
The last statement is what the method returns(outputs) so it ignores your jsonTrans in the output.
The first idea I have would be to reorder it:
#...
Trans = Transaction.new(params[:endUserId],params[:code], params[:referenceCode], params[:callback])
jsonTrans = Trans.to_json
receipts.addReceipt(Trans.transId,jsonTrans)
# fire callback
if Trans.callback then
sleep(10)
Trans.postCallback
end
jsonTrans
end
This should work if you don't expect any output from that RestClient.post-call.
Thanks for the suggestion daddz - actually I managed to solve it by wrapping by spawning a new thread: Thread.new{
if Trans.callback then
sleep(5)
Trans.postCallback
end}
jsonTrans
I have a working TCP/IP socket server that loads 3-flash files in succession. How can I unload previous files and eliminate the garbage build up?
2-Flash clients are active, 1-the loader, 2-the next Flash file being loaded, however "the Flash files don't unload." Maybe there's a "put - kill" method or something similar to addChild removeChild in as3. Any resource would help, since I'm not very familiar with Ruby.
Files involved
POLICY SERVER "server lets Flash files play"
policyserver_little.rb
RUBY TCP/IP SOCKET SERVER "server plays loader, that loads 3-Flash files"
server_little#.rb
FLASH LOADER "client"
loader_little.swf
FLASH MOVIES "numbers_odom.swf, numbers_fruits.swf, $mil.swf"
"msg1, msg2, msg3"
WHAT I SEE
def worker==>end
There's no method to unload.
RUBY SERVER server_little#.rb
require 'socket'
require 'rexml/document'
include Socket::Constants
def create_xml_msg(msg, parent)
el = nil
msg.each do |key, value|
if parent
el = parent.add_element(key)
else
el = REXML::Element.new(key)
end
if value.instance_of?(Hash)
create_xml_msg(value, el)
else
el.text = value.to_s
end
end
return el
end
def worker(client, client_sockaddr, worker_number)
$tid << Thread.new([client, client_sockaddr, worker_number]) do |cl|
Thread.current[:number] = cl[2]
puts("\nThread #{cl[2]} servicing #{Socket.unpack_sockaddr_in(cl[1]).join(':')}")
#2
seq_no = cl[2] * 10000000
loop do
begin
msg1 = {"msg" => {"head" => {"type" => "frctl", "seq_no" => seq_no, "version" => 1.0},
"body" => {"file" => "numbers_odom.swf", "start" => 5,
"end" => 3000, "rate" => 40, "duration" => 60}}}
msg2 = {"msg" => {"head" => {"type" => "frctl", "seq_no" => seq_no, "version" => 1.0},
"body" => {"file" => "numbers_fruits.swf", "start" => 5, "end" => 3000, "rate" => 40, "duration" => 60}}}
msg3 = {"msg" => {"head" => {"type" => "frctl", "seq_no" => seq_no, "version" => 1.0},
"body" => {"file" => "$mil.swf", "start" => 5,
"end" => 3000, "rate" => 40, "duration" => 60}}}
[ msg1, msg2, msg3 ].each do |m|
seq_no += 1
m["msg"]["head"]["seq_no"] = seq_no
xml_msg = create_xml_msg(m, nil)
xml_msg.write(cl[0], 0)
cl[0].putc 0
sleep 10
end
rescue
cl[0].close
puts "\nThread #{Thread.current[:number]} exiting..."
Thread.exit
end
end
end
end
$tid = [] # array of active worker thread ids
$wno = [] # array of active worker numbers
$worker_count = 0
$max_workers = 3
$wlist = Array(1..$max_workers) #array of all possible worker numbers
socket = Socket.new( AF_INET, SOCK_STREAM, 0 )
sockaddr = Socket.pack_sockaddr_in( 1999, '0.0.0.0' )
begin
socket.bind( sockaddr )
socket.listen( 5 )
rescue
print $!.class, " ", $!
sleep 3
retry
end
loop do
begin
$tid.each do |t|
if (t.status == false || t.status == "aborting" )
t.join
$wno.delete(t[:number])
$tid.delete(t)
$worker_count -= 1
puts("\nWorker count #{$worker_count}")
end
end
client, client_sockaddr = socket.accept_nonblock
if (client)
if ($worker_count >= $max_workers)
puts "\n too many clients...\n"
client.puts("<msg>error: too many clients; closing connection...</msg>")
client.close
else
$worker_count += 1
$wlist.each do |w| #find a hole in worker number list
if (!$wno.include?(w))
$wno << w #add new worker number to the active worker num array
worker(client, client_sockaddr, w)
break
end
end
puts("\nWorker count #{$worker_count}")
end
end
rescue Errno::EAGAIN, Errno::ECONNABORTED, Errno::EINTR, Errno::EWOULDBLOCK
IO.select([socket], nil, nil, 1)
retry
end
end
REFERENCE
Ruby version 186-25
http://rubylearning.com/satishtalim/ruby_threads.html
A very good resource for socket programming in ruby:
Ruby Sockets: IBM
It was tough for me to get a feel for your code by reading through it, so I can't give you a direct answer. However I can tell you that if you want a good resource for sockets in ruby that pdf is it.
In general i think the only way to eliminate garbage is to fork off a process, let it do the garbage-y stuff then die. If that's your question. NB that jruby, macruby, and rubinius have more advanced GC's.
-r