I have a situation where I am calling a method on a gem that does its processing asynchronously. I want to be able to wait for the callback to be called before continuing execution on the thread where the method was called.
gem.async_method(args) do |result|
# the callback
end
# wait until callback is called and then continue execution
puts result # somehow have access to the result from the callback
PubNub Ruby SDK http_sync
I think you are just looking for the http_sync parameter when you publish, subscribe', etc. PubNub. https://www.pubnub.com/docs/ruby/api-reference-sdk-v4#publish_args
pubnub.publish(
channel: 'my_channel',
message: { text: 'Hi!' },
http_sync: true
) do |envelope|
puts envelope.status
end
I managed to find the solution to this eventually. The gem I was using returned a Celluloid::Future. If you call future_object.value it will block execution until the value is received.
Related
I'm using EventMachine.defer to handle some long-running processes (an indefinite wait for a response from an outside application). I want to do this in a loop: each time the application responds, I process the response and then immediately want to start waiting for the next response.
My code currently looks like this:
def watch_for_songs_change
EM.defer(
->( ){ `mpc idle playlist` }, # wait for the song list to change
->(_){ update_songs; watch_for_songs_change }
)
end
I realized that this is calling defer from within a callback from defer. Is this valid? Am I spawning one thread from inside another, and will eventually run out of threads? Or does EventMachine invoke the callback after it has returned the thread to the pool?
I've tried to chain calls like this before in EM, and found that using periodic timers is a usually a better design.
#timer = EventMachine.add_periodic_timer( 1 ) { `mpc idle playlist` and update_songs }
Ruby process is single thread. When we start a single process using thin server, why are we still able to handle concurrency request?
require 'sinatra'
require 'thin'
set :server, %w[thin]
get '/test' do
sleep 2 <----
"success"
end
What is inside thin that can handle concurrency request? If it is due to event-machine framework, the code above is actually a sync code which is not for EM used.
Quoting the chapter: "Non blocking IOs/Reactor pattern" in
http://merbist.com/2011/02/22/concurrency-in-ruby-explained/:
"this is the approach used by Twisted, EventMachine and Node.js. Ruby developers can use EventMachine or
an EventMachine based webserver like Thin as well as EM clients/drivers to make non blocking async calls."
The heart of the matter regard EventMachine.defer
*
used for integrating blocking operations into EventMachine's control flow.
The action of defer is to take the block specified in the first parameter (the "operation")
and schedule it for asynchronous execution on an internal thread pool maintained by EventMachine.
When the operation completes, it will pass the result computed by the block (if any)
back to the EventMachine reactor.
Then, EventMachine calls the block specified in the second parameter to defer (the "callback"),
as part of its normal event handling loop.
The result computed by the operation block is passed as a parameter to the callback.
You may omit the callback parameter if you don't need to execute any code after the operation completes.
*
Essentially, in response to an HTTP request, the server executes that you wrote,
invokes the process method in the Connecction class.
have a look at the code in $GEM_HOME/gems/thin-1.6.2/lib/thin/connection.rb:
# Connection between the server and client.
# This class is instanciated by EventMachine on each new connection
# that is opened.
class Connection < EventMachine::Connection
# Called when all data was received and the request
# is ready to be processed.
def process
if threaded?
#request.threaded = true
EventMachine.defer(method(:pre_process), method(:post_process))
else
#request.threaded = false
post_process(pre_process)
end
end
..here is where a threaded connection invoke EventMachine.defer
The reactor
To see where is activated the EventMachine reactor
should follow the initialization of the program:
Notice that for all Sinatra applications and middleware ($GEM_HOME/gems/sinatra-1.4.5/base.rb)
can run the Sinatra app as a self-hosted server using Thin, Puma, Mongrel, or WEBrick.
def run!(options = {}, &block)
return if running?
set options
handler = detect_rack_handler
....
the method detect_rack_handler returns the first Rack::Handler
return Rack::Handler.get(server_name.to_s)
in our test we require thin therefore it returns a Thin rack handler and setup a threaded server
# Starts the server by running the Rack Handler.
def start_server(handler, server_settings, handler_name)
handler.run(self, server_settings) do |server|
....
server.threaded = settings.threaded if server.respond_to? :threaded=
$GEM_HOME/gems/thin-1.6.2/lib/thin/server.rb
# Start the server and listen for connections.
def start
raise ArgumentError, 'app required' unless #app
log_info "Thin web server (v#{VERSION::STRING} codename #{VERSION::CODENAME})"
...
log_info "Listening on #{#backend}, CTRL+C to stop"
#backend.start { setup_signals if #setup_signals }
end
$GEM_HOME/gems/thin-1.6.2/lib/thin/backends/base.rb
# Start the backend and connect it.
def start
#stopping = false
starter = proc do
connect
yield if block_given?
#running = true
end
# Allow for early run up of eventmachine.
if EventMachine.reactor_running?
starter.call
else
#started_reactor = true
EventMachine.run(&starter)
end
end
I'm making a Ruby server using the em-websocket gem. When a client sends some message (e.g. "thread") the server creates two different threads and sends two anwsers to the client in parallel (I'm actually studying multithreading and websockets). Here's my code:
EM.run {
EM::WebSocket.run(:host => "0.0.0.0", :port => 8080) do |ws|
ws.onmessage { |msg|
puts "Recieved message: #{msg}"
if msg == "thread"
threads = []
threads << a = Thread.new {
sleep(1)
puts "1"
ws.send("Message sent from thread 1")
}
threads << b = Thread.new{
sleep(2)
puts "2"
ws.send("Message sent from thread 2")
}
threads.each { |aThread| aThread.join }
end
How it executes:
I'm sending "thread" message to a server
After one second in my console I see printed string "1". After another second I see "2".
Only after that both messages simultaneously are sent to the client.
The problem is that I want to send messages exactly at the same time when debug output "1" and "2" are sent.
My Ruby version is 1.9.3p194.
I don't have experience with EM, so take this with a pinch of salt.
However, at first glance, it looks like "aThread.join" is actually blocking the "onmessage" method from completing and thus also preventing the "ws.send" from being processed.
Have you tried removing the "threads.each" block?
Edit:
After having tested this in arch linux with both ruby 1.9.3 and 2.0.0 (using "test.html" from the examples of em-websocket), I am sure that even if removing the "threads.each" block doesn't fix the problem for you, you will still have to remove it as Thread#join will suspend the current thread until the "joined" threads are finished.
If you follow the function call of "ws.onmessage" through the source code, you will end up at the Connection#send_data method of the Eventmachine module and find the following within the comments:
Call this method to send data to the remote end of the network connection. It takes a single String argument, which may contain binary data. Data is buffered to be sent at the end of this event loop tick (cycle).
As "onmessage" is blocked by the "join" until both "send" methods have run, the event loop tick cannot finish until both sets of data are buffered and thus, all the data cannot be sent until this time.
If it is still not working for you after removing the "threads.each" block, make sure that you have restarted your eventmachine and try setting the second sleep to 5 seconds instead. I don't know how long a typical event loop takes in eventmachine (and I can't imagine it to be as long as a second), however, the documentation basically says that if several "send" calls are made within the same tick, they will all be sent at the same time. So increasing the time difference will make sure that this is not happening.
I think the problem is that you are calling sleep method, passing 1 to the first thread and 2 to the second thread.
Try removing sleep call on both threads or passing the same value on each call.
In all eventmachine code that I've seen, the callbacks / errorbacks were declared after the actual call of the method.
Here's a simple example:
about = EventMachine::HttpRequest.new('http://google.ca/search?q=eventmachine').get
about.callback { # callback nesting, ad infinitum }
about.errback { # error-handling code }
Why are the callbacks and errorbacks declared AFTER ? Is it not possible that the EM::HttpRequest already finished w/ some sort of success or error state? How does EM guarantee that callbacks and errorbacks are actually caught?
The .get call only sets up the request.
The get request method in EM::HttpRequest module.
EM::HttpRequest uses EM::Deferrable module which is sort of a switch.
Add these two together, and you get a functionality where the request is first built and waits until a response is received. So, for the first iteration of the EM.run do..end loop, the connection is setup, the callbacks are registered and when the response is received, which will be processed in the next iteration/whenever the response is received, the set_deferrable_status is set to :succeeded or :failed and the corresponding callback/errback is executed.
Take the following code....
http = EM::HttpRequest.new('http://google.com').get
http.callback {puts "it was a great call"}
http.errback { puts "it was a bad call" }
You might think that if the asynchronous request happens faster than we can set the callback it's possible that the callback will never be called.
The request is happening asynchronously so we might think it's a possibility. But it's not. What if we put some really long running code in between the time we actually set up the callback? I'll demostrate and show that the callbacks still work.
http = EM::HttpRequest.new('http://google.com').get
#Some really long running code
100000000000.times do
#some really long running code
end
http.callback {puts "it was a great call"}
http.errback { puts "it was a bad call" }
The request in this case completes long before the really long running code completes but the callback will still be called? Why?
The reason is because HttpRequest is a Deferrable. It inherits from it. Even though a Deferrable is something that can run asyc Defferables have a status. success or fail and we still have a reference to that deferable in a variable called http.
When we call http.callback {"puts "it was a great call"} We imediately check to see if the status of the deffereable, in this case http, is success. If it is, imediately call the callback. Else set it up so that the defferable calls it whenever it finishes with a "success" status. It's that simple. As long as we have a reference to that defferable we can set the callback at any time.
My guess was confirmed when I actually took a look at the source code for Defferable.
http://eventmachine.rubyforge.org/EventMachine/Deferrable.html#callback-instance_method
Royce
def perform
refund_log = {
success: refund_retry.success?,
amount: refund_amount,
action: "refund"
}
if refund_retry.success?
refund_log[:reference] = refund_retry.transaction.id
refund_log[:message] = refund_retry.transaction.status
else
refund_log[:message] = refund_retry.message
refund_log[:params] = {}
refund_retry.errors.each do |error|
refund_log[:params][error.code] = error.message
end
order_transaction.message = refund_log[:params].values.join('|')
raise "delayed RefundJob has failed"
end
end
When I raise "delayed RefundJob has failed" in the else statement, it creates an Airbrake. I want to run the job again if it ends up in the else section.
Is there any way to re-queue the job without raising an exception? And prevent creating an airbrake?
I am using delayed_job version 1.
The cleanest way would be to re-queue, i.e. create a new job and enqueue it, and then exit the method normally.
To elaborate on #Roman's response, you can create a new job, with a retry parameter in it, and enqueue it.
If you maintain the retry parameter (increment it each time you re-enqueue a job), you can track how many retries you made, and thus avoid an endless retry loop.
DelayedJob expects a job to raise an error to requeued, by definition.
From there you can either :
Ignore your execpetion on airbrake side, see https://github.com/airbrake/airbrake#filtering so it still gets queued again without filling your logs
Dive into DelayedJob code where you can see on https://github.com/tobi/delayed_job/blob/master/lib/delayed/job.rb#L65 that a method named reschedule is available and used by run_with_lock ( https://github.com/tobi/delayed_job/blob/master/lib/delayed/job.rb#L99 ). From there you can call reschedule it manually, instead of raising your exception.
About the later solution, I advise adding some mechanism that still fill an airbrake report on the third or later try, you can still detect that something is wrong without the hassle of having your logs filled by the attempts.