So basically my goal is get some sort of light-weight ruby daemon(or sidekiq/resque worker), that processes jobs and notifies other apps over http. The app itself does not need to receive http requests, so no rack to remain as light-weight as possible. Pretty much a bit of ruby code I can run in loop {}
So trying to not use EventMachine' reactor pattern and using fiber approach instead. Where would I put EM.run or EM.stop in this context Thread.new { EM.run } doesn't seem to be fiber aware so adding it gave no callbacks? Is there a em-synchrony alternative to this?
#slow=true injects a sleep 3, so page 2 callback should output faster
require 'em-http-request'
require 'fiber'
def http_get(url)
f = Fiber.current
http = EventMachine::HttpRequest.new(url).get
# resume fiber once http call is done
http.callback { f.resume(http) }
http.errback { f.resume(http) }
return Fiber.yield
end
puts "fetching some data from database for request params"
EventMachine.run do
Fiber.new{
page = http_get('http://localhost:3000/status?slow=true')
puts "notified external page it responded with: #{page.response_header.status}"
}.resume
Fiber.new{
page = http_get('http://localhost:4000/status')
puts "notified external page 2 it responded with: #{page.response_header.status}"
}.resume
puts "Finishised notification task"
end
puts "Moving on to next task as fast as possible"
Avoid reinventing the wheel, use EM::Synchrony or even better switch to celluloid or celluloid-io as EM seems to have fallen out of maintenance
Related
The example code is from here:
def http_get(url)
f = Fiber.current
http = EventMachine::HttpRequest.new(url).get
# resume fiber once http call is done
http.callback { f.resume(http) }
http.errback { f.resume(http) }
return Fiber.yield
end
EventMachine.run do
Fiber.new{
page = http_get('http://www.google.com/')
puts "Fetched page: #{page.response_header.status}"
if page
page = http_get('http://www.google.com/search?q=eventmachine')
puts "Fetched page 2: #{page.response_header.status}"
end
}.resume
end
So, in the context of the EM run block, the author's creating a fiber and running it immediately with resume. But, I don't understand why the http_get logic is structured in that way. I mean, it's taking the current fiber ( which in this case should be the one created in the EM run block ), it starts a http request which may fail or succeed, and it resumes the current fiber. Afterwards it just calls yield on the fiber. What exactly will be running since he is calling yield? Can someone please explain why http_get is written the way it is?
Fiber is created and triggered in EventMachine
the goal is (a) to fetch a page and (b) work on it
the Fiber should be paused until the page is fetched, this is the role of http_get
http = EventMachine::HttpRequest.new(url).get doesn't trigger anything: EventMachine needs to get the reins back, that's the role of Fiber.yield
Once EventMachine has done the job getting the page, it triggers the callback and resumes the Fiber which was stopped at puts ...
Clearer?
I'm an EM newbie and writing two codes to compare synchronous and asynchronous IO. I'm using Ruby 1.8.7.
The example for sync IO is:
def pause_then_print(str)
sleep 2
puts str
end
5.times { |i| pause_then_print(i) }
puts "Done"
This works as expected, taking 10+ seconds until termination.
On the other hand, the example for async IO is:
require 'rubygems'
require 'eventmachine'
def pause_then_print(str)
Thread.new do
EM.run do
sleep 2
puts str
end
end
end
EventMachine.run do
EM.add_timer(2.5) do
puts "Done"
EM.stop_event_loop
end
EM.defer(proc do
5.times { |i| pause_then_print(i) }
end)
end
5 numbers are shown in 2.x seconds.
Now I explicitly wrote code that EM event loop to be stopped after 2.5 seconds. But what I want is that the program terminates right after printing out 5 numbers. For doing that, I think EventMachine should recognize all 5 threads are done, and then stop the event loop.
How can I do that? Also, please correct the async IO example if it can be more natural and expressive.
Thanks in advance.
A few things about your Async code. EM.defer schedules the code to execute on a thread. You're then creating more threads. There isn't much point to doing that when you could just use EM.defer in your creation loop. This has the added benefit that EM will service the threads from it's internal threadpool which should be a bit faster as there is no thread creation overhead. (Just note, the EM threadpool has, I believe, 20 threads in it so you want to stay below that number). Something like the following should work (although I haven't tested it)
require 'rubygems'
require 'eventmachine'
def pause_then_print(str)
sleep 2
puts str
end
EventMachine.run do
EM.add_timer(2.5) do
puts "Done"
EM.stop_event_loop
end
5.times do |i|
EM.defer { pause_then_print(i) }
end
end
In terms of detecting when the work is done, you can have EM.defer execute a callback when its operation is complete. So, you could have a little bit of code in there that adds the callback when i == 4, or something similar. See the EM docs for how to add the callback: EM.defer
I am trying to implement a request-response pattern using the em-zeromq gem, but I can't get the response socket to send a message back to the request socket in its handler. I have written some very simple code to test it:
em_req.rb
require 'em-zeromq'
client_id = ARGV[0] ? ARGV[0].to_i : 1
message = ARGV[1] || "Foo"
Thread.abort_on_exception = true
class ReqHandler
attr_reader :received
def on_readable(socket, messages)
messages.each do |m|
puts "Received message from server: #{m.copy_out_string}"
end
end
end
trap('INT') do
EM.stop
end
ctx = EM::ZeroMQ::Context.new(1)
EM.run do
conn = ctx.connect(ZMQ::REQ, 'tcp://127.0.0.1:9000', ReqHandler.new, identity: "client#{client_id}")
conn.socket.send_string(message)
end
em_rep.rb
require 'em-zeromq'
Thread.abort_on_exception = true
class ResponseHandler
attr_reader :received
def on_readable(socket, messages)
message = messages.first.copy_out_string
puts "Received message from client: #{message}"
socket.send_msg("re: #{message}")
end
end
trap('INT') do
EM.stop
end
ctx = EM::ZeroMQ::Context.new(1)
EM.run do
socket = ctx.bind(ZMQ::REP, 'tcp://127.0.0.1:9000', ResponseHandler.new)
end
I have written similar code using the push-pull pattern and got that to work, but for request-response all I get is the response code printing "Received message from client1: Foo" but the reply never reaches the request code. I suspect it has to do with writing to the socket in the response code's handler, because the same thing happens when I use a request-router pattern. The only time it works is when I send a message from the server without sending a message from the client first (using push-pull).
Any ideas about what might be causing this? The author of the gem isn't maintaining it anymore, but I thought I would post this issue anyway in the hopes of other developers with similar experiences seeing this.
I am using em-zeromq 0.2.2 on Ruby 1.9.2p290.
I commmited a fix in the master branch which should solve your problem, can you give it a try ?
You can use bundler to easily test it:
Create a file called Gemfile in your application folder:
source :rubygems
gem 'em-zeromq', :git => "git://github.com/andrewvc/em-zeromq.git"
And add this on top of your ruby files:
require 'rubygems'
require 'bundler/setup'
And last run this in the application folder ($ is your prompt):
$ bundle
Now you can execute your ruby files they will use the latest code from github
Edit: I am the new maintainer for the em-zeromq gem.
The em-synchrony documentation links to this article which implies that this code with fiber:
require 'eventmachine'
require 'fiber'
require 'em-http-request'
def http_get(url)
f = Fiber.current
http = EventMachine::HttpRequest.new(url).get
# resume fiber once http call is done
http.callback { f.resume(http) }
http.errback { f.resume(http) }
return Fiber.yield
end
EventMachine.run do
Fiber.new {
page = http_get('http://myurl')
puts "Fetched page: #{page.response}"
EventMachine.stop
}.resume
end
...is equivalent to this much simpler code using em-synchrony:
require 'em-synchrony'
require 'em-http-request'
EventMachine.synchrony do
page = EventMachine::HttpRequest.new("http://myurl").get
p "No callbacks! Fetched page: #{page.response}"
EventMachine.stop
end
However running the two produces different results. In the first the fiber yields until the HTML response comes back, while the second seems to print immediately without waiting for the response and as a result the printed response is empty. Am I misreading or mistyping, or is the article actually suggesting the wrong thing?
You need to use extended version of EventMachine::HttpRequest that knows how to work with EventMachine.synchrony.
Change
require 'em-http-request'
to
require "em-synchrony/em-http"
This in turn will require "em-http-request" and will patch #get, #head, #post, #delete, #put methods of EventMachine::HttpRequest to work with Fibers.
Here is the link to source code of em-synchrony/em-http.
I'm working on creating a background script that uses EventMachine to connect to a server with WebSockets. The script will be run using DelayedJob or Resque. I've been able to get it to talk to the WebSockets server and send messages, but whenever an error is raised within the EventMachine loop it doesn't crash the script - which is what should happen (and what I need to have happen). I don't have to use EventMachine as I'm only sending WebSocket messages and not receiving them - but I'd love any help on this :) thank you!
#!/usr/bin/env ruby
require 'rubygems'
require 'eventmachine'
require 'em-http'
class Job
include EventMachine::Deferrable
def self.perform
job = Job.new
EventMachine.run {
http = EventMachine::HttpRequest.new("ws://localhost:8080/").get :timeout => 0
http.errback { puts "oops" }
http.callback {
puts "WebSocket connected!"
http.send("Hello watcher")
}
http.stream { |msg| }
job.callback { puts "done" }
Thread.new {
job.execute(http)
http.close
EventMachine.stop
}
}
end
def execute(h)
sleep 1
puts "Job Runner!"
h.send("welcome!")
sleep 2
asdsadsa # here I am trying to simulate an error
sleep 1
h.send("we are all done!")
sleep 1
set_deferred_status :succeeded
end
end
Job.perform
Since you're causing an exception inside a thread, you should set Thread.abort_on_exception to true otherwise these errors will not be raised properly.
You don't need to use Thread.new here at all, in fact, it's not thread safe to do so (eventmachine itself is not thread safe, except for EM::Queue, EM::Channel and EM.schedule).
If you wanted to do synchronous things in execute, and you must have that thread, then, you'll want to call h.send via EM.schedule, for example:
EM.schedule { h.send("welcome!") }
If you must have that thread in this way, then, you want to catch exceptions from the thread you spawn yourself. You should then stop and shutdown on your own, or just raise back up in the main (eventmachine) thread:
EM.run do
thread = Thread.new do
raise 'boom'
end
EM.add_periodic_timer(0.1) { thread.join(0) }
end
The above pattern can easily just enumerate an array of threads in the periodic timer instead, if appropriate.
Finally, please note that exception bubbling (correct exception reporting) was only supported in EventMachine > 1.0, which is still in beta. To get usable backtraces when exceptions occur, either gem install eventmachine --pre, or better, use master from the Github repo.