I want to use Ruby EventMachine and em-http-request to run parallel HTTP synchronous requests triggered from different threads.
The idea is to run a single reactor in its own thread and push HTTP requests to complete on its event queue thanks to EM.next_tick.
The pattern for every call could be
def complete_request(url, options)
Thread.new { EM.run } unless EM.reactor_running?
EM.run do
EM.next_tick do
con = EventMachine::HttpRequest.new(url, options[:connection_headers])
http = com.setup_request(verb, head: options[:headers], body: options[:body])
http.errback { }
http.callback { }
end
end
# wait for request completion (but how?)
...
end
reqs = []
responses = []
reqs << Thread.new { responses << complete_request('http://www.stackoverflow.com', verb: get) }
reqs << Thread.new { responses << complete_request('http://www.flickr.com', verb: get) }
reqs.each { |req| req.join }
To make the requests synchronous, I tried to use Fibers but unsuccessfully. Either the request fails to connect or it completes but never exits the event loop.
I don't want to call EM.stop in the callbacks because it would screw up other requests being executed in parallel I guess, and would also stop the reactor while I want it to run until I decide no more requests should be treated.
Does anyone already try to use EventMachine and em-http-request this way ? Can EventMachine support this use case ?
Related
So basically my goal is get some sort of light-weight ruby daemon(or sidekiq/resque worker), that processes jobs and notifies other apps over http. The app itself does not need to receive http requests, so no rack to remain as light-weight as possible. Pretty much a bit of ruby code I can run in loop {}
So trying to not use EventMachine' reactor pattern and using fiber approach instead. Where would I put EM.run or EM.stop in this context Thread.new { EM.run } doesn't seem to be fiber aware so adding it gave no callbacks? Is there a em-synchrony alternative to this?
#slow=true injects a sleep 3, so page 2 callback should output faster
require 'em-http-request'
require 'fiber'
def http_get(url)
f = Fiber.current
http = EventMachine::HttpRequest.new(url).get
# resume fiber once http call is done
http.callback { f.resume(http) }
http.errback { f.resume(http) }
return Fiber.yield
end
puts "fetching some data from database for request params"
EventMachine.run do
Fiber.new{
page = http_get('http://localhost:3000/status?slow=true')
puts "notified external page it responded with: #{page.response_header.status}"
}.resume
Fiber.new{
page = http_get('http://localhost:4000/status')
puts "notified external page 2 it responded with: #{page.response_header.status}"
}.resume
puts "Finishised notification task"
end
puts "Moving on to next task as fast as possible"
Avoid reinventing the wheel, use EM::Synchrony or even better switch to celluloid or celluloid-io as EM seems to have fallen out of maintenance
I'm trying to build a notifications system with Redis and Sinatra streams. However I can't seem to catch when connection closes down, so the blocking Redis subscription block seems to never close down. What is the best way to achieve this?
get '/user/:id/next_notification' do
stream :keep_open do |out|
$redis.subscribe("notifications:#{params[:id]}") { |on|
on.message { |channel, msg|
$redis.unsubscribe
out << msg
}
}
out.callback {
puts "unsub"
# $redis.unsubscribe
}
out.errback {
puts "unsub"
# $redis.unsubscribe
}
end
end
Redis Subscription is a blocking call. So you need to execute it in a separate thread. Dunno how to do it in Ruby. But i'm sure there must be threading library in ruby.
Wrap the Blocking call in a try..catch and you will know when the connection has closed from the server side.
The example code is from here:
def http_get(url)
f = Fiber.current
http = EventMachine::HttpRequest.new(url).get
# resume fiber once http call is done
http.callback { f.resume(http) }
http.errback { f.resume(http) }
return Fiber.yield
end
EventMachine.run do
Fiber.new{
page = http_get('http://www.google.com/')
puts "Fetched page: #{page.response_header.status}"
if page
page = http_get('http://www.google.com/search?q=eventmachine')
puts "Fetched page 2: #{page.response_header.status}"
end
}.resume
end
So, in the context of the EM run block, the author's creating a fiber and running it immediately with resume. But, I don't understand why the http_get logic is structured in that way. I mean, it's taking the current fiber ( which in this case should be the one created in the EM run block ), it starts a http request which may fail or succeed, and it resumes the current fiber. Afterwards it just calls yield on the fiber. What exactly will be running since he is calling yield? Can someone please explain why http_get is written the way it is?
Fiber is created and triggered in EventMachine
the goal is (a) to fetch a page and (b) work on it
the Fiber should be paused until the page is fetched, this is the role of http_get
http = EventMachine::HttpRequest.new(url).get doesn't trigger anything: EventMachine needs to get the reins back, that's the role of Fiber.yield
Once EventMachine has done the job getting the page, it triggers the callback and resumes the Fiber which was stopped at puts ...
Clearer?
I'm working on creating a background script that uses EventMachine to connect to a server with WebSockets. The script will be run using DelayedJob or Resque. I've been able to get it to talk to the WebSockets server and send messages, but whenever an error is raised within the EventMachine loop it doesn't crash the script - which is what should happen (and what I need to have happen). I don't have to use EventMachine as I'm only sending WebSocket messages and not receiving them - but I'd love any help on this :) thank you!
#!/usr/bin/env ruby
require 'rubygems'
require 'eventmachine'
require 'em-http'
class Job
include EventMachine::Deferrable
def self.perform
job = Job.new
EventMachine.run {
http = EventMachine::HttpRequest.new("ws://localhost:8080/").get :timeout => 0
http.errback { puts "oops" }
http.callback {
puts "WebSocket connected!"
http.send("Hello watcher")
}
http.stream { |msg| }
job.callback { puts "done" }
Thread.new {
job.execute(http)
http.close
EventMachine.stop
}
}
end
def execute(h)
sleep 1
puts "Job Runner!"
h.send("welcome!")
sleep 2
asdsadsa # here I am trying to simulate an error
sleep 1
h.send("we are all done!")
sleep 1
set_deferred_status :succeeded
end
end
Job.perform
Since you're causing an exception inside a thread, you should set Thread.abort_on_exception to true otherwise these errors will not be raised properly.
You don't need to use Thread.new here at all, in fact, it's not thread safe to do so (eventmachine itself is not thread safe, except for EM::Queue, EM::Channel and EM.schedule).
If you wanted to do synchronous things in execute, and you must have that thread, then, you'll want to call h.send via EM.schedule, for example:
EM.schedule { h.send("welcome!") }
If you must have that thread in this way, then, you want to catch exceptions from the thread you spawn yourself. You should then stop and shutdown on your own, or just raise back up in the main (eventmachine) thread:
EM.run do
thread = Thread.new do
raise 'boom'
end
EM.add_periodic_timer(0.1) { thread.join(0) }
end
The above pattern can easily just enumerate an array of threads in the periodic timer instead, if appropriate.
Finally, please note that exception bubbling (correct exception reporting) was only supported in EventMachine > 1.0, which is still in beta. To get usable backtraces when exceptions occur, either gem install eventmachine --pre, or better, use master from the Github repo.
I'm trying to run a mock webserver within a thread within a class. I've tried passing the class' #server property to the thread block but as soon as I try to do server.accept the thread stops. Is there some way to make this work? I want to basically be able to run a webserver off of this script while still taking user input via stdin.gets. Is this possible?
class Server
def initialize()
#server = TCPServer.new(8080)
end
def run()
#thread = Thread.new(#server) { |server|
while true
newsock = server.accept
puts "some stuff after accept!"
next if !newsock
# some other stuff
end
}
end
end
def processCommand()
# some user commands here
end
test = Server.new
while true do
processCommand(STDIN.gets)
end
In the above sample, the thread dies on server.accept
In the code you posted, you're not calling Server#run. That's probably just an oversight in making the post. Server.accept is supposed to block a thread, returning only when someone has connected.
Anyone who goes into writing an HTTP server with bright eyes soon learns that it's more fun to let someone else do that work. For quick and dirty HTTP servers, I've got good results enlisting the aid of WEBrick. It's a part of the Ruby library. Here's a WEBrick server that will serve up "Boo!" When you connect your browser to localhost:8080/:
#!/usr/bin/ruby1.8
require 'webrick'
class MiniServer
def initialize
Thread.new do
Thread::abort_on_exception = true
server = WEBrick::HTTPServer.new(:BindAddress=>'127.0.0.1',
:Port=>8080,
:Logger=>WEBrick::Log.new('/dev/stdout'))
server.mount('/', Servlet, self)
server.start
end
end
private
class Servlet < WEBrick::HTTPServlet::AbstractServlet
def initialize(webrick_server, mini_server)
end
def do_GET(req, resp)
resp.body = "<html><head></head><body>Boo!</body></html>"
end
alias :do_POST :do_GET
end
end
server = MiniServer.new
gets
I don't know ruby, but it looks like server.accept is blocking until you get a tcp connection... your thread will continue as soon as a connection is accepted.
You should start the server in your main thread and then spawn a new thread for each connection that you accept, that way your server will immediately go to accept another connection and your thread will service the one that was just accepted.