I know that current timeouts are currently not supported with Http.rb and Celluloid[1], but is there an interim workaround?
Here's the code I'd like to run:
def fetch(url, options = {} )
puts "Request -> #{url}"
begin
options = options.merge({ socket_class: Celluloid::IO::TCPSocket,
timeout_class: HTTP::Timeout::Global,
timeout_options: {
connect_timeout: 1,
read_timeout: 1,
write_timeout: 1
}
})
HTTP.get(url, options)
rescue HTTP::TimeoutError => e
[do more stuff]
end
end
Its goal is to test a server as being live and healthy. I'd be open to alternatives (e.g. %x(ping <server>)) but these seem less efficient and actually able to get at what I'm looking for.
[1] https://github.com/httprb/http.rb#celluloidio-support
You can set timeout on future calls when you fetch for the request
Here is how to use timeout with Http.rb and Celluloid-io
require 'celluloid/io'
require 'http'
TIMEOUT = 10 # in sec
class HttpFetcher
include Celluloid::IO
def fetch(url)
HTTP.get(url, socket_class: Celluloid::IO::TCPSocket)
rescue Exception => e
# error
end
end
fetcher = HttpFetcher.new
urls = %w(http://www.ruby-lang.org/ http://www.rubygems.org/ http://celluloid.io/)
# Kick off a bunch of future calls to HttpFetcher to grab the URLs in parallel
futures = urls.map { |u| [u, fetcher.future.fetch(u)] }
# Consume the results as they come in
futures.each do |url, future|
# Wait for HttpFetcher#fetch to complete for this request
response = future.value(TIMEOUT)
puts "*** Got #{url}: #{response.inspect}\n\n"
end
I am working with ruby 192 p290: under one unit-test script (shown below) throws ThreadError
1) Error:
test_orchpr_pass(TC_MyTest):
ThreadError: deadlock; recursive locking
internal:prelude:8:in `lock'
internal:prelude:8:in `synchronize'
testth.rb:121:in `orchpr_run'
testth.rb:158:in `test_orchpr_pass'
With ruby 187 gives error: Thread tried to join itself.
CODE
def orchpr_run(timeout = 60)
# used by the update function to signal that a final update was
# received from all clients
#update_mutex.lock
# required since we'll have to act as an observer to the DRb server
DRb.start_service
# get configuration objects
run_config_type = DataLayer.get_run_config_type
client_daemon = DataLayer.get_client_daemon_by_branch (run_config_type, #branch)
client_daemon['port_no'] = 9096
#get the servers for this client_daemon
servers = DataLayer.get_servers(run_config_type, client_daemon.id)
servers.each { |server| #pr[server.host_name] = OrchestratedPlatformRun.new(run_config_type, server, timeout)
}
#pr.each_value { |x| x.add_observer(self)
#pr.each_value { |x| x.start(#service_command_pass, true)
# wait for update to receive notifications from all servers # this is the statement causing error:
#update_mutex.synchronize {} end
Another piece of code throwing same error:
require "thread"
require "timeout"
def calc_fib(n)
if n == 0
0
elsif n == 1
1
else
calc_fib(n-1) + calc_fib(n-2)
end
end
lock = Mutex.new
threads = 20.times.collect do
Thread.new do
20.times do
begin
Timeout.timeout(0.25) do
lock.synchronize{ calc_fib(1000) }
end
rescue ThreadError => e
puts "#{e.class}: #{e.message}:\n" + e.backtrace.join("\n") + "\n\n"
rescue Timeout::Error => e
#puts e.class
nil
end
end
end
end
threads.each{ |t| t.join }
Commenting synchronizing Block will cause the error to disappear but then then threads are not able to synchronize. I found some stuff on net, saying bug with ruby 192 , need changes in file prelude.rb and thread.c regarding MUTEX synchronization.
But Under windows installation Unable to find file prelude.rb
If a mutex is locked by a thread then an error will be raised if you try and lock it again from the same thread
This is exactly what you are doing, since synchronize is just a convenience for method for locking the mutex, yielding to the block and then releasing the lock. I'm not sure what you're trying to do, but it feels to me like you might be trying to use mutexes for something other than their intended purposes.
Using threads and locks well is difficult to get right - you might want to look at celluloid for a different approach to concurrency.
I have a rabbitmq queue subscriber that spins up a new thread every time a new message is consumed:
AMQP.start(#conf) do |connection|
channel = AMQP::Channel.new(connection)
requests_queue = channel.queue("one")
requests_queue.subscribe(:ack => true) do |header, body|
puts "we have a message at #{Time.now} and n is #{n}"
url_search = MultiJson.decode(body)
Thread.new do
5.times do
lead = get_lead(n, (n == 5))
puts "message #{n} is_last = #{lead.is_last} at #{Time.now}";
AMQP::Exchange.default.publish(
MultiJson.encode(lead),
:routing_key => header.reply_to,
:correlation_id => header.correlation_id
)
n += 1
sleep(2)
end
end
end
end
My question is, how do I dispose of the thread after the message is handled? Should I be using the threadpool?
I am using JRuby. Does the above code create a Java JVM thread behind the scenes using the normal ruby syntax or should I be explicitly creating a Java thread?
You don't have to manually dispose the thread I think, and you should be using ruby threads, from what I gather they are java threads in jruby, which is from what jruby gets it's nice performance.
A common thing to do is to spin up a couple of threads and then join all of them before continuing if you want to be sure that all are complete before the next step, but it doesn't seem to be required here.
Here's a little test program:
# foo.rb
a = Thread.new { print "a"; sleep(1); print "b"; print "c" }
require 'pp'
pp Thread.list
puts "foo"
sleep(2);
pp Thread.list
puts "bar"
As you can see the spawned background thread is automatically removed. (Tested in jruby as well as 1.9.2
$ ruby foo.rb
a[#<Thread:0x00000100887678 run>, #<Thread:0x0000010086c7d8 sleep>]
foo
bc[#<Thread:0x00000100887678 run>]
bar
I'm playing with EventMachine for some days now which has a steep learn curve IMHO ;-) I try to return a hash by triggering HttpHeaderCrawler.query() which I need within the callback. But what I get in this case is not the hash {'http_status' => xxx, 'http_version' => xxx} but an EventMachine::HttpClient Object itself.
I wanna keep the EM.run block clean and wanna do all logic within own classes / modules so how to return such a value into the main loop to access it by the callback? Many thanks in advance ;-)
#!/usr/bin/env ruby
require 'eventmachine'
require 'em-http-request'
class HttpHeaderCrawler
include EM::Deferrable
def query(uri)
http = EM::HttpRequest.new(uri).get
http.callback do
http_header = {
"http_status" => http.response_header.http_status,
"http_version" => http.response_header.http_version
}
puts "Returns to EM main loop: #{http_header}"
succeed(http_header)
end
end
end
EM.run do
domains = ['http://www.google.com', 'http://www.facebook.com', 'http://www.twitter.com']
domains.each do |domain|
hdr = HttpHeaderCrawler.new.query(domain)
hdr.callback do |header|
puts "Received from HttpHeaderCrawler: #{header}"
end
end
end
This snippet produces the following output:
Returns to EM main loop: {"http_status"=>302, "http_version"=>"1.1"}
Received from HttpHeaderCrawler: #<EventMachine::HttpClient:0x00000100d57388>
Returns to EM main loop: {"http_status"=>301, "http_version"=>"1.1"}
Received from HttpHeaderCrawler: #<EventMachine::HttpClient:0x00000100d551a0>
Returns to EM main loop: {"http_status"=>200, "http_version"=>"1.1"}
Received from HttpHeaderCrawler: #<EventMachine::HttpClient:0x00000100d56280>
I think the problem is #query returns http.callback, which returns the http object itself, whereas it should return self, i.e. the HttpHeaderCrawler. See if this works.
def query(uri)
http = EM::HttpRequest.new(uri).get
http.callback do
http_header = {
"http_status" => http.response_header.http_status,
"http_version" => http.response_header.http_version
}
puts "Returns to EM main loop: #{http_header}"
succeed(http_header)
end
self
end
I am using ruby-smpp and redis to achive a queue based background worker to send SMPP messages.
And i am wondering if I am using eventmachine in the right way. It works but it doesnt feel right.
#!/usr/bin/env ruby
# Sample SMS gateway that can receive MOs (mobile originated messages) and
# DRs (delivery reports), and send MTs (mobile terminated messages).
# MTs are, in the name of simplicity, entered on the command line in the format
# <sender> <receiver> <message body>
# MOs and DRs will be dumped to standard out.
require 'smpp'
require 'redis/connection/hiredis'
require 'redis'
require 'yajl'
require 'time'
LOGFILE = File.dirname(__FILE__) + "/sms_gateway.log"
PIDFILE = File.dirname(__FILE__) + '/worker_test.pid'
Smpp::Base.logger = Logger.new(LOGFILE)
#Smpp::Base.logger.level = Logger::WARN
REDIS = Redis.new
class MbloxGateway
# MT id counter.
##mt_id = 0
# expose SMPP transceiver's send_mt method
def self.send_mt(sender, receiver, body)
if sender =~ /[a-z]+/i
source_addr_ton = 5
else
source_addr_ton = 2
end
##mt_id += 1
##tx.send_mt(('smpp' + ##mt_id.to_s), sender, receiver, body, {
:source_addr_ton => source_addr_ton
# :service_type => 1,
# :source_addr_ton => 5,
# :source_addr_npi => 0 ,
# :dest_addr_ton => 2,
# :dest_addr_npi => 1,
# :esm_class => 3 ,
# :protocol_id => 0,
# :priority_flag => 0,
# :schedule_delivery_time => nil,
# :validity_period => nil,
# :registered_delivery=> 1,
# :replace_if_present_flag => 0,
# :data_coding => 0,
# :sm_default_msg_id => 0
#
})
end
def logger
Smpp::Base.logger
end
def start(config)
# Write this workers pid to a file
File.open(PIDFILE, 'w') { |f| f << Process.pid }
# The transceiver sends MT messages to the SMSC. It needs a storage with Hash-like
# semantics to map SMSC message IDs to your own message IDs.
pdr_storage = {}
# Run EventMachine in loop so we can reconnect when the SMSC drops our connection.
loop do
EventMachine::run do
##tx = EventMachine::connect(
config[:host],
config[:port],
Smpp::Transceiver,
config,
self # delegate that will receive callbacks on MOs and DRs and other events
)
# Let the connection start before we check for messages
EM.add_timer(3) do
# Maybe there is some better way to do this. IDK, But it works!
EM.defer do
loop do
# Pop a message
message = REDIS.lpop 'messages:send:queue'
if message # If there is a message. Process it and check the queue again
message = Yajl::Parser.parse(message, :check_utf8 => false) # Parse the message from Json to Ruby hash
if !message['send_after'] or (message['send_after'] and Time.parse(message['send_after']) < Time.now)
self.class.send_mt(message['sender'], message['receiver'], message['body']) # Send the message
REDIS.publish 'log:messages', "#{message['sender']} -> #{message['receiver']}: #{message['body']}" # Push the message to the redis queue so we can listen to the channel
else
REDIS.lpush 'messages:queue', Yajl::Encoder.encode(message)
end
else # If there is no message. Sleep for a second
sleep 1
end
end
end
end
end
sleep 2
end
end
# ruby-smpp delegate methods
def mo_received(transceiver, pdu)
logger.info "Delegate: mo_received: from #{pdu.source_addr} to #{pdu.destination_addr}: #{pdu.short_message}"
end
def delivery_report_received(transceiver, pdu)
logger.info "Delegate: delivery_report_received: ref #{pdu.msg_reference} stat #{pdu.stat}"
end
def message_accepted(transceiver, mt_message_id, pdu)
logger.info "Delegate: message_accepted: id #{mt_message_id} smsc ref id: #{pdu.message_id}"
end
def message_rejected(transceiver, mt_message_id, pdu)
logger.info "Delegate: message_rejected: id #{mt_message_id} smsc ref id: #{pdu.message_id}"
end
def bound(transceiver)
logger.info "Delegate: transceiver bound"
end
def unbound(transceiver)
logger.info "Delegate: transceiver unbound"
EventMachine::stop_event_loop
end
end
# Start the Gateway
begin
puts "Starting SMS Gateway. Please check the log at #{LOGFILE}"
# SMPP properties. These parameters work well with the Logica SMPP simulator.
# Consult the SMPP spec or your mobile operator for the correct settings of
# the other properties.
config = {
:host => 'server.com',
:port => 3217,
:system_id => 'user',
:password => 'password',
:system_type => 'type', # default given according to SMPP 3.4 Spec
:interface_version => 52,
:source_ton => 0,
:source_npi => 1,
:destination_ton => 1,
:destination_npi => 1,
:source_address_range => '',
:destination_address_range => '',
:enquire_link_delay_secs => 10
}
gw = MbloxGateway.new
gw.start(config)
rescue Exception => ex
puts "Exception in SMS Gateway: #{ex} at #{ex.backtrace.join("\n")}"
end
Some easy steps to make this code more EventMachine-ish:
Get rid of the blocking Redis driver, use em-hiredis
Stop using defer. Pushing work out to threads with the Redis driver will make things even worse as it relies on locks around the socket it's using.
Get rid of the add_timer(3)
Get rid of the inner loop, replace it by rescheduling a block for the next event loop using EM.next_tick. The outer one is somewhat unnecessary. You shouldn't loop around EM.run as well, it's cleaner to properly handle a disconnect by doing a reconnect in your unbound method instead of stopping and restarting the event loop, by calling the ##tx.reconnect.
Don't sleep, just wait. EventMachine will tell you when new things come in on a network socket.
Here's how the core code around EventMachine would look like with some of the improvements:
def start(config)
File.open(PIDFILE, 'w') { |f| f << Process.pid }
pdr_storage = {}
EventMachine::run do
##tx = EventMachine::connect(
config[:host],
config[:port],
Smpp::Transceiver,
config,
self
)
REDIS = EM::Hiredis.connect
pop_message = lambda do
REDIS.lpop 'messages:send:queue' do |message|
if message # If there is a message. Process it and check the queue again
message = Yajl::Parser.parse(message, :check_utf8 => false) # Parse the message from Json to Ruby hash
if !message['send_after'] or (message['send_after'] and Time.parse(message['send_after']) < Time.now)
self.class.send_mt(message['sender'], message['receiver'], message['body'])
REDIS.publish 'log:messages', "#{message['sender']} -> #{message['receiver']}: #{message['body']}"
else
REDIS.lpush 'messages:queue', Yajl::Encoder.encode(message)
end
end
EM.next_tick &pop_message
end
end
end
end
Not perfect and could use some cleaning up too, but this is more what it should be like in an EventMachine manner. No sleeps, avoid using defer if possible, and don't use network drivers that potentially block, implement traditional loop by rescheduling things on the next reactor loop. In terms of Redis, the difference is not that big, but it's more EventMachine-y this way imho.
Hope this helps. Happy to explain further if you still have questions.
You're doing blocking Redis calls in EM's reactor loop. It works, but isn't the way to go. You could take a look at em-hiredis to properly integrate Redis calls with EM.