I'm using Celluloid::IO to do DNS query and below is my code:
require 'celluloid/io'
class MyResolver
include Celluloid::IO
def initialize
#resolver = DNSResolver.new
end
def resolve(domain)
ips = #resolver.resolve domain
#sleep 1
return {domain: domain, ip: ips}
end
end
pool = MyResolver.pool(size: 5)
domains = [
'www.google.com',
## many other record
]
futures = domains.map {|d| pool.future.resolve(d)}
futures.each do |future|
puts "#{future.value}"
end
This code works and finished in few seconds. But when I add the line sleep 1(just for learning purpose), after printing some results, the process blocked forever, which is very strange.
Thanks for any help.
sleep is an overridden keyword in Celluloid, so if you want sleep from Ruby itself, use Kernel.sleep. But that being said, as of 0.17.0-dependent branch of Celluoid::IO this error you describe does not exist ( anymore? ).
I used your reproducible failing case to test the new celluloid-pool gem being released in version 0.17.0 of Celluloid, and it is working no problem with sleep 1 as is.
Something is wrong with DNSResolver at least in that case, but you can use "thread-aware DNS resolver" Resolv from standard ruby library - no any blocks with ~25k domains array. Don't forget to catch exceptions from Resolv.
Related
Here is my current implementation of a safe eval in Ruby:
$mthread = Thread.new {}
class SafeEval
def self.safeEval code
$killed = false
$mthread = Thread.new {
$SAFE = 4
result = begin
eval code
rescue Exception => e
"Error in eval: #{e}"
end
Thread.current[:evalResult] = result
}
Thread.new {
sleep 3
if $mthread.alive?
$killed = true
Thread.kill $mthread
end
}.join
$mthread.join
$killed ? 'Error in eval: Maximum execution time reached' : String($mthread[:evalResult])
end
end
It uses $SAFE = 4. From my understanding, and from this post I've read, that's not enough to stop security vulnerabilities. However, if I set a maximum execution time, and kill the thread running the code after the time expires, is that enough for a safe eval?
If not, why isn't it safe? Are there still any vulnerabilites? Is there any way to prevent these vulnerabilities as well?
Of course setting an execution time is not secure. All you're doing then is making the execution path of whatever is executed less predictable.
Security is not about saying 'Oh, no untrusted code can cause trouble if it runs for less than 4s'. Security starts with not letting untrusted code execute anywhere outside of a strict sandboxed environment.
Why are you using eval here? What are you trying to accomplish?
edit- I'm an idiot, ignore, I read that as a timeout, not as a level. :P That said, this works perfectly well on my local machine:
$mthread = Thread.new {}
class SafeEval
def self.safeEval code
$killed = false
$mthread = Thread.new {
$SAFE = 4
result = begin
eval code
rescue Exception => e
"Error in eval: #{e}"
end
Thread.current[:evalResult] = result
}
Thread.new {
sleep 3
if $mthread.alive?
$killed = true
Thread.kill $mthread
end
}.join
$mthread.join
$killed ? 'Error in eval: Maximum execution time reached' : String($mthread[:evalResult])
end
end
SafeEval.safeEval("`cat /etc/passwd > /Users/usr/development/source/tests/test.txt`")
run that code on a web server that has a mail client or other method of connecting to remote servers, and an attacker can establish the user accounts on your machine and from there engage in social engineering to recover passwords.
Sandboxing is important because it prevents stuff like the above. $SAFE is not enough in and of itself, and this is one of the reasons you never put something like eval() or anything else whose core job is to execute untrusted code in an environment that could be reached by an attacker.
If you consider 'being able to kill the bot' as security vulnerability, then $SAFE = 4 is not safe enough, as we found out while testing it.
People can execute this, without getting the 'unsafe eval' error:
loop { Thread.start { loop{} } }
This starts many threads within 3 seconds, and after enough executions this will have created lots and lots of threads, which has killed the bot while testing.
Or this:
Thread.start { loop { Thread.start { loop {} } } }
It starts a thread which keeps generating other threads. The timeout does not stop this.
I have a pair of Puppet custom functions, one of which, namely am_func_cluster, returns a hash of array of currently running instances (reading a file as input) and the second one call that function, iterate over the array and returns the first successful one that listening to port 22. Here is the 2nd function:
module Puppet::Parser::Functions
newfunction(:am_func_head, :type => :rvalue ) do |args|
Puppet::Parser::Functions.function('am_func_cluster')
mCls = function_am_func_cluster(['/opt/running-inst.txt'])
cls = args[0].to_sym if args[0].is_a? String
require 'socket'
require 'timeout'
mCls[cls].each do |dns|
begin
Timeout::timeout(1) { TCPSocket.new(dns, 22)
return (dns if mCls.key?(cls)) || 'undefined'
}
break
rescue SocketError
rescue Timeout::Error
end
end
end
end
upon running, it returns this error:
Error: Could not retrieve catalog from remote server: Error 400 on
SERVER: return can't jump across threads at
/etc/puppet/manifests/nodes.pp:19 on node ip-10-0-9-130.xxx
It works just fine, if I comment out the begin ... end bit in the script. Google didn't favor much in this case, so far. Does anyone one know what that error means or what am I doing wrong. Still don't find myself very efficient understanding the errors returned by Ruby. Any pointer much appreciated. Cheers!!
As it says, you cannot use return there. Use break to escape from the timeout block:
require "timeout"
Timeout.timeout(3){break "foo"}
# => "foo"
and you should not use break outside of it.
As a general tip, it is easy to confuse return, break, and next. If one of them does not work, try another.
I want to be able to call a method that repeats x amount of times on a separate thread that sends messages such as "still running" every few moments to the console while I am free to call other methods that do the same thing.
This works in my test environment and everything checks out via rspec - but when I move the code into a gem and call it from another script, it appears that the code is working in additional threads, but the strings are never sent to my console (or anywhere that I can tell).
I will put the important parts of the code below, but for a better understanding it is important to know that:
The code will check stock market prices at set intervals with the intent of notifying the user when the value of said stock reaches a specific price.
The code should print to the console a message stating that the code is still running when the price has not been met.
The code should tell the user that the stock has met the target price and then stop looping.
Here is the code:
require "trade_watcher/version"
require "market_beat"
module TradeWatcher
def self.check_stock_every_x_seconds_for_value(symbol, seconds, value)
t1 = Thread.new{(self.checker(symbol, seconds, value))}
end
private
def self.checker(symbol, seconds, value)
stop_time = get_stop_time
pp stop_time
until is_stock_at_or_above_value(symbol, value) || Time.now >= stop_time
pp "#{Time.now} #{symbol} has not yet met your target of #{value}."
sleep(seconds)
end
if Time.now >= stop_time
out_of_time(symbol, value)
else
reached_target(symbol, value)
end
end
def self.get_stop_time
Time.now + 3600 # an hour from Time.now
end
def self.reached_target(symbol, value)
pp "#{Time.now} #{symbol} has met or exceeded your target of #{value}."
end
def self.out_of_time(symbol, value)
pp "#{Time.now} The monitoring of #{symbol} with a target of #{value} has expired due to the time limit of 1 hour being rached."
end
def self.last_trade(symbol)
MarketBeat.last_trade_real_time symbol
end
def self.is_stock_at_or_above_value(symbol, value)
last_trade(symbol).to_f >= value
end
end
Here are the tests (that all pass):
require 'spec_helper'
describe "TradeWatcher" do
context "when comparing quotes to targets values" do
it "can report true if a quote is above a target value" do
TradeWatcher.stub!(:last_trade).and_return(901)
TradeWatcher.is_stock_at_or_above_value(:AAPL, 900).should == true
end
it "can report false if a quote is below a target value" do
TradeWatcher.stub!(:last_trade).and_return(901)
TradeWatcher.is_stock_at_or_above_value(:AAPL, 1000).should == false
end
end
it "checks stock value multiple times while stock is not at or above the target value" do
TradeWatcher.stub!(:last_trade).and_return(200)
TradeWatcher.should_receive(:is_stock_at_or_above_value).at_least(2).times
TradeWatcher.check_stock_every_x_seconds_for_value(:AAPL, 1, 400.01)
sleep(2)
end
it "triggers target_value_reahed when the stock has met or surpassed the target value" do
TradeWatcher.stub!(:last_trade).and_return(200)
TradeWatcher.should_receive(:reached_target).exactly(1).times
TradeWatcher.check_stock_every_x_seconds_for_value(:AAPL, 1, 100.01)
sleep(2)
end
it "returns a 'time limit reached' message once a stock has been monitored for the maximum of 1 hour" do
TradeWatcher.stub!(:last_trade).and_return(200)
TradeWatcher.stub!(:get_stop_time).and_return(Time.now - 3700)
TradeWatcher.check_stock_every_x_seconds_for_value(:AAPL, 1, 100.01)
TradeWatcher.should_receive(:out_of_time).exactly(1).times
sleep(2)
end
end
And here is a very simple script that (in my understanding) should print "{Time.now} AAPL has not yet met your target of 800.54." every 1 second that the method is still running and should at least be visible for 20 seconds (I test this using sleep in rspec and am able to see the strings printed to the console):
require 'trade_watcher'
TradeWatcher.check_stock_every_x_seconds_for_value(:AAPL, 1, 800.54)
sleep (20)
However I get no output - although the program does wait 20 seconds to finish. If I add other lines to print out to the console they work just fine, but nothing within the thread triggered by my TradeWatcher method call actually work.
In short, I'm not understanding how to have threads communicate with each other appropriately - or how to sync them up with each other (I don't think thread.join is appropriate here because it would leave the main thread hanging and unable to accept another method call if I chose to send one at a time in the future). My understanding of Ruby multithreading is weak anyone able to understand what I'm trying to get at here and nudge me in the right direction?
It looks like the pp function is simply not yet loaded by ruby when you go to print. By adding:
require 'pp'
to the top of trade_watcher.rb I was able to get the output you're expecting. You might also want to consider adding:
$stdout.sync = $stderr.sync = true
to your binary/executable script so that your output is not buffered internally by the IO class and instead passed directly to the os.
I have a ruby code that triggers php script over https.
Use case: The php script usually finishes in 5 minutes so I have set up time out for https request after 10 minutes. I need a timer that would trigger code after let's say 7 minutes after the https request started.
I was thinking of using thread that I created just before I initiate https request. I am not sure if this the correct way to approach this. Maybe there is not need to use threads at all. I am using ruby 1.8.7 (2010-08-16 patchlevel 302) [i386-mingw32]. Also I don't now if I can 'kill' the thread on successful finish of https request.
uri = URI.parse(url)
start = Time.new
http_read_timeout=60*10
connection = Net::HTTP.new(uri.host, 443)
connection.use_ssl = true
begin
response = connection.start() do |http|
http.open_timeout = 50
http.read_timeout = http_read_timeout
http.request_get(uri.request_uri)
# here I need to place a code that is triggered
# in case of custom timeout is reached
end
rescue Timeout::Error
# "Connection failed
time_out_message ="security time out - after #{http_read_timeout} sec"
return time_out_message
end
puts "finished"
The basic structure could be like this:
seconds_timer = MyDelay
counter = 0
test_thread = Thread.new do
run_http_php_test
end
while test_thread.alive?
counter += 1
if counter > seconds_timer
handle_custom_timeout_somehow
# if you want to halt run_http_php_test:
test_thread.kill if test_thread.alive?
# otherwise:
break
end
sleep 1
end
# the below doesn't apply if you kill the run_http_php_test thread
test_thread.join if test_thread.alive?
...but of course you could change that sleep 1 to whatever polling interval you like. Polling is nicer than just forcing your original thread to sleep, because the code will finish faster if run_http_php_test is done before you hit your custom timeout value.
Most or all of your code above can be in the run_http_php_test method, or inserted directly...whichever you'd prefer.
ruby 1.9.3 implements timeout module that has a timeout function. you can see it here. if you scroll down you can click show source and see the definition for timeout method. you can copy it if you dont want to upgrade to ruby 1.9.3 (I recommend upgrade since 1.8.7 is very slow compared to 1.9.3)
I am interested in knowing what would be the best way to implement a thread based queue.
For example:
I have 10 actions which I want to execute with only 4 threads. I would like to create a queue with all the 10 actions placed linearly and start the first 4 action with 4 threads, once one of the thread is done executing, the next one will start etc - So at a time, the number of thread is either 4 or less than 4.
There is a Queue class in thread in the standard library. Using that you can do something like this:
require 'thread'
queue = Queue.new
threads = []
# add work to the queue
queue << work_unit
4.times do
threads << Thread.new do
# loop until there are no more things to do
until queue.empty?
# pop with the non-blocking flag set, this raises
# an exception if the queue is empty, in which case
# work_unit will be set to nil
work_unit = queue.pop(true) rescue nil
if work_unit
# do work
end
end
# when there is no more work, the thread will stop
end
end
# wait until all threads have completed processing
threads.each { |t| t.join }
The reason I pop with the non-blocking flag is that between the until queue.empty? and the pop another thread may have pop'ed the queue, so unless the non-blocking flag is set we could get stuck at that line forever.
If you're using MRI, the default Ruby interpreter, bear in mind that threads will not be absolutely concurrent. If your work is CPU bound you may just as well run single threaded. If you have some operation that blocks on IO you may get some parallelism, but YMMV. Alternatively, you can use an interpreter that allows full concurrency, such as jRuby or Rubinius.
There area a few gems that implement this pattern for you; parallel, peach,and mine is called threach (or jruby_threach under jruby). It's a drop-in replacement for #each but allows you to specify how many threads to run with, using a SizedQueue underneath to keep things from spiraling out of control.
So...
(1..10).threach(4) {|i| do_my_work(i) }
Not pushing my own stuff; there are plenty of good implementations out there to make things easier.
If you're using JRuby, jruby_threach is a much better implementation -- Java just offers a much richer set of threading primatives and data structures to use.
Executable descriptive example:
require 'thread'
p tasks = [
{:file => 'task1'},
{:file => 'task2'},
{:file => 'task3'},
{:file => 'task4'},
{:file => 'task5'}
]
tasks_queue = Queue.new
tasks.each {|task| tasks_queue << task}
# run workers
workers_count = 3
workers = []
workers_count.times do |n|
workers << Thread.new(n+1) do |my_n|
while (task = tasks_queue.shift(true) rescue nil) do
delay = rand(0)
sleep delay
task[:result] = "done by worker ##{my_n} (in #{delay})"
p task
end
end
end
# wait for all threads
workers.each(&:join)
# output results
puts "all done"
p tasks
You could use a thread pool. It's a fairly common pattern for this type of problem.
http://en.wikipedia.org/wiki/Thread_pool_pattern
Github seems to have a few implementations you could try out:
https://github.com/search?type=Everything&language=Ruby&q=thread+pool
Celluloid have a worker pool example that does this.
I use a gem called work_queue. Its really practic.
Example:
require 'work_queue'
wq = WorkQueue.new 4, 10
(1..10).each do |number|
wq.enqueue_b("Thread#{number}") do |thread_name|
puts "Hello from the #{thread_name}"
end
end
wq.join