I'm looking to extend a the simple circuter breaker written ruby to work across multiple thread...
And thus far I manage to accomplish something like this ..
## following is a simple cicruit breaker implementation with thread support.
## https://github.com/soundcloud/simple_circuit_breaker/blob/master/lib/simple_circuit_breaker.rb
class CircuitBreaker
class Error < StandardError
end
def initialize(retry_timeout=10, threshold=30)
#mutex = Mutex.new
#retry_timeout = retry_timeout
#threshold = threshold
reset!
end
def handle
if tripped?
raise CircuitBreaker::Error.new('circuit opened')
else
execute
end
end
def execute
result = yield
reset!
result
rescue Exception => exception
fail!
raise exception
end
def tripped?
opened? && !timeout_exceeded?
end
def fail!
#mutex.synchronize do
#failures += 1
if #failures >= #threshold
#open_time = Time.now
#circuit = :opened
end
end
end
def opened?
#circuit == :opened
end
def timeout_exceeded?
#open_time + #retry_timeout < Time.now
end
def reset!
#mutex.synchronize do
#circuit = :closed
#failures = 0
end
end
end
http_circuit_breaker = CircuitBreaker.new
http_circuit_breaker.handle { make_http_request }
but I'm not sure about few things ...
The multithreaded code has always puzzled me hence I'm not the entirely confident about the approach to say that the stuff seems correct.
Read operation are not under mutex:
While (I think, I have ensured that no data race condition every happens between two threads) mutex are applied for the write operation but the read operation is mutex free. Now, since there can be a scenario where a thread 1 has a held mutex while changing the #circuit or #failure variable but the other thread read the stale value.
So, I'm not able to think thorough does by achieving a full consistency(while applying the read lock) is worth a trade-off over here. Where consistency might be 100 % but the execution code as turn a bit slower because of the excessive lock.
it's unclear what you are asking, so i guess your post will be closed.
nevertheless, i think that the only thread-safe-way to implement a circuit-breaker would be to have the mutex around all data operartions which would result in a sequential flow, so it's basically useless.
otherwise you will have race-conditions like
thread-a starts (server does not respond immediately due to network issues)
thread-b starts (10 seconds later)
thread-b finishes all good
thread-a aborts due to a timeout -> opens circuit with stale data
a version that is mentioned in martin fowlers blog is a circuit-breaker in combination with a thread-pool: https://martinfowler.com/bliki/CircuitBreaker.html
Related
I have a ruby method
def get_status(creds)
client = create_client(creds)
status = client.account_status
client.close_session
status
end
Usually, I optimize this kind of code by tap or yield_self, but here I can't find a nice way to optimize it.
The only solution I have come up:
def get_status(creds)
create_client(creds).yeild_self do |client|
[client, client.account_status]
end.yield_self do |client, status|
client.close_session
status
end
end
But it doesn't better than the original solution, is it?
One could write the following.
class Client
def account_status
"Overdrawn!"
end
def close_session
puts "It's closed"
end
end
def create_client(creds)
Client.new
end
def get_status(creds)
begin
client = create_client(creds)
client.account_status
ensure
client.close_session if client
end
end
get_status("Anything")
It's closed
#=> "Overdrawn!"
Do I prefer this to #1 in the question? No.
Do I prefer this to #2 in the question? Yes!
Do I prefer this to #max's answer? No.
I understand a finalizer could be created using the class method ObjectSpace::define_finalizer.
class Client
def initialize
ObjectSpace.define_finalizer(self, proc { puts "It's finalized!" })
end
def account_status
"Overdrawn!"
end
end
def create_client(creds)
Client.new
end
def get_status(creds)
create_client(creds).account_status
end
get_status("Anything")
#=> "Overdrawn!"
exit
It's finalized!
One must be careful when creating finalizers, as explained Here. I understand a technique sometimes used is to have finalizer's proc reference class-level objects. See, for example, this article, #Amadan's comments below and #Matt's comments on the question. I am not advocating the use of a finalizer. I merely thought readers unfamiliar with finalizers (as I was before writing this) would find this useful.
Let's list the goal of the function:
Open connection
Read value (and return it)
Close connection
I would consider this a "temporary connection", and that leads me to think it could be refactored to a separate method.
Reasoning: The get_status method is concerned with getting the status from a connection - it doesn't have to handle the details of actually closing/opening the connection itself.
def open_temporary_connection(creds, &block)
client = create_client(creds)
result = block.call(client)
client.close_session
result
end
def get_status(creds)
open_temporary_connection(creds, &:account_status)
end
Also, I should mention, I think yield_self is a bit of a trap. Unless you're dead set on making all of your code into a single expression, it makes the code look awkward without offering a lot of benefit.
I like your first version because it is short, easy to read, and easy to understand. I would not change it.
Nevertheless, an alternative version using tap might look like this:
def get_status(creds)
client = create_client(creds)
client.account_status.tap { client.close_session }
end
I am encountering an interesting issue with Ruby TCPServer, where once a client connects, it continually uses more and more CPU processing power until it hits 100% and then the entire system starts to bog down and can't process incoming data.
The processing class that is having an issue is designed to be a TCP Client that receives data from an embedded system, processes it, then returns the processed data to be further used (either by other similar data processors, or output to a user).
In this particular case, there is an external piece of code that would like this processed data, but cannot access it from the main parent code (the thing that the original process class is returning it's data to). This external piece may or may not be connected at any point while it is running.
To solve this, I set up a Thread with a TCPServer, and the processing class continually adds to a queue, and the Thread pulls from the queue and sends it to the client.
It works great, except for the performance issues. I am curious if I have something funky going on in my code, or if it's just the nature of this methodology and it will never be performant enough to work.
Thanks in advance for any insight/suggestions with this problem!
Here is my code/setup, with some test helpers:
process_data.rb
require 'socket'
class ProcessData
def initialize
super
#queue = Queue.new
#client_active = false
Thread.new do
# Waiting for connection
#server = TCPServer.open('localhost', 5000)
loop do
Thread.start(#server.accept) do |client|
puts 'Client connected'
# Connection established
#client_active = true
begin
# Continually attempt to send data to client
loop do
unless #queue.empty?
# If data exists, send it to client
begin
until #queue.empty?
client.puts(#queue.pop)
end
rescue Errno::EPIPE => error
# Client disconnected
client.close
end
end
sleep(1)
end
rescue IOError => error
# Client disconnected
#client_active = false
end
end # Thread.start(#server.accept)
end # loop do
end # Thread.new do
end
def read(data)
# Data comes in from embedded system on this method
# Do some processing
processed_data = data.to_i + 5678
# Ready to send data to external client
if #client_active
#queue << processed_data
end
return processed_data
end
end
test_embedded_system.rb (source of the original data)
require 'socket'
#data = '1234'*100000 # Simulate lots of data coming ing
embedded_system = TCPServer.open('localhost', 5555)
client_connection = embedded_system.accept
loop do
client_connection.puts(#data)
sleep(0.1)
end
parent.rb (this is what will create/call the ProcessData class)
require_relative 'process_data'
processor = ProcessData.new
loop do
begin
s = TCPSocket.new('localhost', 5555)
while data = s.gets
processor.read(data)
end
rescue => e
sleep(1)
end
end
random_client.rb (wants data from ProcessData)
require 'socket'
loop do
begin
s = TCPSocket.new('localhost', 5000)
while processed_data = s.gets
puts processed_data
end
rescue => e
sleep(1)
end
end
To run the test in linux, open 3 terminal windows:
Window 1: ./test_embedded_system.rb
Window 2: ./parent.rb
\CPU usage is stable
Window 3: ./random_client.rb
\CPU usage continually grows
I ended up figuring out what the issue was, and unfortunately I lead folks astray with my example.
It turns out my example didn't quite have the issue I was having, and the main difference was the sleep(1) was not in my version of process_data.rb.
That sleep is actually incredibly important, because it is inside of a loop do, and without the sleep, the Thread won't yield the GVL, and will continually eat up CPU resources.
Essentially, it was unrelated to TCP stuff, and more related to Threads and loops.
If you stumble on this question later on, you can put a sleep(0) in your loops if you don't want it to wait, but you want it to yield the GVL.
Check out these answers as well for more info:
Ruby infinite loop causes 100% cpu load
sleep 0 has special meaning?
How to handle exceptions in concurrent-ruby thread pools (http://ruby-concurrency.github.io/concurrent-ruby/file.thread_pools.html)?
Example:
pool = Concurrent::FixedThreadPool.new(5)
pool.post do
raise 'something goes wrong'
end
# how to rescue this exception here
Update:
Here is simplified version of my code:
def process
pool = Concurrent::FixedThreadPool.new(5)
products.each do |product|
new_product = generate_new_product
pool.post do
store_in_db(new_product) # here exception is raised, e.g. connection to db failed
end
end
pool.shutdown
pool.wait_for_terminaton
end
So what I want to achive, is to stop processing (break loop) in case of any exception.
This exception is also rescued at higher level of application and there are executed some cleaning jobs (like setting state of model to failure and sending some notifications).
The following answer is from jdantonio from here https://github.com/ruby-concurrency/concurrent-ruby/issues/616
"
Most applications should not use thread pools directly. Thread pools are a low-level abstraction meant for internal use. All of the high-level abstractions in this library (Promise, Actor, etc.) all post jobs to the global thread pool and all provide exception handling. Simply pick the abstraction that best fits your use case and use it.
If you feel the need to configure your own thread pool rather than use the global thread pool, you can still use the high-level abstractions. They all support an :executor option which allows you to inject your custom thread pool. You can then use the exception handling provided by the high-level abstraction.
If you absolutely insist on posting jobs directly to a thread pool rather than using our high-level abstractions (which I strongly discourage) then just create a job wrapper. You can find examples of job wrappers in all our high-level abstractions, Rails ActiveJob, Sucker Punch, and other libraries which use our thread pools."
So how about an implementation with Promises ?
http://ruby-concurrency.github.io/concurrent-ruby/Concurrent/Promise.html
In your case it would look something like this:
promises = []
products.each do |product|
new_product = generate_new_prodcut
promises << Concurrent::Promise.execute do
store_in_db(new_product)
end
end
# .value will wait for the Thread to finish.
# The ! means, that all exceptions will be propagated to the main thread
# .zip will make one Promise which contains all other promises.
Concurrent::Promise.zip(*promises).value!
There may be a better way, but this does work. You will want to change the error handling within wait_for_pool_to_finish.
def process
pool = Concurrent::FixedThreadPool.new(10)
errors = Concurrent::Array.new
10_000.times do
pool.post do
begin
# do the work
rescue StandardError => e
errors << e
end
end
end
wait_for_pool_to_finish(pool, errors)
end
private
def wait_for_pool_to_finish(pool, errors)
pool.shutdown
until pool.shutdown?
if errors.any?
pool.kill
fail errors.first
end
sleep 1
end
pool.wait_for_termination
end
I've created an issue #634. Concurrent thread pool can support abortable worker without any problems.
require "concurrent"
Concurrent::RubyThreadPoolExecutor.class_eval do
# Inspired by "ns_kill_execution".
def ns_abort_execution aborted_worker
#pool.each do |worker|
next if worker == aborted_worker
worker.kill
end
#pool = [aborted_worker]
#ready.clear
stopped_event.set
nil
end
def abort_worker worker
synchronize do
ns_abort_execution worker
end
nil
end
def join
shutdown
# We should wait for stopped event.
# We couldn't use timeout.
stopped_event.wait nil
#pool.each do |aborted_worker|
# Rubinius could receive an error from aborted thread's "join" only.
# MRI Ruby doesn't care about "join".
# It will receive error anyway.
# We can "raise" error in aborted thread and than "join" it from this thread.
# We can "join" aborted thread from this thread and than "raise" error in aborted thread.
# The order of "raise" and "join" is not important. We will receive target error anyway.
aborted_worker.join
end
#pool.clear
nil
end
class AbortableWorker < self.const_get :Worker
def initialize pool
super
#thread.abort_on_exception = true
end
def run_task pool, task, args
begin
task.call *args
rescue StandardError => error
pool.abort_worker self
raise error
end
pool.worker_task_completed
nil
end
def join
#thread.join
nil
end
end
self.send :remove_const, :Worker
self.const_set :Worker, AbortableWorker
end
class MyError < StandardError; end
pool = Concurrent::FixedThreadPool.new 5
begin
pool.post do
sleep 1
puts "we shouldn't receive this message"
end
pool.post do
puts "raising my error"
raise MyError
end
pool.join
rescue MyError => error
puts "received my error, trace: \n#{error.backtrace.join("\n")}"
end
sleep 2
Output:
raising my error
received my error, trace:
...
This patch works fine for any version of MRI Ruby and Rubinius. JRuby is not working and I don't care. Please patch JRuby executor if you want to support it. It should be easy.
My app freezes when I use emit to update status on GUI.
I want to know the reason or how to avoid this freeze. Thanks for your review.
My test environment
Windows 7 x64
railsinstaller-3.0.0.exe(MD5 : 26889DE0029C01A45AD2AED873708057)
https://github.com/railsinstaller/railsinstaller-windows/releases/download/3.0.0-alpha.2/railsinstaller-3.0.0.exe
qtbindings (4.8.5.2 x86-mingw32)
qtbindings-qt (4.8.5 x86-mingw32)
The demo app is present below.
#!/usr/bin/env ruby
# encoding: UTF-8
#
require 'Qt'
class App < Qt::MainWindow
signals 'test()'
slots 'on_test()'
def initialize
super
#label = Qt::Label.new
self.centralWidget = #label
self.show
connect self, SIGNAL('test()'), SLOT('on_test()')
start_count
end
def start_count
Thread.new do
loop {
emit test()
}
end
end
def on_test()
#label.text = #label.text.to_i + 1
end
end
app = Qt::Application.new(ARGV)
App.new
app.exec
#hyde
Thank you for you answer.
Solution 2 of qtbindings seems to be no help.
connect self, SIGNAL('test()'), SLOT('on_test()')
=>
connect self, SIGNAL('test()'), SLOT('on_test()'), Qt::BlockingQueuedConnection
Solution 1 was tested and the app runs fluently.
The code of Solution 1:
#!/usr/bin/env ruby
# encoding: UTF-8
#
require 'Qt'
class App < Qt::MainWindow
slots 'on_test()'
def initialize
super
#label = Qt::Label.new
self.centralWidget = #label
self.show
#c = Qt::AtomicInt.new
start_count
start_timer
end
def start_count
Thread.new do
loop {
#c.fetchAndAddRelaxed(1)
}
end
end
def start_timer
t = Qt::Timer.new(self)
t.start(16)
connect t, SIGNAL('timeout()'), SLOT('on_test()')
end
def on_test()
#label.text = #c.fetchAndAddRelaxed(0) + 1
end
end
app = Qt::Application.new(ARGV)
App.new
app.exec
Speculation on reason: the Qt main thread event loop never gets to process application events, because it is too busy delivering queued signals. The signals are queued, because they are between threads, so slot gets called in the right thread, independent of the emitting thread (and they must be queued in this case, because slot manipulates GUI objects, which is only allowed from the main thread)
There are a few alternatives to solve this. I don't really know Ruby, or its Qt bindings, so here's just a rough outline:
Solution 1: let the thread loop go around as fast as it can:
Create an atomic variable. Either use some Ruby type, or QAtomicInt (I assume that's possible with the Ruby Qt bindings).
In your thread loop, just increment the atomic variable, instead of emitting a signal.
Add a QTimer for updating the #label.text at desired intervals. If the user is supposed to be able to read the number, I'd suggest something like 500 ms interval. Minimum sensible interval is something like 16 ms for ~60 fps update rate.
Connect the timer timeout to on_test and get the value of the atomic integer to update the text.
That way updating of the numeric value is independent of displaying it.
Solution 2: make the thread block until emitted signal is actually delivered:
Use connection type BlockingQueuedConnection (note: do not use with single thread). Add that connection type to the signal connection statement (however you do it with Ruby Qt). Then the emitting thread will block until the target slot is actually called, so the signals will only be emitted at the rate they can be processed, no faster.
Your application freezes cause you emit signals in the infinite loop but not cause of emit
In RSpec, I have function that creates a new thread, and inside that thread performs some action–in my case, calls TCPSocket#readline. Here's the function as it is right now:
def read
Thread.new do
while line = #socket.readline
#TODO: stuff
end
end
end
Due to thread scheduling, my test will fail if written as such:
it "reads from socket" do
subject.socket.should_receive(:readline)
subject.read
end
Currently the only way I know to hack around this is to use sleep 0.1. Is there a way to properly delay the test until that thread is running?
If your goal is to assert the system state is changed by the execution of your second thread, you should join on the second thread in your main test thread:
it "reads from socket" do
subject.socket.should_receive(:readline)
socket_thread = subject.read
socket_thread.join
end
This is a bit of a hack, but here's a before block you can use in case you'd like the thread to yield but be able to call join at the end of the thread.
before do
allow(Thread).to receive(:new).and_yield.and_return(Class.new { def join; end }.new)
end