Here is my code:
EventMachine.run {
conn = EM::Protocols::HttpClient2.connect request.host, 80
req = conn.get(request.query)
req.callback { |response|
p(response.status)
p(response.headers)
p(response.content)
}
}
The callbacks fire, that is, I get the string outputs of the status, etc.
But what I want it to do is fire the callbacks, then repeat. There is more logic I plan to implement, such as tweaking the URL each time, but for now, I just want it to:
Retrieve the URL
Fire the callbacks
Repeat...
My understanding about this pattern was that everything in that loop fires, then returns, then goes on forever until I do an EM.stop.
Right now, it retrieves the URL data, and just seems to hang.
Do I need to do a return of some sort to continue here? Why is it hanging, and not looping over and over?
If I surround the entire above code block with a loop do ... end it works as expected.. is that the correct way to implement this? I suppose I am confused as I thought everything within EM.run repeats when it completes.
The run block you give runs only once. The event loop is not exposed directly to you but is something that's intended to be invisible. Don't confuse the run block with a while loop. It's run once and once only, but it is run while the event loop is executing.
If you want to repeat an operation you need to create some kind of a stack and work through that, with each callback checking the stack if there's more work to do and then issuing another call. EventMachine applications are built using this callback-chaining method.
You will need to implement something like:
def do_stuff(queue, request = nil)
request ||= queue.pop
return unless (request)
conn = EM::Protocols::HttpClient2.connect request.host, 80
req = conn.get(request.query)
req.callback { |response|
p(response.status)
p(response.headers)
p(response.content)
EventMachine.next_tick do
# This schedules an operation to be performed the next time through
# the event-loop. Usually this is almost immediate.
do_stuff(queue)
end
}
end
Inside your event loop you kick of this chain:
EventMachine.run do
queue = [ ... ] # List of things to do
do_stuff(queue)
end
You can probably find a more elegant way to implement this once you get a better sense of how EventMachine works.
Related
I can't figure out how to handle an event loop such that I can run other code concurrently. I want to make it so when the handler receives data, it prints it without effecting anything else the program is doing.
I have tried wrapping trading_stream.run in an asyncio task, but this produces an error and isn't what I really want. It's like once I run the stream, my program is stuck inside the update_handler function.
from alpaca.trading.stream import TradingStream
trading_stream = TradingStream('api-key', 'secret-key', paper=True)
async def update_handler(data):
# trade updates will arrive in our async handler
print(data)
# subscribe to trade updates and supply the handler as a parameter
trading_stream.subscribe_trade_updates(update_handler)
# start our websocket streaming
trading_stream.run()
Premise: it would probably be best to understand what event loop is TradingStream using and, if possible, schedule tasks on that loop once retrieved, e.g.
trading_stream = TradingStream('api-key', 'secret-key', paper=True)
evt_loop = trading_stream.some_evt_loop_getter()
evt_loop.create_task(my_concurrent_task)
if TradingStream is using asyncio.get_event_loop() under the hood, then the following is also possible.
import asycio
trading_stream = TradingStream('api-key', 'secret-key', paper=True)
evt_loop = asyncio.get_event_loop()
evt_loop.create_task(my_concurrent_task)
Not being able to assess whether either of the above is the case, the following hack does solve your problem, but I would not resort to this unless the alternatives are not viable.
OTHER_LOGIC_FLAG = True
async def my_other_async_logic():
# Concurrent logic here
async def update_handler(data):
global OTHER_LOGIC_FLAG
if OTHER_LOGIC_FLAG:
asyncio.create_task(my_other_async_logic()
OTHER_LOGIC_FLAG = False
# trade updates will arrive in our async handler
print(data)
Again, do try to get a handle to the event loop first.
I'm trying to implement and Authorization module, because currently the authorization logic for a certain resource is separated on two or three different places and even though I'm not sure this is the best approach, at least I think it will provide some encapsulation.
Since I'm checking for several different things
Does the user have the right role
Is the resources in the right state to process the required action
Do the user has the right to perform the required action on this particular resource
So as you can see, there are several checks, I'm not pretending to be completely correct here, but this is pretty close to the real case, so I've decided to use something like a Result Object even though it's actually not an object but a struct and I'm not using Gem but pretty simple custom implementation.
So part of my Authorization module is this:
module Authorization
Result = Struct.new(:successfull?, :error)
extend self
def read(user, resource, message: 'Permission denied')
can_read =
[
condition1,
condition2,
condition3
]
return Result.new(can_read.any?, can_read.any? ? nil : message))
end
However within this Authorization module I have a lot of methods and some of them check read internally like so:
def assign(user, resource, message: 'Permission denied')
return read(user, resource) unless read(user, resource).successfull?
Result.new(true, nil)
end
So my main question is how to avoid this double call to read(user, resource). I guess one option would be to just call it before the check like:
result = read(user, resource)
return result unless result.successfull?
However I'm pretty new to Ruby and I suspect that maybe there is more ruby-like way to do this. Just to inline it somehow by assigning the result from read inside the condition check...However this is just wild guess.
And one more question, that came up while I was writing this. Currently if I want to send nil for message when the authorization passes I'm doing this:
return Result.new(can_read.any?, can_read.any? ? nil : message))
Because message unless can_read.any? is throwing and error even though I thought it would default to nil. So again, is there some more ruby-like way to do this?
First part can be written with Object#yield_self:
def assign(user, resource, message: 'Permission denied')
read(user, resource).yield_self do |res|
res.successful? ? Result.new(true, nil) : res
end
end
successfull? -> successful? for English reasons. I am not convinced this is more readable than using a local variable though. Alternatively:
(res = read(user, resource)).successful? ? Result.new(true, nil) : res
As for your second question, you'll need more parentheses
Result.new(can_read.any?, (message if can_read.none?))
the return is not needed.
I would also advise you to slow down with all the unlesses, try to swap your conditions to if whenever possible -- I find it quite useful to make Result a class and define a failed? method for it. Actually, I'd consider this:
class Result
def initialize(error)
#error = error
end
def successful?
#error.nil?
end
def failed?
!successful?
end
end
That depends on how complicated your Result gets, but for the use case shown, it would be a little cleaner imho.
Is there a way for return a value to a function, then invoking a thread in that function? For example:
def foo
return fast_function
Thread.new do
slow_function
end
end
The reason behind this is that both fast_function and slow_function write to the same resource. But I want to ensure that fast_function runs and complete first, and return its value to foo before slow_function writes to the shared resource. There are some cases where slow_function completes before fast_function and I am hit with a race condition.
EDIT:
More context on the problem. This is related to server-side events I am trying to implement. I am trying to get fast_function to compute an event id and return and html. While slow_function is responsible for notifying the client via event id that the process is done. However, in some cases, slow_function notifies the client before the client event know where to listen, because fast_function did not return the event id yet.
No, a return will exit the function, it would also exit the function in a yield block. In my opinions there are multiple solutions to this problem.
Actually it would be a perfect fit for a Promise of Concurrent Ruby (https://github.com/ruby-concurrency/concurrent-ruby)
you could use it somewhat like this:
def foo
fast = Concurrent::Promise.execute{ fast_function }
slow = promises[:fast].then{ slow_function }
.on_fullfill{ notify_client }
return fast.value
end
As you can guess it will return the value of your fast function.
But it will also call the on_fullfill function (Or a proc) if the slow function has finished. And the most important, it will guarante order.
NOTE: I am not sure if I understood you correctly, if you want to start booth threads at the same time, but ensure that the fast one has finished first. you can do something like this:
fast = Concurrent::Promise.execute{ fast_function }
slow = Concurrent::Promise.execute{ slow_function }
render fast.value # Or what you ever do with the html.
#.value will wait for the Promise to finish.
result slow = slow.value
This way you would start booth functions parallel, but be sure you would get the answer first for the fast one.
Edit 1: I thougth about this, and I am not really sure if you want to have an asynchronous task at all. It is hard to tell since you posted a minimal example (what is correct of coruse).
If you just want to have a function which returns botth function returns in the right order, you could just do a yield:
def foo
yield fast_function
yield slow_function
end
My process has two threads like the following
#semaphore = Mutex.new
.
.
.
.
thread_a = Thread.new {
loop do
#some work
if(some_condition)
#semaphore.synchronize {
#thread_b_running = false
}
end
end
}
thread_b = Thread.new {
while(#semaphore.synchronize { #thread_b_running }) do
#thread_b's work
end
}
thread_a.join
thread_b.join
Basically, thread_a and thread_b do some work in parallel, however when thread_a sees an event happen it needs to shut down thread_b. As you can see right now I am doing it by using a boolean protected by a mutex. I think this approach is not too bad performance wise since thread_b will almost always get the lock without waiting for it. However since I have not written a lot of multithreaded code I was wondering if there is a better way of doing what I'm doing?
If only one of the threads is writing the variable, there is no need for a mutex. So a better way in your example is just removing the mutex.
I have a synchronized queue that provides a condition variable.
That condition variable signals when data is added to the queue.
I have 5 threads:
Thread.new do
loop do
#queue.synchronize {
cond.wait_until { #queue.has_data? || #queue.finished? }
}
# some processing code that can also call #queue.enqueue
end
end
Then I do:
#queue.enqueue some_data
#threads.each(&:join)
MyQueue#enqueue looks like this:
def enqueue(data)
synchronize do
#pending << v unless queued?(data) || processed?(data) || processing?(data)
data_cond.signal
end
end
def finished?
#started && #processing.empty? && #pending.empty?
end
def has_data?
!#pending.empty?
end
And I get on #join
deadlock detected
How exactly does this cause a deadlock and how would one fix it?
I wonder if this is a problem that all of the threads are blocked on the same condition variable, and there isnt a thread available to enqueue data, which would release the other threads.
Based on the comment in this code:
Thread.new do
loop do
#queue.synchronize {
cond.wait_until { #queue.has_data? || #queue.finished? }
}
# some processing code that can also call #queue.enqueue
end
end
Your comment that mentions "some processing code that can also call #queue.enqueue", is this the only place where #queue.enqueue is called? If so, then all of the threads will be blocked on the condition variable and none will be able to get to the point to be able to call enqueue. Im sure Ruby can detect that all threads are locked on the same entity and none are available to release it, thus deadlock.
If you do indeed have a separate thread that only enqueues (which would be a typical producer/consumer situation) make sure that it doesnt also wait on the condition variable, which could also cause deadlock.
It's a little hard to help you because you are only posting code fragments...
You should try the work_queue gem, or at least take a look at the source code.
There is no need to wait for has_data? || finished? in synchronize block. The code should look like:
Thread.new do
loop do
cond.wait_until { #queue.has_data? || #queue.finished? }
enq = nil
#queue.synchronize {
enq = #queue.pop
}
# some processing code that can also call #queue.enqueue
end
end
In that case you lock other threads only when operating with queue content. What you need to do is to synchronize on queue state change, like finished
A better solution is to wrap all thread critical variables with mutex, like here in rails. It'll make code a little bit slower since it eliminate simultaneous variable access.