I am trying to implement master-worker program.
My master has jobs that the workers are going to do. Every time a worker completes a job, he asks for a new job from the master, and the master sends it to him. The workers are calculating minimal paths. When a worker finds a minimum that is better than the global minimum he got, he sends it to everyone including the master.
I plan for the workers and masters to send data using MPI_ISEND. Also, I think that the receive should be blocking. The master has nothing to do when no one has asked for work or has updated the best result, so he should block waiting for a receive. Also, each worker should, after he has done his work, wait on a receive to get a new one.
Nevertheless, I'm not sure of the impact of using non-blocking asynchronous send, and blocking synchronous receive.
An alternative I think is using MPI_IPROBE, but I'm not sure that this will give me any optimization.
Please help me understand whether what I'm doing is right. Is this the right solution?
You can match blocking sends with nonblocking receives and vice versa, that won't cause any problems. However, if the master really has nothing to do while the workers work, and the workers should block after completing their work unit, then there's no reason for non-blocking communication on that front. The master can post a blocking receive with MPI_ANY_SOURCE, and the workers can just use a blocking send to post back their results, since the matching receive at the master will already have been posted.
So, I'd have Send-Recv for exchanging work units between master and worker, and Isend-Irecv for broadcasting the new global minima.
Related
I've been trying to implement a call centre type system using Taskrouter using this guide as a base:
https://www.twilio.com/docs/tutorials/walkthrough/dynamic-call-center/ruby/rails
Project location is Australia, if that affects call details.
This system dials multiple numbers (workers), and I have run into an issue where phones will continue to ring even after the call has been accepted or cancelled.
ie. If Taskrouter calls Workers A and B, and A picks up first they are connected to the customer, but B will continue to ring. If B then picks up the phone they are greeted by a hangup tone. Ringing can continue for at least minutes until B picks up (I haven't checked if it ever times out).
Similar occurs if no one picks up and the call simply times out and is redirected to voicemail. As you can imagine, an endlessly ringing phone is pretty annoying, especially when there's no one on the other end.
I was able to replicate this issue using the above guide without modification (other than the minimum changes to set it up locally). Note that it doesn't dial workers simultaneously, rather it dials the first in line for a few seconds before moving to the next.
My interpretation of what is occurring is that Taskrouter is dialling workers, but not updating them when dialling should end, and simply moving on to the next stage of the workflow. It does update Worker status, so it knows if they've timed out for instance, but that doesn't update the actual call.
I have looked for any solutions to this and havent found much about it except the following:
How to make Twilio stop dialing numbers when hangup() is fired?
https://www.twilio.com/docs/api/rest/change-call-state
These don't specifically apply to Taskrouter, but suggest that a call that needs to be ended can be updated and completed.
I am not too sure if I can implement this however, as it seems to be using the same CallSid for all calls being dialled within a Workflow, makes it hard/impossible to seperate each call, and would end the active call as well.
It also just seems wrong that Taskrouter wouldn't be doing this automatically, so I wanted to ask about this before I tinker too much and break things.
Has anyone run into this issue before, or is able/unable to replicate it using the tutorial code?
When testing I've noticed the problem much more on landline numbers, which may only be because mobiles have their own timeout/redirects. VOIPs seem to immediately answer calls, so they behave a bit differently.
Any help/suggestions appreciated, thanks!
Current suggestion to work around this is to not issue the dequeue instruction immediately, but rather issue a Call instruction on the REST API when the Worker wishes to accept the Inbound Call.
This will create an Outbound Call to bridge the two calls together and thus won’t have many outbound calls for the same inbound caller at once.
Your implementation will depend on the behavior that you want to achieve:
Do you want to simul-dial both Workers?
Do you want to send
the task to both Workers and whoever clicks to Accept the Task first
will have the call routed to them?
If it's #2, this is a scenario where you're saying that the Worker should accept the Reservation (reservation.accepted) before issuing the Call.
If it's #1, you can either issue a Call Instruction or Dequeue Instruction. The key being that you provide a DequeueStatusCallbackUrl or CallStatusCallbackUrl to receive call progress events. Once one of the outbound calls is connected, you will need to complete the other associated call. So you will have to unfortunately track which outbound calls are tied to which Reservation, by using AssignmentCallbacks or EventCallbacks, to make that determination within your app.
I'm implementing a standard MPI master/slave system: there is a master that distributes work, and there are slaves who ask for chunks and process data.
However... if implemented in a naive way (rank==0 is master, the rest are slaves), the master ends up doing no real work, but still takes one core for what needs practically no real computing power. So I tried to implement a separate "scheduler" thread in the master, but that involved sending MPI messages to itself, and didn't really work...
Do you have any ideas how to solve this?
As I realized after some googling: you can send messages to yourself using tags. Tags are a kind of filter: if you do a recv for only tag==1, then you'll receive only those, with later messages being able to overtake eariler ones.
So, as for the solution:
tag the "scheduler to worker" and "worker to scheduler" messages with a different id
if rank==0: start a scheduler thread
afterwards, regardless of the rank, request work.
This way, the rank 0 worker won't receive its own "let's give me work" messages, because they will have a "to be received by the scheduler only" tag.
Edit: this thing doesn't really seem to be thread-safe though... (= it sometimes crashes in "free()" even though it's written in Python...) so I'd be still interested in the real & proven solution :)
First of all, I'm new to ZeroMQ and message queue systems, so what I'm trying to do may be solved through a different approach. I'm designing a messaging system that does the following:
Multiple clients connect to a broker and send the id of an item that needs to be processed. The client disconnects immediately and does not wait for a response.
The broker sends items to workers, one item per worker, to perform some processing. Each return returns a signal that the processing was completed.
I have a rudimentary system setup which is processing requests/replies correctly, but I'd also like to be able to do the following:
Query the broker to see how many processes are actually running on the workers and how many are simply waiting to be run.
Have the broker ensure that only one process per id is running - if a duplicate id arrives and that item is not currently being processed by a worker, do not add it to the queue.
I'm using a poll setup with broker/dealer sockets. The code I'm using is very similar to this example from Ian Barber.
My first inclination (although I'm not sure how to implement it in zmq) is to have the broker keep track of the ids that have been received, and those that are actively being processed by workers. It seems that the broker forwards requests to workers immediately, regardless of whether or not they are available to actually run the processing. The workers then queue up the ids and process them in order. This isn't ideal since I'm looking to be able to monitor and control what is going on in the system centrally to achieve reliability.
Anyways, any hints, tips or examples of this type of setup would be greatly appreciated.
ZeroMQ is, in my opinion, best used in broker-less designs, for which the library is designed. If you want to monitor the number of items in a queue, or throughput, or whatever, you're going to have to build that into the application/device/producer yourself. Since you're new to messaging, that could get out of hand real quick. Given this, I'd suggest looking into RabbitMQ (or a similar broker), which would provide these services for you out of the box. If you do adopt RabbitMQ (or rather, AMQP), I'd suggest using a fanout exchange for the scenario you describe above.
The Python library for ZeroMQ seems to come with a pattern for dealing with this: http://zeromq.github.com/pyzmq/devices.html#monitoredqueue
I'm designing a network sever based on boost::asio. I need to perform long running processing jobs in handlers and think that these processing should be moved from handlers to separate thread pool where I would have better control (e.g. prioritize tasks). Handlers would just enqueue a new task in job queue.
There would be also a response queue where responses would be dequeued and send back to the clients. (client send requests synchronously)
I wonder if this make sense or just miss something.
Short answer is Yes. Long answer it depends. Generally speaking, if you want a higher network though put you should minimize processing that is performed in the handlers and offload it to a thread. This is especially important if you have causality requirements for the data that you receive, since async_receive doesn't guarantee execution order of handlers.
I am using a standard LRU queue as defined by the ZeroMQ guide figure 41, and I am wondering how to add in protection so that I don't send messages to end points that have disappeared (server crash, OOM killer, anything along those lines).
From the documentation I read that XREP will just drop the message if it is going to a non-existant end-point, and there is no way I get notified about that. Is there a way to get such a notification? Should I just send out a "ping" first and if I don't get a response then that "worker" is dead meat to me? How will I know that it is the same client that I just sent the ping to that I am getting the message back from?
Or is my use case not a good one for ZeroMQ? I just want to make sure that a message has been received, I don't want it being dropped on the floor without my knowledge...
Pinging a worker to know if it is alive will cause a race condition: the worker might well answer the ping just before it dies.
However, if you assume that a worker will not die during a request processing (you can do little in this case), you can reverse the flow of communication between the workers and the central queue. Let the worker fetch a request from the queue (using a REQ/REP connection) and have it send the answer along with the original envelope when the processing is done (using the same socket as above, or even better through a separate PUSH/PULL connection).
With this scenario, you know that a dead worker will not be sent requests, as it will be unable to fetch them (being dead…). Moreover, your central queue can even ensure that it receives an answer to every request in a given time. If it does not, it can put the request back in the queue so that a new worker will fetch it shortly after. This way, even if a worker dies while processing a request, the request will eventually be served.
(as a side note: be careful if the worker crashes because of a particular request - you do not want to kill your workers one by one, and might want to put a maximum number of tries for a request)
Edit: I wrote some code implementing the other direction to explain what I mean.