The gist of what I'm trying to accomplish is to have a fan-out type of processing that will return a result to the initial client.
Right now, it is set up as:
[REQ]-->[ROUTER|PUB]-->[SUB|PUSH]-->[PULL|???]
I have it set up as PUB-SUB as the idea is that each SUB node will process a different part of a given manifest. For certain manifests, all SUB nodes are hit. For other manifests, maybe only a subset of the SUB nodes are hit. Using the SUB allows me to implement it without creating a discrete decision point on which nodes to route to.
I've got it to the point where I'm more or less able to bring the results together, but I have no idea how I'm supposed to return a result to the initial caller on the REQ without the caller binding a new socket at the client and then connecting to the socket. Mistakenly, I figured that if I could get the address of the caller at the ROUTER, I could pass that info along and send a message back to the initial REQ.
It seems that it should be possible and what I'm missing is perhaps some device coupled to the ROUTER?
So is it possible to accomplish this and is there a better pattern for this without binding another socket at the caller?
The initial caller REQ expects its reply from ROUTER and cannot accept messages from anywhere else. Therefore, a simple approach would be a broker with three endpoints:
ROUTERfor communication with client
PUB for sending message to all workers
PULL for getting in the results
Routing within the broker would be:
ROUTER -> PUB
PULL -> aggregate_by_client_id() -> ROUTER
The from my point of view tricky part is hidden in aggregate_by_client_id(), which is necessary since you can send only answer to REQ. Do you know how many results from workers to expect?
Related
TL;DR I want to have the functionality where a channel has two extra fields that tell the producer whether it is allowed to send to the channel and if so tell the producer what value the consumer expects. Although I know how to do it with shared memory, I believe that this approach goes against Go's ideology of "Do not communicate by sharing memory; instead, share memory by communicating."
Context:
I wish to have a server S that runs (besides others) three goroutines:
Listener that just receives UDP packets and sends them to the demultplexer.
Demultiplexer that takes network packets and based on some data sends it into one of several channels
Processing task which listens to one specific channel and processes data received on that channel.
To check whether some devices on the network are still alive, the processing task will periodically send out nonces over the network and then wait for k seconds. In those k seconds, other participants of my protocol that received the nonce will send a reply containing (besides other information) the nonce. The demultiplexer will receive the packets from the listener, parse them and send them to the processing_channel. After the k seconds elapsed, the processing task processes the messages pushed onto the processing_channel by the demultiplexer.
I want the demultiplexer to not just blindly send any response (of the correct type) it received onto the the processing_channel, but to instead check whether the processing task is currently even expecting any messages and if so which nonce value it expects. I made this design decision in order to drop unwanted packets a soon as possible.
My approach:
In other languages, I would have a class with the following fields (in pseudocode):
class ActivatedChannel{
boolean flag_expecting_nonce;
int expected_nonce;
LinkedList chan;
}
The demultiplexer would then upon receiving a packet of the correct type simply acquire the lock for the ActivatedChannel processing_channel object, check whether the flag is set and the nonce matches, and if so add the message to the LinkedList chan!
Problem:
This approach makes use of locks and shared memory, which does not align with Golang's "Do not communicate by sharing memory; instead, share memory by communicating" mantra. Hence, I would like to know... :
... whether my approach is "bad" regarding Go in the sense that it relies on shared memory.
... how to achieve the outlined result in a more Go-like way.
Yes, the approach described by you doesn't align with Golang's Idiomatic way of implementation. And you have rightly pointed out that in the above approach you are communicating by sharing memory.
To achieve this in Go's Idiomatic way, one of the approaches could be that your Demultiplexer "remembers" all the processing_channels that are expecting nonce and the corresponding type of the nonce. Whenever a processing_channels is ready to receive a reply, it sends a signal to the Demultiplexe saying that it is expecting a reply.
Since Demultiplexer is at the center of all the communication it can maintain a mapping between a processing_channel & the corresponding nonce it expects. It can also maintain a "registry" of all the processing_channels which are expecting a reply.
In this approach, we are Sharing memory by communicating
For communicating that a processing_channel is expecting a reply, the following struct can be used:
type ChannelState struct {
ChannelId string // unique identifier for processing channel
IsExpectingNonce bool
ExpectedNonce int
}
In this approach, there is no lock used.
I'm experimenting with microservices, event sourcing and CQRS. However, I'm a little bit confused about how I go from issuing a command to performing a query to return the new state, specifically with regard to interactions with a web API gateway.
As an example, the simple application I am attempting to write (which probably doesn't actually need any of these; it is just something to aid my learning) creates a random-graph and then performs some long-running calculations on the graph. I've modelled this as two separate services: the GraphService and the ComputationService. The imagined process flow is as follows:
User requests new random graph.
API gateway constructs CreateGraph command and sends it to the
graph service.
GraphService command handler creates a graph and publishes a
GraphCreated event.
GraphService event handler subscribes to topic for graph events,
processes GraphCreated event and stores graph in persistent read
storage.
Client somehow gets the newly created graph.
ComputationService event handler subscribes to topic for graph
events, processes GraphCreated event and begins potentially
long-running computation, e.g. calculate diameter.
ComputationService publishes DiameterComputed event.
GraphService event handler subscribes to topic for computation
events, processed DiameterComputed event and updates the graph in
persistent read storage.
Client somehow gets updated - easier than getting the new graph, since already have an ID and can poll for changes / websockets / SSE, etc.
That seems relatively simple. However, my confusion lies in how to go about informing the API gateway, and thus the web client, of the new graph (as highlighted in bold above). In a typical CRUD process, the result of the POST request to create a new graph would be to return the URL of the new resource, for instance. However, with CQRS, commands should return nothing or an exception.
How do I pass information back to the client of the service (in this case the API gateway) about the ID of the new graph so that it can perform a query to get the representation of the new resource and send it to the user? Or at least get an ID so that the web client can ask the API gateway, etc?
As I see it at the moment, after sending a command, everyone is just left hanging. There needs to be some sort of subscription model that can be interrogated for the status of the graph creation. I considered having the API gateway generate a request ID which gets embedded with the CreateGraph command, but this then couples the service to the API.
I'm obviously missing something, but have no idea what. None of the examples I've looked at or discussions I've read address this issue and assume that the ID of whatever resource is already known. I couldn't find any discussions here addressing this issue, but if I've just missed them, please point me there rather than duplicating questions. Any pointers would be hugely welcomed.
How do I pass information back to the client of the service (in this case the API gateway) about the ID of the new graph so that it can perform a query to get the representation of the new resource and send it to the user? Or at least get an ID so that the web client can ask the API gateway, etc?
By listening for the echo.
The basic idea behind at least once delivery is that I'm going to send you a message, and keep sending it over and over until I receive a message that proves you've received at least one copy of my message.
Therefore, my protocol looks something like
Establish a mail box where I can collect messages
Encode into the message instructions for delivering to my mailbox
Send the message to you
Check my mailbox
if the answer is there, I'm done
otherwise, I send you another copy of the message
The mail box could be implemented any number of ways -- it could be a callback; it could be a promise, it could be a correlation identifier. You could have the signal dispatched by the command handler, when it gets acknowledgement of the write by the book of record, or by the "read model" when the new resource is available.
I am experimenting with ZeroMQ where I want to create a server that does :
REQ-PIPELINE-REPLY
I want to sequentially receives data query requests, push it through a inproc pipeline to parallelise the data query and the sink merges the data back. After the sink merges the data together, the sink sends the merged data as the reply back to the request.
Is this possible? How would it look? I am not sure if the push/pull will preserve client's address for the REP socket to send back to.
Assuming that each client has only a single request out at any one time.
Is this possible?
Yes, but with different socket types.
How would it look?
(in C)
What you may like to do is shift from a ZMQ_REP socket on the external server socket to a ZMQ_ROUTER socket. The Router/Dealer sockets have identities which can allow you to have multiple requests in your pipeline and still respond correctly to each.
The Asynchronous Client/Server Pattern:
http://zguide.zeromq.org/php:chapter3#The-Asynchronous-Client-Server-Pattern
The only hitch in this is that you will need to manage the multiple parts of the ZMQ message. The first part is the identity. Second is null. Third is the data. As long as you REPLY in the same order as the REQUEST the identity will guide your response's data to the correct client. I wrapped my requests in a struct:
struct msg {
zmq_msg * identity;
zmq_msg * nullMsg;
zmq_msg * data;
};
Make sure to use zmq_msg_more when receiving messages and set the more flag when sending correctly.
I am not sure if the push/pull will preserve client's address for the
REP socket to send back to.
You are correct. A push pull pattern would not allow for specifying of the return address between multiple clients.
I have a REST server on heroku. It will have N-dynos for the REST service and N-dynos for workers.
Essentially, I have some long running rest requests. When these come in I want to delegate them to one of the workers and give the client a redirect to poll the operation and eventually return the result of the operation.
I am going to use JEDIS/REDIS from RedisToGo for this. As far as I can tell there are two ways I can do this.
I can use the PUB/SUB functionality. Have the publisher create unique identities for the work results and return these in a redirect URI to the REST client.
Essentially the same thing but instead of PUB/SUB use RPUSH/BLPOP.
I'm not sure what the advantage is to #1. For example, if I have a task called LongMathOperation it seems like I can simply have a list for this. The list elements are JSON objects that have the math operation arguments as well as a UUID generated by the REST server for where the results should be placed. Then all the worker dynos will just have blocking BLPOP calls and the first one there will get the job, process it, and put the results in REDIS using the key of the UUID.
Make sense? So my question is "why would using PUB/SUB be better than this?" What does PUB/SUB bring to the table here that I am missing?
Thanks!
I would also use lists because pubsub messages are not persistent. If you have no subscribers then the messages are lost. In other words, if for whatever reason you do not have any workers listening then the client won't get served properly. Lists are persistent on the other hand. But pubsub does not take as much memory as lists obviously for the same reason: there is nothing to store.
I am using a standard LRU queue as defined by the ZeroMQ guide figure 41, and I am wondering how to add in protection so that I don't send messages to end points that have disappeared (server crash, OOM killer, anything along those lines).
From the documentation I read that XREP will just drop the message if it is going to a non-existant end-point, and there is no way I get notified about that. Is there a way to get such a notification? Should I just send out a "ping" first and if I don't get a response then that "worker" is dead meat to me? How will I know that it is the same client that I just sent the ping to that I am getting the message back from?
Or is my use case not a good one for ZeroMQ? I just want to make sure that a message has been received, I don't want it being dropped on the floor without my knowledge...
Pinging a worker to know if it is alive will cause a race condition: the worker might well answer the ping just before it dies.
However, if you assume that a worker will not die during a request processing (you can do little in this case), you can reverse the flow of communication between the workers and the central queue. Let the worker fetch a request from the queue (using a REQ/REP connection) and have it send the answer along with the original envelope when the processing is done (using the same socket as above, or even better through a separate PUSH/PULL connection).
With this scenario, you know that a dead worker will not be sent requests, as it will be unable to fetch them (being dead…). Moreover, your central queue can even ensure that it receives an answer to every request in a given time. If it does not, it can put the request back in the queue so that a new worker will fetch it shortly after. This way, even if a worker dies while processing a request, the request will eventually be served.
(as a side note: be careful if the worker crashes because of a particular request - you do not want to kill your workers one by one, and might want to put a maximum number of tries for a request)
Edit: I wrote some code implementing the other direction to explain what I mean.