Best practice to control the flow on a ZMQ Streamer device? - zeromq

Question

Imagine I have a streaming device like the one in the bellow template.
image from here
def worker(backend_port):
context = zmq.Context()
socket = context.socket(zmq.PULL)
socket.connect("tcp://127.0.0.1:%d" % backend_port)
while True:
task = socket.recv()
# do something
def streamer(frontend_port):# it's called streamer because streams files from hdfs
context = zmq.Context()
socket = context.socket(zmq.PUSH)
socket.connect("tcp://127.0.0.1:%d" % frontend_port)
while True:
# get the file
# prepare it
socket.send(msg, copy=False)
number_of_workers = 16
number_streamers = 10
frontend_port = 7559
backend_port = 7560
streamerdevice = ProcessDevice(zmq.STREAMER, zmq.PULL, zmq.PUSH)
streamerdevice.bind_in("tcp://127.0.0.1:%d" % frontend_port )
streamerdevice.bind_out("tcp://127.0.0.1:%d" % backend_port)
streamerdevice.setsockopt_in(zmq.IDENTITY, b'PULL')
streamerdevice.setsockopt_out(zmq.IDENTITY, b'PUSH')
streamerdevice.start()
for i in range(number_of_workers):
p = Process(target=yolo, args=(backend_port,))
p.start()
for i in range(number_streamers):
s = Process(target=streamer, args=(frontend_port,))
s.start()
To give more information messages are images so are on the bigger size. I am using zero-copy. Performance is the most important point in my case since I am running on a very big scale.
Is there a way to multi-thread that streamerdevice?
How can I control the flow? is there a way to know how many messages are waiting to get processed?
one of the main goals is to make the receiving time as fast as possible on the worker side. Any suggestions for that?
Does it help to start the zmq.Context(nIoTHREADs) for nIoTHREADs>1 in worker side?

Answer 1

How to run a multi-threaded ZMQ device?
The simplest way to run multi-threaded ZeroMQ services is to instantiate the respective zmq.Context( nIoTHREADs ) for nIoTHREADs > 1. There are more performance tweaking configuration options for scaling performance. Review the pyzmq wrapper, how does this ZeroMQ tooling apply to actual the pyzmq wrapper code and ask maintainers, whether their setup and/or configuration remained fixed or just unpublished with this regard. ZeroMQ is flexible enough to provide means for doing this.
...device (which is a one way queue) is not scalable as you increase the number of input connections
Well, a device is neither a one way queue, nor is not scalable. Right the very opposite.
Is it possible to run it in a multi-threaded way?
Sure it is.
what is the best way to scale it?
That depends a lot on exact messaging and signaling requirements that your system design ought meet. High volume, ultra-low latency, almost linear scaling, adaptive connection handing, app-domain user-defined protocols are just a few of such properties to take into account.
Is it because the queue gets so big and that is why it gets slow?
The post contained zero pieces of information about neither the sizes, nor about any speed observations and/or expectations. In other, documented, test setups, ZeroMQ is known to be very cute in buffer-management, if zero-copy and other performance motivated approaches are used, so no one can help on a more detailed level.
is there a way to know how big the queue is at any moment?
No, there is no such "built-in" feature, that would spend time and resources on counting this. If such add-on feature is needed in your system, you are free to add such add-on, without making other use-cases wear a single bit of such additional code for use-cases, where a message counting and queue-size reporting is meaningless.
any load balancing strategy for the device without a REQ-REP pattern?
Yes, there are many ready-made Scalable Formal Communication Archetype Patterns plus there are scenarios one may compose for going beyond the ready-made bilateral distributed-behaviour Archetype primitives, that permit to build a custom-specific, controlled as required, load-balancer of one's own wish and taste.