ZeroMQ: very high latency with PUSH/PULL - zeromq

Question

Summary
I see very high latency at the receiver using zmq.PUSH/PULL sockets.
Details
I'm new to ZeroMQ, and am trying to send a few MBytes from the sender to the receiver (my final aim is to send a camera stream from the sender to the receiver). The sender and receiver are different machines on the same local network (one is a Mac and the other is running Ubuntu 18.04, if that matters). The sender is able to send its packets very fast (I'm guessing they get buffered in some zmq/tcp queue), but the receiver receives them very slowly with increasing latency per packet.
Here's the sender :
1 """
2 Sender
3 """
4 import zmq
5 from time import time
6 import sys
7
8 addr = "tcp://192.168.86.33:5555"
9
10 context = zmq.Context()
11 socket = context.socket(zmq.PUSH)
12 socket.bind(addr)
13
14 num = 0
15 while True:
16 frame = [0] * 1000000
17 ts = time()
18 num += 1
19 socket.send_pyobj(dict(num=num, frame=frame, ts=ts))
20
21 delay = time() - ts
22 print("Sender:: pkt_num: {}, delay: {}".format(num, delay))
And the receiver :
1 """
2 Receiver
3 """
4 import zmq
5 from time import time
6
7 addr = "tcp://192.168.86.33:5555"
8 context = zmq.Context()
9
10 socket = context.socket(zmq.PULL)
11 socket.connect(addr)
12
13 while True:
14 msg = socket.recv_pyobj()
15 frame = msg['frame']
16 num = msg['num']
17 ts = msg['ts']
18
19 delay = time() - ts
20
21 if True:
22 print("Receiver:: pkt_num: {} latency: {}".format(num, delay))
When I run this, I see the sender is able to send its packets very quickly :
Sender:: pkt_num: 1, delay: 0.026965618133544922
Sender:: pkt_num: 2, delay: 0.018309354782104492
Sender:: pkt_num: 3, delay: 0.01821303367614746
Sender:: pkt_num: 4, delay: 0.016669273376464844
Sender:: pkt_num: 5, delay: 0.01674652099609375
Sender:: pkt_num: 6, delay: 0.01668095588684082
Sender:: pkt_num: 7, delay: 0.015082836151123047
Sender:: pkt_num: 8, delay: 0.014363527297973633
Sender:: pkt_num: 9, delay: 0.014063835144042969
Sender:: pkt_num: 10, delay: 0.014398813247680664
But the receiver sees very high and growing packet latencies :
Receiver:: pkt_num: 1 latency: 0.1272585391998291
Receiver:: pkt_num: 2 latency: 0.2539491653442383
Receiver:: pkt_num: 3 latency: 0.40800905227661133
Receiver:: pkt_num: 4 latency: 0.5737316608428955
Receiver:: pkt_num: 5 latency: 0.7272651195526123
Receiver:: pkt_num: 6 latency: 0.9418754577636719
Receiver:: pkt_num: 7 latency: 1.0799565315246582
Receiver:: pkt_num: 8 latency: 1.228663682937622
Receiver:: pkt_num: 9 latency: 1.3731486797332764
Receiver:: pkt_num: 10 latency: 1.5067603588104248
I tried swapping the sender and receiver between the Mac and Linux machines, and saw the same behavior. Since my goal is to send a video stream from the sender to the receiver, these high latencies make this unusable for that purpose.
Edit 1
Based on user3666197's suggestion, I edited the sender/receiver test code to remove some overheads. On the sender side keep sending the same dict. I also added more prints.
Sender:
14 num = 0
15 frame = [0] * 1000000
16 payload = dict(num=0, frame=frame, ts=0.0)
17 while True:
18 payload['num'] += 1
19 payload['ts'] = time()
20 socket.send_pyobj(payload)
21
22 delay = time() - payload['ts']
23 print("Sender:: pkt_num: {:>6d}, delay: {:6f}" \
24 .format(payload['num'], delay))
Receiver
10 socket = context.socket(zmq.PULL)
11 socket.connect(addr)
12 clk = zmq.Stopwatch()
13 clk.start()
14
15 while True:
16 iterT = clk.stop()
17 clk.start()
18 msg = socket.recv_pyobj()
19 rcvT = clk.stop()
20 delay = time() - msg['ts']
21
22 print("Server:: pkt_num: {:>6d} latency: {:>6f} iterT: {} rcvT: {}" \
23 .format(msg['num'], delay, iterT, rcvT))
24 clk.start()
The sender's per packet delay has reduced further. An interesting datapoint revealed is that the receiver takes almost 0.15s to receive each packet, which seems to be the main problem.
Sender:: pkt_num: 1, delay: 1.797830
Sender:: pkt_num: 2, delay: 0.025297
Sender:: pkt_num: 3, delay: 0.019500
Sender:: pkt_num: 4, delay: 0.019500
Sender:: pkt_num: 5, delay: 0.018166
Sender:: pkt_num: 6, delay: 0.017320
Sender:: pkt_num: 7, delay: 0.017258
Sender:: pkt_num: 8, delay: 0.017277
Sender:: pkt_num: 9, delay: 0.017426
Sender:: pkt_num: 10, delay: 0.017340
In the receiver's prints, rcvT is the per-packet receive time in microseconds.
Server:: pkt_num: 1 latency: 2.395570 iterT: 1 rcvT: 331601
Server:: pkt_num: 2 latency: 0.735229 iterT: 1 rcvT: 137547
Server:: pkt_num: 3 latency: 0.844345 iterT: 1 rcvT: 134385
Server:: pkt_num: 4 latency: 0.991852 iterT: 1 rcvT: 166980
Server:: pkt_num: 5 latency: 1.089429 iterT: 2 rcvT: 117047
Server:: pkt_num: 6 latency: 1.190770 iterT: 2 rcvT: 119466
Server:: pkt_num: 7 latency: 1.348077 iterT: 2 rcvT: 174566
Server:: pkt_num: 8 latency: 1.460732 iterT: 1 rcvT: 129858
Server:: pkt_num: 9 latency: 1.585445 iterT: 2 rcvT: 141948
Server:: pkt_num: 10 latency: 1.717757 iterT: 1 rcvT: 149666
Edit 2
I implemented the solution pointed out in this answer that uses PUB/SUB, and it works perfectly; I get 30fps video on the receiver. I still do not understand why my PUSH/PULL sample sees a delay.

Answer 1

Q : "ZeroMQ: very high latency with PUSH/PULL"
The ZeroMQ PUSH/PULL archetype is a part of the story. Let's decompose the claimed latency :
1 )
Let's measure the actual payload assembly costs [us] :
aClk = zmq.Stopwatch()
num = 0
while True:
aClk.start() #--------------------------------------- .start() the microsecond timer
frame = [0] * 1000000
ts = time()
num += 1
aPayLOAD = dict( num = num,
frame = frame,
ts = ts
)
_1 = aClk.stop() #---------------------------------- .stop() the microsecond timer
aPayLOAD['ts'] = time()
aClk.start() #-------------------------------------- .start() the microsecond timer
socket.send_pyobj( aPayLOAD )
_2 = aClk.stop() #---------------------------------- .stop() the microsecond timer
delay = time() - aPayLOAD['ts']
print( "Sender:: pkt_num: {0:>6d}, assy: {1:>6d} [us] .send_pyobj(): {2:>6d} [us] 'delay': {3:>10.6f} [s] ".format( num, _1, _2, delay ) )
2 )
Doing the same for the receiving side, you have a clear picture of how many [us] got consumed on the Python side, before the ZeroMQ has ever touched the first byte of the payload.
3 )
Next comes the performance tuning :
Avoid Python garbage collection
Improve the wasted memory-management ( avoid creating new instances, better inject data into memory-efficient "handlers" ( numpy.ndarray.data-zones instead of assigning re-composing list-s into a python variable )
refactor the code to avoid expensive dict-based key-value mapping and rather use compact binary maps, as available in struct-module ( your proposed mapping is both static & trivial )
test adding a compression-step ( may improve a bit the over-the-network hop latency )
4 )
Only now the ZeroMQ part comes :
may test improved performance envelopes from adding more IO-threads into either of Context( nIO_threads )-instances
may go in for .setsockopt( zmq.CONFLATE, 1 ) to ignore all non-"last" frames, as video-streaming may get no added value from "late" re-creation of already "old" frames