How to send byte message with ZeroMQ PUB / SUB setting? - zeromq

So I'm new to ZeroMQ and I am trying to send byte message with ZeroMQ, using a PUB / SUB setting.
Choice of programming language is not important for this question since I am using ZeroMQ for communication between multiple languages.
Here is my server code in python:
import zmq
import time
port = "5556"
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:%s" % port)
while True:
socket.send(b'\x84\xa5Title\xa2hi\xa1y\xcb\x00\x00\x00\x00\x00\x00\x00\x00\xa1x\xcb#\x1c\x00\x00\x00\x00\x00\x00\xa4Data\x08')
time.sleep(1)
and here is my client code in python:
import zmq
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:5556")
total_value = 0
for update_nbr in range (5):
string = socket.recv()
print (string)
My client simply blocks at string = socket.recv().
I have done some study, so apparently, if I were to send string using PUB / SUB setting, I need to set some "topic filter" in order to make it work. But I am not sure how to do it if I were to send some byte message.

ZeroMQ defines protocols, that guarantee cross-platform compatibility of both the behaviours and the message-content .
The root-cause:to start receiving messagesone must change the initial "topic-filter" state for the SUB-socket( which initially is "receive nothing" Zero-subscription )
ZeroMQ is a lovely set of tools, created around smart principles.
One of these says, do nothing on SUB-side, until .setsockopt( zmq.SUBSCRIBE, ... ) explicitly says, what to subscribe to, to start checking the incoming messages ( older zmq-fans remember the initial design, where PUB-side always distributes all messages down the road, towards each connected SUB-"radio-broadcast receivers", where upon each message receival the SUB-side performs "topic-filtering" on it's own. Newer versions of zmq reverse the architecture and perform PUB-side filtering ).
Anyway, the inital-state of the "topic-filter" makes sense. Who knows what ought be received a-priori? Nobody. So receive nothing.
Given you need or wish to start the job, an easy move to subscribe to anything ... let's any message get through.
Yes, that simple .setsockopt( zmq.SUBSCRIBE, "" )
If one needs some key-based processing and the messages are of a reasonable size ( no giga-BLOBs ) one may just simply prefix some key ( or a byte-field if more hacky ) in front of the message-string ( or a payload byte-field ).
Sure, one may save some fraction of the transport-layer overhead in case the zmq-filtering is performed on the PUB-side one ( not valid for the older API versions ), otherwise there is typically not big deal to subscribe to receive "anything" and check the messages for some pre-assembled context-key ( a prefix substring, a byte-field etc ) before the rest of the message payload is being processed.
The Best Next Step:
If your code strives to go into production state, not to remain as just an academia example, there will have to be much more work to be done, to provide surviveability measures for the hostile real-world production environments.
An absolutely great perspective for doing this and a good read for realistic designs with ZeroMQ is Pieter HINTJEN's book "Code Connected, Vol.1" ( may check my posts on ZeroMQ to find the book's direct pdf-link ).
Plus another good read comes from Martin SUSTRIK, the co-father of ZeroMQ, on low-level truths about the ZeroMQ implementation details & scale-ability

Related

Inter-process communication between async and sync tasks using PyZMQ

On a single process I have a tasks running on a thread that produces values and broadcasts them and
several consumer async tasks that run concurrently in an asyncio loop.
I found this issue on PyZMQ's github asking async <-> sync communication
with inproc sockets which is what I also wanted and the answer was to use .shadow(ctx.underlying) when
creating the async ZMQ Context.
I prepared this example and seems to be working fine:
import signal
import asyncio
import zmq
import threading
import zmq.asyncio
import sys
import time
import json
def producer(ctrl):
# delay first push to give asyncio loop time
# to start
time.sleep(1)
ctx = ctrl["ctx"]
s = ctx.socket(zmq.PUB)
s.bind(ctrl["endpoint"])
v = 0
while ctrl["run"]:
payload = {"value": v, "timestamp": time.time()}
msg = json.dumps(payload).encode("utf-8")
s.send(msg)
v += 1
time.sleep(5)
print("Bye")
def main():
endpoint = "inproc://testendpoint"
ctx = zmq.Context()
actx = zmq.asyncio.Context.shadow(ctx.underlying)
ctrl = {"run": True, "ctx": ctx, "endpoint": endpoint, }
th = threading.Thread(target=producer, args=(ctrl,))
th.start()
try:
asyncio.run(amain(actx, endpoint))
except KeyboardInterrupt:
pass
print("Stopping thread")
ctrl["run"] = False
th.join()
async def amain(ctx, endpoint):
s = ctx.socket(zmq.SUB)
s.subscribe("")
s.connect(endpoint)
loop = asyncio.get_running_loop()
def stop():
try:
print("Closing zmq async socket")
s.close()
except:
pass
raise KeyboardInterrupt
loop.add_signal_handler(signal.SIGINT, stop)
while True:
event = await s.poll(1000)
if event & zmq.POLLIN:
msg = await s.recv()
payload = json.loads(msg.decode("utf-8"))
print("%f: %d" % (payload["timestamp"], payload["value"]))
if __name__ == "__main__":
sys.exit(main())
Is it safe to use inproc://* between a thread and asyncio task in this way? The 0MQ
context is thread safe and I'm not sharing sockets between the thread and the
asyncio task, so I would say in general that this is thread safe, right? Or am I
missing something that I should consider?
Q :Is it safe to use inproc://* between a thread and asyncio task in this way?""
A :First and foremost, I might be awfully wrong (not only here), yet having worked with ZeroMQ since native API 2.1.1+ I dare claim that unless newer "improvements" got lost the core principles ( ZeroMQ ZMTP/RFC-documented properties for building legal implementation of the still valid ZMTP-arsenal ), the answer here shall be YES, as much as the newer releases of pyzmq-binding kept all mandatory properties of the inproc:-Transport-Class without a compromise.
Q :" The 0MQ context is thread safe and I'm not sharing sockets between the thread and the asyncio task, so I would say in general that this is thread safe, right? "
A :Here my troubles start - ZeroMQ implementations were since ever developed based on Martin SUSTRIK's & Pieter HINTJENS' Zen-of-Zero -- i.e. also as Zero-sharing -- so never sharing was the principle ( though "share"-zmq.Context-instances were no problem to be used from different threads, to the contrary of the zmq.Socket-instances )
Python (since ever & still valid in 2022-Q1) used to use & still uses a total [CONCURRENT]-code-execution avoider -- prevented by GIL-lock, which principally avoids any & all kinds of problems, arising from [CONCURRENT]-code-execution to never happen insider Python GIL-lock re-[SERIAL]-ised flow of code-execution, so even if the asyncio-part is built as a pythonic (non-destructive) part of the ecosystem, your code shall never "meet" any kind of concurrency-related issue, as the unless it gains GIL-lock, it does nothing but "hanging in NOP-s cracking" ( nuts-cracking in idle loop ).
Being inside the same process, there seems no advantage to spawn another Context-instance at all ( this used to be the rock-solid certainty since ever, not to ever increase any kind of overheads - Zen-of-Zero ( almost )Zero-overhead ... ). The Sig/Msg core engine was, if performance or latency needs required, powered with more zmq.Context( IOthreads ) upon instantiations, yet these were zmq.Context-owned, not Python-GIL-governed/(b)locked threads, so the performance was pretty well scalable, without wasting any RAM/HWM/buffers/...-resources, without growing any overheads and very efficient, as the IO-threads were co-located for only indeed I/O-work, so not needed for inproc:-( protocol-less )-Transport-Class at all )
Q :" Or am I missing something that I should consider? "
A :Mixing asyncio, O/S-signals ( that are well documented how they interact with native ZeroMQ API ) and other layers of complexity is for sure possible, yet it comes at a cost - it makes the use-case less and less readable and more and more prone to conceptual-gaps and similar hard to decode "errors".
I remember using Tkinter-mainloop() as a cost-wise very cheap and a super-stable framework for rapid-prototyping an MVC-{ M-odel, V-isual, C-ontroller }-parts of many-actors' indeed distributed-system applications in Python. There were Zerop-problems to use ZeroMQ with a single Context-instance, passing the references of the respective AccessNodes' into whatever amount of event-handlers, supposing we kept the ZeroMQ Zen-of-Zero, i.e. no to "share" (meaning no two parts "use" (compete to use) one and the same AccessPoint "one-over-another")
This all was designed-in, at "Zero-cost", by the ZeroMQ by-definition, so unless spoilt in some later phase, re-wrapping a re-wrapped native API, all this ought still work in 2022-Q1, ought it not?

How may client can connect to the server socket in MQL?

I want to connect more than 500 hundred client to the MQL (Meta Trader) server socket.
There is no description about it in the documentation: https://www.mql5.com/en/docs/network/socketcreate
How many client can connect to the sever and deal with no problem?
Q :" I want to connect more than 500 hundred client to the MQL (Meta Trader) server ... How many client can connect to the sever and deal with no problem? "
A :Not an easy task, indeed.
As you may already know, all the MetaTrader 4/5 ecosystems are built as a distributed-system, having a Terminal-side ( on your, clients' side(s) ) and a Server-side ( a multi-host platform, located at the Broker DataCenter, who registers users, authenticates & feeds, besides many further noted things, a latency-sensitive, high-volume ( markets Volume-wise times number of active clients-wise ) stream of { CFD | FX | DeFi | * }-Market QUOTE messages (having easily cadence of hundreds ToB-events / messages per millisecond at FX-market) to all auth'ed active { MT4 | MT5 }-Terminal computers & accepts and executes XTO-instructions from auth'ed clients & reports results ( state-changes preformed & client's-funds accounting operations ) from XTO-s back to the respective trader's terminals ). That amount of work is, on the Broker side, split among several MetaTrader 4/5 Server server-infrastructure computers. The web-socket handling gets served by one part of such Broker-side infrastructure.
Closer to your reach goes the MetaTrader 4/5 Terminal, that you can program & control. Even here the amounts of resources are limited, as you can read from your linked, Terminal-side, not Server-side documentation of programming tools available :
You can create a maximum of 128 sockets from one MQL5 program. If the limit is exceeded, the error 5271 (ERR_NETSOCKET_TOO_MANY_OPENED) is written to _LastError.
So, the Server-side is controlled by the Broker ( who owns the license to use the MetaQuotes, Inc. product, that gets configured for expected performance envelopes - being ready or not to handle additional 50.000 web-socket connections for NTO-s might not be the Brokers' core business priority, as they collect fees from XTO-s )
"(...) The question is, do we create new socket for each client to connect? As I know, we create the server socket just one time on the Oninit function, then on a timer or chart event handler, do accepting incoming client connection request. So, there is just one socket and many client connect to this socket. Am I right #user3666197 ? – Behzad 23 hours ago"
-&-
"I think my question is not clear. I have done this project. I bought a VPS then install a MT5 on it with the EA that has played the server role. The sever EA could accept 500 client without any problem. It can send and receive messages as well as one connection. For clients, on my pc create a loop to connect 500 connection to the server. One socket on the server EA. – Behzad 4 hours ago"
Given you call MT5-Client-Terminal a "server" in a sense ( just a VPS-hosted MT5-Client-Terminal, running a user-defined MQL5-ExpertAdviser-code ), there seems to be some magic :
(A)you claim to be able to "(...) accept 500 client without any problem.", which is in a direct contradiction to the official MQL5-documented limit of not more than 128 sockets ever opened from an MQL5-{ EA | Script }-code
(B) the official MQL5-documentation does not present a way, how an MT5-Client-Terminal running an MQL5-{ EA | Script }-code can receive connections arriving asynchronously from remote clients ( yet without specifying how that might ever happen, as the official MQL5-Documentation is strict on practically avoiding such to happen if using the MQL5-language functions as of 2022-Q1 )
(C) the official MQL5-documentation confirms, one can SocketConnect() from inside an MT5-Client-Terminal MQL5-{ EA | Script }-code to a known TCP/IP:PORT address :
string KNOWN_ADDRESS = "some.known.FQDN";
int KWOWN_PORT = 80,
TimeoutMILLIS = 1000;
bool FLAG_ExtTLS = false;
//+------------------------------------------------------------------+
...
int MyOUTGOINGsocket = SocketCreate(); //--- check the handle
if ( MyOUTGOINGsocket != INVALID_HANDLE )
{
if ( SocketConnect( MyOUTGOINGsocket, //--- from MT5-Terminal
KNOWN_ADDRESS, // to <_address_>
KNOWN_PORT, // on <_port_>
TimeoutMILLIS // try <_millis_>
) // else FAIL
)
{
Print( "INF: Established connection to ",
KNOWN_ADDRESS, ":",
KNOWN_PORT
);
...
}
else
{
Print( "ERR: Connection to ",
KNOWN_ADDRESS, ":",
KNOWN_PORT,
" failed, error ",
GetLastError()
);
...
}
SocketClose( MyOUTGOINGsocket ); //--- close a socket to release RAM/resources
}
else
{ Print( "ERR: Failed to even create a socket, error was ",
GetLastError()
);
...
}
...
...
//+------------------------------------------------------------------+
One may use, for sure some other, DLL-#import-ed tools for the similar tasks, yet as no MCVE-formulated problem description was presented so far, it is so hard to tell anything more, except for the facts already described above
You can using webrequest method with API from mql client
There is an article explaining how to create a server on MT5:
Working with sockets in MQL, or How to become a signal provider
https://www.mql5.com/en/articles/2599

Why do almost all ZeroMQ code samples contain sleep() operations?

I'm learning ZeroMQ and trying to build a simple message queue in Python.
I noticed basically all code samples contain some kind of sleep() operation.
Even the hello world example on the ZeroMQ guide does, with the comment "Do some work".
I find this a little unclear, is the motivation to simulate the act of processing the message? Why is this necessary?
import time
import zmq
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
while True:
# Wait for next request from client
message = socket.recv()
print("Received request: %s" % message)
# Do some 'work'
time.sleep(1)
# Send reply back to client
socket.send(b"World")
is the motivation to simulate the act of processing the message ?
Sort of yes. Launching while True: "without" any handbrake would be soon pretty ugly on screen with a literally endlessly running river of print()-s, wouldn't it?
Why is this necessary ?
Just a cheap SLOC / convenience-trick. Except for cases, where some latency needs to get injected, there is no technical reason for sleep()-s

Topics in ZeroMQ

How better to separate topics using ZeroMQ - just by specifying different ports or by using prefixes like here: ZeroMQ and multiple subscribe filters in Python
It looks simpler to specify ports.
I do not mean to connect to multiple topics. I mean that different parts of application will connect to different ports.
No, using different port numbers will not work the way you feel.
aSubTypeSOCKET.setsockopt( zmq.SUBSCRIBE, aLeftMatchTopicFILTERasSTRING )
aSubTypeSOCKET.setsockopt( zmq.SUBSCRIBE, anotherTopicFILTERasSTRING )
...
aSubTypeSOCKET.setsockopt( zmq.UNSUBSCRIBE, aLeftMatchTopicFILTERasSTRING )
...
aSubTypeSOCKET.setsockopt( zmq.SUBSCRIBE, yetAnotherTopicFILTERasSTRING )
is the standard mechanism how to setup Topic-filter matching condition(s) to effectively work with.
Multiple connect()-s are possible, but
aSubTypeSOCKET.connect( "tcp://123.123.123.123:12345" )
aSubTypeSOCKET.connect( "tcp://123.123.123.123:23456" )
aSubTypeSOCKET.connect( "tcp://123.123.123.123:34567" )
will make aSubTypeSOCKET indeed .connect()-ed to several port-numbers, but the sender side yet decides, if a subscription "mechanics" will deliver a content "specialisation" that you might have expected to use instead of the common Topic-filter(s) subscribed to using .setsockopt().
Next, if multiple .connect()-ed PUB-s deliver messages, the SUB-side is getting these in through an incoming "routing" strategy, by a "Fair-queued" manner, so the SUB-side will have to turn and check the round-robin wheel a complete turn around, just to see, if any present/not-present and any non-empty subscription Topic-filter setup will apply to all PUB-sides symmetrically, similarly the first empty-string Topic-filter will "unblock" / "short-cut" all senders to deliver any messages from all PUB-access-nodes.
I would not call that any simpler.
APPs typically build a complex signalling/messaging plane
an indeed as persistent as possible infrastructure, where many socket-Access-Points do .bind() / .connect() to many individual access-point addresses, as desired for, using some mix of many transport-classes available { inproc:// | ipc:// | tcp:// | pgm:// | epgm:// | vmci:// }.
This is one of many strengths the designs, based on ZeroMQ, can and do enjoy.

Publisher finishes before subscriber and messages are lost - why?

Fairly new to zeromq and trying to get a basic pub/sub to work. When I run the following (sub starting before pub) the publisher finishes but the subscriber hangs having not received all the messages - why ?
I think the socket is being closed but the messages have been sent ? Is there a way of ensuring all messages are received ?
Publisher:
import zmq
import random
import time
import tnetstring
context=zmq.Context()
socket=context.socket(zmq.PUB)
socket.bind("tcp://*:5556")
y=0
for x in xrange(5000):
st = random.randrange(1,10)
data = []
data.append(random.randrange(1,100000))
data.append(int(time.time()))
data.append(random.uniform(1.0,10.0))
s = tnetstring.dumps(data)
print 'Sending ...%d %s' % (st,s)
socket.send("%d %s" % (st,s))
print "Messages sent: %d" % x
y+=1
print '*** SERVER FINISHED. # MESSAGES SENT = ' + str(y)
Subscriber :-
import sys
import zmq
import tnetstring
# Socket to talk to server
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:5556")
filter = "" # get all messages
socket.setsockopt(zmq.SUBSCRIBE, filter)
x=0
while True:
topic,data = socket.recv().split()
print "Topic: %s, Data = %s. Total # Messages = %d" % (topic,data,x)
x+=1
In ZeroMQ, clients and servers always try to reconnect; they won't go down if the other side disconnects (because in many cases you'd want them to resume talking if the other side comes up again). So in your test code, the client will just wait until the server starts sending messages again, unless you stop recv()ing messages at some point.
In your specific instance, you may want to investigate using the socket.close() and context.term(). It will block until all the messages have been sent. You also have the problem of a slow joiner. You can add a sleep after the bind, but before you start publishing. This works in a test case, but you will want to really understand what is the solution vs a band-aid.
You need to think of the PUB/SUB pattern like a radio. The sender and receiver are both asynchronous. The Publisher will continue to send even if no one is listening. The subscriber will only receive data if it is listening. If the network goes down in the middle, the data will be lost.
You need to understand this in order to design your messages. For example, if you design your messages to be "idempotent", it doesn't matter if you lose data. An example of this would be a status type message. It doesn't matter if you have any of the previous statuses. The latest one is correct and message loss doesn't matter. The benefits to this approach is that you end up with a more robust and performant system. The downsides are when you can't design your messages this way.
Your example includes a type of message that requires no loss. Another type of message would be transactional. For example, if you just sent the deltas of what changed in your system, you would not be able to lose the messages. Database replication is often managed this way which is why db replication is often so fragile. To try to provide guarantees, you need to do a couple things. One thing is to add a persistent cache. Each message sent needs to be logged in the persistent cache. Each message needs to be assigned a unique id (preferably a sequence) so that the clients can determine if they are missing a message. A second socket (ROUTER/REQ) needs to be added for the client to request the missing messages individually. Alternatively, you could just use the secondary socket to request resending over the PUB/SUB. The clients would then all receive the messages again (which works for the multicast version). The clients would ignore the messages they had already seen. NOTE: this follows the MAJORDOMO pattern found in the ZeroMQ guide.
An alternative approach is to create your own broker using the ROUTER/DEALER sockets. When the ROUTER socket saw each DEALER connect, it would store its ID. When the ROUTER needed to send data, it would iterate over all client IDs and publish the message. Each message should contain a sequence so that the client can know what missing messages to request. NOTE: this is a sort of reimplementation of Kafka from linkedin.

Resources