How to set up a ZeroMQ request-reply between a c# and python application - zeromq

I'm trying to communicate between a c#(5.0) and a python (3.9) application via ZeroMQ. For .Net I'm using NetMQ and for python PyZMQ.
I have no trouble letting two applications communicate, as long as they are in the same language
c# app to c# app;
python -> python;
java -> java,
but trouble starts when I try to connect between different languages.
java -> c# and reverse works fine as well [edited]
I do not get any errors, but it does not work either.
I first tried the PUB-SUB Archetype pattern, but as that didn't work, I tried REQ-REP, so some remainders of the "PUB-SUB"-version can still be found in the code.
My Python code looks like this :
def run(monitor: bool):
loop_counter: int = 0
context = zmq.Context()
# socket = context.socket(zmq.PUB)
# socket.bind("tcp://*:5557")
socket = context.socket(zmq.REP)
socket.connect("tcp://localhost:5557")
if monitor:
print("Connecting")
# 0 = Longest version, 1 = shorter version, 2 = shortest version
length_version: int = 0
print("Ready and waiting for incoming requests ...")
while True:
message = socket.recv()
if monitor:
print("Received message:", message)
if message == "long":
length_version = 0
elif message == "middle":
length_version = 1
else:
length_version = 2
sys_info = get_system_info(length_version)
"""if not length_version == 2:
length_version = 2
loop_counter += 1
if loop_counter == 15:
length_version = 1
if loop_counter > 30:
loop_counter = 0
length_version = 0"""
if monitor:
print(sys_info)
json_string = json.dumps(sys_info)
print(json_string)
socket.send_string(json_string)
My C# code :
static void Main(string[] args)
{
//using (var requestSocket = new RequestSocket(">tcp://localhost:5557"))
using (var requestSocket = new RequestSocket("tcp://localhost:5557"))
{
while (true) {
Console.WriteLine($"Running the server ...");
string msg = "short";
requestSocket.SendFrame(msg);
var message = requestSocket.ReceiveFrameString();
Console.WriteLine($"requestSocket : Received '{message}'");
//Console.ReadLine();
Thread.Sleep(1_000);
}
}
}

Seeing the period of your problems maybe it's because of versions.
I run fine a program for long time with communications from Windows/C# with NTMQ 4.0.0.207 239,829 7/1/2019 on one side and Ubuntu/Python with zeromq=4.3.1 and pyzmq=18.1.0.
I just tried updating to use same NETMQ version but with new versions zeromq=4.3.3 and pyzmq=20.0.0 but there is a problem/bug somewhere and it doesn't run well anymore.
So your code doesn't look bad may be it's software versions issues not doing well try with NTMQ 4.0.0.207 on c# side and zeromq=4.3.1 with pyzmq=18.1.0 on python side

Q : "How to set up a ZeroMQ request-reply between a c# and python application"
The problem starts with the missed understanding of how REQ/REP archetype works.
Your code uses a blocking-form of the .recv()-method, so you remain yourselves hanging Out-of-the-Game, forever & unsalvageable, whenever a REQ/REP two-step gets into troubles (as no due care was taken to prevent this infinite live-lock).
Rather start using .poll()-method to start testing a presence / absence of a message in the local AccessNode-side of the queue and this leaves you in a capability to state-fully decide what to do next, if a message is already or is not yet present, so as to keep the mandatory sequence of an API-defined need to "zip" successful chainings ofREQ-side .send()-.recv()-.send()-.recv()-... with REP-side .recv()-.send()-.recv()-.send()-... calls, are the REQ/REP archetype works as a distributed-Finite-State-Automaton (dFSA), that may easily deadlock itself, due to "remote"-side not being compliant with the local-side expectations.
Having a code, that works in a non-blocking, .poll()-based mode avoids falling into these traps, as you may handle each of these unwanted circumstances while being still in a control of the code-execution paths (which a call to a blocking-mode method in a blind belief it will return at some future point in time, if ever, simply is not capable of).
Q.E.D.
If in doubts, one may use a PUSH/PULL archetype, as the PUB/SUB-archetype may run into problems with non-matching subscriptions ( topic-list management being another, version dependent detail ).
There ought be no other problem for any of the language-bindings, if they passed all the documented ZeroMQ API features without creating any "shortcuts" - some cases were seen, where language-specific binding took "another" direction for PUB/SUB, when sending a pure message, transformed into a multi-part message, putting a topic into a first frame and the message into the other. That is an example of a binding not compatible with the ZeroMQ API, where a cross-language / non-matching binding-version system problems are clear to come.

Your port numbers do not match, the python code is 55557 and the c# is 5557

I might be late, but this same thing happened to me. I have a python Subscriber using pyzmq and a C# Publisher using NetMQ.
After a few hours, it occurred to me that I needed to let the Publisher some time to connect. So a simple System.Threading.Thread.Sleep(500); after the Connect/Bind did the trick.

Related

MassTransit timeouts under load on .NETFramework under IIS

Under load in production we receive "RabbitMQ.Client.Exceptions.ConnectFailureException" connection failed and "MassTransit.RequestTimeoutException" timeout waiting for response. The consumer does receive the message and send it back. It's like the web app isn't listening, or unable to accept the connection.
We're running an ASP.NET web application ( not MVC ) on .NET Framework 4.6.2 on Windows Server 2019 on IIS. We're using MassTransit 7.0.4. In production, under load, we can get some exceptions dealing with sockets on RabbitMQ or timeouts from masstransit. It's difficult to reproduce them in Dev. RabbitMQ is in a mirror, it seems to happen once we turn on a high-load service that bumps from 140 message/sec to 250 message/sec.
I have a few questions about the code architecture, and then if anyone else is running into these kinds of timeout issues.
Questions:
Should I have static scope for the IBusControl? IE, should it be static inside Global asax? And does it matter at all if it's a singleton underneath?
Should I create a new IBusControl and start it per request ( maybe stick it in Application BeginRequest ). Would that make a difference?
Would adding another worker process affect the total number of open connections I'm able to make -- If this is a resource issue ( exhausting threads, connections or some resource ).
Exceptions:
MassTransit.RequestTimeoutException
Timeout Waiting for response
Stacktrace:
System.Runtime.ExceptionServices.ExceptionDispathInfo.Throw
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
MassTransit.Clients.ResponseHandlerConnectionHandle`1+<GetTask>d_11.MoveNext
System.Threading.ExecutionContext.RunInternal
RabbitMQ.Client.Exceptions.ConnectFailureException
Connection failed
Statcktrace:
RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail
RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingAddressFamily
RabbitMQ.Client.Impl.SocketFrameHandler..ctor
RabbitMQ.Client.ConnectionFactory.CreateFrameHandler
RabbitMQ.Client.EndPointResolverExtensions.SelectOne
RabbitMQ.Client.ConnectionFactory.CreateConnection
How Our Code Works ( overview )
Static IBusControl that is instantiated the first time someone tries to produce a message. The whole connection and send code is a little large to put in here ( connection factory and other metric classes, but below are the interesting parts ).
Static IBusControl B;
B = Bus.Factory.CreateUsingRabbitMq(x =>
{
hostAddress = host.HostAddress;
x.Host(new Uri(host.HostAddress), h =>
{
h.Username(host.UserName);
h.Password(host.Password);
});
x.Durable = false;
x.SetQueueArgument("x-message-ttl", 600000);
});
B.Start(new TimeSpan(0, 0, 10));
// Then send the Actual Messages
// Generic with TRequest and TResponse : class BaseMessage
// Pulling the code out of a few different classes
string serviceAddressString = string.Format("{0}/{1}?durable={2}", HostAddress, ChkMassTransit.QueueName(typeof(TRequest), typeof(TResponse)), false ? "true" : "false");
Uri serviceAddress = new Uri(serviceAddressString);
RequestTimeout rt = RequestTimeout.After(0, 0, 0, 0, timeout.Value);
IRequestClient<TRequest> reqClient = B.CreateRequestClient<TRequest>(serviceAddress, rt);
var v = reqClient.GetResponse<TResponse>(request, sendInfo.CT, sendInfo.RT);
if ( v.Wait(timeoutMS) ) { /*do some stuff*/ }
First, I find your lack of async disturbing. Using Wait or anything like it on TPL-based code is a recipe for death and destruction, pain and suffering, dogs and cats living together, etc.
Yes, you should have a single bus instance that is started when the application starts. Since you're doing request/response, set AutoStart = true on the bus configurator to make sure it's all warmed up and ready.
Never, no, one bus only!
Each bus instance only has a single connection, so you shouldn't see any resource issues related to capacity on RabbitMQ.
MassTransit 7.0.4 is really old, you might consider the easy upgrade 7.3.1 and see if that improves things for you. It's the last version of the v7 codebase available.

Publishing protocol buffer messages over websockets in Julia

I'm working on a project where I want to have a Julia server process periodically publishing messages that involve some binary data. The initial client will be written in javascript, but we'd ultimately like the capability of having multiple clients, implemented in various ways. For that reason, I'd like to define the message using protocol buffers. I've turned up 3 Julia websocket implementations: WebSockets.jl, SimpleSockets.jl and HTTP.jl's WebSockets. I've done some naive experimenting with WebSockets.jl and got an error ("WebSockets does not support byte I/O"). My inclination is to shift my focus to the HTTP.jl implementation; I get the impression that it is under more active development than WebSockets.jl.
Update: I've continued with my experimentation. I was guided to a previous SO question, unable to write binary data in websocket, that was instructive. I modeled my server implementation after that in the link, yielding:
include("testmessage_pb.jl")
text = "A man spekith"
msg = TestMessage(someText=text)
function server(port)
#async HTTP.listen(Sockets.localhost, port) do http::HTTP.Stream
if HTTP.WebSockets.is_upgrade(http.message)
HTTP.WebSockets.upgrade(http, binary=true) do ws
while !eof(ws)
data = readavailable(ws)
IOExtras.startwrite(ws)
writeproto(ws, msg)
IOExtras.closewrite(ws)
end
end
end
end
end
Running it, I got an error very much like the previous one I reported:
HTTP.WebSockets.WebSocket{HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}} does not support byte I/O
I'm wondering if anyone reading this has any experience with this particular cluster of technologies (julia, protobuf, websockets) and suggestions on how to proceed ("don't try" would count as useful feedback).
HTTP.jl websockets are working fine. Havn't tried it with protobuf, but hardly there should be any problem with that. It's basically
HTTP.WebSockets.open(data_url) do ws
x = readavailable(ws)
# Do protobuf related things, generate response
write(ws, response)
end
readavailable returns you UInt8[] so you can do whatever you want with it.
I was able to tweak the server implementation that I showed in the question, and it now works. It's a little clunky, and I'll hold off from accepting my own answer in hopes that a nicer approach comes along. In any event, here's the new version:
function server(port)
#async HTTP.listen(Sockets.localhost, port) do http::HTTP.Stream
if HTTP.WebSockets.is_upgrade(http.message)
HTTP.WebSockets.upgrade(http, binary=true) do ws
while !eof(ws)
data = readavailable(ws)
iob = PipeBuffer()
writeproto(iob, msg)
write(ws, take!(iob))
end
end
end
end
end

Network throttling with chrome and selenium

Google Chrome 38 introduced the new "Device Mode & Mobile Emulation" functionality in devtools. In addition to choosing a device for emulation, it is also possible to emulate different network conditions:
Optimizing your site's performance under varying network conditions is
a key aspect of developing for a mobile audience.
Device mode's network conditioning allows you to test your site on a
variety of network connections, including Edge, 3G, and even offline.
Select a connection from the preset dropdown to apply network
throttling and latency manipulation.
For example, we can set it to be like in good old days - GPRS 50 Kbps:
Now we have a good use case for it - we have an internal application for network speed testing. And this new emulation functionality is very helpful for manual testing. But, we'd like to automate it.
Question is:
Is it possible to start chrome via selenium with specified network conditions? Is it something that can be controlled through chrome preferences or command-line arguments?
There are certainly multiple options to simulate slow internet connection, but the question is specifically about chrome+selenium.
The API to control network emulation were added to ChromeDriver. And should be available for quite a while now. According to comment in the linked issue you should use version at least 2.26 because of some bugfix.
According to Selenium changelog bindings are available for these languages:
JavaScript as of version 3.4.0 (commit)
Python as of version 3.5.0 (commit)
Ruby as of version 3.11.0 (commit)
C# as of version 4 (commit)
If you need these binding in other languages you should probably open issue/contribute implementation similar to one of the above.
Example usage from Python is below:
driver.set_network_conditions(
offline=False,
latency=5, # additional latency (ms)
download_throughput=500 * 1024, # maximal throughput
upload_throughput=500 * 1024) # maximal throughput
No, it is not possible to control Network Connectivity Emulation through Chrome preferences or command-line arguments. Network Connectivity Emulation is part of the build-in Chrome debugger. One way way in solving this is to control the debugger. This can be done via an extension or by directly controlling the debugger, see explanation. However, this will not work with WebDriver. The reason for this is that there can only be one "debug" session and WebDriver is already using it, see explanation. Since there is no public interface, there is also no way to control it via WebDriver.
For Device Mode & Mobile Emulation which is also part of the build-in debugger, there is a public interface (details), and thus can be controlled. This can be done through WebDriver Capabilities. Two options 1) Specify a device name 2) Enter your own parameters (limited).
You can use this method to run your test case in specified network conditions
protected void networkThrotting() throws IOException {
Map map = new HashMap();
map.put("offline", false);
map.put("latency", 5);
map.put("download_throughput", 500);
map.put("upload_throughput", 1024);
CommandExecutor executor = ((ChromeDriver)driver).getCommandExecutor();
Response response = executor.execute(
new Command(((ChromeDriver)driver).getSessionId(), "setNetworkConditions", ImmutableMap.of("network_conditions", ImmutableMap.copyOf(map)))
);
}
Indeed C# Selenium latest (3.11) has NetworkConditions added. Now you can use it like this:
var driver = new ChromeDriver(pathToDriver);
driver.NetworkConditions = new ChromeNetworkConditions()
{ DownloadThroughput = 5000, UploadThroughput = 5000, Latency = TimeSpan.FromMilliseconds(5) };
The problem is it's not yet usable because of the bug
https://github.com/SeleniumHQ/selenium/issues/5693
So .Net guys will have to wait until 3.12 Selenium Release.
While this is a very welcome and useful bit of functionality, for serious testing I think the conventional methods of network simulation are still the way to go.
I am aware of 2 solutions in addition to those already linked - the Charles web proxy (very useful tool - commercial) and implementing your own recipe using Linux Traffic Control (e.g. see chapter 6 of LAMPe2e).
By interfering with the network connections rather than the browser, you then get a proper measure of the impact independently of the browser in use.
Why do you just want to use the Chrome functionality?
Let's consider two different approaches,
one where we can throttle the entire network and one where we can specify which network requests to throttle specifically.
Approach 1: throttle the entire network
const { Builder } = require("selenium-webdriver")
async function throttleNetwork() {
let driver = await new Builder().forBrowser("chrome").build();
await driver.setNetworkConditions({
offline: false,
latency: 5000, // Additional latency (ms).
download_throughput: 50 * 1024, // Maximal aggregated download throughput.
upload_throughput: 50 * 1024, // Maximal aggregated upload throughput.
});
driver.get("http://www.google.com/");
}
thanks to Yaroslav for pointing out the commit.
This has a downside where we can't specify a specific network request to throttle and the rest to go unthrottled.
Let's fix this downside in our next approach.
Approach 2: throttle a specific network request
Here we'd be using an npm package from requestly called Requestly for Selenium.
We need to create a rule first in their client application and get the link by creating a shared list.
For example, let's throttle network request to google.com
require("chromedriver");
const { Builder } = require("selenium-webdriver");
const chrome = require("selenium-webdriver/chrome");
const {
getRequestlyExtension,
importRequestlySharedList,
} = require("#requestly/selenium");
const sharedListUrl = "YOUR_SHARED_LIST_LINK_HERE" // For example, use "https://app.requestly.io/rules/#sharedList/1631611216670-delay"
async function throttleGoogle() {
const options = new chrome.Options().addExtensions(
getRequestlyExtension("chrome") // This installs requestly chrome extension in your testing instance
);
const driver = new Builder()
.forBrowser("chrome")
.setChromeOptions(options)
.build();
await importRequestlySharedList(driver, sharedListUrl); // Here we import the shared list we created some time back
driver.get("http://www.google.com/");
}
This was a high-level overview of how we can overcome the downsides of the selenium-only approach. I've written a blog on the same where I go into depth on how to create a rule, shared list, and so on. You can read it here.
The below issue has now been fixed in this commit
For anyone like me in the C# world wondering why the upload/download throughput does not work as expected, it seems the tooltips for these properties are mislabelled. The tooltip states the data rate is measured in kb/s but in my own experience it is actually bytes per second so if you want to use a more familiar measurement like Mbps you will have to multiply by 125,000:
int latencyInMilliseconds = 20;
long downloadLimitMbps = 20;
long uploadLimitMbps = 5;
_driver.NetworkConditions = new ChromeNetworkConditions()
{
Latency = new TimeSpan(0, 0, 0, 0, latencyInMilliseconds),
DownloadThroughput = downloadLimitMbps * 125000, // Mbps to bytes per second
UploadThroughput = uploadLimitMbps * 125000, // Mbps to bytes per second
IsOffline = false,
};
Using these settings and looking at network traffic while my tests are running I can see they result in exactly 20Mbps down and 5Mbps up.
It looks like it's coming soon to Selenium (C#). The commit was on 01/28/2018:
https://github.com/SeleniumHQ/selenium/blob/ef156067a583fe84b66ec338d969aeff6504595d/dotnet/src/webdriver/Chrome/ChromeNetworkConditions.cs
I know this is an old question, but I recently had to solve for this problem and this page came up at the top of my Google search. Here are the main bits from how I did it in C#. Hope this helps someone in the future.
var networkConditions = new ChromeNetworkConditions();
networkConditions.Latency = new TimeSpan(150);
networkConditions.IsOffline = false;
networkConditions.DownloadThroughput = 120 * 1024;
networkConditions.UploadThroughput = 150 * 1024;
Driver.NetworkConditions = networkConditions;
Inspired by the answer from TridentTrue, here is an updated version for Selenium 4.0.0 in C#. If anyone knows how to use it for alpha7 and upwards without beeing version specific feel free to update this. :)
public void LimitNetwork(int latencyInMilliseconds, long downloadLimitMbps, long uploadLimitMbps)
{
IDevTools devTools = driver as IDevTools;
session = devTools.CreateDevToolsSession();
session.Network.Enable(new EnableCommandSettings());
EmulateNetworkConditionsCommandSettings command = new EmulateNetworkConditionsCommandSettings();
command.Latency = latencyInMilliseconds;
command.DownloadThroughput = downloadLimitMbps * 125000; // Mbps to bytes per second
command.UploadThroughput = uploadLimitMbps * 125000; // Mbps to bytes per second
command.Offline = false;
session.Network.EmulateNetworkConditions(command);
}
Update: After I had implemented this for my own, I found a really good article to get an overview in Selenium 4.0, also in Emulating network conditions.
Update 2: My issue was that I forgot to add the Network.Enable command, so don't forget to call it before you do the other stuff.
I have updated the code. :)

ZeroMQ: How to initialize a SUB and PUSH socket in same code? i.e. black box pattern but not using different machines

I have this code
context = zmq.Context()
app_worker = context.socket(zmq.PUSH)
app_worker.bind("tcp://127.0.0.1:9005")
app_sub = context.socket(zmq.SUB)
app_sub.connect("tcp://127.0.0.1:9004")
app_sub.setsockopt(zmq.SUBSCRIBE,'sometopic')
while True:
msg = app_sub.recv()
msg_data = msg.split(' ',1)
app_worker.send_json(msg_data[1])
print msg_data[1]
but when i run this, it is unable to receive any message from the publisher but when i comment this lines
app_worker = context.socket(zmq.PUSH)
app_worker.bind("tcp://127.0.0.1:9005")
it suddenly works. it is stated in the zeromq guide chapter 5 black box pattern that this is possible. if so, what am i doing wrong here?
You didn't supply enough data to solve this question with 100% assurances.
But based on what you did post the most obvious problem is that the port 9005 was already binded by someone else.
Its very likely your app_worker.send_json(msg_data[1]) is blocking (the entire thread) if there are no downstream nodes to PULL the messages.
Set the send_json to non blocking mode and check the error/exception returned
app_worker.send_json(msg_data[1], zmq.NOBLOCK)
The reason it "works" when you comment out the bind is because the send is just failing and not blocking.

Synchronizing client calls in an xml-rpc-light server in OCaml

I'm writing an XML-RPC module in OCaml using the xml-rpc-light library. I'd like to have a server which can receive concurrent requests and gather the data sent by all the requests in a common 'state'. To be concrete, but simplifying the real problem, suppose the server provides the function
send : int -> bool
which sends an integer and returns true on success and false on failure, and that the server wants to keep a list of all the integers that were ever called since it started (including duplicates).
I understand the XML-RPC part. The part I'm having trouble to implement in OCaml is synchronizing the various client calls on the server side so that they are thread safe.
I've written a client class
class client =
object
val xr_client = new XmlRpc.client "http://localhost:11111"
method send n = xr_client#call "send_integer" [`Int n] in
end
which calls the appropriate send_integer remote method with some integer n.
I've also written a server class
class server =
object(self)
val mutable state = (ref [] : int list ref)
val xr_server = new XmlRpcServer.netplex ()
val m = Mutex.create ()
method send n =
let send_sync s n = (Mutex.lock m; s := n::!s; Mutex.unlock m) in
(send_sync state n; `Boolean true)
method init () =
xr_server#register "send_integer"
~signatures:[[`Int; `Boolean]]
(function
| [`Int x] -> self#send x;
| _ -> XmlRpcServer.invalid_params ());
xr_server#run ()
end
which has a mutable state to keep all the integers and a send method to actually execute the client calls. For each client call, I just update the state. But if a second call comes before the second is finished, I run into thread safety problems. That's why I've added the Mutex bit which uses a single class field mutex to acquire a lock on updating the state.
My questions are:
Would synchronization be achieved by this?
Are there neater ways of doing it?
Is synchronization built-in in the XML-RPC server already, making all of this unnecessary?
After solving compilation problems with this code, I realised that we can't use the threads library together with xmlrpc-light; at least directly. So, we can't use the Mutex module anyway.
Synchronisation is obtained by configuring the XML-RPC server to allow only one thread at a time during runtime. This can be achieved by providing our server with a configuration file similar to the adder example but setting max_threads = 1 in the last line. If we allow, say, 2 threads, then sometimes the change is made in the mutable state of one thread, sometimes it is made in the mutable state of the other.

Resources