Is there a way to programatically know when a pgm zeromq socket has stopped forwarding information because the ZMQ_RATE limit has been reached or if it is dropping data because the ZMQ_SNDHWM limit has been reached. There is a zmq_socket_monitor function call that allows the user to see events like client connect and client disconnect. I am thinking that there should be a similar construct for the rate limit.
Q :"Is there a way to programatically know when a ... zeromq socket has stopped forwarding ... because ..." ?
A : to the best of my knowledge and knowing limits thereof, there is no such way implemented so far.
If bets on reasoning for not having a way, I'd put my few cents on
(a)such feature having zero-( if not negative )-effects on ZeroMQ primary goals, that are maximum performance, minimised resources needs to achieve performance and minimum latency
(b)anyone, capable of providing a stable & acceptable implementation of this feature into the core & c-API could've been warm welcome to implement it, yet no one has put one's efforts into developing & testing & accepting it so far, so we stay in 2022-Q1 still in square number one
Related
Imagine I have a streaming device like the one in the bellow template.
image from here
def worker(backend_port):
context = zmq.Context()
socket = context.socket(zmq.PULL)
socket.connect("tcp://127.0.0.1:%d" % backend_port)
while True:
task = socket.recv()
# do something
def streamer(frontend_port):# it's called streamer because streams files from hdfs
context = zmq.Context()
socket = context.socket(zmq.PUSH)
socket.connect("tcp://127.0.0.1:%d" % frontend_port)
while True:
# get the file
# prepare it
socket.send(msg, copy=False)
number_of_workers = 16
number_streamers = 10
frontend_port = 7559
backend_port = 7560
streamerdevice = ProcessDevice(zmq.STREAMER, zmq.PULL, zmq.PUSH)
streamerdevice.bind_in("tcp://127.0.0.1:%d" % frontend_port )
streamerdevice.bind_out("tcp://127.0.0.1:%d" % backend_port)
streamerdevice.setsockopt_in(zmq.IDENTITY, b'PULL')
streamerdevice.setsockopt_out(zmq.IDENTITY, b'PUSH')
streamerdevice.start()
for i in range(number_of_workers):
p = Process(target=yolo, args=(backend_port,))
p.start()
for i in range(number_streamers):
s = Process(target=streamer, args=(frontend_port,))
s.start()
To give more information messages are images so are on the bigger size. I am using zero-copy. Performance is the most important point in my case since I am running on a very big scale.
Is there a way to multi-thread that streamerdevice?
How can I control the flow? is there a way to know how many messages are waiting to get processed?
one of the main goals is to make the receiving time as fast as possible on the worker side. Any suggestions for that?
Does it help to start the zmq.Context(nIoTHREADs) for nIoTHREADs>1 in worker side?
How to run a multi-threaded ZMQ device?
The simplest way to run multi-threaded ZeroMQ services is to instantiate the respective zmq.Context( nIoTHREADs ) for nIoTHREADs > 1. There are more performance tweaking configuration options for scaling performance. Review the pyzmq wrapper, how does this ZeroMQ tooling apply to actual the pyzmq wrapper code and ask maintainers, whether their setup and/or configuration remained fixed or just unpublished with this regard. ZeroMQ is flexible enough to provide means for doing this.
...device (which is a one way queue) is not scalable as you increase the number of input connections
Well, a device is neither a one way queue, nor is not scalable. Right the very opposite.
Is it possible to run it in a multi-threaded way?
Sure it is.
what is the best way to scale it?
That depends a lot on exact messaging and signaling requirements that your system design ought meet. High volume, ultra-low latency, almost linear scaling, adaptive connection handing, app-domain user-defined protocols are just a few of such properties to take into account.
Is it because the queue gets so big and that is why it gets slow?
The post contained zero pieces of information about neither the sizes, nor about any speed observations and/or expectations. In other, documented, test setups, ZeroMQ is known to be very cute in buffer-management, if zero-copy and other performance motivated approaches are used, so no one can help on a more detailed level.
is there a way to know how big the queue is at any moment?
No, there is no such "built-in" feature, that would spend time and resources on counting this. If such add-on feature is needed in your system, you are free to add such add-on, without making other use-cases wear a single bit of such additional code for use-cases, where a message counting and queue-size reporting is meaningless.
any load balancing strategy for the device without a REQ-REP pattern?
Yes, there are many ready-made Scalable Formal Communication Archetype Patterns plus there are scenarios one may compose for going beyond the ready-made bilateral distributed-behaviour Archetype primitives, that permit to build a custom-specific, controlled as required, load-balancer of one's own wish and taste.
I have an architecture where I have a ROUTER socket and multiple DEALER sockets.
Some DEALER sockets will only send data, some will only receive data and others can do a mixture of both.
I have a scenario, where I have one DEALER socket, that is sending data at an extremely fast rate. This data is received by another DEALER, that will process this as fast as it can. The send rate is always going to be higher than the receive.
In my current setup the ZMQ_SNDHWM on my ROUTER socket gets hit for the receive client and will silently drop messages. I do not want this to be the case.
What is the best way to do so as to deal with this scenario?
I have looked at DEALER->DEALER on a different port, but this could be hard to maintain, depending on the number of sessions that are created I could potentially have to have one port per session.
The other way I can think of solving this is to do some pipe-lining in which the receiving DEALER socket will tell the sender when it is ready to receive but this seems to add a lot of complication to the overall protocol and a lot more state management. It also seems to defeat the ability to be able to naturally block on DEALER sockets which is really what I need in this case; the DEALER sockets will never have to communicate with any other socket.
Do not rely on blocking, the less on uncontrolled use of resources
In distributed-system there is not much space for optimistic beliefs and expectations. In many posts I advocate, where possible, not to rely on blocking states, as your code gets out-of-control and you cannot do anything about leaving such a state, but pray for a message to come any time soon, if ever.
Rather bear the responsibility, end-to-end, which in distributed-system means that you need to also design strategies how to survive a "remote" death and similar situations, that are outside of the range of your-domain-of-control, but which your code design has no right to abstract from.
Even if not willing to, an explicit flow-management is the way to go
Late 90-ies have demonstrated many flow-control strategies for distributed systems, so this is definitely not a new field.
Increasing the box-of-worm size does not help to manage the un-controlled / un-managed flow of events. While ZMQ_???HMW + ZMQ_???BUF may help somehow tweak non-brutal cases, where having a bit more space may temporarily postpone the principal weakness of un-controlled message-flows, yet the problem is like remaining stand still with closed eyes right in the middle of the cross-fire shooting. Such agent may survive, but it's survival is not a result of it's design cleverness, but just an accidental luck.
Token-passing seems to be the cheapest way how to throttle the flow so as to remain acceptable / process-able on the slowest node. Increasing such a node-processing performance may be done with a use of a static, an incrementally expanded or a fully adaptive pooling-proxy, so even this known bottleneck remains manageable and under your design-control.
The highest layer of robustness is in making your distributed-system's design resilient to spurious bursts of events ( be it messages or .connect() attempts ). So, independently of selecting the building blocks, designer has also the responsibility to design in all the survival-strategies. Not doing this leaves your system vulnerable to capacity-directed vector of attack or other sort of unhandled exploits of these kinds of known weakness vulnerabilites.
Is it possible to gauge a web browsers upload and/or download speed by monitoring normal http requests? Ideally a web application would be able to tell the speed of a client without any modifications and without client-side scripting like JavaScript/Java/Flash. So even if a client was accessing the service with a library like Curl it would still work. If this is possible, how? If its not possible, why? How accurate can this method be?
(If it helps assume PHP/Apache, but really this is a platform independent question. Also being able to gauge the upload speed is more important to me.)
Overview
You're asking for what is commonly called "passive" available bandwidth (ABW) measurement along a path (versus measuring a single link's ABW). There are a number of different techniques1 that estimate bandwidth using passive observation, or low-bandwidth "Active" ABW probing techniques. However, the most common algorithms used in production services are active ABW techniques; they observe packet streams from two different end-points.
I'm most familiar with yaz, which sends packets from one side and measures variation in delay on the other side. The one-sided passive path ABW measurement techniques are considered more experimental; there aren't solid implementations of the algorithms AFAIK.
Discussion
The problem with the task you've asked for is that all non-intrusive2 ABW measurement techniques rely on timing. Sadly, timing is a very tricky thing when working with http...
You have to deal with the reality of object caching (for instance, akamai) and http proxies (which terminate your TCP session prematurely and often spoof the web-server's IP address to the client).
You have to deal with web-hosts which may get intermittently slammed
Finally, active ABW techniques rely on a structured packet stream (wrt packet sizes and timing), unlike what you see in a standard http transfer.
Summary
In summary, unless you set up dedicated client / server / protocol just for ABW measurement, I think you'll be rather frustrated with the results. You can keep your ABW socket connections on TCP/80, but the tools I have seen won't use http3.
Editorial note: My original answer suggested that ABW with http was possible. On further reflection, I changed my mind.
END-NOTES:
---
See Sally Floyd's archive of end-to-end TCP/IP bandwidth estimation tools
The most common intrusive techniques (such as speedtest.net) use a flash or java applet in the browser to send & receive 3-5 parallel TCP streams to each endpoint for 20-30 seconds. Add the streams' average throughput (not including lost packets requiring retransmission) over time, and you get that path's tx and rx ABW. This is obviously pretty disruptive to VoIP calls, or any downloads in progress. Disruptive meausurements are called bulk transfer capacity (BTC). See RFC 3148: A Framework for Defining Empirical Bulk Transfer Capacity Metrics. BTC measurements often use HTTP, but BTC doesn't seem to be what you're after.
That is good, since it removes the risk of in-line caching by denying http caches an object to cache; although some tools (like yaz) are udp-only.
Due to the way TCP connections adapt to available bandwidth, no this is not possible. Requests are small and typically fit within one or two packets. You need a least a dozen full-size packets to get even a coarse bandwidth estimate, since TCP first has to scale up to available bandwidth ("TCP slow start"), and you need to average out jitter effects. If you want any accuracy, you're probably talking hundreds of packets required. That's why upload rate measurement scripts typically transfer several megabytes of data.
OTOH, you might be able to estimate round-trip delay from the three-way handshake and the timing of acks. But download speed has at least as much impact as upload speed.
There's no support in javascript or any browser component to measure upload performance.
The only way I can think of is if you are uploading to a page/http handler, and the page is receiving the incoming bytes, it can measure how many bytes it is receiving per second. Then store that in some application wide dictionary with a session ID.
Then from the browser you can periodically poll the server to get the value in the dictionary using the session ID and show it to user. This way you can tell how's the upload speed.
You can use AJAXOMeter, a JavaScript library which meassures your up- and download speed. You can see a live demo here.
That is not feasible in general as in-bound and out-bound bandwidth frequently is not symmetric. Different ISPs have significantly different ratios here that can vary on even time of the day basis.
Lately I've asked this question. But the answer doesn't suit my demands, and I know that file hosting providers do manage to limit the speed. So I'm wondering what's the general algorithm/method to do that (I do mean downloading technique) - in particular limiting single connection/user download speed.
#back2dos I want to give a particular user a particular download speed (corresponding to hardware capabilities of course) or in other words give user ability to download some particular file with lets say 20kb/s. Surely I want to have an ability to change that to some other value.
You could use a token bucket ( http://en.wikipedia.org/wiki/Token_bucket)
Without mention of platform/language, it's difficult to answer, but a "leaky bucket" algorithm would probably be the best fit:
http://en.wikipedia.org/wiki/Leaky_bucket
Well, since this answer is really general, here's a very simple approach for plain TCP:
You put the resource handlers of all download connection into a list, paired up with information about what data is requested, and loop through it. Then you write a chunk of the required data onto the socket, maybe about 1.5K, which is the most commonly used maximum segment size, as far as I know. When you're at the and of the list, you start over. Before starting over, simply wait to get the desired average bandwidth.
Please note, if too many clients have lower bandwidth than you allow, then your TCP buffer is likely explode. some TCP bindings permit finding the size of currently buffered data for one socket. if it exceeds a threshold, you can simply skip the socket.
Also, if too many clients are connected, you will actually not have enough time to write to all the sockets, thus after one loop, you "have to wait for a negative time". Increasing the chunk size might speed up things in such scenarios, but at some point your server will stop getting faster.
A more simple approach is to do this on the client side, but this may generate a lot of overhead. The dead simple idea is to have the client request 1K every 50ms (assuming you want 20KB/s). You can even do that over HTTP, although I strongly suggest bigger chunk size, since HTTP has enourmous overheads.
My guess is, the best is to try to find a webserver capable of doing such things out of the box. I think Apache has a number of modules for al kinds of quota.
greetz
back2dos
This question is related with Microcontroller programming but anyone may suggest a good algorithm to handle this situation.
I have a one central console and set of remote sensors. The central console has a receiver and the each sensor has a transmitter operates on same frequency. So we can only implement Simplex communication.
Since the transmitters work on same frequency we cannot have 2 sensors sending data to central console at the same time.
Now I want to program the sensors to perform some "polling". The central console should get some idea about the existence of sensors (Whether the each sensor is responding or not)
I can imagine several ways.
Using a same interval between the poll messages for each sensor and start the sensors randomly. So they will not transmit at the same time.
Use of some round mechanism. Sensor 1 starts polling at 5 seconds the second at 10 seconds etc. More formal version of method 1.
The maximum data transfer rate is around 4800 bps so we need to consider that as well.
Can some one imagine a good way to resolve this with less usage of communication links. Note that we can use different poll intervals for each sensors if necessary.
I presume what you describe is that the sensors and the central unit are connected to a bus that can deliver only one message at a time.
A normal way to handle this is to have collision detection. This is e.g. how Ethernet operates as far as I know. You try to send a message; then attempt to detect collision. If you detect a collision, wait for a random amount (to break ties) and then re-transmit, of course with collision check again.
If you can't detect collisions, the different sensors could have polling intervals that are all distinct prime numbers. This would guarantee that every sensor would have dedicated slots for successful polling. Of course there would be still collisions, but they wouldn't need to be detected. Here example with primes 5, 7 and 11:
----|----|----|----|----|----|----|----| (5)
------|------|------|------|------|----- (7)
----------|----------|----------|-:----- (11)
* COLLISION
Notable it doesn't matter if the sensor starts "in phase" or "out of phase".
I think you need to look into a collision detection system (a la Ethernet). If you have time-based synchronization, you rely on the clocks on the console and sensors never drifting out of sync. This might be ok if they are connected to an external, reliable time reference, or if you go to the expense of having a battery backed RTC on each one (expensive).
Consider using all or part of an existing protocol, unless protocol design is an end in itself - apart from saving time you reduce the probability that your protocol will have a race condition that causes rare irreproducible bugs.
A lot of protocols for this situation have the sensors keeping quiet until the master specifically asks them for the current value. This makes it easy to avoid collisions, and it makes it easy for the master to request retransmissions if it thinks it has missed a packet, or if it is more interested in keeping up to date with one sensor than with others. This may even give you better performance than a system based on collision detection, especially if commands from the master are much shorter than sensor responses. If you end up with something like Alohanet (see http://en.wikipedia.org/wiki/ALOHAnet#The_ALOHA_protocol) you will find that the tradeoff between not transmitting very often and having too many collisions forces you to use less than 50% of the available bandwidth.
Is it possible to assign a unique address to each sensor?
In that case you can implement a Master/Slave protocol (like Modbus or similar), with all devices sharing the same communication link:
Master is the only device which can initiate communication. It can poll each sensor separately (one by one), by broadcasting its address to all slaves.
Only the slave device which was addressed will reply.
If there is no response after a certain period of time (timeout), device is not available and Master can poll the next device.
See also: List of automation protocols
I worked with some Zigbee systems a few years back. It only had two sensors so we just hard-coded them with different wait times and had them always respond to requests. But since Zigbee has systems However, we considered something along the lines of this:
Start out with an announcement from the console 'Hey everyone, let's make a network!'
Nodes all attempt to respond with something like 'I'm hardware address x, can I join?'
At first it's crazy, but with some random retry times, eventually the console responds to all nodes: 'Yes hardware address x, you can join. You are node #y and you will have a wait time of z milliseconds from the time you receive your request for data'
Then it should be easy. Every time the console asks for data the nodes respond in their turn. Assuming transmission of all of the data takes less time than the polling period you're set. It's best not to acknowledge the messages. If the console fails to respond, then very likely the node will try to retransmit just when another node is trying to send data, messing both of them up. Then it snowballs into complete failure...