Suppose that several machines are interacting together using python's zeroMQ client.
These messages are naturally formatted as strings.
Is there a limit to the length of a message (string)?
There is no limit to the size of messages being sent however small messages are handled differently than large messages (see here).
The max size of a small messages is defined in the source code at 30 bytes (see here, look for ZMQ_MAX_VSM_SIZE).
There is the socket option ZMQ_MAXMSGSIZE which causes a peer sending an oversized message to be disconnected, but the default is "no limit".
No limit
As for small size messages transmitted within zmq_msg_t structures, their limit is 29 bytes (for zmq version 3.2.2)
"max_vsm_size = 29," quoted from https://github.com/zeromq/libzmq/blob/master/src/msg.hpp
Some socket types support up to 2^64, but some less than 2^31.
You should build a protocol that keeps chunks below that size anyway, but this is the real answer.
https://github.com/zeromq/libzmq/issues/1332
Related
I have been testing the SDK with code based on the Azure-ESP-Starter code.
When I try to send a larger message body the message fails. I am using an ESP32 Wroom and the SDK installed in VScode.
Message length of 5438 transmits ok, but 5458 fails.
I was looking for a reason and noticed that in the TCP settings, the default send buffer size = 5744
If I change this to 15744 I can send a 10,000 byte message.
The SDK seems to be failing when trying to fragment & send larger messages.
At first I thought it may have something to do with the TLS maximum outgoing fragment length setting (default 4096), but increasing this did not resolve the problem. The TCP buffer setting does allow larger messages.
I thought that Azure allows a 256k message. Has anyone else noticed this issue?
The following document section lists the limits associated with the different service tiers S1, S2, S3, and F1. Device-to-cloud messages can be at most 256 KB, and can be grouped in batches to optimize sends. Batches can be at most 256 KB.
I would suggest making use of a different Azure IoT Library or a different device which you can repro the same issue or maybe with another protocol?
Throttling details
IoT Hub measures message size in a protocol-agnostic way, considering only the actual payload. The size in bytes is calculated as the sum of the following values:
The body size in bytes.
The size in bytes of all the values of the message system properties.
The size in bytes of all user property
names and values.
device-to-cloud messages can be at most 256 KB
The total message size, including the enrichments, can't exceed 256 KB. If a message size exceeds 256 KB, the IoT Hub will drop the message. You can use IoT Hub metrics to identify and debug errors when messages are dropped. For example, you can monitor the telemetry messages incompatible (d2c.telemetry.egress.invalid) metric in the routing metrics. To learn more, see Monitor IoT Hub.
We are developing a project using Angular in the front and Spring at the backend. Nothing new. But we have set-up the backend to use HTTP2 and from time to time we find weird problems.
Today I started playing with "Network Log Export" from chrome and I found this interesting piece of information in the HTTP2_SESSION line of the log.
t=43659 [st=41415] HTTP2_SESSION_RECV_GOAWAY
--> active_streams = 4
--> debug_data = "Connection [263], Too much overhead so the connection will be closed"
--> error_code = "11 (ENHANCE_YOUR_CALM)"
--> last_accepted_stream_id = 77
--> unclaimed_streams = 0
t=43659 [st=41415] HTTP2_SESSION_CLOSE
--> description = "Connection closed"
--> net_error = -100 (ERR_CONNECTION_CLOSED)
t=43661 [st=41417] HTTP2_SESSION_POOL_REMOVE_SESSION
t=43661 [st=41417] -HTTP2_SESSION
It looks like the root of the problem for the ERR_CONNECTION_CLOSED is the server decides there are too much overhead from the same client and closes the connection.
The question is ¿Can we tune the server to accept overhead up to a certain limit? ¿how? I believe this is something we should be able to tune up in Spring or tomcat or somewhere there.
Cheers
Ignacio
The overhead protection was put in place in response to a collection of CVE's reported against HTTP/2 in the middle of 2019. While Tomcat wasn't directly affected (the malicious input didn't trigger excessive load) we did take steps to block input that matched the malicious profile.
From your GitHub comment, you see issues with POSTs. That strongly suggests that the client is sending the POST data in multiple small packets rather than a smaller number of larger packets. Some clients (e.g. Chrome) are know to do this occasionally due to they way they buffer data.
A number of the HTTP/2 DoS attacks could be summarized as sending more overhead than data. While Tomcat wasn't directly affected, we took the decision to monitor for clients operating in this way and drop connections if any were found on the grounds that the client was likely to be malicious.
Generally, data packets reduce the overhead count, non-data packets increase the overhead count and (potentially) malicious packets increase the overhead count significantly. The idea is that an established, generally well-behaved, connection should be able to survive the occasional 'suspect' packet but any more than that will quickly trigger the connection to be closed.
In terms of small POST packets the key configuration setting is:
overheadCountFactor
overheadDataThreshold
The overhead count starts at -10. For every DATA frame received it is reduced by 1. For every SETTINGS, PRIORITY and PING frame it is increased by overheadCountFactor.If the overhead count goes above 0, the connection is closed.
In addition, if the average size of a received non-final DATA frame and the previously received DATA frame (on that same stream) is less than overheadDataThreshold then the overhead count is increased by overheadDataThreshold/(average size of current and previous DATA frames). In this way, the smaller the DATA frame, the greater the increase in the overhead. A small number of small non-final DATA frames should be enough to trigger connection closure.
The averaging is there so buffering such as exhibited by Chrome does not trigger the overhead protection.
To diagnose this problem you need to look at the logs to see what size non-final DATA frames are being sent by the client. I suspect that will show a series of non-final DATA frames with size less than 1024 (the default for overheadDataThreshold).
To fix the issue my recommendation is to look at the client first. Why is it sending small non-final DATA frames and what can be done to stop it?
If you need an immediate mitigation then you can reduce overheadDataThreshold. The information you get on DATA frame sizes sent by the client should guide you as to what to set this to. It needs to be smaller than DATA frames being sent by the client. In extremis you can set overheadDataThreshold to zero to disable the protection.
Hi guys I was just wondering if the websocket protocol already handles the sending of large data in chunks. At least knowing that it does will save me the time of doing so myself.
According to RFC-6455 base framing, has a maximum size limit of 2^63 bytes which means it actually depends on your client library implementation.
I was just wondering if the websocket protocol already handles the sending of large data in chunks...
Depends what you mean by that.
The WebSockets protocol is frame based (not stream based)
If what you're wondering about is "will a huge payload arrive in one piece?" - the answer is always "yes".
The WebSockets protocol is a frame / message based protocol - not a streaming protocol. Which means that the protocols wraps and unwraps messages in a way that's designed to grantee message ordering and integrity. A messages will not get...
...truncated in the middle (unlike TCP/IP, which is a streaming based protocol, where ordering is preserved, but not message boundaries).
The WebSockets protocol MAY use fragmented "packets"
According to the standard, the protocol may break large messages to smaller chunks. It doesn't have too.
There's a 32 bit compatibility concern that makes some clients / servers fragment messages into smaller fragments and later put them back together on the receiving end (before the onmessage callback is called).
Application layer "chunking" is required for multiplexing
Sending large payloads over a single WebSocket connection will cause a pipelining issue, where other messages will have to wait until the huge payload is sent, received and (if required) re-assembled.
In practice, this means that large payloads should be fragmented by the application layer. This "chunked" application layer approach will enable multiplexing the single WebSocket connection.
In the following JBoss/HornetQ user manual page you can see how HornetQ provides a mechanism for streaming data to a Message for a Queue using a java.io.InputStream. A JMS version of the same code is given. Has anyone come across an equivalent using IBM MQSeries / WebsphereMQ?
Say I have a large amount of data to place in the JMS Message which to me is just a stream of bytes. In the Hornet example, the stream is only read when the message is sent, so if it is, say a FileInputStream, then we only need enough memory to buffer a chunk of the bytes. I can use a javax.jms.BytesMessage to send in chunks of bytes and use the BytesMessage to buffer them. The problem with this is that the IBM implementation of BytesMessage (com.ibm.msg.client.jms.internal.JmsBytesMessageImpl) has to cache them until the Message is sent and if that is a large amount of data it is a problem. Worse it appears that although I am only sending bytes, the IBM implementation appears to keep duplicate copies, one in a BytesArrayOutputStream the other in a DataOutputStream.
In WebSphere MQ the closest thing to what you describe is a reference message. The method described in the Infocenter requires custom programming of channel exits to grab the filesystem object and put it into a message before it is transmitted over the channel. A complementary exit on the remote side saves the payload to a file and puts a reference to the file in the message that is returned to the app.
We also have programs in WMQ that take STDIN or a pipe at one end and put messages to a queue on the other end. A pair of these can act as a pipe through which line-oriented ASCII data flows between processes on separate machines. However, there's no JMS implementation of this and it doesn't work too well for binary data.
In WMQ, we have concept of Group and Segment.
Segmentation is supported in all OS except Z/OS.
Check for details here Segmentation In WMQ
Make use of GroupId, MsgSeqNumber, and Offset while putting the message.
While getting the message if you give MQGMO_COMPLETE_MSG in GMO, then all segments are joined automatically according to the MsgSeqNumber and
you will get a single message on the recieving application with a
single GET.
Hi all ActiveMQ experts!
I had a look via WireShark what's happening under the hood when ActiveMQ "/examples" producer sends messages and it revealed that every TextMessage shorter than 1000 bytes is padded with spaces (' ' or HEX 20) until it fills exactly 1000 bytes.
(using the ActiveMQ's "native" transport - TCP "OpenWire")
Wonder what is doing that?
(I presume the ActivemQ JMS Client implementation)
but WHY??
And most importantly, is there a way to optimize it so that sending short messages does not incur almost 1K overhead of unnecessary spaces?
Thank you!
cheers,
O.K.
Your are correct that the client implementation is adding the spaces to the messages (see the createMessageText method). This is simply so that the messages are an even size. You can either change the size when running the example (use the arg: -DmessageSize=<size>) or you can change the ProducerTool so that it does not pad each message by simply editing the code and running it again (Ant will compile it before running it). I've done this many times to remove the spaces altogether, to add extra text to the message, to add message headers, to format the message body using XML, etc.
Bruce