Actually i'm trying to evaluate the performance and efficiency of different protocols. I'm aware that TCP implementation considers a packet as "lost" when either the ack timeout or 3 duplicate acks are received. But this cannot tell when the packet is actually lost among the network.
Now i'm capturing packets on both sides of the connection via tcpdump, and get two pcap files. Can i get which exact packets are lost by comparing the two pcap files? And is this worth trying?
A direct attempting would be differing the ip packets' IDs to get the unmatched ones, which is also my first try. The problem is that the network adapter splits and combines some packets so libpcap cannot capture the actual ip packets through the network.
If i don't turn off features such like "generic receive offload" on NIC, which are on by default, what i captured on sender side and receiver side will not match exactly, so simply differing the ids will lead to wrong conclusion; But turning off the features that splitting/combining packets may affect transporting performance badly and thus my conclusion in the other way. So i'm in the middle of a dilemma.
A related post about how "GRO", "LRO" features behave on NIC:
different tcp packets captured on sender and receiver
Related
I have an NDIS driver where i send received packets to the user service, then the service marks those packets that are OK (not malicious), then i iterate over the packets that are good to receive then i send them one by one by by converting each of them back to a proper NetBufferList with one NetBuffer and then i indicate them using NdisFIndicateReceiveNetBufferLists.
This caused a problem that in large file transfers through SMB (copying files from shares), which reduced the transfer speed significantly.
As a workaround, i now chain all of the NBLs that are OK altogether (instead of sending them one by one), and then send all of them at once via NdisFIndicateReceiveNetBufferLists.
My question is, will this change cause any issue? Any difference between sending X number of NBLs one by one vs chaining them together and sending all of them at once? (since most of them might be related to different flows/apps)
Also, the benefit of chaining packets together is much greater in multi packet receive compared to multi packet send via FilterSendNetBufferLists, why is that?
An NET_BUFFER represents one single network frame. (With some appropriate hand-waving for LSO/RSC.)
An NET_BUFFER_LIST is a collection of related NET_BUFFERs. Each NET_BUFFER on the same NET_BUFFER_LIST belong to the same "traffic flow" (more on that later), they have all the same metadata and will have all the same offloads performed on them. So we use the NET_BUFFER_LIST to group related packets and to have them share metadata.
The datapath generally operates on batches of multiple NET_BUFFER_LISTs. The entire batch is only grouped together for performance reasons; there's not a lot of implied relation between multiple NBLs within a batch. Exception: most datapath routines take a Flags parameter that can hold flags that make some claims about all the NBLs in a batch, for example, NDIS_RECEIVE_FLAGS_SINGLE_ETHER_TYPE.
So to summarize, you can indeed safely group multiple NET_BUFFER_LISTs into a single indication, and this is particularly important for perf. You can group unrelated NBLs together, if you like. However, if you are combining batches of NBLs, make sure you clear out any NDIS_XXX_FLAGS_SINGLE_XXX style flags. (Unless, of course, you know that the flags' promise still holds. For example, if you're combining 2 batches of NBLs that both had the NDIS_RECEIVE_FLAGS_SINGLE_ETHER_TYPE flag, and if you verify that the first NBL in each batch has the same EtherType, then it is actually safe to preserve the NDIS_RECEIVE_FLAGS_SINGLE_ETHER_TYPE flag.)
However note that you generally cannot combine multiple NET_BUFFERs into the same NET_BUFFER_LIST, unless you control the application that generated the payload and you know that the NET_BUFFERs' payloads belong to the same traffic flow. The exact semantics of a traffic flow are a little fuzzy down in the NDIS layer, but you can imagine it means that any NDIS-level hardware offload can safely treat each packet as the same. For example, an IP checksum offload needs to know that each packet has the same pseudo-header. If all the packets belong to the same TCP or UDP socket, then they can be treated as the same flow.
Also, the benefit of chaining packets together is much greater in multi packet receive compared to multi packet send via FilterSendNetBufferLists, why is that?
Receive is the expensive path, for two reasons. First, the OS has to spend CPU to demux the raw stream of packets coming in from the network. The network could send us packets from any random socket, or packets that don't match any socket at all, and the OS has to be prepared for any possibility. Secondly, the receive path handles untrusted data, so it has to be cautious about parsing.
In comparison, the send path is super cheap: the packets just fall down to the miniport driver, who sets up a DMA and they're blasted to hardware. Nobody in the send path really cares what's actually in the packet (the firewall already ran before NDIS saw the packets, so you don't see that cost; and if the miniport is doing any offload, that's paid on the hardware's built-in processor, so it doesn't show up on any CPU you can see in Task Manager.)
So if you take a batch of 100 packets and break it into 100 calls of 1 packet on the receive path, the OS has to grind through 100 calls of some expensive parsing functions. Meanwhile, 100 calls through the send path isn't great, but it'll be only a fraction of the CPU costs of the receive path.
Given that UDP packets don't actually send acks, how does a program like iperf measure their one-way performance, i.e., how can it confirm that the packets actually reached:
within a time frame
intact, and uncorrupted
To contrast, Intuitively, to me, it seems that TCP packets, which have an ack signal sent back to allow rigorous benchmarking of their movement across a network can be done very reliably from a client.
1/ "how can it confirm that the packets actually reached [...] intact, and uncorrupted"
UDP is an unfairly despised protocol, but come on, this is going way too far here! :-)
UDP have checksum, just like TCP:
https://en.wikipedia.org/wiki/User_Datagram_Protocol#Checksum_computation
2/ "how can it confirm that the packets actually reached [...] within a time frame"
It does not, because this is not what UDP is about, nor TCP by the way.[*]
As can be seen from its source code here:
https://github.com/esnet/iperf/blob/master/src/iperf_udp.c#L55
...what it does though, is check for out of order packets. A "pcount" is set in the sending side, and checked at the receiving side here:
https://github.com/esnet/iperf/blob/master/src/iperf_udp.c#L99
...and somewhat calculate a bogus jitter:
https://github.com/esnet/iperf/blob/master/src/iperf_udp.c#L110
(real life is more complicated than this, you not only have jitter, but also drift)
[*]:
For semi-garanteed, soft "within a time frame" / real time layer 3 and above protocols, look at RTP, RTSP and such. But neither TCP nor UDP inherently have this.
For real, serious hard real-time garantee, you've got to go to layer 2 protocols such as Ethernet-AVB:
https://en.wikipedia.org/wiki/Audio_Video_Bridging
...which were designed because IP and above simply cannot. make. hard. real. time. guaranteed. delivery. Period.
EDIT:
This is another debate, but...
The first thing you need for "within a time frame", is a shared wall clock on sending/receiving systems (else, how could you tell that such received packet is out of date?)
From Layer 3 (IP) and above, NTP precision target is about 1ms. It can be less than that on a LAN (but accross IP networks, it's just taking a chance and hope the best).
On layer 2, aka "LAN" the layer 2 PTP (Precision Time Protocol) IEEE 1588 is for sub-microsecond range. That's a 1000 times more accurate. Same goes for the derived IEEE 802.1AS, "Timing and Synchronization for Time-Sensitive Applications (gPTP)" used In Ethernet AVB.
Conclusion on this sub-topic:
TCP/IP, though very handy and powerful, is not designed to "guarantee delivery within a time frame". Be it TCP or UDP. Get this idea out of your head.
The obvious way would be to connect to a server that participates in the testing.
The client starts by (for example) connecting to an NTP server to get an accurate time base.
Then the UDP client sends a series of packets to the server. In its payload, each packet contains:
a serial number
a timestamp when it was sent
a CRC
The server then looks these over and notes whether any serial numbers are missing (after some reasonable timeout) and compares the time it received each packet to the time the client sent the packet. After some period of time, the server sends a reply indicating how many packets it received, the mean and standard deviation of the transmission times, and an indication of how many appeared to be corrupted based on the CRCs they contained.
Depending on taste, you might also want to set up a simultaneous TCP connection from the client to the server to coordinate testing of the UDP channel and (possibly) return results.
I am new to networking. I have a small doubt.
I am sending an alarm using SNMP to a target, but the alarm is not received at the target within the specified amount of time. I feel that the data may be lost or dropped.
Now my question is : on what basis should I conclude that there is a loss or drop?
Or will there be any other reason for the trap not to be received?
If I assume your definition of "lost" means one of the network equipment (switch, firewall, ...) didn't forward it to the next hop, and "dropped" means your network board didn't deliver it to your application (e.g. input buffer full, ...).
Under those assumptions, you have no way to know, in your application, that the packet has been "lost" or "dropped". If you want to be sure, you can install network sniffer such as Wireshark on your computer to make sure your packet is delivered (but maybe not processed by your application), or configure your network appliance (if you can) to log packets dropping (meaning "loss" accross the network).
I have a windows app consuming large amounts of incoming udp traffic and sending a small number of udp packets 'keep alive' messages. I'm seeing a small amount of drops on both incoming and outgoing. I was surprised that the small amount of outgoing data was experiencing drops so I captured the packets using netMon and see them all being sent though out of the server, 3 frames sent only 2 arrive at the linux server.
I'd like to know the following:
1. Is NetMon a clone on the sock_buffer and therefore the data may be dropped at the packet buffer and not actually be being sent of the server?
2. Is the packet buffer memory the same for both send and receive (ie. if receive packet buffers are using all the buffer memory preallocated could this cause packet loss on the small amount of outgoing traffic)?
First thing: The send and receive packet buffer have separate memory.
Second thing: NetMon works on lower network layer, not the socket layer.
third thing: Keep in mind that UDP is unreliable protocol and you can not ensure that all packets sent from one end will be received on other end. If you need reliability, you should consider TCP or some other reliable protocol.
By the way both sender and receiver are on same LAN or Internet?? How they are connected? if you can describe it then maybe someone can suggest something else to debug the issue further.
I'm using 2 computers with an application to send and receive udp datagrams. There is no flow control and ICMP is disabled. Frequently when I send a file as UDP datagrams via the application, I get two packets changing their order and therefore - packet loss.
I've disabled and kind of firewall and there is no hardware switch connected between the computers (they are directly wired).
Is there a way to make sure Winsock and send() will send the packets the same way they got there?
Or is the OS doing that?
Or network device configuration needed?
UDP is a lightweight protocol that by design doesn't handle things like packet sequencing. TCP is a better choice if you want robust packet delivery and sequencing.
UDP is generally designed for applications where packet loss is acceptable or preferable to the delay which TCP incurs when it has to re-request packets. UDP is therefore commonly used for media streaming.
If you're limited to using UDP you would have to develop a method of identifying the out of sequence packets and resequencing them.
UDP does not guarantee that your packets will arrive in order. (It does not even guarantee that your packets will arrive at all.) If you need that level of robustness you are better off with TCP. Alternatively you could add sequence markers to your datagrams and rearrange them at the other end, but why reinvent the wheel?
is there a way to make sure winsock and send() will send the packets the same way they got there?
It's called TCP.
Alternatively try a reliable UDP protocol such as UDT. I'm guessing you might be on a small embedded platform so you want a more compact protocol like Bell Lab's RUDP.
there is no flow control (ICMP disabled)
You can implement your own flow control using UDP:
Send one or more UDP packets
Wait for acknowledgement (sent as another UDP packets from receiver to sender)
Repeat as above
See Sliding window protocol for further details.
[This would be in addition to having a sequence number in the packets which you send.]
There is no point in trying to create your own TCP like wrapper. We love the speed of UPD and that is just going to slow things down. Your problem can be overcome if you design your protocol so that every UDP datagram is independent of each other. Our packets can arrive in any order so long as the header packet arrives first. The header says how many packets are suppose to arrive. Also, UPD has become a lot more reliable since this post was created over a decade ago. Don't try to
This question is 12 years old, and it seems almost a waste to answer it now. Even as the suggestions that I have have already been posed. I dealt with this issue back in 2002, in a program that was using UDP broadcasts to communicate with other running instances on the network. If a packet got lost, it wasn't a big deal. But if I had to send a large packet, greater than 1020 bytes, I broke it up into multiple packets. Each packet contained a header that described what packet number it was, along with a header that told me it was part of a larger overall packet. So, the structure was created, and the payload was simply dropped in the (correct) place in the buffer, and the bytes were subtracted from the overall total that was needed. I knew all the packets had arrived once the needed byte total reached zero. Once all of the packets arrived, that packet got processed. If another advertisement packet came in, then everything that had been building up was thrown away. That told me that one of the fragments didn't make it. But again, this wasn't critical data; the code could live without it. But, I did implement an AdvReplyType in every packet, so that if it was a critical packet, I could reply to the sender with an ADVERTISE_INCOMPLETE_REQUEST_RETRY packet type, and the whole process could start over again.
This whole system was designed for LAN operation, and in all of my debugging/beta testing, I rarely ever lost a packet, but on larger networks I would often get them out of order...but I did get them. Being that it's now 12 years later, and UDP broadcasting seems to be frowned upon by a lot of IT Admins, UDP doesn't seem like a good, solid system any longer. ChrisW mentioned a Sliding Window Protocol; this is sort of what I built...without the sliding part! More of a "Fixed Window Protocol". I just wasted a few more bytes in the header of each of the payload packets to tell how many total bytes are in this Overlapped Packet, which packet this was, and the unique MsgID it belonged to so that I didn't have to get the initial packet telling me how many packets to expect. Naturally, I didn't go as far as implementing RFC 1982, as that seemed like overkill for this. As long as I got one packet, I'd know the total length, unique Message Id, and which packet number this one was, making it pretty easy to malloc() a buffer large enough to hold the entire Message. Then, a little math could tell me where exactly in the Message this packet fits into. Once the Message buffer was filled in...I knew I got the whole message. If a packet arrived that didn't belong to this unique Message ID, then we knew this was a bust, and we likely weren't going to ever get the remainder of the old message.
The only real reason I mention this today, is that I believe there still is a time and a place to use a protocol like this. Where TCP actually involves too much overhead, on slow, or spotty networks; but it's those where you also have the most likelihood of and possible fear of packet loss. So, again, I'd also say that "reliability" cannot be a requirement, or you're just right back to TCP. If I had to write this code today, I probably would have just implemented a Multicast system, and the whole process probably would have been a lot easier on me. Maybe. It has been 12 years, and I've probably forgotten a huge portion of the implementation details.
Sorry, if I woke a sleeping giant here, that wasn't my intention. The original question intrigued me, and reminded me of this turn-of-the-century Windows C++ code I had written. So, please try to keep the negative comments to a minimum--if at all possible! (Positive comments, of course...always welcome!) J/K, of course, folks.