I'm writing a program in python3 to analyze a .pcap file and I'm just wondering if the endianness (little endian or big endian) applies to all packets, or just the main header.
Thanks!
It applies to:
the file header;
the per-packet header (time stamp, captured length, on-the-network length);
some meta-data headers in the packet contents (LINKTYPE_NULL, LINKTYPE_USB_LINUX, LINKTYPE_USB_LINUX_MMAPPED, LINKTYPE_IEEE802_11_PRISM, LINKTYPE_NFLOG, LINKTYPE_PFLOG);
some fields in the CANbus header in LINKTYPE_LINUX_SLL/LINKTYPE_LINUX_SLL2 captures).
Other fields in the packet contents are in whatever byte order the protocol containing the field in question specifies - for example, multi-byte integral-valued fields in the IPv4, IPv6, UDP, and TCP headers are big-endian regardless of the endianness of the file and per-packet headers.
Related
I've found what code parses IP (v4) packets in the kernel source tree. This function, ip_rcv, can to a high degree of certainty detect whether a packet is correct or not, as is outlined in one of the comments:
Length at least the size of an ip header
Version of 4
Checksums correctly. [Speed optimisation for later, skip loopback checksums]
Doesn't have a bogus length
Malformed packets are simply dropped. This function seems to get a bunch of bytes that should resemble an IP packet, but what if some malicious actor would sneak an extra byte on the line? If not handled correctly, all the chunks of bytes that ip_rcv receives from now on will start 1 byte off and no correct IP packet can be reconstructed anymore. I assume the kernel does something smarter than to try all different byte offsets at which to start parsing an IP packet. What exactly, I'm unable to find. Could anyone shed some light on this?
I haven't taken the time to look at the kernel code but most protocol stacks are going to work by parsing data immediately following the previous stack location and not by searching for data.
In the case of Ethernet, an Ethernet frame header is typically 14 bytes in size. It can vary but the header itself indicates the different length in the etherType field when necessary. In this example, the NIC (Network Interface Card) will receive an Ethernet frame. If the frame is destined for this NIC then the data passed from the NIC driver to the IP stack will be an Ethernet frame containing this 14-byte header followed immediately by the IP header (first nibble will be 4 if it is a version 4 IP header for instance).
Again, I didn't look at the network stack code but there are two common cases here:
1) The IP stack is told this is an Ethernet frame and only needs to parse the Ethernet frame header for its length and the very next byte must be an IP header or the data is deemed not an IP frame.
2) The IP stack is given a pointer to the beginning of the data immediately following the Ethernet frame header and the IP stack then starts parsing at that location.
I want to know whether this parameter is sent by decoder and used to limit the encoder.
For example, a client receives a SETTINGS frame from a server with a parameter SETTINGS_HEADER_TABLE_SIZE = 1024 bytes. Does this parameter mean that the max memory, which the client can use to deflate the request headers, is 1024 bytes?
The SETTINGS_HEADER_TABLE_SIZE parameter indicates the size of the dynamic table as specified in RFC 7541 (also known as HPACK encoding).
The particular section you want to look at to understand what is the header table size is RFC 7541, section 4.
This parameter is related to the max memory that is allocated for each connection, but its value does not represent the actual number of bytes of memory used by the HPACK encoding.
This parameter can be changed dynamically by either peer as specified in RFC 7540, section 6.5.2.
I ParseFromArray the protocol buffer's protocol, the protocol is not lack any filed. But the ParseFromArray function returns false. Why?
I'm assuming you are using C++. ParseFromArray() fails if:
The input data is not in valid protobuf format.
The input data is lacking a required field.
If you are sure that all required fields are set, then it must be the case that your input data is corrupted. You should verify that the bytes and size you are passing into ParseFromArray() are exactly the bytes and size that you got from SerializeToArray() and ByteSize() on the sending side. You will probably find that you are losing some bytes somewhere, or that some bytes got corrupted.
Common reasons for corruption include:
Passing the encoded bytes over a text-only channel. E.g. if you write the data to (or read it from) a file that is not opened in "binary" mode, or if you at some point store the bytes in a Java String, the data will become corrupted, as these channels expect text, and encoded protobufs are not text.
Passing the bytes as a char*, i.e. assuming NUL-termination. Encoded protobufs can contain '\0' bytes, meaning that you cannot represent one as a char* alone -- you must include the size separately.
Serializing to an array that is larger than needed, and then forgetting to pay attention to how much data was actually written. When you call SerializeToArray(), you must also call ByteSize() to see how large the message is, and you must make sure the receiving end receives that size and passes it to ParseFromArray(). Otherwise, the parser will think that the extra bytes at the end of the buffer are part of the message, and will fail to parse them.
This is 2 example:
How to append data on a packet from kernel space?
How to route the splitted packets using netfilter hooks in kernel space
I just want change data coming server at hook LOCAL_IN, this is similar spllitted example.
At append data example, that is ok. But splitted example, that is not work.
I think problem is update length, checksum udp,ip packet(example: the value offset in calculating checksum at hook LOCAL_IN and LOCAL_OUT is different( int offset = skb_transport_offset(skb)) because when a packet goes in, packet is processed before go to udp layer).I try to alter htons -> ntohs but that is not work.
Anyone have idea to solve? Thanks
the problem is different function checksum.
In side sender, when you update udp checksum at hook(POST_ROUTING or LOCAL_OUT), checksum just for pseudo header, not include udp datagrams.
In side receiver, when you updata udp checksum at hook(PRE_ROUTING or LOCAL_IN), checksum must include pseudo header+udp datagrams.
I'm programming with sockets in Mac 10.6.8. Whenever I receive a packet, it starts with the IP header. I've been using Wireshark to analyze incoming packets, and I noticed that my machine's socket implementation will consistently change the "total length" field in the IP header. Specifically, it will subtract the IP header length and reverse the bytes (from network to host order).
For example, here's the beginning of an IP header as reported by Wireshark:
45 c0 00 38 ...
That breaks down as follows:
4 bits (0x4): IP version: 4
4 bits (0x5): IP header length: 5 words (20 bytes)
8 bits (0xc0): differentiated services flags
16 bits (0x0038): total length: 56 bytes
However, when I print the contents of the buffer filled by recvfrom for the same packet, I get a different lede:
ssize_t recvbytes = recvfrom(sock->fd, buffer, size, /*flags=*/0,
(struct sockaddr*)src, &src_len);
returns
45 c0 24 00 ...
4 bits (0x4): IP version: 4
4 bits (0x5): IP header length: 5 words (20 bytes)
8 bits (0xc0): differentiated services flags
16 bits (0x2400): total length: 9216 bytes (?!)
I figured out that before I get access to the buffer, the socket implementation is reading the total length, subtracting the IP header length, and then writing it back in host order (little endian on my machine) rather than network order (big endian). In this example, that means:
read the total length: 0x0038 = 56
subtract the header length: 56 - 20 = 36
write back in host order: 36 = 0x0024 (big endian) = 0x2400 (little endian = host order on my machine)
The problem gets worse. It won't just change the total length of the outermost IP header. It will also change the total length fields of internal IP headers, e.g., the one buried in an ICMP "time exceeded" message (which must include the original IP header of the dropped packet). Funnier still, it won't subtract the IP header length from the internal headers; it just reverses the byte order.
Is this happening to anyone else? Is it part of a standard I'm unaware of? Is there a way to fix my machine's socket implementation to stop tampering with packets? How is Wireshark able to get around this problem?
Thanks in advance for your consideration.
EDIT: My code and Makefile are available on GitHub. I wrote a fixip_osx function to allow verifying IP checksums:
https://github.com/thejohnfreeman/netutils/blob/master/lib/ip.c
void fixip_osx(struct ip* ip) {
/* Something on my Mac subtracts the header length from `ip_len` and stores
* it in host order (little endian). */
u16_t ip_hdrlen = ip->ip_hl << 2;
u16_t ip_totlen = ip->ip_len + ip_hdrlen;
ip->ip_len = htons(ip_totlen);
}
However, it's still a problem for verifying ICMP checksums when the payload contains another IP header.
The problem exists whether I compile with Clang 3.2 (built from trunk) or GCC 4.7 (MacPorts port), so I assume the problem lies in either the sockets implementation (packaged with Mac OS) or in Mac OS X itself.
The BSD suite of platforms (excluding OpenBSD) present the IP offset and length in host byte order. All other platforms present in the received network byte order. This is a "feature", and is referenced in the man page for IP(4) - Internet Protocol (FreeBSD, OS X).
The ip_len and ip_off fields must be provided in host byte order .
All other fields must be provided in network byte order.
IP length can equal packet length - IP header length in FreeBSD/NetBSD.
Reference: Stevens/Fenner/Rudolph, Unix Network Programming Vol.1, p.739
I have to deal with these anomalies with a user space implementation of the PGM network protocol, specific code:
https://code.google.com/p/openpgm/source/browse/trunk/openpgm/pgm/packet_parse.c#76
It's actually quite annoying to detect for AutoConf, I think all packages have this hard coded on a per-platform basis. I've seen a bug report (header byte order config options detected incorrectly) raised this week on this very issue.
It is very unlikely that Mac itself is doing that. That would fundamentally break the IP protocol if it were. More likely whatever is capturing the packets and delivering them to recvfrom() (presumably you are doing a promiscuous network capture, right?) is what is transforming the data after Mac is done processing it. ireshark operates on a lower level and has access to the actual network data.