What is the most efficient protobuf type (in C++) for storing ipv4 or ipv6 address? My address is a boost::asio::ip::address_v4 (or v6) - c++11

I read that protobuf has a type called "bytes" which can store arbitrary number of bytes and is the equivalent of "C++ string". The reason why I don't prefer to use "bytes" is that it expects input as a C++ string i.e., boost IP will need to be converted to a string. Now my concern lies here : I want to perform serialize and send the encoded protobuf message over TCP socket. I want to ensure that the encoded message size is as small as possible.
Currently, I am using the below .proto file :
syntax = "proto2";
message profile
{
repeated **uint32** localEndpoint = 1;
repeated **uint32** remoteEndpoint = 2;
}
In order to save boost IP in the protobuf message, I am first converting boost IP into byte-format array by using "boost::asio::ip::address_v4::to_bytes()". So for a v4 IP, resultant array size is 4. Then I am converting 1st 4 bytes from the resultant byte-array into one uint32_t number and then storing in "localEndpoint" field of the protobuf message. Likewise, I repeat for the next 4 bytes (for v6). I am taking 4 bytes at a time so as to utilize full 32 bits of the uint32.
Hence for a v4 address, 1 occurrence of "localEndpoint" field is used.
Similarly, for a v6 address, 4 occurrence of "localEndpoint" field is used.
Please allow me to highlight that if I had used "bytes" here, my input string itself would have been of size 15 bytes for a v4 ip like 111.111.111.111
Using uint32 instead of "bytes" does save me some encoded-data-size but I am looking for a more efficient protobuf type requiring lesser number of bytes.
Sorry for a long description but I wanted to explain my query in details. Please help me.. Thanks a lot in advance :)

An ipv4 address should require exactly 4 bytes. If you're somehow getting 8, you're doing something wrong - are you perhaps hex-encoding it? You don't need that here. Likewise, ipv6 should be 16 bytes.
4 bytes with a usually-set high byte is most effectively stored as fixed32 - varint would be overhead here, due to the high bits. 16 bytes is more subtle - I'd go with bytes (field header plus length), since it is simpler to form into a union, and if the field-number is large, it avoids having to pay for multiple multi-byte field headers (a length prefix of 16 will always be single-byte).
I'd then create a union of these via oneof:
oneof ip_addr {
fixed32 v4 = 1;
bytes v6 = 2;
}

Related

UDP packet creation in goLang

I am trying to create a raw UDP packet. UDP packet consists of 3 parts:
MessageId
Payload
checksum
Message-ID is defined as:
The first 4 bytes are the client IP, the next 2 bytes are the port number, the next 2 bytes are randomly generated, and the last 8 bytes the time the request was generated (e.g., in nanoseconds). Alternatively, one could use UUIDs.
My questions is : Are there any libraries in Golang like which converts IP address into a byte array? If so, what I am assuming is that to create MessageID in my case, I need to have 4 bytes arrays corresponding to IP address, 2 bytes port number, 2 bytes random generated number, and 8 bytes long time stamp. Thereafter, I will merge all of them in one array to form 16 byte-long Message-ID. Please help me to know if my understanding is correct here or not?
I would highly appreciate the support.
Are there any libraries in Golang like which converts IP address into a byte array?
The Go type for an IP address is net.IP. A net.IP is a slice of bytes.
Use the builtin copy function to initialize a byte array from a byte slice:
var myArray [4]byte
copy(myArray[:], myIPAddress)

How do I interpret a python byte string coming from F1 2020 game UDP packet?

Title may be wildly incorrect for what I'm trying to work out.
I'm trying to interpret packets I am recieving from a racing game in a way that I understand, but I honestly don't really know what I'm looking at, or what to search to understand it.
Information on the packets I am recieving here:
https://forums.codemasters.com/topic/54423-f1%C2%AE-2020-udp-specification/?tab=comments#comment-532560
I'm using python to print the packets, here's a snippet of the output, which I don't understand how to interpret.
received message: b'\xe4\x07\x01\x03\x01\x07O\x90.\xea\xc2!7\x16\xa5\xbb\x02C\xda\n\x00\x00\x00\xff\x01\x00\x03:\x00\x00\x00 A\x00\x00\xdcB\xb5+\xc1#\xc82\xcc\x10\t\x00\xd9\x00\x00\x00\x00\x00\x12\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00$tJ\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01
I'm very new to coding, and not sure what my next step is, so a nudge in the right direction will help loads, thanks.
This is the python code:
import socket
UDP_IP = "127.0.0.1"
UDP_PORT = 20777
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(4096)
print ("received message:", data)
The website you link to is describing the data format. All data represented as a series of 1's and 0's. A byte is a series of 8 1's and 0's. However, just because you have a series of bytes doesn't mean you know how to interpret them. Do they represent a character? An integer? Can that integer be negative? All of that is defined by whoever crafted the data in the first place.
The type descriptions you see at the top are telling you how to actually interpret that series of 1's and 0's. When you see "unit8", that is an "unsigned integer that is 8 bits (1 byte) long". In other words, a positive number between 0 and 255. An "int8" on the other hand is an "8-bit integer", or a number that can be positive or negative (so the range is -128 to 127). The same basic idea applies to the *16 and *64 variants, just with 16 bits or 64 bits. A float represent a floating point number (a number with a fractional part, such as 1.2345), generally 4 bytes long. Additionally, you need to know the order to interpret the bytes within a word (left-to-right or right-to-left). This is referred to as the endianness, and every computer architecture has a native endianness (big-endian or little-endian).
Given all of that, you can interpret the PacketHeader. The easiest way is probably to use the struct package in Python. Details can be found here:
https://docs.python.org/3/library/struct.html
As a proof of concept, the following will interpret the first 24 bytes:
import struct
data = b'\xe4\x07\x01\x03\x01\x07O\x90.\xea\xc2!7\x16\xa5\xbb\x02C\xda\n\x00\x00\x00\xff\x01\x00\x03:\x00\x00\x00 A\x00\x00\xdcB\xb5+\xc1#\xc82\xcc\x10\t\x00\xd9\x00\x00\x00\x00\x00\x12\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00$tJ\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'
#Note that I am only taking the first 24 bytes. You must pass data that is
#the appropriate length to the unpack function. We don't know what everything
#else is until after we parse out the header
header = struct.unpack('<HBBBBQfIBB', data[:24])
print(header)
You basically want to read the first 24 bytes to get the header of the message. From there, you need to use the m_packetId field to determine what the rest of the message is. As an example, this particular packet has a packetId of 7, which is a "Car Status" packet. So you would look at the packing format for the struct CarStatus further down on that page to figure out how to interpret the rest of the message. Rinse and repeat as data arrives.
Update: In the format string, the < tells you to interpret the bytes as little-endian with no alignment (based on the fact that the documentation says it is little-endian and packed). I would recommend reading through the entire section on Format Characters in the documentation above to fully understand what all is happening regarding alignment, but in a nutshell it will try to align those bytes with their representation in memory, which may not match exactly the format you specify. In this case, HBBBBQ takes up 2 bytes more than you'd expect. This is because your computer will try to pack structs in memory so that they are word-aligned. Your computer architecture determines the word alignment (on a 64-bit computer, words are 64-bits, or 8 bytes, long). A Q takes a full word, so the packer will try to align everything before the Q to a word. However, HBBBB only requires 6 bytes; so, Python will, by default, pad an extra 2 bytes to make sure everything lines up. Using < at the front both ensures that the bytes will be interpreted in the correct order, and that it won't try to align the bytes.
Just for information if someone else is looking for this. In python there is the library f1-2019-telemetry existing. On the documentation, there is a missing part about the "how to use" so here is a snippet:
from f1_2020_telemetry.packets import *
...
udp_socket = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM)
udp_socket.bind((host, port))
while True:
udp_packet = udp_socket.recv(2048)
packet = unpack_udp_packet(udp_packet)
if isinstance(packet, PacketSessionData_V1): # refer to doc for classes / attribute
print(packet.trackTemperature) # for example
if isinstance(packet, PacketParticipantsData_V1):
for i, participant in enumerate(packet.participants):
print(DriverIDs[participant.driverId]) # the library has some mapping for pilot name / track name / ...
Regards,
Nicolas

Extract custom data format in TCP payload using golang

I have an industrial sensor which is connected to a Interface Module that provides information to the end user via telnet.
Telnet Configuration
Port: 10001
Data Format: Little-Endian
Data payload
Preamble - 32 bit
Article number - 32 bit
Serial number - 32 bit
Channels - 64 bit
Status - 32 bit
Frame number M / - 16 bit/ 16 bit
bytes per frame
Measuring value counter - 32 bit
Measuring value frame 1 - N * 32 bit
..
..
This should be available after the TCP headers (if I understand correctly) i.e. Payload.
I am trying to learn golang with more practical examples and I am thinking in the following direction:
Create a TCP server for listening to the sensor's Interface module IP and port number
Extract the payload and just print it out as seperate parameters like preamble, article number etc. as a beginners task
I have gone through some blog-posts about TCP server/client creations but they only address string in their payload. How can I extract such a byte orientated custom payload as mentioned above?
P.S.
I can understand that this can be achieved in python probably via struct.pack and struct.unpack hence some correlation as understanding would be appreciated
In section 6.2 of your linked pdf document above there is a table listing the data you seem to list in the question
I would convert that table into a struct and then read into it with the encoding/binary module. See https://golang.org/pkg/encoding/binary/#example_Read for a relevant example
"All measuring values are transmitted as Int32, Uint32 or Float depending on the connected sensor" probably easiest to get them as uint32 and then convert them later

using int64 type for snmp v2c oid?

I am debugging some snmp code for an integer overflow problem. Basically we use an integer to store disk/raid capacity in KB. However when a disk/raid of more than 2TB is used, it'll overflow.
I read from some internet forums that snmp v2c support integer64 or unsigned64. In my test it'll still just send the lower 32 bits even though I have set the type to integer64 or unsigned64.
Here is how I did it:
a standalone program will obtain the capacity and write the data to a file. example lines for raid capacity
my-sub-oid
Counter64
7813857280
/etc/snmp/snmpd.conf has a clause to pass thru the oids:
pass_persist mymiboid /path/to/snmpagent
in the mysnmpagent source, read the oidmap into oid/type/value structure from the file, and print to stdout.
printf("%s\n", it->first.c_str());
printf("%s\n", it->second.type.c_str());
printf("%s\n", it->second.value.c_str());
fflush(stdout);
use snmpget to get the sub-oid, and it returns:
mysuboid = Counter32: 3518889984
I use tcpdump and the last segment of the value portion is:
41 0500 d1be 0000
41 should be the tag, 05 should be the length, and the value is only carrying the lower 32-bit of the capacity. (note 7813857280 is 0x1.d1.be.00.00)
I do find that using string type would send correct value (in octetstring format). But I want to know if there is a way to use 64-bit integer in snmp v2c.
I am running NET-SNMP 5.4.2.1 though.
thanks a lot.
Update:
Found the following from snmpd.conf regarding pass (and probably also pass_persist) in net-snmp doc page. I guess it's forcing the Counter64 to Counter32.
Note:
The SMIv2 type counter64 and SNMPv2 noSuchObject exception are not supported.
You are supposed to use two Unsigned32 for lower and upper bytes of your large number.
Counter64 is not meant to be used for large numbers this way.
For reference : 17 Common MIB Design Errors (last one)
SNMP SMIv2 defines a new type Counter64,
https://www.rfc-editor.org/rfc/rfc2578#page-24
which is in fact unsigned 64 bit integer. So if your data fall into the range, using Counter64 is proper.
"In my test it'll still just send the lower 32 bits even though I have set the type to integer64 or unsigned64" sounds like a problem, but unless you show more details (like showing some code) on how you tested it out and received the result, nobody might help further.

How do I convert bytes to seconds?

I have a socket server listening for UDP packets from a GSM device. Some of the data comes in as multibytes, such as time, which requires multibytes for accuracy. This is an example:
179,248,164,14
The bytes are represented in decimal notation. My goal is to convert that into seconds:
245692595
I am trying to do that and was told:
"You must take those 4 bytes and place them into a single long integer in little endian format. If you are using Python to read and encode the data, you will need to look at using the .read() and struct.unpack() methods to successfully convert it to an integer. The resulting value is the number of seconds since 01/01/2000."
So, I tried to do this:
%w(179 248 164 14).sort.map(&:to_i).inject(&:+)
=> 605
And I obviously am getting the wrong answer.
You should use the pack and unpack methods to do this:
[179,248,164,14].pack('C*').unpack('I')[0]
# => 245692595
It's not about adding them together, though. You're doing the math wrong. The correct way to do it with inject is this:
[179,248,164,14].reverse.inject { |s,v| s * 256 + v }
# => 245692595
Note you will have to account for byte ordering when representing binary numbers that are more than one byte long.
If the original data is already a binary string you won't have to perform the pack operation and can proceed directly to the unpack phase.

Resources