I ParseFromArray the protocol buffer's protocol, the protocol is not lack any filed. But the ParseFromArray function returns false. Why?
I'm assuming you are using C++. ParseFromArray() fails if:
The input data is not in valid protobuf format.
The input data is lacking a required field.
If you are sure that all required fields are set, then it must be the case that your input data is corrupted. You should verify that the bytes and size you are passing into ParseFromArray() are exactly the bytes and size that you got from SerializeToArray() and ByteSize() on the sending side. You will probably find that you are losing some bytes somewhere, or that some bytes got corrupted.
Common reasons for corruption include:
Passing the encoded bytes over a text-only channel. E.g. if you write the data to (or read it from) a file that is not opened in "binary" mode, or if you at some point store the bytes in a Java String, the data will become corrupted, as these channels expect text, and encoded protobufs are not text.
Passing the bytes as a char*, i.e. assuming NUL-termination. Encoded protobufs can contain '\0' bytes, meaning that you cannot represent one as a char* alone -- you must include the size separately.
Serializing to an array that is larger than needed, and then forgetting to pay attention to how much data was actually written. When you call SerializeToArray(), you must also call ByteSize() to see how large the message is, and you must make sure the receiving end receives that size and passes it to ParseFromArray(). Otherwise, the parser will think that the extra bytes at the end of the buffer are part of the message, and will fail to parse them.
Related
While trying to implement a code of encryption, I got this thing in terminal
cms/protocol: ASN.1 Error — unexpected trailing data
What does it signify?
I am not familiar with CMS or exactly what you are doing, but based on the "unexpected trailing data" message and my knowledge of ASN.1, I would say the error message suggests that there was more data available after the end of the ASN.1 data, which suggests that the data does not represent what it was claimed to represent. For example, if you have 10 bytes of data that are supposedly a PER encoding of type X, but after decoding a value of X out of that data, only 9 bytes were read, then those 10 bytes were not a PER encoding of X.
Title may be wildly incorrect for what I'm trying to work out.
I'm trying to interpret packets I am recieving from a racing game in a way that I understand, but I honestly don't really know what I'm looking at, or what to search to understand it.
Information on the packets I am recieving here:
https://forums.codemasters.com/topic/54423-f1%C2%AE-2020-udp-specification/?tab=comments#comment-532560
I'm using python to print the packets, here's a snippet of the output, which I don't understand how to interpret.
received message: b'\xe4\x07\x01\x03\x01\x07O\x90.\xea\xc2!7\x16\xa5\xbb\x02C\xda\n\x00\x00\x00\xff\x01\x00\x03:\x00\x00\x00 A\x00\x00\xdcB\xb5+\xc1#\xc82\xcc\x10\t\x00\xd9\x00\x00\x00\x00\x00\x12\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00$tJ\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01
I'm very new to coding, and not sure what my next step is, so a nudge in the right direction will help loads, thanks.
This is the python code:
import socket
UDP_IP = "127.0.0.1"
UDP_PORT = 20777
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(4096)
print ("received message:", data)
The website you link to is describing the data format. All data represented as a series of 1's and 0's. A byte is a series of 8 1's and 0's. However, just because you have a series of bytes doesn't mean you know how to interpret them. Do they represent a character? An integer? Can that integer be negative? All of that is defined by whoever crafted the data in the first place.
The type descriptions you see at the top are telling you how to actually interpret that series of 1's and 0's. When you see "unit8", that is an "unsigned integer that is 8 bits (1 byte) long". In other words, a positive number between 0 and 255. An "int8" on the other hand is an "8-bit integer", or a number that can be positive or negative (so the range is -128 to 127). The same basic idea applies to the *16 and *64 variants, just with 16 bits or 64 bits. A float represent a floating point number (a number with a fractional part, such as 1.2345), generally 4 bytes long. Additionally, you need to know the order to interpret the bytes within a word (left-to-right or right-to-left). This is referred to as the endianness, and every computer architecture has a native endianness (big-endian or little-endian).
Given all of that, you can interpret the PacketHeader. The easiest way is probably to use the struct package in Python. Details can be found here:
https://docs.python.org/3/library/struct.html
As a proof of concept, the following will interpret the first 24 bytes:
import struct
data = b'\xe4\x07\x01\x03\x01\x07O\x90.\xea\xc2!7\x16\xa5\xbb\x02C\xda\n\x00\x00\x00\xff\x01\x00\x03:\x00\x00\x00 A\x00\x00\xdcB\xb5+\xc1#\xc82\xcc\x10\t\x00\xd9\x00\x00\x00\x00\x00\x12\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00$tJ\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'
#Note that I am only taking the first 24 bytes. You must pass data that is
#the appropriate length to the unpack function. We don't know what everything
#else is until after we parse out the header
header = struct.unpack('<HBBBBQfIBB', data[:24])
print(header)
You basically want to read the first 24 bytes to get the header of the message. From there, you need to use the m_packetId field to determine what the rest of the message is. As an example, this particular packet has a packetId of 7, which is a "Car Status" packet. So you would look at the packing format for the struct CarStatus further down on that page to figure out how to interpret the rest of the message. Rinse and repeat as data arrives.
Update: In the format string, the < tells you to interpret the bytes as little-endian with no alignment (based on the fact that the documentation says it is little-endian and packed). I would recommend reading through the entire section on Format Characters in the documentation above to fully understand what all is happening regarding alignment, but in a nutshell it will try to align those bytes with their representation in memory, which may not match exactly the format you specify. In this case, HBBBBQ takes up 2 bytes more than you'd expect. This is because your computer will try to pack structs in memory so that they are word-aligned. Your computer architecture determines the word alignment (on a 64-bit computer, words are 64-bits, or 8 bytes, long). A Q takes a full word, so the packer will try to align everything before the Q to a word. However, HBBBB only requires 6 bytes; so, Python will, by default, pad an extra 2 bytes to make sure everything lines up. Using < at the front both ensures that the bytes will be interpreted in the correct order, and that it won't try to align the bytes.
Just for information if someone else is looking for this. In python there is the library f1-2019-telemetry existing. On the documentation, there is a missing part about the "how to use" so here is a snippet:
from f1_2020_telemetry.packets import *
...
udp_socket = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM)
udp_socket.bind((host, port))
while True:
udp_packet = udp_socket.recv(2048)
packet = unpack_udp_packet(udp_packet)
if isinstance(packet, PacketSessionData_V1): # refer to doc for classes / attribute
print(packet.trackTemperature) # for example
if isinstance(packet, PacketParticipantsData_V1):
for i, participant in enumerate(packet.participants):
print(DriverIDs[participant.driverId]) # the library has some mapping for pilot name / track name / ...
Regards,
Nicolas
If I have a definition which is only repeated strings, I can find the length of the packed buffers via the get_packed_size call. However, if I am on the receiving side of the exchange, how do I know how many bytes to read to form a complete message? (Since there are a variable number of entries, it isn't known apriori.)
Sender:
length = <name>_get_packed_size(&message)
buffer = malloc(length)
<name>_pack(&message, buffer)
write(fd, buffer, length)
Receiver:
read(fd, buffer, ???) // what is '???' if 'fd' is a stream socket?
If I am in datagram mode, I can issue the read for something like 64K bytes and just get the entire message. However, if I am in stream mode, how do I do this without short changing the message or reading part of the next message?
See this answer for a typical solution to this common problem: https://stackoverflow.com/a/5586945/618259
I use a small buffer (e.g. 128 bytes) and I want to use "async_read_until" with big incoming messages on the TCP connection (discarding all but last 128 bytes prior to the delimiter).
How can this be done? The ASIO docs are not very clear what happens when the provided buffer is not big enough.
Here is my code for the read initiazation
typedef boost::shared_ptr<boost::asio::streambuf >streambuf_ptr;
streambuf_ptr inBuf(new boost::asio::streambuf (128));
boost::asio::async_read_until(*sock, *inBuf, "\r\n\r\n", boost::bind(my_read_handler, sock, inBuf, boost::asio::placeholders::error, boost::asio::placeholders::bytes_transferred));
When the provided buffer is not big enough, async_read_until fills it in completely and then invokes the read handler with the error code asio::error::not_found, meaning that the delimiter was not found.
At that point you can .consume() some (or all) data from the buffer and call async_read_until again. It may be difficult to guarantee, with a 128-byte buffer, that when the delimiter is finally found, it is in the exact last position in the buffer (and even then, with a four-byte delimiter, you will only have the last 124 bytes prior to it). It may be best to use a larger buffer and buffer.consume(buffer.size()-128) in the not_found error handler, to make sure there's at least 128 bytes free at all time.
I'm processing Midi on the iPad and everything is working fine and I can log everything that comes in and all works as expected. However, in trying to recieve long messages (ie Sysex), I can only get one packet with a maximum of 256 bytes and nothing afterwards.
Using the code provided by Apple:
MIDIPacket *packet = &packetList->packet[0];
for (int i = 0; i > packetList->numPackets; ++i) {
// ...
packet = MIDIPacketNext (packet);
}
packetList->numPackets is always 1. After I get that first message, no other callback methods are called until a 'new' sysex message is sent. I don't think that my MIDI processing method would be called with the full packetList (which could potentially be any size). I would have thought I would recieve the data as a stream. Is this correct?
After digging around the only thing I could find was this: http://lists.apple.com/archives/coreaudio-api/2010/May/msg00189.html, which mentions the exact same thing but was not much help. I understand I probably need to implement buffering, but I can't even see anything past the first 256 bytes so I'm not sure where to even start with it.
My gut feeling here is that the system is either cramming the entire sysex message into one packet, or breaking it up into multiple packets. According to the CoreMidi documentation, the data field of the MIDIPacket structure has some interesting properties:
A variable-length stream of MIDI messages. Running status is not allowed. In the case of system-exclusive messages, a packet may only contain a single message, or portion of one, with no other MIDI events.
The MIDI messages in the packet must always be complete, except for system-exclusive.
(This is declared to be 256 bytes in length so clients don't have to create custom data structures in simple situations.)
So basically, you should look at the declared length field of the MIDIPacket and see if it is larger than 256. According to the spec, 256 bytes is just the standard allocation, but that array can hold more if necessary. You might find that the entire message has been crammed into that array.
Otherwise, it seems that the system is breaking the sysex messages up into multiple packets. Since the spec says that running status is not allowed, then it would have to send multiple packets, each with a leading 0xF0 byte. You would then need to create your own internal buffer to store the contents of these messages, stripping away the status bytes or header as necessary, and appending the data to your buffer until you read a 0xF7 byte which denotes the end of the sequence.
I had a similar issue on iOS. You are right MIDI packets number is always 1.
In my case, when receiving multiple MIDI events with the same timestamp (MIDI events received at the same time), iOS does not split those multiple MIDI events in multiple packets, as expected.
But, fortunately nothing is lost ! Indeed instead of receiving multiple packets with their correct number of bytes, you will receive a single packet with multiple events in it and the number of bytes will be increased accordingly.
So here what you have to do is:
In your MIDI IN callback, parse all packets received (always 1 for iOS), then for each packet received you must check the length of the packet as well as the MIDI status, then loop into that packet to retrieve all MIDI events in the current packet.
For instance, if the packet contains 9 bytes, and the MIDI status is a note ON (3 bytes message), that means your current packet contains more than a single note ON, you must then parse the first Note ON (bytes 0 to 2) then check the following MIDI status from byte 3 and so on ..
Hope this helps ...
Jerome
There is a good reference of how to walk through a MIDI packet in this file of a GitHub project : https://github.com/krevis/MIDIApps/blob/master/Frameworks/SnoizeMIDI/SMMessageParser.m
(Not mine, but it helped me solve the problems that got me to this thread)