Reading binary data - firemonkey

I am trying to read data from a binary file. One block of data is 76 bytes long (this varies with the number of the 2-byte "main data items" in the middle of the block). The first datum is 4 bytes, second is 4 bytes, and then there are a bunch of 2 byte main data items, and at the end are 2 more 2-byte pieces of data.
Based on this Delphi sample I've learned how to read the file with the code below:
short AShortInt; // 16 bits
int AInteger; // 32 bits
try
{
infile=new TFileStream(myfile,fmOpenRead); // myfile is binary
BR = new TBinaryReader(infile, TEncoding::Unicode, false);
for (int rows = 0; rows < 5; rows++) { // just read the first 5 blocks of data for testing
AInteger = BR->ReadInt32(); // read first two 4 byte integers for this block
AInteger = BR->ReadInt32();
for (int i = 0; i < 32; i++) { // now read the 32 2-byte integers from this block
AShortInt = BR->ReadInt16();
}
AShortInt = BR->ReadInt16(); // read next to last 2-byte int
AShortInt = BR->ReadInt16(); // read the last 2-byte int
}
delete infile;
delete BR;
Close();
}
catch(...)
{
delete infile; // closes the file, doesn't delete it.
delete BR;
ShowMessage("Can't open file!");
Close();
}
But, what i would like to do is use a 76-byte wide buffer to read the entire block, and then pick the various datum out of that buffer. I put together the following code based on this question and i can read a whole block of data into the buffer.
UnicodeString myfile = System::Ioutils::TPath::Combine(System::Ioutils::TPath::GetDocumentsPath(), "binaryCOM.dat");
TFileStream*infile=0;
try
{
infile=new TFileStream(myfile,fmOpenRead);
const int bufsize=76;
char*buf=new char[bufsize];
int a = 0;
while(int bytesread=infile->Read(buf,bufsize)) {
a++; // just a place to break on Run to Cursor
}
delete[]buf;
}
catch(...)
{
delete infile;
ShowMessage("Can't open file!");
Close();
}
But i can't figure out how to piece together subsets out of the bytes in the buffer. Is there a way to concatenate bytes? So i could read a block of data into a 76 byte buffer and then do something like this below?
unsigned int FirstDatum = buf[0]+buf[1]+buf[2]+buf[3]; // concatenate the 4 bytes for the first piece of data
This will be an FMX app for Win32, iOS, and Android built in C++Builder 10.3.2.

Here is my modified code using Remy's suggestion of TMemoryStream.
UnicodeString myfile = System::Ioutils::TPath::Combine(System::Ioutils::TPath::GetDocumentsPath(), "binaryCOM.dat");
TMemoryStream *MS=0;
TBinaryReader *BR=0;
std::vector<short> myArray;
short AShortInt;
int AInteger;
int NumDatums = 32; // the variable number of 2-byte main datums
try
{
MS = new TMemoryStream();
MS->LoadFromFile(myfile);
BR = new TBinaryReader(MS, TEncoding::Unicode, false);
for (int rows = 0; rows < 5; rows++) { // testing with first 5 blocks of data
AInteger = BR->ReadInt32(); // read first two 4 byte integers
AInteger = BR->ReadInt32(); // here
for (int i = 0; i < NumDatums; i++) { // read the main 2-byte data
AShortInt = BR->ReadInt16();
myArray.push_back(AShortInt); // push it into vector
}
AShortInt = BR->ReadInt16(); // read next to last 2-byte int
AShortInt = BR->ReadInt16(); // read the last 2-byte int
// code here to do something with this block of data just read from file
}
}
delete MS;
delete BR;
}
catch(...)
{
delete MS;
delete BR;
ShowMessage("Can't open file.");
}

Related

Split encrypted messages into chunks and put them together again

I want to send GPG encrypted data via GET request of known format.
Issue #1: Data block size in the request is limited (4096 symbols), and it is not enough for a typical GPG message. So, I need to chunk it.
Issue #2: Chunks may be sent in the wrong order. Each chunk must have a unique message ID and serial number, so the messages can be put together.
GPG has the method to send encrypted data in text format (armoring). RFC 2440 standard allows chunking armored messages:
BEGIN PGP MESSAGE, PART X/Y
Used for multi-part messages, where the armor is split amongst Y
parts, and this is the Xth part out of Y.
BEGIN PGP MESSAGE, PART X
Used for multi-part messages, where this is the Xth part of an
unspecified number of parts. Requires the MESSAGE-ID Armor Header
to be used.
But, unfortunately, I've found no evidence that this feature is implemented in GPG.
And no word about chunking of public keys, which, actually, can be huge too.
So I turned down the idea of using native GPG armors for chunking.
My current home-made solution: binary encrypted data are splitted into chunks, then each chunk is put into a block, which contains UUID (MessageID analog), the serial number of the block, the total number of blocks, and CRC checksum of the block.
Like that:
[ UUID ][ Number ][ Total ][ Chunk of encrypted data ][ Checksum ]
Putting the message together out of that blocks is a bigger challenge, but doable as well.
But I want more clear solution, preferably on C++.
Could you help me?
Qt provides very simple methods for data serialization. I created a class to chunk, store, and rebuild binary data, and for now I don't think I need something more simple.
But, if someone knows a better solution, please share it with me.
#include <QByteArrayView>
#include <QDataStream>
#include <QException>
#include <QUuid>
enum CHUNKER {
MESSAGE_READY = 0,
BLOCK_ADDED
};
struct ChunkedMessage {
QUuid UUID;
QByteArray Data;
};
class Chunker {
public:
Chunker();
~Chunker();
static quint16 GetChecksum(QByteArray *Block);
static QByteArrayList ArmorData(QByteArray *Data, qsizetype *ChunkSize);
CHUNKER AddBlock(QByteArray *Block, ChunkedMessage *Message);
private:
struct MessageBlock {
QUuid UUID;
quint32 Number;
quint32 Total;
QByteArray Data;
};
QMap<QUuid, quint32> Sizes;
QMap<QUuid, QMap<quint32, Chunker::MessageBlock>*> Stack;
MessageBlock DearmorChunk(QByteArray *Block);
bool CheckIntegrity(QUuid *UUID, QByteArray *Reconstructed);
};
Chunker::Chunker() { }
Chunker::~Chunker() { }
quint16 Chunker::GetChecksum(QByteArray *Block) { return qChecksum(QByteArrayView(*Block), Qt::ChecksumIso3309); }
QByteArrayList Chunker::ArmorData(QByteArray *Data, qsizetype *ChunkSize) {
QByteArrayList Result;
QUuid UUID = QUuid::createUuid();
qsizetype RealChunkSize = (*ChunkSize) - sizeof(UUID.toRfc4122()) - sizeof(quint32) - sizeof(quint32) - sizeof(quint16);
const quint32 ChunkCount = ((*Data).length() / RealChunkSize) + 1;
for (auto Pos = 0; Pos < ChunkCount; Pos++) {
QByteArray Block;
QDataStream Stream(&Block, QIODeviceBase::WriteOnly);
Stream << UUID.toRfc4122() << (Pos + 1) << ChunkCount << (*Data).mid(Pos * RealChunkSize, RealChunkSize);
Stream << Chunker::GetChecksum(&Block);
Result.push_back(Block);
}
return Result;
}
Chunker::MessageBlock Chunker::DearmorChunk(QByteArray *Block) {
Chunker::MessageBlock Result;
QDataStream Stream(Block, QIODeviceBase::ReadOnly);
QByteArray ClearBlock = (*Block).chopped(sizeof(quint16));
QByteArray BytesUUID;
quint16 Checksum;
Stream >> BytesUUID >> Result.Number >> Result.Total >> Result.Data >> Checksum;
Result.UUID = QUuid::fromRfc4122(QByteArrayView(BytesUUID));
if (Chunker::GetChecksum(&ClearBlock) != Checksum) throw std::runtime_error("Checksums are not equal");
return Result;
}
bool Chunker::CheckIntegrity(QUuid *UUID, QByteArray *Reconstructed) {
quint32 Size = this->Sizes[*UUID];
if (this->Stack[*UUID]->size() > Size) throw std::runtime_error("Corrupted message blocks");
if (this->Stack[*UUID]->size() < Size) return false;
for (quint32 Counter = 0; Counter < Size; Counter++) {
if (!(this->Stack[*UUID]->contains(Counter + 1))) return false;
(*Reconstructed).append((*(this->Stack[*UUID]))[Counter + 1].Data);
}
return true;
}
CHUNKER Chunker::AddBlock(QByteArray *Block, ChunkedMessage *Message) {
Chunker::MessageBlock DecodedBlock = Chunker::DearmorChunk(Block);
if (!this->Sizes.contains(DecodedBlock.UUID)) {
this->Sizes[(QUuid)DecodedBlock.UUID] = (quint32)DecodedBlock.Total;
this->Stack[(QUuid)DecodedBlock.UUID] = new QMap<quint32, Chunker::MessageBlock>;
}
(*(this->Stack[DecodedBlock.UUID]))[(quint32)(DecodedBlock.Number)] = Chunker::MessageBlock(DecodedBlock);
QByteArray ReconstructedData;
if (this->CheckIntegrity(&DecodedBlock.UUID, &ReconstructedData)) {
(*Message).UUID = (QUuid)(DecodedBlock.UUID);
(*Message).Data = (QByteArray)ReconstructedData;
this->Sizes.remove(DecodedBlock.UUID);
delete this->Stack[DecodedBlock.UUID];
this->Stack.remove(DecodedBlock.UUID);
return CHUNKER::MESSAGE_READY;
}
return CHUNKER::BLOCK_ADDED;
}

How to convert a byte array into a string in CAPL?

I have a byte array and I need to print the elements in a single line.
I tried using 'snprintf()' but it won't take a byte array as its input parameter.
I tried copying the byte array into an integer array and then used the snprintf(), but instead of printing the HEX values, corresponding ASCII values are printed.
You can try this code :
variables
{
int ar[100];
}
on diagResponse TCCM.*
{
char tmp[8]; // Temporary buffer containing single HEX value
char out[301]; // Bigger output string and "local" to function
// Better to place them there (i and response) if they are not global
int i;
byte response[100];
out[0] = 0; // Clear output string
s1 = DiagGetPrimitiveData(this, response, elcount(response));
for (i = 0; i < s1; i++)
{
ar[i] = response[i];
snprintf(tmp, elcount(tmp), "%.2X ", response[i]); // byte to HEX convert
strncat(out, tmp, elcount(out)); // Concatenate HEX value to output string
}
write("HEX Response : %s", out);
}
Olivier

SAFEARRAY data to unsigned char*

I am trying to convert a SAFEARRAY data pointer to unsinged char*. However I am not getting the expected data. Here is a snippet.
SafeArrayLock(psaFrameData);
psaFrameData->rgsabound->cElements;
int nCount = psaFrameData->rgsabound->cElements - psaFrameData->rgsabound->lLbound + 1;
frameData = new unsigned char[nCount];
memset(frameData, 0, nCount);
for (int i = 0; i < nCount; ++i)
{
frameData[i] = ((unsigned char*)(psaFrameData)->pvData)[i];
}
SafeArrayUnlock(psaFrameData);
Do not manually lock the array and then access its pvData (or any of its other data members) directly. Use the various accessors functions instead, such as SafeArrayAccessData():
Increments the lock count of an array, and retrieves a pointer to the array data.
Try something more like this:
// safety check: make sure the array has only 1 dimension...
if (SafeArrayGetDim(psaFrameData) != 1)
{
// handle the error ...
}
else
{
// safety check: make sure the array contains byte elements...
VARTYPE vt = 0;
SafeArrayGetVartype(psaFrameData, &vt);
if (vt != VT_UI1)
{
// handle the error ...
}
else
{
// get a pointer to the array's byte data...
unsigned char *data;
if (FAILED(SafeArrayAccessData(psaFrameData, (void**)&data)))
{
// handle the error ...
}
else
{
// calculate the number of bytes in the array...
LONG lBound, uBound;
SafeArrayGetLBound(psaFrameData, 1, &lBound);
SafeArrayGetUBound(psaFrameData, 1, &uBound);
long nCount = uBound - lBound + 1;
// copy the bytes...
frameData = new unsigned char[nCount];
memcpy(frameData, data, nCount);
// release the pointer to the array's byte data...
SafeArrayUnaccessData(psaFrameData);
}
}
}

Non-blockings reads/writes to stdin/stdout in C on Linux or Mac

I have two programs communicating via named pipes (on a Mac), but the buffer size of named pipes is too small. Program 1 writes 50K bytes to pipe 1 before reading pipe 2. Named pipes are 8K (on my system) so program 1 blocks until the data is consumed. Program 2 reads 20K bytes from pipe 1 and then writes 20K bytes to pipe2. Pipe2 can't hold 20K so program 2 now blocks. It will only be released when program 1 does its reads. But program 1 is blocked waiting for program 2. deadlock
I thought I could fix the problem by creating a gasket program that reads stdin non-blocking and writes stdout non-blocking, temporarily storing the data in a large buffer. I tested the program using cat data | ./gasket 0 | ./gasket 1 > out, expecting out to be a copy of data. However, while the first invocation of gasket works as expected, the read in the second program returns 0 before all the data is consumed and never returns anything other than 0 in follow on calls.
I tried the code below both on a MAC and Linux. Both behave the same. I've added logging so that I can see that the fread from the second invocation of gasket starts getting no data even though it has not read all the data written by the first invocation.
#include <stdio.h>
#include <fcntl.h>
#include <time.h>
#include <stdlib.h>
#include <unistd.h>
#define BUFFER_SIZE 100000
char buffer[BUFFER_SIZE];
int elements=0;
int main(int argc, char **argv)
{
int total_read=0, total_write=0;
FILE *logfile=fopen(argv[1],"w");
int flags = fcntl(fileno(stdin), F_GETFL, 0);
fcntl(fileno(stdin), F_SETFL, flags | O_NONBLOCK);
flags = fcntl(fileno(stdout), F_GETFL, 0);
fcntl(fileno(stdout), F_SETFL, flags | O_NONBLOCK);
while (1) {
int num_read=0;
if (elements < (BUFFER_SIZE-1024)) { // space in buffer
num_read = fread(&buffer[elements], sizeof(char), 1024, stdin);
elements += num_read;
total_read += num_read;
fprintf(logfile,"read %d (%d) elements \n",num_read, total_read); fflush(logfile);
}
if (elements > 0) { // something in buffer that we can write
int num_written = fwrite(&buffer[0],sizeof(char),elements, stdout); fflush(stdout);
total_write += num_written;
fprintf(logfile,"wrote %d (%d) elements \n",num_written, total_write); fflush(logfile);
if (num_written > 0) { // copy data to top of buffer
for (int i=0; i<(elements-num_written); i++) {
buffer[i] = buffer[i+num_written];
}
elements -= num_written;
}
}
}
}
I guess I could make the gasket multi-threaded and use blocking reads in one thread and blocking writes in the other, but I would like to understand why non-blocking IO seems to break for me.
Thanks!
My general solution to any IPC project is to make the client and server non-blocking I/O. To do so requires queuing data both on writing and reading, to handle cases where the OS can't read/write, or can only read/write a portion of your message.
The code below will probably seem like EXTREME overkill, but if you get it working, you can use it the rest of your career, whether for named pipes, sockets, network, you name it.
In pseudo-code:
typedef struct {
const char* pcData, * pcToFree; // pcData may no longer point to malloc'd region
int iToSend;
} DataToSend_T;
queue of DataToSend_T qdts;
// Caller will use malloc() to allocate storage, and create the message in
// that buffer. MyWrite() will free it now, or WritableCB() will free it
// later. Either way, the app must NOT free it, and must not even refer to
// it again.
MyWrite( const char* pcData, int iToSend ) {
iSent = 0;
// Normally the OS will tell select() if the socket is writable, but if were hugely
// compute-bound, then it won't have a chance to. So let's call WritableCB() to
// send anything in our queue that is now sendable. We have to send the data in
// order, of course, so can't send the new data until the entire queue is done.
WritableCB();
if ( qdts has no entries ) {
iSent = write( pcData, iToSend );
// TODO: check error
// Did we send it all? We're done.
if ( iSent == iToSend ) {
free( pcData );
return;
}
}
// OK, either 1) we had stuff queued already meaning we can't send, or 2)
// we tried to send but couldn't send it all.
add to queue qdts the DataToSend ( pcData + iSent, pcData, iToSend - iSent );
}
WritableCB() {
while ( qdts has entries ) {
DataToSend_T* pdts = qdts head;
int iSent = write( pdts->cData, pdts->iToSend );
// TODO: check error
if ( iSent == pdts->iToSend ) {
free( pdts->pcToFree );
pop the front node off qdts
else {
pdts->pcData += iSent;
pdts->iToSend -= iSent;
return;
}
}
}
// Off-subject but I like a TINY buffer as an original value, that will always
// exercise the "buffer growth" code for almost all usage, so we're sure it works.
// If the initial buffer size is like 1M, and almost never grows, then the grow code
// may be buggy and we won't know until there's a crash years later.
int iBufSize = 1, iEnd = 0; iEnd is the first byte NOT in a message
char* pcBuf = malloc( iBufSize );
ReadableCB() {
// Keep reading the socket until there's no more data. Grow buffer if necessary.
while (1) {
int iRead = read( pcBuf + iEnd, iBufSize - iEnd);
// TODO: check error
iEnd += iRead;
// If we read less than we had space for, then read returned because this is
// all the available data, not because the buffer was too small.
if ( iRead < iBufSize - iEnd )
break;
// Otherwise, double the buffer and try reading some more.
iBufSize *= 2;
pcBuf = realloc( pcBuf, iBufSize );
}
iStart = 0;
while (1) {
if ( pcBuf[ iStart ] until iEnd-1 is less than a message ) {
// If our partial message isn't at the front of the buffer move it there.
if ( iStart ) {
memmove( pcBuf, pcBuf + iStart, iEnd - iStart );
iEnd -= iStart;
}
return;
}
// process a message, and advance iStart by the size of that message.
}
}
main() {
// Do your initial processing, and call MyWrite() to send and/or queue data.
while (1) {
select() // see man page
if ( the file handle is readable )
ReadableCB();
if ( the file handle is writable )
WritableCB();
if ( the file handle is in error )
// handle it;
if ( application is finished )
exit( EXIT_SUCCESS );
}
}

Extract trailing int from string containing other characters

I have a problem in regards of extracting signed int from string in c++.
Assuming that i have a string of images1234, how can i extract the 1234 from the string without knowing the position of the last non numeric character in C++.
FYI, i have try stringstream as well as lexical_cast as suggested by others through the post but stringstream returns 0 while lexical_cast stopped working.
int main()
{
string virtuallive("Images1234");
//stringstream output(virtuallive.c_str());
//int i = stoi(virtuallive);
//stringstream output(virtuallive);
int i;
i = boost::lexical_cast<int>(virtuallive.c_str());
//output >> i;
cout << i << endl;
return 0;
}
How can i extract the 1234 from the string without knowing the position of the last non numeric character in C++?
You can't. But the position is not hard to find:
auto last_non_numeric = input.find_last_not_of("1234567890");
char* endp = &input[0];
if (last_non_numeric != std::string::npos)
endp += last_non_numeric + 1;
if (*endp) { /* FAILURE, no number on the end */ }
auto i = strtol(endp, &endp, 10);
if (*endp) {/* weird FAILURE, maybe the number was really HUGE and couldn't convert */}
Another possibility would be to put the string into a stringstream, then read the number from the stream (after imbuing the stream with a locale that classifies everything except digits as white space).
// First the desired facet:
struct digits_only: std::ctype<char> {
digits_only(): std::ctype<char>(get_table()) {}
static std::ctype_base::mask const* get_table() {
// everything is white-space:
static std::vector<std::ctype_base::mask>
rc(std::ctype<char>::table_size,std::ctype_base::space);
// except digits, which are digits
std::fill(&rc['0'], &rc['9'], std::ctype_base::digit);
// and '.', which we'll call punctuation:
rc['.'] = std::ctype_base::punct;
return &rc[0];
}
};
Then the code to read the data:
std::istringstream virtuallive("Images1234");
virtuallive.imbue(locale(locale(), new digits_only);
int number;
// Since we classify the letters as white space, the stream will ignore them.
// We can just read the number as if nothing else were there:
virtuallive >> number;
This technique is useful primarily when the stream contains a substantial amount of data, and you want all the data in that stream to be interpreted in the same way (e.g., only read numbers, regardless of what else it might contain).

Resources