bash/unix toolchain binary stream processing/slicing - bash

Question

I have a binary stream on standard input, it's in a fixed size format, a continuos stream of packets, each packet has a header with length X and a body with length Y.
So if X=2 Y=6 then it's something like 00abcdef01ghijkl02mnopqr03stuvwx, but it's binary and both the header and data can contain any "characters" (including '\0' and newline), the example is just for readability.
I want to get rid of the header data so the output looks like this: abcdefghijklmnopqrstuvwx.
Are there any commands in the unix toolchain that allow me to do this? And in general are there any tools for handling binary data? The only tool I could think of is od/hexdump but how do you convert the result back to binary?

Answer 1

Use xxd which goes to and from a hexdump.
xxd -c 123 -ps
will output your stream with 123 bytes per line. To reverse use
xxd -r -p
You should now be able to put this together with cut to drop characters since you can do something like
cut -c 3-
to get all characters from 3 to the end of a line. Do not forget to use a number of characters equal to 2X to account for two hex characters per byte.
So something along the lines of
xxd -c X+Y -ps | cut -c 2X+1- | xxd -r -p
where X+Y and 2X+1 are replaced with actual numerical values. You'll need to put your datastream somewhere appropriate in to the above command.

Answer 2

Perl is a pretty standard unix tool. Pipe it to perl. If its fixed length byte aligned a simple substr operation should work. Here is a perl sample that should work.
#!/usr/bin/env perl
use strict;
use warnings;
my $buf;
my $len = 8;
my $off = 2;
while(sysread(STDIN,$buf,$len) != 0 ){
print substr($buf,$off);
}
exit 0;

Answer 3

As a one-liner, I'd write:
perl -00 -ne 'chomp; while (/(?:..)(......)/sg) {print $1}'
example:
echo '00abcdef01ghijkl02mnopqr03stuvw
00abcdef01ghi
kl02mnopqr' | perl -00 -ne 'chomp; while (/(?:..)(......)/sg) {print $1}' | od -c
produces
0000000 a b c d e f g h i j k l m n o p
0000020 q r s t u v w \n a b c d e f g h
0000040 i \n k l m n o p q r
0000052

Answer 4

There's also bbe - binary block editor, which is kind of binary sed for handling binary data the Unix way.
http://bbe-.sourceforge.net

Answer 5

The binary stream editor is a tool written in java for handling streams.
It can be used from java as well as command line.
https://sourceforge.net/projects/bistreameditor/
DISCLAIMER : i am the author of this tool.
Unlike new-line based tools like sed, it allows custom traversing and data-storage via the traversal and buffer. Binary data can be treated as one byte chars and string operations/matches allowed. It can write to multiple outputs and use different encodings. Because of this flexibility, currently the command line has a lot of parameters, which needs to be simplified.
The bse.zip file should be downloaded and used.
For the above example, we would simply need to do a substr(2) on the input of len 8. The full command line is
java -classpath "./bin:$CMN_LIB_PATH/commons-logging-1.1.1.jar:$CMN_LIB_PATH/commons-io-2.1.jar:$CMN_LIB_PATH/commons-jexl-2.1.1.jar:$CMN_LIB_PATH/commons-lang3-3.1.jar"
-Dinputsrc=file:/fullpathtofile|URL|System.in
-Dtraverser=org.milunsagle.io.streameditor.FixedLengthTraverser
-Dtraversercons=size -Dtraverserconsarg0=8
-Dbuffer=org.milunsagle.io.streameditor.CircularBuffer
-Dbuffercons=size -Dbufferconsarg0=8
-Dcommands='PRN V $$__INPUT.substring(2)'
org.milunsagle.io.streameditor.BinaryStreamEditorInvoker