Splitting a binary file on binary delimiter? - bash

Question

I'm working on a shell script to convert MPO stereographic 3D images into standard JPEG images. A MPO file is just two JPEG images, concatenated together.
As such, you can split out the JPEG files by finding the byte offset of the second JPEG's magic number header (0xFFD8FFE1). I've done this manually using hexdump/xxd, grep, head, and tail.
The problem here is grep: what can I use to search a binary directly for a specific magic number, and get back a byte offset? Or should I not use a shell script for this at all? Thanks.

Answer 1

You can do this using bbe (http://bbe-.sourceforge.net/) which is a sed like program for binary files:
In order to extract the first JPEG use:
bbe -b '/\xFF\xD8\xFF\xE1/:' -e 'D 2' -o first_jpeg mpo_file
And for the second one:
bbe -b '/\xFF\xD8\xFF\xE1/:' -e 'D 1' -o second_jpeg mpo_file
Note that this will not work if the JPEG's magic number occurs somewhere else in the MPO file.

Answer 2

I think that Bart is on to your biggest problem.. If that binary sequence repeats during the process, you will get partial JPEGs.
I did a quick test by concatenating some JPEGs and then extracting them with awk (please note that the magic number in my files ended in 0xE0 and not 0xE1):
# for i in *.jpg ; do cat $i ; done > test.mpo
# awk 'BEGIN {RS="\xFF\xD8\xFF\xE0"; FILENUM=-1} {FILENUM++; if (FILENUM == 0) {next}; FILENAME="image0"FILENUM".jpg"; printf "%s",RS$0 > FILENAME;}' test.mpo
# file image0*.jpg
image01.jpg: JPEG image data, JFIF standard 1.01
image010.jpg: JPEG image data, JFIF standard 1.01
image011.jpg: JPEG image data, JFIF standard 1.01
This seemed to work ok for me, but the above mentioned issues are still unhandled and very real.

Answer 3

I've found a much better explanation of MPO file structure (and how to process it correctly) at http://www.davidglover.org/2010/09/using-the-fuji-finepix-real-3d-w3-camera-on-a-mac-or-linuxunix.html
Edit, October 2019:
Since the blog entry now 404s, here is the script that I wrote based on it. I haven't used it in many years.
#!/usr/bin/env bash
# Script to convert 3D MPO files, as used in the Fuji FinePix series of 3D cameras, into standard JPEG files.
# Based on work by David Glover, posted at http://www.davidglover.org/2010/09/using-the-fuji-finepix-real-3d-w3-camera-on-a-mac-or-linuxunix.html
# This script requires exiftool and ImageMagick.
FULLNAME="$1"
FILENAME="$(basename $FULLNAME)"
DIRNAME="$(dirname $FULLNAME)"
BASENAME="${FILENAME%.*}"
# Create output directories
mkdir -p "$DIRNAME"/stereoscopic-rl/
mkdir -p "$DIRNAME"/stereoscopic-mpo/
mkdir -p "$DIRNAME"/stereoscopic-anaglyph/
mkdir -p "$DIRNAME"/monoscopic-l/
mkdir -p "$DIRNAME"/monoscopic-r/
# Create separate left and right images
exiftool -trailer:all= "$FULLNAME" -o "$DIRNAME"/monoscopic-l/"$BASENAME"-left.jpg
exiftool "$FULLNAME" -mpimage2 -b > "$DIRNAME"/monoscopic-r/"$BASENAME"-right.jpg
# Move the MPO file to its new home
mv "$FULLNAME" "$DIRNAME"/stereoscopic-mpo/
# Determine parallax value and create cropped images for stereo generation
# 36 is only appropriate for 4:3 or 3:2 images
parallax=$(exiftool -b -Parallax "$DIRNAME"/monoscopic-r/"$BASENAME"-right.jpg)
parallax=$(echo "$parallax"*36+0.5 | bc | cut -d . -f 1)
# The above pipeline can't deal with a parallax of zero
# In theory, this fix doesn't cover values between zero and -1
# TODO improve the calculation
if [ ! $parallax ]; then
parallax=0
fi
echo $parallax
if [ $parallax -ge 0 ]; then
convert "$DIRNAME"/monoscopic-l/"$BASENAME"-left.jpg -crop +"$parallax"+0 "$DIRNAME"/monoscopic-l/"$BASENAME"-left-cropped.jpg
convert "$DIRNAME"/monoscopic-r/"$BASENAME"-right.jpg -crop -"$parallax"+0 "$DIRNAME"/monoscopic-r/"$BASENAME"-right-cropped.jpg
else
convert "$DIRNAME"/monoscopic-l/"$BASENAME"-left.jpg -crop -"$((-1*$parallax))"+0 "$DIRNAME"/monoscopic-l/"$BASENAME"-left-cropped.jpg
convert "$DIRNAME"/monoscopic-r/"$BASENAME"-right.jpg -crop +"$((-1*$parallax))"+0 "$DIRNAME"/monoscopic-r/"$BASENAME"-right-cropped.jpg
fi
# Create stereoscopic images for cross-eye (right-left) and anaglyph (red-cyan) viewing
convert "$DIRNAME"/monoscopic-r/"$BASENAME"-right-cropped.jpg "$DIRNAME"/monoscopic-l/"$BASENAME"-left-cropped.jpg +append "$DIRNAME"/stereoscopic-rl/"$BASENAME"-stereoscopic-rl.jpg
composite -stereo 0 "$DIRNAME"/monoscopic-r/"$BASENAME"-right-cropped.jpg "$DIRNAME"/monoscopic-l/"$BASENAME"-left-cropped.jpg "$DIRNAME"/stereoscopic-anaglyph/"$BASENAME"-stereoscopic-anaglyph.jpg
# Clean up separated parallax-corrected images
rm "$DIRNAME"/monoscopic-l/"$BASENAME"-left-cropped.jpg
rm "$DIRNAME"/monoscopic-r/"$BASENAME"-right-cropped.jpg
exit 0

Answer 4

I think a very simple home brew approach will be your best bet. The code for doing this would be very small, depending on all the special cases of your binary file format.
Use mmap to get a convenient view of your file in memory.
Start scanning, and save the byte-offset in a variable, say start.
Scan until you reach your delimiter, saving the ending offset, in say end.
Create a new file
Memory-map the new file
Copy the byte-range from start to end into the new file.
Close the new file and start scanning again.

Answer 5

FFE1 is not part of the some jpeg "magic number", it's the APP1 marker. And it's not guaranteed to come right after the SOI marker FFD8. Also, you should be careful that some jpeg images embed a thumbnail jpeg in an EXIF block. That will most likely also contain an APP1 marker.