Reasonably performant hex decoding in native bash? - bash

Question

I have a 2MB file which is a sequence of hex values delimited by spaces. For example:
3F 41 56 00 00
Easy peasy to do this in Bash:
cat hex.txt | tr -s " " $'\n' | while read a; do
echo $a | xxd -r -p | tee -a ascii
done
or
f=$(cat hex.txt)
for a in $f; do
echo $a | xxd -r -p | tee -a ascii
done
Both are excruciatingly slow.
I whipped up a C program which converted the file in about about two seconds and later realized that I could have just done this:
cat hex.txt | xxd -r -p
As I've already converted the file and found an optimal solution, my question isn't about the conversion process itself but rather how to optimize my first two attempts as if the third were not possible. Is there anything to be done to speed up these one-liners or is Bash just too slow for this?

Answer 1

Try the following - unfortunately, the solution varies by awk implementation used:
# BSD/OSX awk
xargs printf '0x%s ' < hex.txt | awk -v RS=' ' '{ printf "%c", $0 }' > ascii
# GNU awk; option -n needed to support hex. numbers
xargs printf '0x%s ' < hex.txt | awk -n -v RS=' ' '{ printf "%c", $0 }' > ascii
# mawk - sadly, printf "%c" only works with letters and numbers if the input is *hex*
awk -v RS=' ' '{ printf "%c", int(sprintf("%d", "0x" $0)) }' < hex.txt
With a 2MB input file, the timings on my late-2012 iMac with 3.2 GHz Intel Core i5 and a Fusion Drive, running OSX 10.10.3 are as follows:
BSD/OSX awk: ca. 1s
GNU awk: ca. 0.6s
mawk: ca. 0.5s
Contrast this with PSkocik's optimized-bash-loop solution: ca. 11s
It's tempting to think that the mawk solution, given that it's a single command without a pipeline, should be the faster solution with all awk implementations, but in practice it is not. Here's a version that works with all three implementations with -n for GNU awk provided on demand: awk $([[ $(gawk --version 2>/dev/null) = GNU* ]] && printf %s -n) -v RS=' ' '{ printf "%c", int(sprintf("%d", "0x" $0)) }' < hex.txt
The speed increase comes from avoiding bash loops altogether and letting utilities do the work:
xargs printf '0x%s ' < hex.txt prefixes all values in hex.txt with 0x so that awk will later recognize them as hexadecimals.
Note that, depending on your platform, the command line that xargs constructs with all stdin input tokens as arguments may exceed the maximum command-line length as reported by getconf ARG_MAX - fortunately, xargs is smart enough to then invoke the command multiple times, fitting as many arguments as possible on the command line each time.
awk -v RS=' ' '{ printf "%c", $0 }'
awk -v RS=' ' reads each space-separated token - i.e., each hex. value - as a separate input record
printf "%c", $0 then simply converts each record into its ASCII-character equivalent using printf.
Generally speaking:
Bash loops with large iteration counts are intrinsically slow.
It gets much worse if you also call an external utility in every iteration.
See Charles Duffy's comments below.
For good performance with large iteration counts, avoid bash loops and let external utilities do the iteration work.

Answer 2

It's slow because you're invoking two programs,
xxd and tee,
in each iteration of the loop.
Using the printf builtin should be more loop-friendly, and you only need one instance of tee:
tr -s " " '\n' < hex.txt |
while read seq; do printf "\x$seq"; done |
tee -a ascii
(You might not be needing the -a switch to tee anymore).
(
If you want to use a use a scripting language, ruby is another good choice beside awk:
tr -s " " '\n' < hex.txt | ruby -pe '$_ = $_.to_i(16).chr'
(Much faster than the in-bash version).
)

Answer 3

Well, you can drop the first cat and replace it by tr < hex.txt. Then you can also build a static conversion table and drop echo and xxd. But the loop will still be slow, and you can't get rid of that, I think.