I have a file with binary data and I need to replace a few bytes in a certain position. I've come up with the following to direct bash to the offset and show me that it found the place I want:
dd bs=1 if=file iseek=24 conv=block cbs=2 | hexdump
Now, to use "file" as the output:
echo anInteger | dd bs=1 of=hextest.txt oseek=24 conv=block cbs=2
This seems to work just fine, I can review the changes made in a hex editor. Problem is, "anInteger" will be written as the ASCII representation of that integer (which makes sense) but I need to write the binary representation.
I want to use bash for this and the script should run on as many systems as possible (I don't know if the target system will have python or whatever installed).
How do I tell the command to convert the input to binary (possibly from a hex)?
printf is more portable than echo. This function takes a decimal integer and outputs a byte with that value:
echobyte () {
if (( $1 >= 0 && $1 <= 255 ))
then
printf "\\x$(printf "%x" $1)"
else
printf "Invalid value\n" >&2
return 1
fi
}
$ echobyte 97
a
$ for i in {0..15}; do echobyte $i; done | hd
00000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
00000010
You can use echo to emit specific bytes using hex or octal. For example:
echo -n -e \\x30
will print ascii 0 (0x30)
(-n remove trailing newline)
xxd is the better way. xxd -r infile outfile will take ascii hex-value in infile to patch outfile, and you can specify the specific position in infile by this: 1FE:55AA
Worked like a treat. I used the following code to replace 4 bytes at byte 24 in little endian with two integers (1032 and 1920). The code does not truncate the file.
echo -e \\x08\\x04\\x80\\x07 | dd of=<file> obs=1 oseek=24 conv=block,notrunc cbs=4
Thanks again.
I have a function to do this:
# number representation from 0 to 255 (one char long)
function chr() { printf "\\$(printf '%03o' "$1")" ; return 0 ; }
# from 0 to 65535 (two char long)
function word_litleendian() { chr $(($1 / 256)) ; chr $(($1 % 256)) ; return 0 ; }
function word_bigendian() { chr $(($1 % 256)) ; chr $(($1 / 256)) ; return 0 ; }
# from 0 to 4294967295 (four char long)
function dword_litleendian() { word_lilteendian $(($1 / 65536)) ; word_litleendian $(($1 % 65536)) ; return 0 ; }
function dword_bigendian() { word_bigendian $(($1 / 65536)) ; word_bigendian $(($1 % 65536)) ; return 0 ; }
You can use piping or redirection to catch the result.
If you're willing to rely on bc (which is fairly common)
echo -e "ibase=16\n obase=2 \n A1" | bc -q
might help.
With bash, "printf" has the "-v" option, and all shell has logical operators.
So here is simplier form in bash :
int2bin() {
local i=$1
local f
printf -v f '\\x%02x\\x%02x\\x%02x\\x%02x' $((i&255)) $((i >> 8 & 255)) $((i >> 16 & 255)) $((i >> 24 & 255))
printf "$f"
}
You might put the desired input into a file and use the "if=" option to dd to insert exactly the input you desire.
In my case, I needed to go from a decimal numeric argument to the actual unsigned 16-bit big endian value. This is probably not the most efficient way, but it works:
# $1 is whatever number (0 to 65535) the caller specifies
DECVAL=$1
HEXSTR=`printf "%04x" "$DECVAL"`
BYTEONE=`echo -n "$HEXSTR" | cut -c 1-2`
BYTETWO=`echo -n "$HEXSTR" | cut -c 3-4`
echo -ne "\x$BYTEONE\x$BYTETWO" | dd of="$FILENAME" bs=1 seek=$((0xdeadbeef)) conv=notrunc
Related
I need to pad NUL bytes at the end of a byte stream exceeding available storage & memory, so output length is divisible by N. Context of the function I am implementing:
#!/bin/sh
generate_arbitrary_length | paddingN | work_with_padded
Working code for N=8192:
padding8192(){ dd status=none bs=8192 conv=sync ; }
But reducing copy block size is orders of magnitude slower for small N, this did not finish:
padding4(){ dd status=none bs=4 conv=sync ; }
I can express the counting & padding using wc and dd, after duplicating the input stream:
padding4(){ { { tee /dev/fd/3 >&2 ; } 3>&1 | wc -c | { read -r isize ; pad=$(( 4 - isize % 4)) ; [ 0 -lt $pad ] && dd status=none if=/dev/zero bs=$pad count=1 >&2 ; } } 2>&1 ; }
Much faster already. But very difficult to read - who could even tell why padding ends up at EOF?
Any better approach?
Though I only need to keep as much state as needed to store byte count modulo word size, I cannot think of a simple yet performant implementation using shell builtins. Dependencies should remain minimal: using GNU coreutils/cpio/tar, no compiler/perl/features that would differ between busybox/dash/bash. I have not come up with an awk solution as I failed to make it perform well (G/s) on binary input not evenly NL/NUL-separated into lines.
Since you mention there's a compiler available, here's a tiny, portable C program. It does not get any faster and memory-economic. It's even readable for most people in the programming community. If not, you can always sprinkle /* Comments! */. :-)
#!/bin/sh
#
# pad.sh - pad input, reading in large blocks from stdin, writing stdout.
# padding $1:padchar $2:alignment $3:blocksize
padding () {
aout="./a$$.out"
cc -x c -o "$aout" - <<EOF
#include <stdio.h>
int main (void) {
size_t align = $2, nwritten = 0, nread;
char buffer[$3];
while ((nread = fread (buffer, 1, sizeof buffer, stdin)) > 0)
nwritten += fwrite (buffer, 1, nread, stdout);
if ((nwritten % align) != 0)
for (align -= nwritten % align; align != 0; --align)
putchar ($1);
return 0;
}
EOF
"$aout" && rm "$aout"
}
printf '%s' 123456789 | padding 0 4 16384 | od -c
printf '%s' abcdefghi | padding "'\n'" 16 BUFSIZ | od -c
printf '%s' PAGE_SIZE | padding 65 32 "$(getconf PAGE_SIZE)" | od -c
In action:
$ ./pad.sh
0000000 1 2 3 4 5 6 7 8 9 \0 \0 \0
0000014
0000000 a b c d e f g h i \n \n \n \n \n \n \n
0000020
0000000 P A G E _ S I Z E A A A A A A A
0000020 A A A A A A A A A A A A A A A A
0000040
If you are concerned about the non-POSIXly compiler option -x c you can easily write the C program to pad.c and compile it from there. Advanced error handling for fwrite, fread and putchar left to the reader.
Note how the here-document avoids main having to parse arguments. You can even pass strings like PAGE_SIZE if your stdio makes them available by default.
I just realized that compiling C like this is not much different from a nifty awk script -- awk also compiles an internal program and then executes it. What's better than compiling to the machine's CPU and running the executable?
The POSIX thing to do would be to use a temporary file.
padding() (
tmpf=$(mktemp) &&
trap 'rm "$tmpf"' EXIT &&
tee "$tmpf" &&
isize=$(wc -c <"$tmpf") &&
pad=$(( $1 - isize % $1 )) &&
if [ "$pad" -ne 0 ]; then
dd status=none if=/dev/zero bs="$pad" count=1
fi
)
I have this function:
function convert_ascii_string_to_decimal {
ascii=$1
unset converted_result
while IFS="" read -r -n 1 char; do
decimal=$(printf '%d' "'$char")
echo $decimal
converted_result="$converted_result $decimal"
done < <(printf %s "$ascii")
converted_result=$(echo $converted_result | xargs) #strip leading and trailing
}
It is meant to take an ascii string variable, loop through every character, and concatenate the ascii decimal representation to a string. However, this while loop seems to ignore null chars, ie characters with ascii 0. I want to be able to read every single ascii there is, including null.
To get all characters of a string as decimal number, you can use hexdump to parse a string:
echo -e "hello \x00world" | hexdump -v -e '1/1 "%d "'
104 101 108 108 111 32 0 119 111 114 108 100 10
This also works for parsing a file:
echo '05 04 03 02 01 00 ff' | xxd -r -ps > file
hexdump --no-squeezing --format '1/1 "%d "' file
5 4 3 2 1 0 255
hexdump explanation:
options -v and --no-squeezing prints all bytes (without skipping duplicated bytes)
options -e and --format allows giving a specific format
format is 1/1 "%d " which means
Iteration count = 1 (process the byte only once)
Byte count = 1 (apply this format for each byte)
Format = "%d" (convert to decimal)
You can't store the null character in a bash variable, which is happening in your script with the $char variable.
I suggest using xxd instead of writing your own script:
echo -ne "some ascii text" | xxd -p
If we echo a null charcter:
$ echo -ne "\0" | xxd -p
00
I am trying to dump the decimal integer values from one file in a hex format.
I do have a file with integer values in decimal.
$ more test.dat_trim
2 9
0 -11
7 -17
14 -1
I am trying to print this integer in hex. I know also that the integer values are small enough to fit on 2 bytes. I want the output to be on 2 bytes. But then when i am trying:
declare -i i;for i in $(<test.dat_trim);do printf "%.2x\n" $i; done;
02
09
00
fffffffffffffff5
07
ffffffffffffffef
0e
ffffffffffffffff
Basically printf "%.2x\n" it is only working for positive number. How can i make it work for negative also?
Just to clarify what i am expecting: The result should be like this:
02
09
00
f5
07
ef
0e
ff
meaning that i want for the negative values to be sign extended only on 1 byte.
Printing signed hex values is uncommon, so there is no conversion specifier providing this.
You could use the following work around:
for i in $(<test.dat_trim); do
if [ $i -ge 0 ]; then
printf " 0x%02x\n" $i;
else
printf "%c0x%02x\n" '-' $[$i * -1];
fi
done;
Referrig the update to the question:
Just replace this line
printf "%c0x%02x\n" '-' $[$i * -1];
with this
printf " 0x%02x\n" $[256 + $i];
This however, only works for the numbers >= -256.
It can be done in awk, that handles negative numbers also:
awk '{printf "0x%x%s0x%x\n", $1, OFS, $2}' OFS='\t' file
0x2 0x9
0x0 0xfffffff5
0x7 0xffffffef
0xe 0xffffffff
Kinda silly but what the heck:
xargs -a test.dat_trim bash -c 'printf %.2s\\n $(printf %02x\\n $* | rev) | rev' _
Have you tried printf("%04x\n",i)?
I'm new to BASH scripting, but I need a quick way to convert a signed hexadecimal to signed decimal. For instance FF should be -1 not 256. The msbit should be sign bit.
so far I have.. and I need and signed decimal from a signed hex word.
#! /bin/sh
ADDR=`echo $1 | tr a-z A-Z`
ADDR2=`echo "ibase=16; obase=10; $ADDR+1" | bc`
VARmsb=`./mpu-6050-getbyte $ADDR;`
VARlsb=`./mpu-6050-getbyte $ADDR2;`
echo $VARmsb$VARlsb
Probably not canonical, but this function should do it:
function conv() {
res=$(printf "%d" "0x$1")
(( res > 127 )) && (( res -= 256 ))
echo $res
}
For instance:
conv 1
conv 0A
conv 7F
conv 80
conv FF
1
10
127
-128
-1
This should do the trick
signed_dec=$(printf "%d" $signed_hex)
If you need the result to fit inside a certain range then you can apply a modulo operator via bash arithmetic e.g:
signed_dec=$(($(printf "%d" 0xFAB) % 256))
How can I create a binary file with consequent binary values in Bash?
Like:
hexdump testfile
0000000 0100 0302 0504 0706 0908 0b0a 0d0c 0f0e
0000010 1110 1312 1514 1716 1918 1b1a 1d1c 1f1e
0000020 2120 2322 2524 2726 2928 2b2a 2d2c 2f2e
0000030 ....
In C, I do:
fd = open("testfile", O_RDWR | O_CREAT);
for (i=0; i< CONTENT_SIZE; i++)
{
testBufOut[i] = i;
}
num_bytes_written = write(fd, testBufOut, CONTENT_SIZE);
close (fd);
This is what I wanted:
#! /bin/bash
i=0
while [ $i -lt 256 ]; do
h=$(printf "%.2X\n" $i)
echo "$h"| xxd -r -p
i=$((i-1))
done
There's only one byte you cannot pass as an argument in a Bash command line: 0
For any other value, you can just redirect it. It's safe.
echo -n $'\x01' > binary.dat
echo -n $'\x02' >> binary.dat
...
For the value 0, there's another way to output it to a file
dd if=/dev/zero of=binary.dat bs=1c count=1
To append it to file, use
dd if=/dev/zero oflag=append conv=notrunc of=binary.dat bs=1c count=1
Take a look at xxd:
xxd: creates a hex dump of a given file or standard input. It can
also
convert a hex dump back to its original binary form.
If you don't mind to not use an existing command and want to describe you data in a text file, you can use binmake. That is a C++ program that you can compile and use like following:
First get and compile binmake (the binary will be in bin/):
git clone https://github.com/dadadel/binmake
cd binmake
make
Create your text file file.txt:
big-endian
00010203
04050607
# Separated bytes not concerned by endianness
08 09 0a 0b 0c 0d 0e 0f
Generate your binary file file.bin:
./binmake file.txt file.bin
hexdump file.bin
0000000 0100 0302 0504 0706 0908 0b0a 0d0c 0f0e
0000008
Note: you can also use it with standard input and standard output.
Use the below command,
i=0; while [ $i -lt 256 ]; do echo -en '\x'$(printf "%0x" $i)'' >> binary.dat; i=$((i+1)); done