I am trying to dump the decimal integer values from one file in a hex format.
I do have a file with integer values in decimal.
$ more test.dat_trim
2 9
0 -11
7 -17
14 -1
I am trying to print this integer in hex. I know also that the integer values are small enough to fit on 2 bytes. I want the output to be on 2 bytes. But then when i am trying:
declare -i i;for i in $(<test.dat_trim);do printf "%.2x\n" $i; done;
02
09
00
fffffffffffffff5
07
ffffffffffffffef
0e
ffffffffffffffff
Basically printf "%.2x\n" it is only working for positive number. How can i make it work for negative also?
Just to clarify what i am expecting: The result should be like this:
02
09
00
f5
07
ef
0e
ff
meaning that i want for the negative values to be sign extended only on 1 byte.
Printing signed hex values is uncommon, so there is no conversion specifier providing this.
You could use the following work around:
for i in $(<test.dat_trim); do
if [ $i -ge 0 ]; then
printf " 0x%02x\n" $i;
else
printf "%c0x%02x\n" '-' $[$i * -1];
fi
done;
Referrig the update to the question:
Just replace this line
printf "%c0x%02x\n" '-' $[$i * -1];
with this
printf " 0x%02x\n" $[256 + $i];
This however, only works for the numbers >= -256.
It can be done in awk, that handles negative numbers also:
awk '{printf "0x%x%s0x%x\n", $1, OFS, $2}' OFS='\t' file
0x2 0x9
0x0 0xfffffff5
0x7 0xffffffef
0xe 0xffffffff
Kinda silly but what the heck:
xargs -a test.dat_trim bash -c 'printf %.2s\\n $(printf %02x\\n $* | rev) | rev' _
Have you tried printf("%04x\n",i)?
Related
I have this function:
function convert_ascii_string_to_decimal {
ascii=$1
unset converted_result
while IFS="" read -r -n 1 char; do
decimal=$(printf '%d' "'$char")
echo $decimal
converted_result="$converted_result $decimal"
done < <(printf %s "$ascii")
converted_result=$(echo $converted_result | xargs) #strip leading and trailing
}
It is meant to take an ascii string variable, loop through every character, and concatenate the ascii decimal representation to a string. However, this while loop seems to ignore null chars, ie characters with ascii 0. I want to be able to read every single ascii there is, including null.
To get all characters of a string as decimal number, you can use hexdump to parse a string:
echo -e "hello \x00world" | hexdump -v -e '1/1 "%d "'
104 101 108 108 111 32 0 119 111 114 108 100 10
This also works for parsing a file:
echo '05 04 03 02 01 00 ff' | xxd -r -ps > file
hexdump --no-squeezing --format '1/1 "%d "' file
5 4 3 2 1 0 255
hexdump explanation:
options -v and --no-squeezing prints all bytes (without skipping duplicated bytes)
options -e and --format allows giving a specific format
format is 1/1 "%d " which means
Iteration count = 1 (process the byte only once)
Byte count = 1 (apply this format for each byte)
Format = "%d" (convert to decimal)
You can't store the null character in a bash variable, which is happening in your script with the $char variable.
I suggest using xxd instead of writing your own script:
echo -ne "some ascii text" | xxd -p
If we echo a null charcter:
$ echo -ne "\0" | xxd -p
00
I have to convert decimal number to hexadecimal, but with filling eventual voids with zeroes:
Example:
I've tried this:
printf "%X\n" 190
Output is:
BE
i need it to look like this:
00BE
In short, output should have 4 hex symbols, if less, it should be filled with zeroes at the beginning
How to do that in bash?
Use format specifiers:
$ printf "%04X\n" 190
00BE
$ printf "%04X\n" 1
0001
$ printf "%04X\n" 42
002A
I have a decimal number in each line of a file.txt:
1
2
3
I am trying (for too long now) to write a one-liner script to have an output where each row has a column with the decimal, hexadecimal and the binary. To ease the task we can say that the original number is expressed in a byte. So the maximum value is 255.
I first try to decode each number as a bynary with prepended 0 so to have an 8 bits pattern:
awk '{print "ibase=10;obase=2;" $1}' $1 | bc | xargs printf "%08d\n"
where the outer $1 in the awk statement is file.txt. The output is :
00000001
00000010
00000011
Same for hex with one prepended 0
awk '{printf("0x%02x\n", $1)}' $1
Same as before. The Output is :
0x01
0x02
0x03
Well, the decimal should be just a print:
1
2
3
What I'd like to have is one liner where I have:
1 00000001 0x01
2 00000001 0x02
so basically to put 1. 2. and 3. in each line of the output.
I tried to execute bc (and other command) within awk using system() without success. And a zillion other ways. What is the way you would do it?
The following one-liner should work:
printf "%s %08d 0x%02x\n" "$1" $(bc <<< "ibase=10;obase=2;$1") "$1"
Example output:
$ for i in {1..10}; do printf "%s %08d 0x%02x\n" "$i" $(bc <<< "ibase=10;obase=2;$i") "$i"; done
1 00000001 0x01
2 00000010 0x02
3 00000011 0x03
4 00000100 0x04
5 00000101 0x05
6 00000110 0x06
7 00000111 0x07
8 00001000 0x08
9 00001001 0x09
10 00001010 0x0a
So I searched for a short and elegant awk binary converter. Not satisfied considered this as a challenge, so here you are. A little bit optimzed for size, so I put a readable version below.
The printf at the end specifies how large the numbers should be. In this case 8 bits.
Is this bad code? Hmm, yeah... it's awk :-)
Does of course not work with very huge numbers.
67 characters long awk code:
awk '{r="";a=$1;while(a){r=((a%2)?"1":"0")r;a=int(a/2)}printf"%08d\n",r}'
Edit: 55 characters awk code
awk '{r="";a=$1;while(a){r=a%2r;a=int(a/2)}printf"%08d\n",r}'
Readable version:
awk '{r="" # initialize result to empty (not 0)
a=$1 # get the number
while(a!=0){ # as long as number still has a value
r=((a%2)?"1":"0") r # prepend the modulos2 to the result
a=int(a/2) # shift right (integer division by 2)
}
printf "%08d\n",r # print result with fixed width
}'
And the asked one liner with bin and hex
awk '{r="";a=$1;while(a){r=a%2r;a=int(a/2)}printf"%08d 0x%02x\n",r,$1}'
You don't need bc. Here's a solution using only awk:
Fetch the bits2str function available in the manual
Add this minimal script:
{
printf("%s %s %x\n", $1, bits2str($1), $1)
}
This produces:
$ awk -f awkscr.awk nums
1 00000001 1
2 00000010 2
3 00000011 3
I am writing a bash script that needs to get the header (first 10 bytes) of a file and then in another section get everything except the first 10 bytes. These are binary files and will likely have \0's and \n's throughout the first 10 bytes. It seems like most utilities work with ASCII files. What is a good way to achieve this task?
To get the first 10 bytes, as noted already:
head -c 10
To get all but the first 10 bytes (at least with GNU tail):
tail -c+11
head -c 10 does the right thing here.
You can use the dd command to copy an arbitrary number of bytes from a binary file.
dd if=infile of=outfile1 bs=10 count=1
dd if=infile of=outfile2 bs=10 skip=1
How to split a stream (or a file) under bash
Two answer here!
Reading SO request:
get the header (first 10 bytes) of a file and then in another section get everything except the first 10 bytes.
I understand:
How to split a file at specific point
As all answers here does access same file two time, instead of just split it!!
Here is my two cents:
The interesting thing using Un*x is considering every whole job as a filter, it's easy to a split stream using unbuffered I/O. Most of standard un*x tools (cat, grep, awk, sed, python, perl ...) work as filters.
1. Using head or dd but in a single pass
{ head -c 10 >head_part; cat >tail_part;} <file
This is the more efficient, as your file is read only 1 time, the first 10 byte goes to head_part and the rest goes to tail_part.
Note: second redirection >tail_part could be place outside of whole list ({ ...;}) as well...
You could do same, using dd:
{ dd count=1 bs=10 of=head_part; cat;} <file >tail_part
This stay more efficient than running two process of dd to open same file two times.
...And still use standard block size for the rest of file:
Another sample based on read by line:
Split HTTP (or mail) stream on near empty line (line containing only carriage return: \r):
nc google.com 80 <<<$'GET / HTTP/1.0\r\nHost: google.com\r\n\r' |
{ sed -u '/^\r$/q' >/tmp/so_head.raw; cat;} >/tmp/so_body.raw
or, to drop empty last head line:
nc google.com 80 <<<$'GET / HTTP/1.0\r\nHost: google.com\r\n\r' |
{ sed -nu '/^\r$/q;p' >/tmp/so_head.raw; cat;} >/tmp/so_body.raw
This will produce two files:
ls -l so_*.raw
-rw-r--r-- 1 root root 307 Apr 25 11:40 so_head.raw
-rw-r--r-- 1 root root 219 Apr 25 11:40 so_body.raw
grep www so_*.raw
so_body.raw:here.
so_head.raw:Location: http://www.google.com/
2. Pure bash way:
If the goal is to obtain values of first 10 bytes in a usable bash variable, here is a nice and efficient way:
Because ten byte are few, fork to head could be avoided. from Read a file by bytes in BASH:
read8() {
local _r8_var=${1:-OUTBIN} _r8_car LANG=C IFS=
read -r -d '' -n 1 _r8_car || { printf -v $_r8_var '';return 1;}
printf -v $_r8_var %02X "'"$_r8_car
}
{
first10=()
for i in {0..9};do
read8 first10[i] || break
done
cat
} < "$infile" >"$outfile"
This will create an array ${first10[#]} containing hexadecimal values of first ten bytes of $infile and store rest of data into $outfile.
declare -p first10
declare -a first10=([0]="25" [1]="50" [2]="44" [3]="46" [4]="2D" [5]="31" [6]="2E"
[7]="34" [8]="0A" [9]="25")
This was a PDF (%PDF -> 25 50 44 46)... Here's another sample:
{
first10=()
for i in {0..9};do
read8 first10[i] || break
done
cat
} <<<"Hello world!"
d!
As I didn't redirect output, string d! will be output on terminal.
echo ${first10[#]}
48 65 6C 6C 6F 20 77 6F 72 6C
printf '%b%b%b%b%b%b%b%b%b%b\n' ${first10[#]/#/\\x}
Hello worl
About binary
You said:
These are binary files and will likely have \0's and \n's throughout the first 10 bytes.
{
first10=()
for i in {0..9};do
read8 first10[i] || break
done
cat
} < <(gzip <<<"Hello world!") >/dev/null
echo ${first10[#]}
1F 8B 08 00 00 00 00 00 00 03
( Sample with a \n at bottom of this ;)
As a function
read8() { local _r8_var=${1:-OUTBIN} _r8_car LANG=C IFS=
read -r -d '' -n 1 _r8_car || { printf -v $_r8_var '';return 1;}
printf -v $_r8_var %02X "'"$_r8_car ;}
get10() {
local -n result=${1:-first10} # 1st arg is array name
local -i _i
result=()
for ((_i=0;_i<${2:-10};_i++));do # 2nd arg is number of bytes
read8 result[_i] || { unset result[_i] ; return 1 ;}
done
cat
}
Then (here, I use the special character ⛶ for: there was no newline. ).
get10 pdf 4 <$infile >$outfile
printf %b ${pdf[#]/#/\\x}
%PDF⛶
echo $(( $(stat -c %s $infile) - $(stat -c %s $outfile) ))
4
get10 test 8 <<<'Hello world'
rld!
printf %b ${test[#]/#/\\x}
Hello Wo⛶
get10 test 24 <<<'Hello World!'
printf %b ${test[#]/#/\\x}
Hello World!
( And the last character printed is a \n! ;)
Final binary demo:
get10 test 256 < <(gzip <<<'Hello world!')
printf '%b' ${test[#]/#/\\x} | gunzip
Hello world!
printf " %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s\n" ${test[#]}
1F 8B 08 00 00 00 00 00 00 03 F3 48 CD C9 C9 57
28 CF 2F CA 49 51 E4 02 00 41 E4 A9 B2 0D 00 00
00
Note!! This work fine and is very quick while number of byte to read stay low, even processing large files. This could be used for file recognition, for sample. But for spliting files on larger parts, you have to use split, head, tail and/or dd.
I have a file with binary data and I need to replace a few bytes in a certain position. I've come up with the following to direct bash to the offset and show me that it found the place I want:
dd bs=1 if=file iseek=24 conv=block cbs=2 | hexdump
Now, to use "file" as the output:
echo anInteger | dd bs=1 of=hextest.txt oseek=24 conv=block cbs=2
This seems to work just fine, I can review the changes made in a hex editor. Problem is, "anInteger" will be written as the ASCII representation of that integer (which makes sense) but I need to write the binary representation.
I want to use bash for this and the script should run on as many systems as possible (I don't know if the target system will have python or whatever installed).
How do I tell the command to convert the input to binary (possibly from a hex)?
printf is more portable than echo. This function takes a decimal integer and outputs a byte with that value:
echobyte () {
if (( $1 >= 0 && $1 <= 255 ))
then
printf "\\x$(printf "%x" $1)"
else
printf "Invalid value\n" >&2
return 1
fi
}
$ echobyte 97
a
$ for i in {0..15}; do echobyte $i; done | hd
00000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
00000010
You can use echo to emit specific bytes using hex or octal. For example:
echo -n -e \\x30
will print ascii 0 (0x30)
(-n remove trailing newline)
xxd is the better way. xxd -r infile outfile will take ascii hex-value in infile to patch outfile, and you can specify the specific position in infile by this: 1FE:55AA
Worked like a treat. I used the following code to replace 4 bytes at byte 24 in little endian with two integers (1032 and 1920). The code does not truncate the file.
echo -e \\x08\\x04\\x80\\x07 | dd of=<file> obs=1 oseek=24 conv=block,notrunc cbs=4
Thanks again.
I have a function to do this:
# number representation from 0 to 255 (one char long)
function chr() { printf "\\$(printf '%03o' "$1")" ; return 0 ; }
# from 0 to 65535 (two char long)
function word_litleendian() { chr $(($1 / 256)) ; chr $(($1 % 256)) ; return 0 ; }
function word_bigendian() { chr $(($1 % 256)) ; chr $(($1 / 256)) ; return 0 ; }
# from 0 to 4294967295 (four char long)
function dword_litleendian() { word_lilteendian $(($1 / 65536)) ; word_litleendian $(($1 % 65536)) ; return 0 ; }
function dword_bigendian() { word_bigendian $(($1 / 65536)) ; word_bigendian $(($1 % 65536)) ; return 0 ; }
You can use piping or redirection to catch the result.
If you're willing to rely on bc (which is fairly common)
echo -e "ibase=16\n obase=2 \n A1" | bc -q
might help.
With bash, "printf" has the "-v" option, and all shell has logical operators.
So here is simplier form in bash :
int2bin() {
local i=$1
local f
printf -v f '\\x%02x\\x%02x\\x%02x\\x%02x' $((i&255)) $((i >> 8 & 255)) $((i >> 16 & 255)) $((i >> 24 & 255))
printf "$f"
}
You might put the desired input into a file and use the "if=" option to dd to insert exactly the input you desire.
In my case, I needed to go from a decimal numeric argument to the actual unsigned 16-bit big endian value. This is probably not the most efficient way, but it works:
# $1 is whatever number (0 to 65535) the caller specifies
DECVAL=$1
HEXSTR=`printf "%04x" "$DECVAL"`
BYTEONE=`echo -n "$HEXSTR" | cut -c 1-2`
BYTETWO=`echo -n "$HEXSTR" | cut -c 3-4`
echo -ne "\x$BYTEONE\x$BYTETWO" | dd of="$FILENAME" bs=1 seek=$((0xdeadbeef)) conv=notrunc