How to manipulate hexadecimal value in Bash? - bash

I have a variable with an hexadecimal value: in this example, a byte with value 0xfe:
echo $MYVAR | hexdump
0000000 0afe
0000002
I want to use this value on my bash script, in particular I need to:
use as a string (echo X$MYVAR should give me Xfe)
increment it, (0xff)
convert back to the original format (I need to save the incremented value for future use)
Maybe it would be easier if I convert it into integer format?
EDIT: here is how I initialize the var:
printf "\xfe" | dd bs=1 of=tempfile seek=8001
MYVAR=`dd if=tempfile skip=8001 count=1 bs=1`

To print hex number as string you can
printf 0x%X $MYVAR
to increment it and print it back in hex you can do, for example
printf 0x%X `echo $(( 0xfe + 1 ))`
For "convert back" to the original format I think you mean keep the integer value, in this case you can simply use $MYVAR without format conversion.
Hope this helps,
Regards.
EDIT :
To follow your question editing, I'll add my answer below.
You could set MYVAR in this way:
read dummy MYVAR <<EOF
`dd if=tempfile skip=8001 count=1 bs=1|od -x`
EOF
Now you have hex value of the byte read from file stored in MYVALUE.
You can now print it directly with echo, printf or whatever.
$ echo $MYVAR
00fe
You can perform math on it as said before:
$ printf %X $((0x$MYVAR + 1))
FF
(thanks to fedorqui for the shortest version)
Regards.

Assuming you stream represents constantly changed numeric parameter, it may be converted by hexdump or od from coreutils. Note, stdbuf is highly recomended for real-time stream processing.
# Print decimal byte values per line
stdbuf -oL hexdump -v -e '1/1 "%u\n"'
# Floating point values
stdbuf -oL hexdump -v -e '1/4 "%f\n"'
Using od from coreutils, for input endianess correction:
# unsigned 8-bit
stdbuf -oL od -v -An -w7 -tu1 --endian=little
# signed 32-bit
stdbuf -oL od -v -An -w4 -tu4 --endian=little
# Float 32-bit
stdbuf -oL od -v -An -w4 -tf4 --endian=little
Unless endianess is issue, hexdump could be more preferable due to good format support - almost like printf:
cat /dev/urandom \
| stdbuf -oL hexdump -v -e '1/4 "%i" 1/4 " %i" 1/4 " %f\n"' \
| stdbuf -oL awk '{ print "("$1" + "$2") * "$3" = ", $1 + $2 * $3 }'
some pointless 8-bit stream math, for example
# Replace the >/dev/null safeguard to actually try it
cat /dev/urandom \
| stdbuf -oL hexdump -e '1/1 " %i" 1/1 " %i" "\n"' \
| {
while read a b; do
r=$(( (a + b) / 2 ))
printf "printf \"\%03o\"" $r
done
} | sh >/dev/null

Related

Syntax error while trying to generate a 2048bit long prime number

I'm trying to generate a 2048 bit long prime number, this is my code so far:
#!/bin/bash
generate_random() {
hex=$(head -c 256 /dev/urandom | xxd -p)
bc <<< "ibase=16; $hex"
}
p=$(generate_random)
echo "$p"
While running the script I get (standard_in) 1: syntax error followed by random zeroes.
Anyone knows what is causing this error and how can I fix it? I've tried with bash -x, but it doesn't add any useful information.
First, bc understands only upper-case letters as hex digits (at least by default). Second, you have separators in your xxd output, so you generate multiple numbers with bc later.
This should work:
#!/bin/bash
generate_random() {
hex=$(head -c 256 /dev/urandom | xxd -p -u | tr -d '\n')
bc <<< "ibase=16; $hex"
}
p=$(generate_random)
echo "$p"
-u flag to xxd instructs it to output upper-case letters as digits, and tr removes separators.
Example output:
84404284040092528807148386035025161100484110236893077703095592941720\
00537078513504880246726730474236368181068985417211434943913923235822\
01284401417146606673073772989889733010524123703686975444423088406509\
44767677616371794606797386146855833950295071249000795855185540560405\
62673903614333076371092344026999031152809898928396395497832309795471\
93897215963003601022703133486344387720277877558264139632520964120681\
97764906669023878701319760947789227343517474218584987497204300184084\
62846775760153647010072072799120566180042021620262646969602253704108\
06274157727080642084167983313757899766696995668747042179553171962777\
5716
To remove newline separators and backslashes, you can do
p_joined=$(echo "$p" | sed -z 's=\\\n==g')
echo "$p_joined"
instead.
An alternative way might be
printf -v hex '%s' $(od -v -An -N256 -x /dev/urandom)
read dec < <(bc <<< "ibase=16; ${hex^^}")
echo $dec

Take first 16 character and covert it into hex string

I have UUID, 3abbea88-c77d-11eb-b8bc-0242ac130003 and I want to take first 16 character of this string and want Hexadecimal string of first 16 characters using shell script.
I tried,
code=$(echo -n ${${ID##*:}:0:16} | od -A n -t x1)
HEX_ID=$(echo ${code//[[:blank:]]/})
Any better way ?
Expected Output : 33616262656138382d633737642d3131
Using od you can simply limit the number of read characters using the -N option:
HEX_ID=$(od -A n -t x1 -N 16 <<< ${ID##*:} | tr -dc '[:xdigit:]')
Edit: tr is used to suppress non-hexadecimal characters, namely whitespaces and potential newlines.
Perl to the rescue!
perl -le 'print unpack "H32", shift' 3abbea88-c77d-11eb-b8bc-0242ac130003
-l adds newlines to print
unpack takes a string and expands it to a list of values based on a template. H32 means "take characters and interpret them as 32 hex values".
shift reads the first command line argument.
Or, using xxd and head:
echo 3abbea88-c77d-11eb-b8bc-0242ac130003 | xxd -p | head -c32
That's certainly a useless echo.
Probably avoid uppercase for your private variables.
uuid='3abbea88-c77d-11eb-b8bc-0242ac130003'
tmp=${uuid//-/}
hex_id=$(od -A n -t x1 <<<${tmp:0:13})
hex_id=${hex_id//[[:blank:]]/}
hex_id=${hex_id%0a}
The here string unattractively supplies trailing newline to od which we have to trim off.
Bash-only:
while read -r -N 1 c # read input string 1 char at a time
do [[ "$c" == " " ]] || # skip embedded spaces
printf "%02X" "$( # output the hexidecimal value of
printf "%d" \'$c # the ASCII decimal ordinal of $c
)"
done <<< "${text##*:}" # ignoring the leading trash to the :
echo # newline-teminate the output
All in one line:
while read -rn1 c;do [[ "$c" == " " ]]||printf %02X $(printf "%d" \'$c);done<<<"${text##*:}";echo
This is not the fastest approach...
hexdump does it all:
hexdump -n 16 -ve '1/1 "%.2x"'
-n 16 means only process the first 16 bytes
-e '1/1 "%.2x"' means display each byte using given printf format
-v means display normally (without this, it replaces dupe sections with * 🤷)
echo '3abbea88-c77d-11eb-b8bc-0242ac130003' | hexdump -n 16 -ve '1/1 "%.2x"'
output:
33616262656138382d633737642d3131

Convert 32/64 bit number to 4/8 character string

Given that:
$ printf "love" | od -td4 -A n
1702260588
$ printf "lovehate" | od -td8 -A n
7310575196135911276
Is there a concise (ideally without loops, awk, sed, perl or python) way in Bash to convert the numbers 1702260588 and 7310575196135911276 to love and lovehate respectively?
Here's what I came up with:
alpha() {
(($1)) && printf "\x"$(printf "%02x" $(($1%256)))$(alpha $(($1/256)))"\n"
}
alpha 1702260588
alpha 7310575196135911276
Output:
love
lovehate
Edit: Here's an answer using the xxd utility:
# The echo is only necessary to get a newline at the end.
echo $(printf "%x" 1702260588 | xxd -r -p | rev)
echo $(printf "%x" 7310575196135911276 | xxd -r -p | rev)
Output:
love
lovehate

How to get a random string of 32 hexadecimal digits through command line?

I'd like to put together a command that will print out a string of 32 hexadecimal digits. I've got a Python script that works:
python -c 'import random ; print "".join(map(lambda t: format(t, "02X"), [random.randrange(256) for x in range(16)]))'
This generates output like:
6EF6B30F9E557F948C402C89002C7C8A
Which is what I need.
On a Mac, I can even do this:
uuidgen | tr -d '-'
However, I don't have access to the more sophisticated scripting languages ruby and python, and I won't be on a Mac (so no uuidgen). I need to stick with more bash'ish tools like sed, awk, /dev/random because I'm on a limited platform. Is there a way to do this?
If you have hexdump then:
hexdump -vn16 -e'4/4 "%08X" 1 "\n"' /dev/urandom
should do the job.
Explanation:
-v to print all data (by default hexdump replaces repetition by *).
-n16 to consume 16 bytes of input (32 hex digits = 16 bytes).
4/4 "%08X" to iterate four times, consume 4 bytes per iteration and print the corresponding 32 bits value as 8 hex digits, with leading zeros, if needed.
1 "\n" to end with a single newline.
If you are looking for a single command and have openssl installed, see below. Generate random 16 bytes (32 hex symbols) and encode in hex (also -base64 is supported).
openssl rand -hex 16
There three ways that I know of:
#!/bin/bash
n=16
# Read n bytes from urandom (in hex):
xxd -l "$n" -p /dev/urandom | tr -d " \n" ; echo
od -vN "$n" -An -tx1 /dev/urandom | tr -d " \n" ; echo
hexdump -vn "$n" -e ' /1 "%02x"' /dev/urandom ; echo
Use one, comment out the other two.
Try:
xxd -u -l 16 -p /dev/urandom
Example output:
C298212CD8B55F2E193FFA16165E95E3
And to convert it back to binary:
echo -n C298212CD8B55F2E193FFA16165E95E3 | xxd -r -p
Here are a few more options, all of which have the nice property of providing an obvious and easy way to directly select the length of the output string. In all the cases below, changing the '32' to your desired string length is all you need to do.
#works in bash and busybox, but not in ksh
tr -dc 'A-F0-9' < /dev/urandom | head -c32
#works in bash and ksh, but not in busybox
tr -dc 'A-F0-9' < /dev/urandom | dd status=none bs=1 count=32
#works in bash, ksh, AND busybox! w00t!
tr -dc 'A-F0-9' < /dev/urandom | dd bs=1 count=32 2>/dev/null
EDIT: Tested in different shells.
If you want to generate output of arbitrary length, including even/odd number of characters:
cat /dev/urandom | hexdump --no-squeezing -e '/1 "%x"' | head -c 31
Or to maximize efficiency over readability/composeability:
hexdump --no-squeezing -e '/1 "%x"' -n 15 /dev/urandom
Here is a version not using dev/random:
awk -v len=32 'BEGIN {
srand('$RANDOM');
while(len--) {
n=int(rand()*16);
printf("%c", n+(n>9 ? 55 : 48));
};}'
you can also use od command like this
od -N32 -x < /dev/urandom | head -n1 | cut -b9- | sed 's/ //gi'
good luck

Ascii/Hex convert in bash

I'm now doing it this way:
[root#~]# echo Aa|hexdump -v
0000000 6141 000a
0000003
[root#~]# echo -e "\x41\x41\x41\x41"
AAAA
But it's not exactly behaving as I wanted,
the hex form of Aa should be 4161,but the output is 6141 000a,which seems not making sense.
and when performing hex to ascii,is there another utility so that I don't need the prefix \x ?
The reason is because hexdump by default prints out 16-bit integers, not bytes. If your system has them, hd (or hexdump -C) or xxd will provide less surprising outputs - if not, od -t x1 is a POSIX-standard way to get byte-by-byte hex output. You can use od -t x1c to show both the byte hex values and the corresponding letters.
If you have xxd (which ships with vim), you can use xxd -r to convert back from hex (from the same format xxd produces). If you just have plain hex (just the '4161', which is produced by xxd -p) you can use xxd -r -p to convert back.
For the first part, try
echo Aa | od -t x1
It prints byte-by-byte
$ echo Aa | od -t x1
0000000 41 61 0a
0000003
The 0a is the implicit newline that echo produces.
Use echo -n or printf instead.
$ printf Aa | od -t x1
0000000 41 61
0000002
For single line solution:
echo "Hello World" | xxd -ps -c 200 | tr -d '\n'
It will print:
48656c6c6f20576f726c640a
or for files:
cat /path/to/file | xxd -ps -c 200 | tr -d '\n'
For reverse operation:
echo '48656c6c6f20576f726c640a' | xxd -ps -r
It will print:
Hello World
$> printf "%x%x\n" "'A" "'a"
4161
With bash :
a=abcdefghij
for ((i=0;i<${#a};i++));do printf %02X \'${a:$i:1};done
6162636465666768696A
I use:
> echo Aa | tr -d '\n' | xxd -p
4161
> echo 414161 | tr -d '\n' | xxd -r -p
AAa
The tr -d '\n' will trim any possible newlines in your input
I don't know how it crazy it looks but it does the job really well
ascii2hex(){ a="$#";s=0000000;printf "$a" | hexdump | grep "^$s"| sed s/' '//g| sed s/^$s//;}
Created this when I was trying to see my name in HEX ;)
use how can you use it :)
Text2Conv="Aa"
for letter in $(echo "$Text2Conv" | sed "s/\(.\)/'\1 /g");do printf '%x' "$letter";done
4161
The trick is using sed to parse the Text2Conv to format we can then seperate anf loop using for.
Finally got the correct thing
echo "Hello, world!" | tr -d '\n' | xxd -ps -c 200
here a little script I wrote to convert ascii to hex. hope it helps:
echo '0x'"`echo 'ASCII INPUT GOES HERE' | hexdump -vC | awk 'BEGIN {IFS="\t"} {$1=""; print }' | awk '{sub(/\|.*/,"")}1' | tr -d '\n' | tr -d ' '`" | rev | cut -c 3- | rev
SteinAir's answer above was helpful to me -- thank you! And below is a way it inspired, to convert hex strings to ascii:
for h in $(echo "4161" | sed "s/\(..\)/\1 /g"); do printf `echo "\x$h"`;done
Aa
echo -n Aa | hexdump -e '/1 "%02x"'; echo
according to http://mylinuxbook.com/hexdump/ you might use the hexdump format parameter
echo Aa | hexdump -C -e '/1 "%02X"'
will return 4161
to add an extra linefeed at the end, append another formatter.
BUT: the format given above will give multiplier outputs for repetitive characters
$ printf "Hello" | hexdump -e '/1 "%02X"'
48656C*
6F
instead of
48656c6c6f
jcomeau#aspire:~$ echo -n The quick brown fox jumps over the lazy dog | python -c "print raw_input().encode('hex'),"
54686520717569636b2062726f776e20666f78206a756d7073206f76657220746865206c617a7920646f67
jcomeau#aspire:~$ echo -n The quick brown fox jumps over the lazy dog | python -c "print raw_input().encode('hex')," | python -c "print raw_input().decode('hex'),"
The quick brown fox jumps over the lazy dog
it could be done with Python3 as well, but differently, and I'm a lazy dog.
echo append a carriage return at the end.
Use
echo -e
to remove the extra 0x0A
Also, hexdump does not work byte-per-byte as default. This is why it shows you bytes in a weird endianess and why it shows you an extra 0x00.

Resources