Is there any shell command to convert large numbers from one base to another
for ex:
Converting 1024-bit binary number into hexadecimal number
You could have a look at bc, here. E.g. convert 12 to binary (output base = 2)
echo 'obase=2;12' | bc
1100
And also dc, here.
And also printf. E.g.
printf "%x" 32
20
Or you can Perl with bigint or bignum. See here. E.g.
perl -e '$line="1101111010101101101111101110111111011110101011011011111011101111110111101010110110111110111011111101111010101101101111101110111111011110101011011011111011101111110111101010110110111110111011111101111010101101101111101110111111011110101011011011111011101111110111101010110110111110111011111101111010101101101111101110111111011110101011011011111011101111110111101010110110111110111011111101111010101101101111101110111111011110101011011011111011101111110111101010110110111110111011111101111010101101101111101110111111011110101011011011111011101111110111101010110110111110111011111101111010101101101111101110111111011110101011011011111011101111110111101010110110111110111011111101111010101101101111101110111111011110101011011011111011101111110111101010110110111110111011111101111010101101101111101110111111011110101011011011111011101111110111101010110110111110111011111101111010101101101111101110111111011110101011011011111011101111110111101010110110111110111011111101111010101101101111101110111111011110101011011011111011101111";$hex=unpack("H*",pack("B*",$line));print $hex'
Output:
deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef
Also, you can use xxd to convert hex to binary:
echo -n $'\x02\x02' | xxd -b
0000000: 00000010 00000010
Related
This question already has answers here:
hexdump output order
(2 answers)
Closed 10 months ago.
Why does hexdump print 00ff here? I expected it to print ff00 , like it got in stdin, but
$ printf "\xFF\x00" | hexdump
0000000 00ff
0000002
hexdump decided to reverse it? why?
This is because hexdump is dumping 16-bit WORDS (=2-bytes-hex) and x86 processors stores words in little-endian format (you're probably using this processor).
From Wikipedia, Endianness:
A big-endian system stores the most significant byte of a word at the
smallest memory address and the least significant byte at the largest.
A little-endian system, in contrast, stores the least-significant byte
at the smallest address.
Notice that, when you use hexdump without specifiyng a parameter, the output is similar to -x.
From hexdump, man page:
If no format strings are specified, the default
display is very similar to the -x output format (the
-x option causes more space to be used between format
units than in the default output).
...
-x, --two-bytes-hex
Two-byte hexadecimal display. Display the input offset in
hexadecimal, followed by eight space-separated, four-column,
zero-filled, two-byte quantities of input data, in
hexadecimal, per line.
If you want to dump single bytes in order, use -C parameter or specify your custom formatting with -e.
$ printf "\xFF\x00" | hexdump -C
00000000 ff 00 |?.|
00000002
$ printf "\xFF\x00" | hexdump -e '"%07.7_ax " 8/1 "%02x " "\n"'
0000000 ff 00
From the hexdump(1) man page:
If no format strings are specified, the default display is very similar to the -x output format
-x, --two-bytes-hex
Two-byte hexadecimal display. Display the input offset in hexadecimal, followed by eight space-separated, four-column, zero-filled, two-byte quantities of input data, in hexadecimal, per line.
On a little-endian host the most significant byte (here 0xFF) is listed last.
There already is a beautiful trick in this thread
to write bytes to binary file at desired address with dd ,is there any way to swap bytes(e.g swap 0x00 and 0xFF), or replace bytes with common tools (such as dd)?
Would you please try the following:
xxd -p input_file | fold -w2 | perl -pe 's/00/ff/ || s/ff/00/' | xxd -r -p > output_file
xxd -p file dumps the binary data file in continuous hexdump style.
fold -w2 wraps the input lines by every two characters (= every bytes).
perl -pe 's/00/ff/ || s/ff/00/' swaps 00 and ff in the input string.
The || logic works as if .. else .. condition. Otherwise the input 00
is once converted to ff and immediately converted back to 00 again.
xxd -r -p is the reversed version of xxd -p which converts the input
hexadecimal strings into binaries.
I'd like to run a command similar to:
# echo 00: 0123456789abcdef | xxd -r | od -tx1
0000000 01 23 45 67 89 ab cd ef
0000010
That is, I'd like to input a hex string and have it converted to bytes on stdout. However, I'd like it to respect byte order of the machine I'm on, which is little endian. Here's the proof:
# lscpu | grep Byte.Order
Byte Order: Little Endian
So, I'd like it to work as above if my machine was big-endian. But since it isn't, I'd like to see:
# <something different here> | od -tx1
0000000 ef cd ab 89 67 45 23 01
0000010
Now, xxd has a "-e" option for little endianess. But 1) I want machine endianess, because I'd like something that works on big or little endian machines, and 2) "-e" isn't support with "-r" anyway.
Thanks!
What about this —
$ echo 00: 0123456789abcdef | xxd -r | xxd -g 8 -e | xxd -r | od -tx1
0000000 ef cd ab 89 67 45 23 01
0000010
According to man xxd:
-e
Switch to little-endian hexdump. This option treats byte groups as words in little-endian byte order. The default grouping of 4 bytes may be changed using -g. This option only applies to hexdump, leaving the ASCII (or EBCDIC) representation unchanged. The command line switches -r, -p, -i do not work with this mode.
-g bytes | -groupsize bytes
Separate the output of every bytes bytes (two hex characters or eight bit-digits each) by a whitespace. Specify -g 0 to suppress grouping. Bytes defaults to 2 in normal mode, 4 in little-endian mode and 1 in bits mode. Grouping does not apply to postscript or include style.
I need to check my string variable for presence of extended ASCII characters, one byte, decimal code 128-255. If any is there, replace it with multiple character hex equivalent, ready for further grep command etc.
Example string: "Ørsted\ Salg", I need it to be converted to "\xD8rsted\ Salg".
I know the way to do it with hastable in Bash 4:
declare -A symbolHashTable=(
["Ø"]="D8"
);
currSearchTerm="Ørsted\ Salg"
for curRow in "${!symbolHashTable[#]}"; do
currSearchTerm=$(echo $currSearchTerm | sed s/$curRow/'\\x'${symbolHashTable[$curRow]}/)
done
, but that seems too tedious for 127 cases. There should be a way to do it shorter and probably faster, without writing all the symbols.
I can detect whether the string has any of the characters in it with:
echo $currSearchTerm | grep -P "[\x80-\xFF]"
I am almost sure there is a way to make sed do it, but I get lost somewhere in the "replace with" part.
You can easily do this with Perl:
#!/bin/bash
original='Ørsted'
replaced=$(perl -pe 's/([\x80-\xFF])/"\\x".unpack "H*", $1/eg' <<< "$original")
echo "The original variable's hex encoding is:"
od -t x1 <<< "$original"
echo "Therefore I converted $original into $replaced"
Here's the output when the file and terminal is ISO-8859-1:
The original variable's hex encoding is:
0000000 d8 72 73 74 65 64 0a
0000007
Therefore I converted Ørsted into \xd8rsted
Here's the output when the file and terminal is UTF-8:
The original variable's hex encoding is:
0000000 c3 98 72 73 74 65 64 0a
0000010
Therefore I converted Ørsted into \xc3\x98rsted
In both cases it works as expected.
I have a csv file which is just a simple comma-separated list of numbers. I want to convert this csv file into a binary file (just a sequence of bytes, with each interpreted number being a number from the csv file).
The reason I am doing this is to be able to import audio data from a spreadsheet of values. In my import (I am using audacity), I have a few formats to choose from for the binary file:
Encoding:
Signed 8, 24, 16, or 32 bit PCM
Unsigned 8 bit PCM
32 bit or 64 bit float
U-Law
A-Law
GSM 6.10
12, 16, or 24 bit DWVW
VOX ADPCM
Byte Order:
No endianness
Big endian
Little endian
I was moving along the lines of big endian 32-bit float to keep things simple. I wanted to keep things as simple as possible, so I was thinking bash would be the optimal tool.
I have a csv file which is just a simple comma-separated list of numbers. I want to convert this csv file into a binary file [...]
I was moving along the lines of big endian 32-bit float to keep things simple.
Not sure how to do it in pure bash (actually doubt that it is doable, since float as binary is non-standard conversion).
But here it is with a simple Perl one-liner:
$ cat example1.csv
1.0
2.1
3.2
4.3
$ cat example1.csv | perl -ne 'print pack("f>*", split(/\s*,\s*/))' > example1.bin
$ hexdump -C < example1.bin
00000000 3f 80 00 00 40 06 66 66 40 4c cc cd 40 89 99 9a |?...#.ff#L..#...|
00000010
It uses the Perl's pack function with f to convert floats to binary, and < to convert them into BE. (I have also added the split in case of multiple numbers per CSV line.)
P.S. The command to convert to integers to 16-bit shorts with native endianness:
perl -ne 'print pack("s*", split(/\s*,\s*/))'
Use "s>*" for BE, or "s<*" for LE, instead of the "s*".
P.P.S. If it is audio data, you can also check the sox tool. Haven't used it in ages, but IIRC it could convert anything PCM-like from literally any format to any format, while also applying effects.
I would recommend Python over bash. For this particular task, it's simpler/saner IMO.
#!/usr/bin/env python
import array
with open('input.csv', 'rt') as f:
text = f.read()
entries = text.split(',')
values = [int(x) for x in entries]
# do a scalar here: if your input goes from [-100, 100] then
# you may need to translate/scale into [0, 2^16-1] for
# 16-bit PCM
# e.g.:
# values = [(val * scale) for val in values]
with open('output.pcm', 'wb') as out:
pcm_vals = array.array('h', values) # 16-bit signed
pcm_vals.tofile(out)
You could also use Python's wave module instead of just writing raw PCM.
Here's how the example above works:
$ echo 1,2,3,4,5,6,7 > input.csv
$ ./so_pcm.py
$ xxd output.pcm
0000000: 0100 0200 0300 0400 0500 0600 0700 ..............
xxd shows the binary values. It used my machine's native endianness (little).