The question is based on the output of the following command:
$ hexdump -C acpid.pid
00000000 36 39 37 0a |697.|
00000004
As expected, 0x36 0x39 0x37 are resolved to their associated symbols 6 9 7.
Since 0x0A is a line feed their is no ordinary symbol to respresent it (according to the ASCII table), but
Why is 0x0A getting resolved to a dot?
My operating system is Ubuntu 18.04.3.
All the unprintable characters are showed as dots in the ASCII column. See e.g. the following printout of the bytes 0x00 to 0xff:
$ for ((i=0; i<=255; ++i)); do printf "\x$(printf %x $i)"; done | hexdump -C
00000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
00000010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................|
00000020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f | !"#$%&'()*+,-./|
00000030 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f |0123456789:;<=>?|
00000040 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f |#ABCDEFGHIJKLMNO|
00000050 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f |PQRSTUVWXYZ[\]^_|
00000060 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f |`abcdefghijklmno|
00000070 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f |pqrstuvwxyz{|}~.|
00000080 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f |................|
00000090 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f |................|
000000a0 a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af |................|
000000b0 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf |................|
000000c0 c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf |................|
000000d0 d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df |................|
000000e0 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef |................|
000000f0 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff |................|
00000100
Related
I am trying to post a public key from a server with Bash to Github as part of an automation set-up. I am using cURL for that like so:
# Make API call to Github
api_token="some string"
pub_key="$(cat /home/${project_name}/.ssh/id_rsa.pub)"
echo $pub_key
curl -H "Authorization: token ${api_token}" -H "Content-Type: application/json" -X POST -d '{"title":"'"$project_name"'","key":"'"$pub_key"'"}' https://api.github.com/user/keys
As you can see I echo the pub_key just to make sure its getting it correctly, this is the output and the result of the API call to Github:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDi62x5maR2U42+asddDSF322dsdkjTNeG69QvV3crOeDgjiZZJTCwqoxkk9lDbfkfycS6QBazA8cPv4scxL1K1bWgXQ6p8/9TWP89/oUJcf+C9+bMzIFmJ6jOGA2ikA0+K6l7Gr4Y7SZR9UuctawFR56vFqWj9YEW7JhBQEVES9TNb+WJLZ6QPwl+PaTOt/6QXIvrnh0ffI5uuTkMg1m7XGMNuTnBnHYa2OQ0GIfL0zG8CEqLWch4AkP2gkBOFH3LnIwucS00ii9xpPVBLRv5O5WCHM9m6A7RtHtKyELu5Z6IOtLVHC6SRPnVdUaw4oEmt3aekqEEnXkq4Jom8arpiULWCYB9d5C4x6CGRrKqophHAfumoQFFh6GMtNrDLcVGUXmIa8qapOQNvsSzcDeVkyCbNu6RlurwyWYG8MH48XCrFDU3L+hmYptGEARgGd9IZSctMng/elUOIz1DL3ZGH43flMelXEZ8Umy8tAkat7/T8Yuu9tU8N8DKqKV16s= stackoverflowtest#rasenberg
{
"id": 46506612,
"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDi62x5maR2U42+asddDSF322dsdkjTNeG69QvV3crOeDgjiZZJTCwqoxkk9lDbfkfycS6QBazA8cPv4scxL1K1bWgXQ6p8/9TWP89/oUJcf+C9+bMzIFmJ6jOGA2ikA0+K6l7Gr4Y7SZR9UuctawFR56vFqWj9YEW7JhBQEVES9TNb+WJLZ6QPwl+PaTOt/6QXIvrnh0ffI5uuTkMg1m7XGMNuTnBnHYa2OQ0GIfL0zG8CEqLWch4AkP2gkBOFH3LnIwucS00ii9xpPVBLRv5O5WCHM9m6A7RtHtKyELu5Z6IOtLVHC6SRPnVdUaw4oEmt3aekqEEnXkq4Jom8arpiULWCYB9d5C4x6CGRrKqophHAfumoQFFh6GMtNrDLcVGUXmIa8qapOQNvsSzcDeVkyCbNu6RlurwyWYG8MH48XCrFDU3L+hmYptGEARgGd9IZSctMng/elUOIz1DL3ZGH43flMelXEZ8Umy8tAkat7/T8Yuu9tU8N8DKqKV16s=",
"url": "https://api.github.com/user/keys/46506612",
"title": "stackoverflowtest",
"verified": true,
"created_at": "2020-09-27T04:23:30Z",
"read_only": false
}
As you can see in the API call, Curl cuts of the = stackoverflowtest#rasenberg part at the end, and therefore I post an invalid pub key to Github, resulting it in not working. What is going wrong?
As https://stackoverflow.com/users/3266847/benjamin-w commented (!), the comment portion of an OpenSSH-format pubkey is optional and not needed, and was almost certainly removed by github (after receipt) not by curl. However, your key is in fact invalid and I'm a bit surprised github considers it verified.
$ printf AAAAB3NzaC1yc2EAAAADAQABAAABgQDi62x5maR2U42+asddDSF322dsdkjTNeG69QvV3crOeDgjiZZJTCwqoxkk9lDbfkfycS6QBazA8cPv4scxL1K1bWgXQ6p8/9TWP89/oUJcf+C9+bMzIFmJ6jOGA2ikA0+K6l7Gr4Y7SZR9UuctawFR56vFqWj9YEW7JhBQEVES9TNb+WJLZ6QPwl+PaTOt/6QXIvrnh0ffI5uuTkMg1m7XGMNuTnBnHYa2OQ0GIfL0zG8CEqLWch4AkP2gkBOFH3LnIwucS00ii9xpPVBLRv5O5WCHM9m6A7RtHtKyELu5Z6IOtLVHC6SRPnVdUaw4oEmt3aekqEEnXkq4Jom8arpiULWCYB9d5C4x6CGRrKqophHAfumoQFFh6GMtNrDLcVGUXmIa8qapOQNvsSzcDeVkyCbNu6RlurwyWYG8MH48XCrFDU3L+hmYptGEARgGd9IZSctMng/elUOIz1DL3ZGH43flMelXEZ8Umy8tAkat7/T8Yuu9tU8N8DKqKV16s= |openssl base64 -d -A|od -Ax -tx1
000000 00 00 00 07 73 73 68 2d 72 73 61 00 00 00 03 01
000010 00 01 00 00 01 81 00 e2 eb 6c 79 99 a4 76 53 8d
000020 be 6a c7 5d 0d 21 77 db 67 6c 76 48 d3 35 e1 ba
000030 f5 0b d5 dd ca ce 78 38 23 89 96 49 4c 2c 2a a3
000040 19 24 f6 50 db 7e 47 f2 71 2e 90 05 ac c0 f1 c3
000050 ef e2 c7 31 2f 52 b5 6d 68 17 43 aa 7c ff d4 d6
000060 3f cf 7f a1 42 5c 7f e0 bd f9 b3 33 20 59 89 ea
000070 33 86 03 68 a4 03 4f 8a ea 5e c6 af 86 3b 49 94
000080 7d 52 e7 2d 6b 01 51 e7 ab c5 a9 68 fd 60 45 bb
000090 26 10 50 11 51 12 f5 33 5b f9 62 4b 67 a4 0f c2
0000a0 5f 8f 69 33 ad ff a4 17 22 fa e7 87 47 df 23 9b
0000b0 ae 4e 43 20 d6 6e d7 18 c3 6e 4e 70 67 1d 86 b6
0000c0 39 0d 06 21 f2 f4 cc 6f 02 12 a2 d6 72 1e 00 90
0000d0 fd a0 90 13 85 1f 72 e7 23 0b 9c 4b 4d 22 8b dc
0000e0 69 3d 50 4b 46 fe 4e e5 60 87 33 d9 ba 03 b4 6d
0000f0 1e d2 b2 10 bb b9 67 a2 0e b4 b5 47 0b a4 91 3e
000100 75 5d 51 ac 38 a0 49 ad dd a7 a4 a8 41 27 5e 4a
000110 b8 26 89 bc 6a ba 62 50 b5 82 60 1f 5d e4 2e 31
000120 e8 21 91 ac aa a8 a6 11 c0 7e e9 a8 40 51 61 e8
000130 63 2d 36 b0 cb 71 51 94 5e 62 1a f2 a6 a9 39 03
000140 6f b1 2c dc 0d e5 64 c8 26 cd bb a4 65 ba bc 32
000150 59 81 bc 30 7e 3c 5c 2a c5 0d 4d cb fa 19 98 a6
000160 d1 84 01 18 06 77 d2 19 49 cb 4c 9e 0f de 95 43
000170 88 cf 50 cb dd 91 87 e3 77 e5 31 e9 57 11 9f 14
000180 9b 2f 2d 02 46 ad ef f4 fc 62 eb bd b5 4f 0d f0
000190 32 aa 29 5d 7a
000195
The length of n, 00 00 01 81 at byte offsets 0x12-0x15, would correspond to a 3072-bit RSA key (with the sign byte required by SSH mpint) but implies the total length of the blob (after base64 decoding, or before encoding) should be 0x197 and instead it's actually 0x195, making it invalid and unusable. Check whatever program or process you used to create this key; there's a bug somewhere.
I was showing my kid some basic scripting with ffmpeg to show him how much easier it is with automation.
Which, of course, failed miserably.
Specifically, we wrote a simple python script that outputs something like:
ffmpeg -ss 0:00:00 -t 1:00:00 -i wholebook.mp3 'Part 01.opus'
ffmpeg -ss 1:00:00 -t 1:00:00 -i wholebook.mp3 'Part 02.opus'
ffmpeg -ss 2:00:00 -t 1:00:00 -i wholebook.mp3 'Part 03.opus'
.
.
.
Which works fine, so then I piped it to bash: python script.py | bash
ffmpeg converted the first part, then started spewing errors and data dumps for most of the rest. (Ok, it got part 15 right, too.)
Copy/pasting each command in turn works fine.
Changing the output to .mp3 didn't matter; it seems like it's erroring out while reading the input.
Why is piping these commands to bash failing?
Update: it's not just piping it to bash. If, instead of cutting & pasting each command and waiting for it to finish, he cuts & pastes the whole series of commands into the terminal (running bash), it fails in the same way.
The error output starts with:
stream #0:
keyframe=1
duration=0.026
dts=426.005 pts=426.005
size=418
stream #0:
keyframe=1
duration=0.026
dts=426.031 pts=426.031
size=418
stream #0:
keyframe=1
duration=0.026
dts=426.057 pts=426.057
size=418
and continues for many thousands of lines. At some point, it switches to:
stream #0:
keyframe=1
duration=0.026
dts=764.473 pts=764.473
size=418
00000000 ff fb 92 64 f0 89 f5 f8 6c 3f 83 2c 4b 52 00 00 ...d....l?.,KR..
00000010 0d 20 00 00 01 19 55 b0 fe ad 31 37 08 00 00 34 . ....U...17...4
00000020 80 00 00 04 71 63 57 34 87 7a a1 ad 69 ab 77 53 ....qcW4.z..i.wS
00000030 fd fb 88 28 d3 10 b2 66 9f 99 dc 82 46 27 44 a2 ...(...f....F'D.
00000040 75 a8 5d b1 83 34 9a 5e bd 72 59 b1 11 1e 0a b8 u.]..4.^.rY.....
00000050 6d e9 d1 61 94 5c b6 92 ac cf 16 4b 74 d4 7c b7 m..a.\.....Kt.|.
00000060 ab c7 7a 6a 6b 7e f1 4e 55 fc 69 dd 2a f0 e1 21 ..zjk~.NU.i.*..!
00000070 09 10 25 09 a8 79 fe d1 da cb 34 06 8a 08 28 21 ..%..y....4...(!
00000080 47 6d 5d b6 06 f1 57 af 59 7b 75 86 60 a8 dd 99 Gm]...W.Y{u.`...
00000090 a4 87 45 aa 90 e1 67 74 78 30 a1 83 08 ce d9 ee ..E...gtx0......
000000a0 4f cf 22 f8 a3 46 4a a1 7c 7b a1 e6 59 df 76 86 O."..FJ.|{..Y.v.
000000b0 53 c5 5e 85 1d 39 57 ac a3 0e c5 03 93 51 f6 c6 S.^..9W......Q..
000000c0 86 2b 9c e1 33 27 59 c8 96 76 db 1c 96 42 58 93 .+..3'Y..v...BX.
000000d0 ce 46 83 44 a2 48 e5 66 0a 60 30 70 27 bc 3f 2a .F.D.H.f.`0p'.?*
000000e0 2d d9 6d 4d d9 79 41 7c d8 fc a9 64 35 af 88 eb -.mM.yA|...d5...
000000f0 15 a4 32 b2 73 0b 18 88 04 81 c9 60 48 ba 97 54 ..2.s......`H..T
00000100 24 89 28 65 e4 8b 0e ce 11 a1 19 39 05 0e 0f 31 $.(e.......9...1
00000110 62 e7 9a 3d 5c 3e b9 64 69 4f f6 f1 1d 19 bb 8f b..=\>.diO......
00000120 e1 29 60 b8 d6 ca 35 f8 ce 50 d4 9f dd 4a e3 d7 .)`...5..P...J..
00000130 79 d3 75 8e 15 67 c9 8b ac d9 a5 6a fa 43 47 10 y.u..g.....j.CG.
00000140 d1 46 fd 9c 93 d8 a3 33 74 2e 80 00 82 94 4c f3 .F.....3t.....L.
00000150 0e 02 8c 07 41 31 48 95 d8 f2 85 89 38 5e 65 37 ....A1H.....8^e7
00000160 76 dd 2b 2e c5 1c 39 13 9b 9b ae ab 25 cc 79 45 v.+...9.....%.yE
00000170 1e 60 1d b0 20 cf d2 12 cc 80 57 9b 99 65 78 9c .`.. .....W..ex.
00000180 82 a3 03 c0 b2 11 6b a5 49 c2 85 bd 3f 90 d5 73 ......k.I...?..s
00000190 13 f4 9e c5 29 23 0a 2a 25 c5 9d eb 5e 9c 95 c9 ....)#.*%...^...
000001a0 ec aa ..
stream #0:
keyframe=1
duration=0.026
dts=764.500 pts=764.500
size=418
00000000 ff fb 92 64 ea 80 f5 14 69 41 cb 0c 34 72 00 00 ...d....iA..4r..
00000010 0d 20 00 00 01 1b 49 b0 fc ac bd 91 88 00 00 34 . ....I........4
00000020 80 00 00 04 5b 32 ac c2 16 a8 6d 65 8d 09 c5 3b ....[2....me...;
00000030 64 81 da 43 d5 27 4a 3d b1 8d c1 f5 95 ba e9 61 d..C.'J=.......a
00000040 62 a7 6a 5e 5d ab 58 7a 23 b7 8a f5 3f 2a 59 62 b.j^].Xz#...?*Yb
...
ffmpeg reads from stdin too, to allow you to change options or exit interactively. This means that the commands you meant for bash are instead read by ffmpeg, and e.g. a q will cause it to quit.
You can instead:
Run the command directly from Python, rather than writing commands to bash. This is generally more robust, since you don't have to do any shell escaping.
Use ffmpeg -nostdin ... or ffmpeg < /dev/null ... to prevent ffmpeg from consuming stdin.
Prepend a line with { and append a line with }. That way, bash will read all commands as a single group up front, so there's no remaining stdin to be had for ffmpeg.
I have an old application that stored images (scanned B&W page) in a large binary file. I have worked out how to single out the individual pages, but I can't decipher the binary for each page. As far as I know, the original pages were TIFF images. The resulting binary is also 1/16th the size, so has obviously been compressed. I'm also not sure of the "endianness" of the files. How would I go about identifying how this file has been stored/compressed or even possibly encrypted? I've included the binary for one of the (smallest) pages (This page should be white with a small line of text in the middle). The original was 58k. Incidentally, all of the pages have the same 14 byte header.
01 05 22 28 8E A9 F0 70 BD EF 03 2E EE 0F D8 B8 31 9B B8 03 D0 85 21 81 81 05 74 71 31 25 40 17 34 1E EF 07 A0 8B BD 29 A0 5B D6 C8 B8 B9 0A 31 30 01 26 48 12 B8 30 88 91 70 B0 D0 71 E0 62 63 E2 C2 80 CB 81 2B E2 2E AF F5 82 8B F4 39 AA 98 C5 FD 0B 4F 80 0C F5 37 8E 17 A0 B2 BF 5B 69 09 0E CC 2F 70 B6 44 07 BA 5D C3 E3 F8 55 41 81 EB 68 D6 E2 C7 53 14 1A 5C 74 B8 3C B8 88 F9 C8 A4 97 27 6A E2 69 95 A5 20 B1 EC A3 04 2B 5A E7 4F 93 AD EB 12 27 62 1F 9C 28 7D 60 C4 B4 E9 23 F4 68 8D A1 47 AA 8F 1B B1 3E 6E A4 3E 92 88 7D 24 A5 8F 62 91 6D 35 5A 7C 75 19 97 76 6E 0D A3 B6 86 B2 DC 61 80 18 D2 F6 58 6C 96 98 F5 0F 2A 9F 25 41 DD 9E CF D9 15 E1 FA 3E FC 24 F1 38 98 98 F9 F1 1F 8E C7 C6 DE DD 18 F4 99 6D 85 83 8B 91 1F FF 61 75 DB 69 C4 42 47 C7 7D 4E D7 C9 02 82 5B 0E 22 B8 18 30 96 39 F0 67 51 2C D5 93 1C AA ED 29 CE 16 4C DB 36 BC 69 D3 94 AD 4C C7 40 F0 96 A1 6C 62 A1 D0 8D C5 6C 82 BB 8E 41 F4 65 24 E8 70 B1 B1 56 6C 0B 8C B4 16 FF 8A 5A 45 90 60 83 28 0C 7C BF 50 02 63 93 0E D6 5F 98 F8 78 6A 7E 61 6D 6D CD 6A A9 E1 AC 3E 56 E9 9F BB 4F FF 8D 6E 0C F8 23 4D 32 0B AD 4E BB C2 03 FE 01
The following code fails to generate binary numbers if backticks are replaced by dollar-parenthesis syntax:
#!/bin/bash
rm test.bin 2>/dev/null
for character in {0..255}
do
char=`printf '\\\\x'"%02x" $character`
printf "$char" >> test.bin
done
hexdump -C test.bin
Result:
00000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
00000010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................|
00000020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f | !"#$%&'()*+,-./|
00000030 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f |0123456789:;<=>?|
00000040 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f |#ABCDEFGHIJKLMNO|
00000050 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f |PQRSTUVWXYZ[\]^_|
00000060 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f |`abcdefghijklmno|
00000070 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f |pqrstuvwxyz{|}~.|
00000080 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f |................|
00000090 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f |................|
000000a0 a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af |................|
000000b0 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf |................|
000000c0 c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf |................|
000000d0 d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df |................|
000000e0 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef |................|
000000f0 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff |................|
That's ok so far. Let's replace backticks and see what we get:
#!/bin/bash
rm test.bin 2>/dev/null
for character in {0..255}
do
char=$(printf '\\\\x'"%02x" $character)
printf "$char" >> test.bin
done
hexdump -C test.bin
Result:
00000000 5c 78 30 30 5c 78 30 31 5c 78 30 32 5c 78 30 33 |\x00\x01\x02\x03|
00000010 5c 78 30 34 5c 78 30 35 5c 78 30 36 5c 78 30 37 |\x04\x05\x06\x07|
00000020 5c 78 30 38 5c 78 30 39 5c 78 30 61 5c 78 30 62 |\x08\x09\x0a\x0b|
00000030 5c 78 30 63 5c 78 30 64 5c 78 30 65 5c 78 30 66 |\x0c\x0d\x0e\x0f|
00000040 5c 78 31 30 5c 78 31 31 5c 78 31 32 5c 78 31 33 |\x10\x11\x12\x13|
00000050 5c 78 31 34 5c 78 31 35 5c 78 31 36 5c 78 31 37 |\x14\x15\x16\x17|
00000060 5c 78 31 38 5c 78 31 39 5c 78 31 61 5c 78 31 62 |\x18\x19\x1a\x1b|
00000070 5c 78 31 63 5c 78 31 64 5c 78 31 65 5c 78 31 66 |\x1c\x1d\x1e\x1f|
.
.
While I prefer dollar-parenthesis syntax it appears to fail in this case but why ?
Credits for the code snippet:
http://code.activestate.com/recipes/578441-a-building-block-bash-binary-file-manipulation
You are running into one of the reasons that $() is preferred to the backtick notation. The shell parsing of $() is more consistent (as it introduces a new parsing context as I understand it).
So your escaping, while correct for the backtick code, is excessive for the $() code.
Try this:
$ : > test.bin; for character in {0..255}
do
char=$(printf '\\x'"%02x" $character)
printf "$char" >> test.bin
done; hexdump -C test.bin
00000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
00000010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................|
00000020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f | !"#$%&'()*+,-./|
00000030 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f |0123456789:;<=>?|
00000040 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f |#ABCDEFGHIJKLMNO|
00000050 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f |PQRSTUVWXYZ[\]^_|
00000060 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f |`abcdefghijklmno|
00000070 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f |pqrstuvwxyz{|}~.|
00000080 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f |................|
00000090 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f |................|
000000a0 a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af |................|
000000b0 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf |................|
000000c0 c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf |................|
000000d0 d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df |................|
000000e0 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef |................|
000000f0 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff |................|
00000100
A little more clearly compare this
$ printf %s\\n `printf %s "\\\\ff"`
\ff
$ printf %s\\n `printf %s '\\\\ff'`
\\ff
to this
$ printf %s\\n $(printf %s "\\\\ff")
\\ff
$ printf %s\\n $(printf %s '\\\\ff')
\\\\ff
This is the difference:
echo `echo '\\'`
\
echo $(echo '\\')
\\
From the manual, Command substitution section:
When the old-style backquoted form of substitution is used, backslash retains its literal meaning except when followed by "$", "`", or "\".
When using the "$(COMMAND)" form, all characters between the parentheses make up the command; none are treated specially.
$() doesn't need extra escaping for \. Use:
char=$(printf '\\x'"%02x" $character)
Would someone please explain why this script sometime return only 15 bytes in hex string representation?
for i in {1..10}; do
API_IV=`openssl rand 16`; API_IV_HEX=`echo -n "$API_IV" | od -vt x1 -w16 | awk '{$1="";print}'`; echo $API_IV_HEX;
done
like this:
c2 2a 09 0f 9a cd 64 02 28 06 43 f8 13 80 a5 04
fa c4 ac b1 95 23 7c 36 95 2d 5e 0e bf 05 fe f4
38 55 d3 b4 32 bb 61 f4 fd 17 92 67 e2 9b b4 04
6d a7 f8 46 e9 99 bd 89 87 f9 7f 2b 15 5a 17 8a
11 c8 89 f4 8f 66 93 f1 6d b9 2b 64 7e 01 61 68
93 e3 9d 28 95 e1 c8 92 e5 62 d9 bf 20 b3 1c dd
37 64 ef b0 2f da c7 60 1c c8 20 b8 28 9d f9
29 f0 5a e9 cc 36 66 de 02 82 fc 8e 36 bf 5d d1
b2 57 d8 79 21 df 73 1c af 07 e9 80 0a 67 c6 15
ba 77 cb 92 39 42 39 f9 a4 57 c8 c4 be 62 19 54
If pipe the "openssl rand 16" directly to the od command then it works fine, but I need the binary value. Thanks for your help.
echo, like various other standard commands, considers \x00 as an end-of-string marker. So stop displaying after it.
Maybe you are looking to the -hex option of openssl rand:
sh$ openssl rand 16 -hex
4248bf230fc9dd927ab53f799e2a9708
Given that option is available on your system, your example could be rewritten:
sh$ openssl version
OpenSSL 1.0.1e 11 Feb 2013
sh$ for i in {1..10}; do
openssl rand 16 -hex | sed -e 's|..|& |g' -e 's| $||'
done
20 cb 6b 7a 85 2d 0b fe 9e c7 d0 4b 91 88 1b bb
5d 74 99 5e 05 c9 7d 9d 37 dd 02 f3 23 bb c5 b7
51 e9 0f dc 58 04 5e 30 e3 6b 9f 63 aa fc 95 05
fc 6b b8 cb 05 82 53 85 78 0e 59 13 3b e7 c1 4b
cf fa fc d9 1a 25 df e0 f8 59 71 a6 2c 64 c5 87
93 1a 29 b4 5a 52 77 bb 3f bb 1d 0a 46 5d c8 b4
0c bb c2 b2 b4 89 d4 37 1c 86 0a 7a 58 b8 64 e2
ee fc a7 ec 6c f8 7f 51 04 43 d6 00 d8 79 65 43
b9 73 9e cc 4b 42 9e 64 9d 5b 21 6a 20 b7 c3 16
06 8a 15 22 6a d5 ae ab 9a d2 9f 60 f1 a9 26 bd
If you need to later convert from hex to bytes use this perl one-liner:
echo MY_HEX_STRING |
perl -ne 's/([0-9a-f]{2})/print chr hex $1/gie' |
my_tool_reading_binary_input_from_stdin
(from https://stackoverflow.com/a/1604770/2363712)
Please note I pipe to the client program and o not use a shell variable here, as I think it cannot properly handle the \x00.
As the bash cannot properly deal with binary strings containing \x00 your best bet if you absolutely want to stick with shell programming is to use an intermediate file to store binary data. And not a variable:
This is the idea. Feel free to adapt to your needs:
for i in {1..10}; do
openssl rand 16 > api_iv_$i # better use `mktemp` here
API_IV_HEX=`od -vt x1 -w16 < api_iv_$i | awk '{$1="";print}'`
echo $API_IV_HEX;
done
sh$ bash t.sh
cf 06 ab ab 86 fd ef 22 1a 2c bd 7f 8c 45 27 e5
2a 01 9c 7a fa 15 d3 ea 40 89 8b 26 d5 4f 97 08
55 2e c9 d3 cd 0d 3a 6f 1b a0 fe 38 6d 0e 20 07
fe 60 35 62 17 80 f2 db 64 7a af da 81 ff f7 e0
74 9a 5c 39 0e 1a 6b 89 a3 21 65 01 a3 de c4 1c
c3 11 45 e3 e0 dc 66 a3 e8 fb 5b 8a bd d0 7d 43
a4 ee 80 f8 c8 8b 4e 50 5c dd 21 00 3b d0 bc cf
e2 d5 11 d4 7d 98 08 a7 16 7b 8c 56 44 ba 6d 53
ad 63 65 fd bf 3f 1f 4a a1 c5 d0 58 23 ae d1 47
80 74 f1 d0 b9 00 e5 1d 50 74 53 96 4b ce 59 50
sh$ hexdump -C ./api_iv_10
00000000 80 74 f1 d0 b9 00 e5 1d 50 74 53 96 4b ce 59 50 |.t......PtS.K.YP|
00000010
As a personal opinion, if you really have a lot of binary data processing, I would recommend to switch to some other language more data oriented (Python, ...)
Because the missing byte was an ASCII NUL '\0' '\x00'. The echo command stops printing its argument(s) when it comes across a null byte in each argument.