I have an old application that stored images (scanned B&W page) in a large binary file. I have worked out how to single out the individual pages, but I can't decipher the binary for each page. As far as I know, the original pages were TIFF images. The resulting binary is also 1/16th the size, so has obviously been compressed. I'm also not sure of the "endianness" of the files. How would I go about identifying how this file has been stored/compressed or even possibly encrypted? I've included the binary for one of the (smallest) pages (This page should be white with a small line of text in the middle). The original was 58k. Incidentally, all of the pages have the same 14 byte header.
01 05 22 28 8E A9 F0 70 BD EF 03 2E EE 0F D8 B8 31 9B B8 03 D0 85 21 81 81 05 74 71 31 25 40 17 34 1E EF 07 A0 8B BD 29 A0 5B D6 C8 B8 B9 0A 31 30 01 26 48 12 B8 30 88 91 70 B0 D0 71 E0 62 63 E2 C2 80 CB 81 2B E2 2E AF F5 82 8B F4 39 AA 98 C5 FD 0B 4F 80 0C F5 37 8E 17 A0 B2 BF 5B 69 09 0E CC 2F 70 B6 44 07 BA 5D C3 E3 F8 55 41 81 EB 68 D6 E2 C7 53 14 1A 5C 74 B8 3C B8 88 F9 C8 A4 97 27 6A E2 69 95 A5 20 B1 EC A3 04 2B 5A E7 4F 93 AD EB 12 27 62 1F 9C 28 7D 60 C4 B4 E9 23 F4 68 8D A1 47 AA 8F 1B B1 3E 6E A4 3E 92 88 7D 24 A5 8F 62 91 6D 35 5A 7C 75 19 97 76 6E 0D A3 B6 86 B2 DC 61 80 18 D2 F6 58 6C 96 98 F5 0F 2A 9F 25 41 DD 9E CF D9 15 E1 FA 3E FC 24 F1 38 98 98 F9 F1 1F 8E C7 C6 DE DD 18 F4 99 6D 85 83 8B 91 1F FF 61 75 DB 69 C4 42 47 C7 7D 4E D7 C9 02 82 5B 0E 22 B8 18 30 96 39 F0 67 51 2C D5 93 1C AA ED 29 CE 16 4C DB 36 BC 69 D3 94 AD 4C C7 40 F0 96 A1 6C 62 A1 D0 8D C5 6C 82 BB 8E 41 F4 65 24 E8 70 B1 B1 56 6C 0B 8C B4 16 FF 8A 5A 45 90 60 83 28 0C 7C BF 50 02 63 93 0E D6 5F 98 F8 78 6A 7E 61 6D 6D CD 6A A9 E1 AC 3E 56 E9 9F BB 4F FF 8D 6E 0C F8 23 4D 32 0B AD 4E BB C2 03 FE 01
Related
I am trying to post a public key from a server with Bash to Github as part of an automation set-up. I am using cURL for that like so:
# Make API call to Github
api_token="some string"
pub_key="$(cat /home/${project_name}/.ssh/id_rsa.pub)"
echo $pub_key
curl -H "Authorization: token ${api_token}" -H "Content-Type: application/json" -X POST -d '{"title":"'"$project_name"'","key":"'"$pub_key"'"}' https://api.github.com/user/keys
As you can see I echo the pub_key just to make sure its getting it correctly, this is the output and the result of the API call to Github:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDi62x5maR2U42+asddDSF322dsdkjTNeG69QvV3crOeDgjiZZJTCwqoxkk9lDbfkfycS6QBazA8cPv4scxL1K1bWgXQ6p8/9TWP89/oUJcf+C9+bMzIFmJ6jOGA2ikA0+K6l7Gr4Y7SZR9UuctawFR56vFqWj9YEW7JhBQEVES9TNb+WJLZ6QPwl+PaTOt/6QXIvrnh0ffI5uuTkMg1m7XGMNuTnBnHYa2OQ0GIfL0zG8CEqLWch4AkP2gkBOFH3LnIwucS00ii9xpPVBLRv5O5WCHM9m6A7RtHtKyELu5Z6IOtLVHC6SRPnVdUaw4oEmt3aekqEEnXkq4Jom8arpiULWCYB9d5C4x6CGRrKqophHAfumoQFFh6GMtNrDLcVGUXmIa8qapOQNvsSzcDeVkyCbNu6RlurwyWYG8MH48XCrFDU3L+hmYptGEARgGd9IZSctMng/elUOIz1DL3ZGH43flMelXEZ8Umy8tAkat7/T8Yuu9tU8N8DKqKV16s= stackoverflowtest#rasenberg
{
"id": 46506612,
"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDi62x5maR2U42+asddDSF322dsdkjTNeG69QvV3crOeDgjiZZJTCwqoxkk9lDbfkfycS6QBazA8cPv4scxL1K1bWgXQ6p8/9TWP89/oUJcf+C9+bMzIFmJ6jOGA2ikA0+K6l7Gr4Y7SZR9UuctawFR56vFqWj9YEW7JhBQEVES9TNb+WJLZ6QPwl+PaTOt/6QXIvrnh0ffI5uuTkMg1m7XGMNuTnBnHYa2OQ0GIfL0zG8CEqLWch4AkP2gkBOFH3LnIwucS00ii9xpPVBLRv5O5WCHM9m6A7RtHtKyELu5Z6IOtLVHC6SRPnVdUaw4oEmt3aekqEEnXkq4Jom8arpiULWCYB9d5C4x6CGRrKqophHAfumoQFFh6GMtNrDLcVGUXmIa8qapOQNvsSzcDeVkyCbNu6RlurwyWYG8MH48XCrFDU3L+hmYptGEARgGd9IZSctMng/elUOIz1DL3ZGH43flMelXEZ8Umy8tAkat7/T8Yuu9tU8N8DKqKV16s=",
"url": "https://api.github.com/user/keys/46506612",
"title": "stackoverflowtest",
"verified": true,
"created_at": "2020-09-27T04:23:30Z",
"read_only": false
}
As you can see in the API call, Curl cuts of the = stackoverflowtest#rasenberg part at the end, and therefore I post an invalid pub key to Github, resulting it in not working. What is going wrong?
As https://stackoverflow.com/users/3266847/benjamin-w commented (!), the comment portion of an OpenSSH-format pubkey is optional and not needed, and was almost certainly removed by github (after receipt) not by curl. However, your key is in fact invalid and I'm a bit surprised github considers it verified.
$ printf AAAAB3NzaC1yc2EAAAADAQABAAABgQDi62x5maR2U42+asddDSF322dsdkjTNeG69QvV3crOeDgjiZZJTCwqoxkk9lDbfkfycS6QBazA8cPv4scxL1K1bWgXQ6p8/9TWP89/oUJcf+C9+bMzIFmJ6jOGA2ikA0+K6l7Gr4Y7SZR9UuctawFR56vFqWj9YEW7JhBQEVES9TNb+WJLZ6QPwl+PaTOt/6QXIvrnh0ffI5uuTkMg1m7XGMNuTnBnHYa2OQ0GIfL0zG8CEqLWch4AkP2gkBOFH3LnIwucS00ii9xpPVBLRv5O5WCHM9m6A7RtHtKyELu5Z6IOtLVHC6SRPnVdUaw4oEmt3aekqEEnXkq4Jom8arpiULWCYB9d5C4x6CGRrKqophHAfumoQFFh6GMtNrDLcVGUXmIa8qapOQNvsSzcDeVkyCbNu6RlurwyWYG8MH48XCrFDU3L+hmYptGEARgGd9IZSctMng/elUOIz1DL3ZGH43flMelXEZ8Umy8tAkat7/T8Yuu9tU8N8DKqKV16s= |openssl base64 -d -A|od -Ax -tx1
000000 00 00 00 07 73 73 68 2d 72 73 61 00 00 00 03 01
000010 00 01 00 00 01 81 00 e2 eb 6c 79 99 a4 76 53 8d
000020 be 6a c7 5d 0d 21 77 db 67 6c 76 48 d3 35 e1 ba
000030 f5 0b d5 dd ca ce 78 38 23 89 96 49 4c 2c 2a a3
000040 19 24 f6 50 db 7e 47 f2 71 2e 90 05 ac c0 f1 c3
000050 ef e2 c7 31 2f 52 b5 6d 68 17 43 aa 7c ff d4 d6
000060 3f cf 7f a1 42 5c 7f e0 bd f9 b3 33 20 59 89 ea
000070 33 86 03 68 a4 03 4f 8a ea 5e c6 af 86 3b 49 94
000080 7d 52 e7 2d 6b 01 51 e7 ab c5 a9 68 fd 60 45 bb
000090 26 10 50 11 51 12 f5 33 5b f9 62 4b 67 a4 0f c2
0000a0 5f 8f 69 33 ad ff a4 17 22 fa e7 87 47 df 23 9b
0000b0 ae 4e 43 20 d6 6e d7 18 c3 6e 4e 70 67 1d 86 b6
0000c0 39 0d 06 21 f2 f4 cc 6f 02 12 a2 d6 72 1e 00 90
0000d0 fd a0 90 13 85 1f 72 e7 23 0b 9c 4b 4d 22 8b dc
0000e0 69 3d 50 4b 46 fe 4e e5 60 87 33 d9 ba 03 b4 6d
0000f0 1e d2 b2 10 bb b9 67 a2 0e b4 b5 47 0b a4 91 3e
000100 75 5d 51 ac 38 a0 49 ad dd a7 a4 a8 41 27 5e 4a
000110 b8 26 89 bc 6a ba 62 50 b5 82 60 1f 5d e4 2e 31
000120 e8 21 91 ac aa a8 a6 11 c0 7e e9 a8 40 51 61 e8
000130 63 2d 36 b0 cb 71 51 94 5e 62 1a f2 a6 a9 39 03
000140 6f b1 2c dc 0d e5 64 c8 26 cd bb a4 65 ba bc 32
000150 59 81 bc 30 7e 3c 5c 2a c5 0d 4d cb fa 19 98 a6
000160 d1 84 01 18 06 77 d2 19 49 cb 4c 9e 0f de 95 43
000170 88 cf 50 cb dd 91 87 e3 77 e5 31 e9 57 11 9f 14
000180 9b 2f 2d 02 46 ad ef f4 fc 62 eb bd b5 4f 0d f0
000190 32 aa 29 5d 7a
000195
The length of n, 00 00 01 81 at byte offsets 0x12-0x15, would correspond to a 3072-bit RSA key (with the sign byte required by SSH mpint) but implies the total length of the blob (after base64 decoding, or before encoding) should be 0x197 and instead it's actually 0x195, making it invalid and unusable. Check whatever program or process you used to create this key; there's a bug somewhere.
The question is based on the output of the following command:
$ hexdump -C acpid.pid
00000000 36 39 37 0a |697.|
00000004
As expected, 0x36 0x39 0x37 are resolved to their associated symbols 6 9 7.
Since 0x0A is a line feed their is no ordinary symbol to respresent it (according to the ASCII table), but
Why is 0x0A getting resolved to a dot?
My operating system is Ubuntu 18.04.3.
All the unprintable characters are showed as dots in the ASCII column. See e.g. the following printout of the bytes 0x00 to 0xff:
$ for ((i=0; i<=255; ++i)); do printf "\x$(printf %x $i)"; done | hexdump -C
00000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
00000010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................|
00000020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f | !"#$%&'()*+,-./|
00000030 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f |0123456789:;<=>?|
00000040 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f |#ABCDEFGHIJKLMNO|
00000050 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f |PQRSTUVWXYZ[\]^_|
00000060 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f |`abcdefghijklmno|
00000070 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f |pqrstuvwxyz{|}~.|
00000080 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f |................|
00000090 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f |................|
000000a0 a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af |................|
000000b0 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf |................|
000000c0 c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf |................|
000000d0 d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df |................|
000000e0 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef |................|
000000f0 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff |................|
00000100
The following code fails to generate binary numbers if backticks are replaced by dollar-parenthesis syntax:
#!/bin/bash
rm test.bin 2>/dev/null
for character in {0..255}
do
char=`printf '\\\\x'"%02x" $character`
printf "$char" >> test.bin
done
hexdump -C test.bin
Result:
00000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
00000010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................|
00000020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f | !"#$%&'()*+,-./|
00000030 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f |0123456789:;<=>?|
00000040 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f |#ABCDEFGHIJKLMNO|
00000050 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f |PQRSTUVWXYZ[\]^_|
00000060 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f |`abcdefghijklmno|
00000070 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f |pqrstuvwxyz{|}~.|
00000080 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f |................|
00000090 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f |................|
000000a0 a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af |................|
000000b0 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf |................|
000000c0 c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf |................|
000000d0 d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df |................|
000000e0 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef |................|
000000f0 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff |................|
That's ok so far. Let's replace backticks and see what we get:
#!/bin/bash
rm test.bin 2>/dev/null
for character in {0..255}
do
char=$(printf '\\\\x'"%02x" $character)
printf "$char" >> test.bin
done
hexdump -C test.bin
Result:
00000000 5c 78 30 30 5c 78 30 31 5c 78 30 32 5c 78 30 33 |\x00\x01\x02\x03|
00000010 5c 78 30 34 5c 78 30 35 5c 78 30 36 5c 78 30 37 |\x04\x05\x06\x07|
00000020 5c 78 30 38 5c 78 30 39 5c 78 30 61 5c 78 30 62 |\x08\x09\x0a\x0b|
00000030 5c 78 30 63 5c 78 30 64 5c 78 30 65 5c 78 30 66 |\x0c\x0d\x0e\x0f|
00000040 5c 78 31 30 5c 78 31 31 5c 78 31 32 5c 78 31 33 |\x10\x11\x12\x13|
00000050 5c 78 31 34 5c 78 31 35 5c 78 31 36 5c 78 31 37 |\x14\x15\x16\x17|
00000060 5c 78 31 38 5c 78 31 39 5c 78 31 61 5c 78 31 62 |\x18\x19\x1a\x1b|
00000070 5c 78 31 63 5c 78 31 64 5c 78 31 65 5c 78 31 66 |\x1c\x1d\x1e\x1f|
.
.
While I prefer dollar-parenthesis syntax it appears to fail in this case but why ?
Credits for the code snippet:
http://code.activestate.com/recipes/578441-a-building-block-bash-binary-file-manipulation
You are running into one of the reasons that $() is preferred to the backtick notation. The shell parsing of $() is more consistent (as it introduces a new parsing context as I understand it).
So your escaping, while correct for the backtick code, is excessive for the $() code.
Try this:
$ : > test.bin; for character in {0..255}
do
char=$(printf '\\x'"%02x" $character)
printf "$char" >> test.bin
done; hexdump -C test.bin
00000000 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f |................|
00000010 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f |................|
00000020 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f | !"#$%&'()*+,-./|
00000030 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f |0123456789:;<=>?|
00000040 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f |#ABCDEFGHIJKLMNO|
00000050 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f |PQRSTUVWXYZ[\]^_|
00000060 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f |`abcdefghijklmno|
00000070 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f |pqrstuvwxyz{|}~.|
00000080 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f |................|
00000090 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f |................|
000000a0 a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af |................|
000000b0 b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf |................|
000000c0 c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf |................|
000000d0 d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df |................|
000000e0 e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef |................|
000000f0 f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff |................|
00000100
A little more clearly compare this
$ printf %s\\n `printf %s "\\\\ff"`
\ff
$ printf %s\\n `printf %s '\\\\ff'`
\\ff
to this
$ printf %s\\n $(printf %s "\\\\ff")
\\ff
$ printf %s\\n $(printf %s '\\\\ff')
\\\\ff
This is the difference:
echo `echo '\\'`
\
echo $(echo '\\')
\\
From the manual, Command substitution section:
When the old-style backquoted form of substitution is used, backslash retains its literal meaning except when followed by "$", "`", or "\".
When using the "$(COMMAND)" form, all characters between the parentheses make up the command; none are treated specially.
$() doesn't need extra escaping for \. Use:
char=$(printf '\\x'"%02x" $character)
Would someone please explain why this script sometime return only 15 bytes in hex string representation?
for i in {1..10}; do
API_IV=`openssl rand 16`; API_IV_HEX=`echo -n "$API_IV" | od -vt x1 -w16 | awk '{$1="";print}'`; echo $API_IV_HEX;
done
like this:
c2 2a 09 0f 9a cd 64 02 28 06 43 f8 13 80 a5 04
fa c4 ac b1 95 23 7c 36 95 2d 5e 0e bf 05 fe f4
38 55 d3 b4 32 bb 61 f4 fd 17 92 67 e2 9b b4 04
6d a7 f8 46 e9 99 bd 89 87 f9 7f 2b 15 5a 17 8a
11 c8 89 f4 8f 66 93 f1 6d b9 2b 64 7e 01 61 68
93 e3 9d 28 95 e1 c8 92 e5 62 d9 bf 20 b3 1c dd
37 64 ef b0 2f da c7 60 1c c8 20 b8 28 9d f9
29 f0 5a e9 cc 36 66 de 02 82 fc 8e 36 bf 5d d1
b2 57 d8 79 21 df 73 1c af 07 e9 80 0a 67 c6 15
ba 77 cb 92 39 42 39 f9 a4 57 c8 c4 be 62 19 54
If pipe the "openssl rand 16" directly to the od command then it works fine, but I need the binary value. Thanks for your help.
echo, like various other standard commands, considers \x00 as an end-of-string marker. So stop displaying after it.
Maybe you are looking to the -hex option of openssl rand:
sh$ openssl rand 16 -hex
4248bf230fc9dd927ab53f799e2a9708
Given that option is available on your system, your example could be rewritten:
sh$ openssl version
OpenSSL 1.0.1e 11 Feb 2013
sh$ for i in {1..10}; do
openssl rand 16 -hex | sed -e 's|..|& |g' -e 's| $||'
done
20 cb 6b 7a 85 2d 0b fe 9e c7 d0 4b 91 88 1b bb
5d 74 99 5e 05 c9 7d 9d 37 dd 02 f3 23 bb c5 b7
51 e9 0f dc 58 04 5e 30 e3 6b 9f 63 aa fc 95 05
fc 6b b8 cb 05 82 53 85 78 0e 59 13 3b e7 c1 4b
cf fa fc d9 1a 25 df e0 f8 59 71 a6 2c 64 c5 87
93 1a 29 b4 5a 52 77 bb 3f bb 1d 0a 46 5d c8 b4
0c bb c2 b2 b4 89 d4 37 1c 86 0a 7a 58 b8 64 e2
ee fc a7 ec 6c f8 7f 51 04 43 d6 00 d8 79 65 43
b9 73 9e cc 4b 42 9e 64 9d 5b 21 6a 20 b7 c3 16
06 8a 15 22 6a d5 ae ab 9a d2 9f 60 f1 a9 26 bd
If you need to later convert from hex to bytes use this perl one-liner:
echo MY_HEX_STRING |
perl -ne 's/([0-9a-f]{2})/print chr hex $1/gie' |
my_tool_reading_binary_input_from_stdin
(from https://stackoverflow.com/a/1604770/2363712)
Please note I pipe to the client program and o not use a shell variable here, as I think it cannot properly handle the \x00.
As the bash cannot properly deal with binary strings containing \x00 your best bet if you absolutely want to stick with shell programming is to use an intermediate file to store binary data. And not a variable:
This is the idea. Feel free to adapt to your needs:
for i in {1..10}; do
openssl rand 16 > api_iv_$i # better use `mktemp` here
API_IV_HEX=`od -vt x1 -w16 < api_iv_$i | awk '{$1="";print}'`
echo $API_IV_HEX;
done
sh$ bash t.sh
cf 06 ab ab 86 fd ef 22 1a 2c bd 7f 8c 45 27 e5
2a 01 9c 7a fa 15 d3 ea 40 89 8b 26 d5 4f 97 08
55 2e c9 d3 cd 0d 3a 6f 1b a0 fe 38 6d 0e 20 07
fe 60 35 62 17 80 f2 db 64 7a af da 81 ff f7 e0
74 9a 5c 39 0e 1a 6b 89 a3 21 65 01 a3 de c4 1c
c3 11 45 e3 e0 dc 66 a3 e8 fb 5b 8a bd d0 7d 43
a4 ee 80 f8 c8 8b 4e 50 5c dd 21 00 3b d0 bc cf
e2 d5 11 d4 7d 98 08 a7 16 7b 8c 56 44 ba 6d 53
ad 63 65 fd bf 3f 1f 4a a1 c5 d0 58 23 ae d1 47
80 74 f1 d0 b9 00 e5 1d 50 74 53 96 4b ce 59 50
sh$ hexdump -C ./api_iv_10
00000000 80 74 f1 d0 b9 00 e5 1d 50 74 53 96 4b ce 59 50 |.t......PtS.K.YP|
00000010
As a personal opinion, if you really have a lot of binary data processing, I would recommend to switch to some other language more data oriented (Python, ...)
Because the missing byte was an ASCII NUL '\0' '\x00'. The echo command stops printing its argument(s) when it comes across a null byte in each argument.
Trying to configure Kerberos auth via the login form. I was able to send the TGT and after that DummyUserDetailsService is executed but at the end I still get status 500.
Below I attach logs and configuration
09:14:13,316 DEBUG FilterChainProxy:176 - Converted URL to lowercase, from: '/j_spring_security_check'; to: '/j_spring_security_check'
09:14:13,317 DEBUG FilterChainProxy:183 - Candidate is: '/j_spring_security_check'; pattern is /**; matched=true
09:14:13,318 DEBUG FilterChainProxy:351 - /j_spring_security_check at position 1 of 9 in additional filter chain; firing Filter: 'org.springframework.security.web.context.SecurityContextPersistenceFilter#1e5b739'
09:14:13,318 DEBUG HttpSessionSecurityContextRepository:145 - HttpSession returned null object for SPRING_SECURITY_CONTEXT
09:14:13,319 DEBUG HttpSessionSecurityContextRepository:91 - No SecurityContext was available from the HttpSession: weblogic.servlet.internal.session.MemorySessionData#1d28f13. A new one will be created.
09:14:13,319 DEBUG FilterChainProxy:351 - /j_spring_security_check at position 2 of 9 in additional filter chain; firing Filter: 'org.springframework.security.web.authentication.UsernamePasswordAuthenticationProcessingFilter#6016b9'
09:14:13,319 DEBUG UsernamePasswordAuthenticationProcessingFilter:194 - Request is to process authentication
09:14:13,320 DEBUG ProviderManager:124 - Authentication attempt using org.springframework.security.extensions.kerberos.KerberosAuthenticationProvider
09:14:13,321 DEBUG SunJaasKerberosClient:55 - Trying to authenticate Login with Kerberos
09:14:13,453 DEBUG SunJaasKerberosClient:63 - Kerberos authenticated user: Subject:
Principal: Login#CORPO.DOMAIN.COM
Private Credential: Ticket (hex) =
0000: 61 82 07 29 30 82 07 25 A0 03 02 01 05 A1 13 1B a..)0..%........
0010: 11 43 4F 52 50 4F 2E 54 2D 4D 4F 42 49 4C 45 2E .CORPO.DOMAIN.COM.
0020: 50 4C A2 26 30 24 A0 03 02 01 02 A1 1D 30 1B 1B ..&0$.......0..
0030: 06 6B 72 62 74 67 74 1B 11 43 4F 52 50 4F 2E 54 .krbtgt..CORPO.
0040: 2D 4D 4F 42 49 4C 45 2E 50 4C A3 82 06 DF 30 82 .DOMAIN.COM....0.
0050: 06 DB A0 03 02 01 12 A1 03 02 01 02 A2 82 06 CD ................
0060: 04 82 06 C9 22 60 2B 12 13 B8 B5 AB 60 90 52 25 ...."`+.....`.R%
0070: 3F C8 9A D4 A7 19 C7 8D BF ED 97 A8 7D ED 94 33 ?..............3
0080: D7 9F 89 83 34 A0 49 24 80 06 04 F6 32 06 84 A5 ....4.I$....2...
0090: 72 AF 8C 05 7E 8C 55 1A C6 52 A9 97 46 4A A9 D0 r.....U..R..FJ..
00A0: E7 52 5E 37 64 B9 0E E4 32 18 1D 38 90 43 1A B8 .R^7d...2..8.C..
00B0: 88 CF BC 89 0B 08 2D 49 4B 04 91 27 97 8A C7 71 ......-IK..'...q
00C0: 27 FD 7E CA E0 19 E0 F8 42 50 E4 14 93 95 A2 96 '.......BP......
00D0: A5 90 E2 90 90 52 EB B0 EE C4 1F D5 78 19 C7 2A .....R......x..*
00E0: A1 F0 0F D8 A8 A2 84 A7 7D DB 20 DA 47 04 52 1D .......... .G.R.
00F0: B8 B7 A8 96 C7 11 7C 95 16 30 19 10 68 81 EC 2C .........0..h..,
0100: 16 BE 73 58 62 AA BF 85 55 B0 97 75 99 12 2F B8 ..sXb...U..u../.
0110: 87 53 A3 D1 17 A7 A0 16 9B FF 98 2F 15 2B DF 95 .S........./.+..
0120: 25 A5 07 0A D0 67 A1 49 E2 CE 3C 28 23 B5 48 0C %....g.I..<(#.H.
0130: 01 F2 5F 24 89 30 21 2B B0 84 E1 E3 0A 79 F4 2B .._$.0!+.....y.+
0140: 2D 36 AF B3 02 AA 56 69 19 45 57 84 14 8A AF 7F -6....Vi.EW.....
0150: B9 D1 F2 9E ED 1A 30 F4 5B E1 3A AA FE DB 2F 4D ......0.[.:.../M
0160: 4B 6C 7A B1 62 F6 7D E4 C4 43 75 CA 81 8E A6 2F Klz.b....Cu..../
0170: 88 34 60 D1 EA C6 0C FB 29 C0 7E 67 C2 1B 1E F1 .4`.....)..g....
0180: 4B 1C F2 B0 9E 8B E8 9D 8D C5 21 B5 B7 21 81 A9 K.........!..!..
0190: 80 F1 75 A4 53 5D E7 CC 20 97 48 04 F2 18 75 C8 ..u.S].. .H...u.
01A0: 91 93 FF 6D 44 8D E1 DF 10 D0 E2 6E E5 7F C7 E2 ...mD......n....
01B0: 48 1B 3A C5 AA 4C 68 99 10 F5 49 00 84 A2 48 82 H.:..Lh...I...H.
01C0: 48 A6 B0 FF 46 91 9C CD 25 1D 64 B2 73 51 0A C3 H...F...%.d.sQ..
01D0: 82 E1 F9 E9 56 FC 45 73 51 C8 08 91 42 97 08 2D ....V.EsQ...B..-
01E0: 89 38 66 96 48 EA 5F A3 1A B2 13 CF BA 7B C6 33 .8f.H._........3
01F0: E0 F6 E2 0F 18 EB 44 96 44 9F A2 F9 D6 BE 9C A9 ......D.D.......
0200: 7D 10 F2 98 E8 5E 3A 39 BC EA C2 7F 8D F0 75 EF .....^:9......u.
0210: 0D 25 2C 08 C4 7F 92 12 80 08 F1 1F 06 62 FA C4 .%,..........b..
0220: CC E4 A4 A8 BC A1 A9 7E 49 32 09 15 6A 8A 1B 89 ........I2..j...
0230: 82 53 E2 64 A4 E9 85 2A CE AC 99 0E A2 29 6F C4 .S.d...*.....)o.
0240: 04 14 A4 96 67 F9 FE 46 CC 2B F4 B7 76 43 6F 18 ....g..F.+..vCo.
0250: CA 92 89 77 31 AB CD 73 CD F6 1D FB 40 DF 38 DF ...w1..s....#.8.
0260: B4 78 98 5A 1B 48 26 EA D5 93 0C 04 B6 AD 18 B4 .x.Z.H&.........
0270: 4E B2 B5 17 E8 D2 1F E7 E8 C7 89 36 FC B3 95 5E N..........6...^
0280: A4 4E D1 F2 56 79 9D 92 37 F5 03 94 06 62 68 C2 .N..Vy..7....bh.
0290: 56 1D 68 4B 87 21 17 4F 02 86 33 D1 E5 A9 F4 8D V.hK.!.O..3.....
02A0: 45 4E FB FC 7C F6 BA 28 7A F2 F8 50 0E 8A 34 20 EN.....(z..P..4
02B0: A1 5C 92 F2 3F 61 32 E4 15 27 04 B3 8C 7E C1 25 .\..?a2..'.....%
02C0: 54 14 66 BF BA 52 8C D8 4D 47 BB B7 37 61 10 B9 T.f..R..MG..7a..
02D0: 4A EC BE D1 2F 08 8B B4 54 34 71 39 6D 47 A4 0C J.../...T4q9mG..
02E0: C0 68 55 5B B5 2B 9E 6D 22 30 76 BF 78 65 9B A1 .hU[.+.m"0v.xe..
02F0: 03 D0 6E 45 38 A9 22 A9 44 43 20 DA 6D B6 C1 61 ..nE8.".DC .m..a
0300: 0B 04 B0 72 D9 FC 23 FE D1 6F 30 EC 51 34 70 38 ...r..#..o0.Q4p8
0310: 1A 60 FA F8 FE 58 A2 06 A2 83 5E 38 36 7B 2C 38 .`...X....^86.,8
0320: D0 AF 7F 24 96 8D EB F9 90 DE EE 32 EA 3D 57 3D ...$.......2.=W=
0330: 07 3E FE 8D 0C 96 F3 2E FB E2 09 D8 32 2C EF 65 .>..........2,.e
0340: EC 53 1B 08 F6 D2 37 63 45 AB FB 04 B3 79 1C 03 .S....7cE....y..
0350: 1B 21 3B 67 F7 C3 21 2D 58 BB D6 AF D4 6A FD 30 .!;g..!-X....j.0
0360: 60 7D AF 3E 1F 13 A7 8B C6 25 D0 8E CE FD 8C FA `..>.....%......
0370: 09 D4 47 F5 60 19 B2 2F E1 D6 D4 B5 F5 0B 98 5F ..G.`../......._
0380: 08 F4 5B E8 F4 F3 91 A5 E3 5D DA 7C 43 69 30 58 ..[......]..Ci0X
0390: 9E 33 68 B9 A9 2B 89 8C F5 12 42 D9 2C 0F 46 11 .3h..+....B.,.F.
03A0: 6B 96 14 AE 28 0B 2E 72 3A 12 CE A8 C4 C2 8C B9 k...(..r:.......
03B0: 41 FD 69 33 C4 2B 70 B9 C8 E8 02 B6 EF F9 84 B8 A.i3.+p.........
03C0: 45 28 0D CB 61 7C 4B 08 32 22 D8 E6 D6 45 3D 67 E(..a.K.2"...E=g
03D0: 85 4D CD 49 79 C7 10 61 BC 79 B8 30 D0 46 2B A0 .M.Iy..a.y.0.F+.
03E0: BB 56 77 48 13 E8 66 30 A0 09 F3 C0 45 0E CE 0E .VwH..f0....E...
03F0: C6 BB B4 3C 19 E6 CF 5B 84 FF D8 92 5B 13 5D FC ...<...[....[.].
0400: B8 E3 24 09 C7 37 AA E7 3B AE 8C 31 3E 0F 5A 3A ..$..7..;..1>.Z:
0410: D8 C5 08 7E 05 C8 B6 3A 60 24 38 61 B1 00 2D 25 .......:`$8a..-%
0420: 1F 37 22 65 28 ED C1 7E 33 02 DE 3F 5F 2E D9 CF .7"e(...3..?_...
0430: 10 D3 33 CE E1 C8 FF 64 83 10 FC 92 9A 35 C5 13 ..3....d.....5..
0440: 7B B6 52 DC 48 EC 06 2D F7 52 53 93 6F D8 63 E9 ..R.H..-.RS.o.c.
0450: 82 AC 77 B3 2B B0 FB DA 04 FF D6 BA C6 1F 8C AC ..w.+...........
0460: EE AD 2F AD 32 C9 EE 06 0E 0D EF F9 DC C1 CE 0A ../.2...........
0470: 70 6C 49 94 D3 8D F9 B9 27 5A 15 34 50 51 EC 0B plI.....'Z.4PQ..
0480: 80 99 70 31 96 3E C4 D8 E8 AD 07 A6 FE 6D B4 7E ..p1.>.......m..
0490: 44 B4 61 2A 68 2D 74 FF 0C B7 98 65 D0 EF 31 A0 D.a*h-t....e..1.
04A0: 61 9B 3D 89 2C B0 DA 7E 15 1A 0B 0D 50 3C B5 E4 a.=.,.......P<..
04B0: 13 80 9C 27 2A B4 4F 95 A3 96 A9 5C D4 A0 B4 1B ...'*.O....\....
04C0: 42 EB 52 33 1C C5 FD 53 21 DC 42 9C 2B 83 6B 5D B.R3...S!.B.+.k]
04D0: B2 E4 E7 F4 16 46 7A 84 41 2C 27 3F D6 6C 5D B5 .....Fz.A,'?.l].
04E0: 54 E5 CA 7F FD 6C 68 97 B9 E2 8C 47 BF 90 A0 51 T....lh....G...Q
04F0: A1 11 C5 CC 75 AC 1B 18 78 18 46 79 88 16 FA F0 ....u...x.Fy....
0500: 7E 63 59 13 BF 8A 98 83 72 57 60 47 EA E0 0A AA .cY.....rW`G....
0510: 5F 33 AA F3 7F 05 EC FB 85 47 15 0B 7C 69 AD 02 _3.......G...i..
0520: D6 83 2C A0 01 EB 17 3B BA B9 24 25 78 4F 20 D0 ..,....;..$%xO .
0530: 96 82 83 3E 97 F8 6E 60 71 CA 1C 00 14 59 1B 7F ...>..n`q....Y..
0540: F3 46 A6 95 69 0E 52 B9 E9 8E 2A 58 D8 48 65 9D .F..i.R...*X.He.
0550: 7E 5D 4D DE 7B DC FD 5E 41 ED 0E 4F 15 95 D7 4C .]M....^A..O...L
0560: A1 A3 B2 12 5A 7D AC 68 FD 98 2D BF 42 AE 29 84 ....Z..h..-.B.).
0570: EB 27 65 92 EC 92 73 7C 5D B2 6B 79 7D 73 E6 51 .'e...s.].ky.s.Q
0580: E7 CF 76 5E 8B 66 D6 D3 9A B8 6C C7 FD FD F3 05 ..v^.f....l.....
0590: 31 66 4E 5F FD B3 F1 40 A4 89 2D 9A 99 F2 4A 87 1fN_...#..-...J.
05A0: 77 C7 29 94 AF 4D 09 2D C4 3C 5B CC 22 27 63 97 w.)..M.-.<[."'c.
05B0: 2E 4C 38 BF 19 BB 0D 28 3B 9E 06 5D 30 C0 7B FC .L8....(;..]0...
05C0: CF E7 85 AE 73 4D 46 35 55 58 0E D3 D0 AC CE 76 ....sMF5UX.....v
05D0: 83 87 0B 7D C5 6A 18 57 EB 37 D1 B4 33 05 BA C9 .....j.W.7..3...
05E0: 1E 4C C8 71 68 6E 0A 4B DD CE DC 85 96 FD C3 30 .L.qhn.K.......0
05F0: 15 42 1F 0A C0 F4 3E 7E 5C 44 4E 45 62 F3 F6 5F .B....>.\DNEb.._
0600: 07 25 4E 98 C7 22 60 C5 3D 72 6A 56 87 8A 70 2D .%N.."`.=rjV..p-
0610: C1 95 BE 09 4D 0B F6 2B FD B0 E0 27 31 C4 2D 45 ....M..+...'1.-E
0620: 0C C0 D7 F2 BB 82 E9 F5 38 05 ED 04 F2 00 C0 37 ........8......7
0630: EF 6E 9C BE 17 CA FA A9 4C A6 56 65 03 37 D0 6B .n......L.Ve.7.k
0640: 7C D8 25 07 42 49 1F 6E 7E CA E6 DE 6D A1 E6 32 ..%.BI.n....m..2
0650: 4D 25 2A 95 13 4B 31 01 33 91 BD FF 51 20 42 C7 M%*..K1.3...Q B.
0660: E7 A0 EA 8D 6F 5A 5A 57 2F 63 5E 5B 98 6B D5 64 ....oZZW/c^[.k.d
0670: A6 0A F1 26 07 7F 2F B9 57 CD F5 E9 CB D8 6D FA ...&../.W.....m.
0680: FC 4F F4 28 F9 51 2D 83 25 AE ED 66 57 52 4B 1F .O.(.Q-.%..fWRK.
0690: 73 DC C7 4D 0F 90 4F 5F 77 B4 2C 45 89 97 64 47 s..M..O_w.,E..dG
06A0: 62 3D A4 C5 C9 76 D7 34 37 2C B2 12 EF 5D C0 5A b=...v.47,...].Z
06B0: 2E 44 FE D4 96 F1 60 58 6C 11 CE EF 09 C8 29 E9 .D....`Xl.....).
06C0: 2E 98 36 80 BD 21 5A 41 E8 35 29 02 DF A0 90 D5 ..6..!ZA.5).....
06D0: BE 47 8B BA D8 46 D7 26 00 14 98 F8 4C B0 9D 03 .G...F.&....L...
06E0: 06 58 17 87 D3 3D BD D3 1F 1F C7 E6 9D 5D CC 11 .X...=.......]..
06F0: F8 68 C9 39 E8 72 7C 6B 1A 59 95 77 69 CC 6D 5F .h.9.r.k.Y.wi.m_
0700: F4 1A C4 32 4C F7 8A EC C2 9A 69 2E 0B 48 34 C2 ...2L.....i..H4.
0710: 49 A1 77 B8 4D 93 18 2A AE 73 50 FE 27 6F 47 CA I.w.M..*.sP.'oG.
0720: D5 1A BA D3 94 E9 57 A8 2F AB 16 33 38 ......W./..38
Client Principal = Login#CORPO.DOMAIN.COM
Server Principal = krbtgt/CORPO.DOMAIN.COM#CORPO.DOMAIN.COM
Session Key = EncryptionKey: keyType=23 keyBytes (hex dump)=
0000: B2 BE 18 1F BB 83 62 CE C9 A2 D9 C5 96 20 18 DB ......b...... ..
Forwardable Ticket false
Forwarded Ticket false
Proxiable Ticket false
Proxy Ticket false
Postdated Ticket false
Renewable Ticket false
Initial Ticket false
Auth Time = Thu Jul 24 09:14:13 CEST 2014
Start Time = Thu Jul 24 09:14:13 CEST 2014
End Time = Thu Jul 24 19:14:13 CEST 2014
Renew Till = null
Client Addresses Null
Private Credential: Kerberos Principal Login#CORPO.DOMAIN.COMKey Version 0key EncryptionKey: keyType=23 keyBytes (hex dump)=
0000: 2F B2 A5 13 00 0C BF E8 9E 37 4A D9 B0 60 2D 54 /........7J..`-T
Private Credential: Kerberos Principal Login#CORPO.DOMAIN.COMKey Version 0key EncryptionKey: keyType=17 keyBytes (hex dump)=
0000: 50 F5 48 E9 99 D1 78 B0 27 A4 E4 10 C5 5B D5 DA P.H...x.'....[..
09:14:13,455 INFO DummyUserDetailsService:20 - DummyUserDetailsService !!!! EXECUTE
09:14:13,456 DEBUG SecurityContextPersistenceFilter:90 - SecurityContextHolder now cleared, as request processing completed
Spring Security Conf
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:sec="http://www.springframework.org/schema/security"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security-3.0.xsd">
<sec:http entry-point-ref="spnegoEntryPoint" auto-config="false">
<sec:intercept-url pattern="/login*" access="IS_AUTHENTICATED_ANONYMOUSLY" />
<sec:intercept-url pattern="/j_spring_security_check*" access="IS_AUTHENTICATED_ANONYMOUSLY" />
<sec:intercept-url pattern="/**" access="ROLE_USER" />
<sec:custom-filter ref="spnegoAuthenticationProcessingFilter" position="BASIC_PROCESSING_FILTER" />
<sec:form-login login-page="/login" default-target-url="/source/hello" always-use-default-target="true"/>
</sec:http>
<bean id="spnegoEntryPoint" class="org.springframework.security.extensions.kerberos.web.SpnegoEntryPoint" />
<bean id="spnegoAuthenticationProcessingFilter" class="org.springframework.security.extensions.kerberos.web.SpnegoAuthenticationProcessingFilter">
<property name="authenticationManager" ref="authenticationManager" />
</bean>
<!-- LDAP Authentication provider -->
<sec:authentication-manager alias="authenticationManager">
<sec:authentication-provider ref="kerberosServiceAuthenticationProvider" />
<sec:authentication-provider ref="kerberosAuthenticationProvider"/>
</sec:authentication-manager>
<bean id="kerberosServiceAuthenticationProvider"
class="org.springframework.security.extensions.kerberos.KerberosServiceAuthenticationProvider">
<property name="ticketValidator">
<bean
class="org.springframework.security.extensions.kerberos.SunJaasKerberosTicketValidator">
<property name="servicePrincipal"
value="HTTP/webappserver.linux.domain.com#CORPO.DOMAIN.COM" />
<property name="keyTabLocation" value="file:/apps/bin/krb5/test.keytab" />
<property name="debug" value="true" />
</bean>
</property>
<property name="userDetailsService" ref="dummyUserDetailsServices" />
</bean>
<bean id="kerberosAuthenticationProvider" class="org.springframework.security.extensions.kerberos.KerberosAuthenticationProvider">
<property name="kerberosClient">
<bean class="org.springframework.security.extensions.kerberos.SunJaasKerberosClient">
<property name="debug" value="true" />
</bean>
</property>
<property name="userDetailsService" ref="dummyUserDetailsService" />
</bean>
<bean
class="org.springframework.security.extensions.kerberos.GlobalSunJaasKerberosConfig">
<property name="debug" value="true" />
<property name="krbConfLocation" value="/apps/bin/krb5/krb5.conf" />
</bean>
<bean id="dummyUserDetailsService" class="com.web.skorpion.ldap.DummyUserDetailsService" />
</beans>
Now configuration work.
I have changed BASIC_PROCESSING_FILTER to BASIC_AUTH_FILTER but first i had changed version of spring securito. to 3.2.4.RELEASE.