I have a file containing hex representations of code from a small program, and am trying to actually convert it into the program itself.
For example, here is a sample of such text, stored in a file, input.txt:
8d
00
a1
21
53
57
43
48
0e
00
bb
I am using the following BASh snippet to convert the file to a binary file:
rm outfile; while read h; do echo -n ${h}; echo -ne \\x${h} >> outfile; done < input.txt
After opening the output file in VIM:
¡!SWCH»
And then converting it to hex representation via xxd:
0000000: 8d00 a121 5357 4348 0e00 bb0a ...!SWCH....
This is all good, except for one thing: There is a trailing byte, 0a, trailing at the end of my binary output file. This happens for every program file I work with. How is the trailing 0a being appending to every output binary file? It's not present in my input file.
Thank you.
Simply, use xxd directly from a bash like
xxd outfile > outfile.hex
and you will see, here isn't any 0a.
The 0a is appended somewhere when the vim sends a line to xxd command. If you want convert inside vim - try use
vim -b outfile
what open the outfile in binary mode.
Related
I'm trying to extract the icon from an xattr of a file.
Using xattr to get the "com.apple.ResourceFork" in hex format i use:
xattr -px com.apple.ResourceFork file
and save the output to a variable
var="$(xattr -px com.apple.ResourceFork file)"
then using variable expansion i remove the first bytes until i reach 69 (icns magic number is 69 63 6E 73)
var=${var#*69 63 6E 73}
next i output the variable and append "69 63 6E 73" to the beginning to restore the magic number.
echo "69 63 6E 73$var" > output.txt
if i take the hex data from the output.txt and insert it into a hexeditor to save it then it works and the .icns is created.
i want to do this programmatically in bash/zsh without having to save it manually.
tried using
touch icon.icns
to create an empty file then
echo "69 63 6E 73$var" > icon.icns
just transforms the output file into an ASCII file.
i'm not stuck to my method, any working method is acceptable to me.
I have access to my Mac again... strangely (to me) it seems xxd works differently when given parameters all together rather than individually, so rather than what I suggested in the comments:
xxd -rp ...
you would need:
xxd -r -p ...
As I don't have an icon.icns file to hand, I'll take a JPEG (which is just as binary), convert it to readable hex and reconstruct it from the hex with xxd.
Here's a JPEG, converted to hex:
xxd x.jpg | more
00000000: ffd8 ffe0 0010 4a46 4946 0001 0100 0001 ......JFIF......
00000010: 0001 0000 ffdb 0043 0003 0202 0202 0203 .......C........
...
...
Then take the hex and give reconstruct the first few bytes of the JPEG:
printf "ff d8 ff e0" | xxd -r -p > recreated.jpg
And look at the recreated file:
xxd recreated.jpg
00000000: ffd8 ffe0
So the process for a while file would be:
hex=$(xxd -p x.jpg)
printf "$hex" | xxd -r -p > recreated.jpg
There already is a beautiful trick in this thread
to write bytes to binary file at desired address with dd ,is there any way to swap bytes(e.g swap 0x00 and 0xFF), or replace bytes with common tools (such as dd)?
Would you please try the following:
xxd -p input_file | fold -w2 | perl -pe 's/00/ff/ || s/ff/00/' | xxd -r -p > output_file
xxd -p file dumps the binary data file in continuous hexdump style.
fold -w2 wraps the input lines by every two characters (= every bytes).
perl -pe 's/00/ff/ || s/ff/00/' swaps 00 and ff in the input string.
The || logic works as if .. else .. condition. Otherwise the input 00
is once converted to ff and immediately converted back to 00 again.
xxd -r -p is the reversed version of xxd -p which converts the input
hexadecimal strings into binaries.
Since my reputation is too low to post an image I will reproduce the terminal
output where my question originated from:
username#computer:/run$ cat rsyslogd.pid
599username#computer:/run$ cat acpid.pid
636
username#computer:/run$
cat acpid.pid
comes with a linebreak whereas
cat rsyslog.pid
doesn't.
But if I open both files there is no visible difference (e.g. the file
acpid.pid
doesn't have an additional blank line)
The Question is: Why does one .pid file come with a linebreak and the other one doesn't?
Addditional Information: My operating system is Ubuntu 18.04.3
The rsyslogd.pid file probably doesn't end with a newline character (ASCII 0x0A).
You didn't mention how you opened the files, but, I suspect you used a text editor which will not display non-printable characters (like newline and backspace). Rather than using a text editor try looking at the raw file with the hexdump tool. Then compare the hex values against an ASCII table. I think you will find that the non-printable characters after the 599 and 636 are different.
hexdump -C rsyslogd.pid
hexdump -C acpid.pid
The following sequence of commands reproduces your output. The key is to use the -n flag for the echo command to create a file without a newline character at the end.
$ echo -n test > file_no_new_line.txt
$ echo test > file_with_new_line.txt
$ cat file_no_new_line.txt
test$ cat file_with_new_line.txt
test
$
Here is the output of hexdump for the two files shown in my example.
$ hexdump -C file_no_new_line.txt
00000000 74 65 73 74 |test|
00000004
$ hexdump -C file_with_new_line.txt
00000000 74 65 73 74 0a |test.|
00000005
$
The command output, in this case from cat, and the shell prompt ($) running into each other is also shell dependent. If the behavior can't be reproduce with the steps above try another shell (e.g. /bin/sh)
I need to check my string variable for presence of extended ASCII characters, one byte, decimal code 128-255. If any is there, replace it with multiple character hex equivalent, ready for further grep command etc.
Example string: "Ørsted\ Salg", I need it to be converted to "\xD8rsted\ Salg".
I know the way to do it with hastable in Bash 4:
declare -A symbolHashTable=(
["Ø"]="D8"
);
currSearchTerm="Ørsted\ Salg"
for curRow in "${!symbolHashTable[#]}"; do
currSearchTerm=$(echo $currSearchTerm | sed s/$curRow/'\\x'${symbolHashTable[$curRow]}/)
done
, but that seems too tedious for 127 cases. There should be a way to do it shorter and probably faster, without writing all the symbols.
I can detect whether the string has any of the characters in it with:
echo $currSearchTerm | grep -P "[\x80-\xFF]"
I am almost sure there is a way to make sed do it, but I get lost somewhere in the "replace with" part.
You can easily do this with Perl:
#!/bin/bash
original='Ørsted'
replaced=$(perl -pe 's/([\x80-\xFF])/"\\x".unpack "H*", $1/eg' <<< "$original")
echo "The original variable's hex encoding is:"
od -t x1 <<< "$original"
echo "Therefore I converted $original into $replaced"
Here's the output when the file and terminal is ISO-8859-1:
The original variable's hex encoding is:
0000000 d8 72 73 74 65 64 0a
0000007
Therefore I converted Ørsted into \xd8rsted
Here's the output when the file and terminal is UTF-8:
The original variable's hex encoding is:
0000000 c3 98 72 73 74 65 64 0a
0000010
Therefore I converted Ørsted into \xc3\x98rsted
In both cases it works as expected.
I have a script that has a variable that might contain some weird characters: 🍿 ✔. I need to remove them but, honestly, I don't even know where to begin to match those characters. I can't copy and paste them into my script, they just show up as ?? ?. How can a match those characters with sed or awk? I don't have the ability to use perl or php or anything much beyond sed or awk due to system availability.
First, put some flag strings around your special chars and then hexdump -C so you can easily see them. Then use HEX code to write the sed command. For example:
[STEP 118] # cat file
>>>🍿 ✔<<<
[STEP 119] # hexdump -C file
00000000 3e 3e 3e f0 9f 8d bf 20 e2 9c 94 3c 3c 3c 0a |>>>.... ...<<<.|
^^^^^^^^^^^^^^^^^^^^^^^^
[STEP 120] # sed -e $'s/\xf0\x9f\x8d\xbf\x20\xe2\x9c\x94//g' file # need to use the $'...'
>>><<<
[STEP 121] #
Then remove the added flag strings when all is done.
Try this - (file contain some control M and the character that you have mentioned in the question and I am trying to print only the alphanumeric character)
$cat f
hello vipin
street1
pin 12345
🍿 ✔
$awk '/[[:alnum:]]/ {print }' f
hello vipin
street1
pin 12345
Looks like control M character is getting disappeared after saving the input file on SO.
$ cat file
some weird characters: 🍿 ✔. I need to remove
second line of some weird characters: 🍿 ✔. I need to remove
$ tr -c -d '[:print:][:space:]' < file
some weird characters: . I need to remove
second line of some weird characters: . I need to remove
The solution I ended up using was just changing the encoding of the script to UTF-8 instead of ASCII. I did this with notepad++. Then I could work with the character directly instead of some roundabout way of converting to hex (which I couldn't do anyway as the variable is an environmental variable and not from a file) or something else. I also didn't need to use awk or sed as the following was much simpler:
cleaned_var=${environmental_variable//" 🍿 ✔"}