This question already has answers here:
Conversion hex string into ascii in bash command line
(7 answers)
Closed 3 years ago.
I need to output a symbol using variable that contains code of that symbol.
I already understood that in my code example bash thinks that i'm giving it a string. So according to specificator it outputs "4".
symbolValue=41
printf '%c' $symbolValue
I expect outputing of "A" symbol in fixed code. Please help me with it.
Could you please try the following.
symbolValue=41
echo "$symbolValue" | xxd -p -r
# OR
xxd -p -r <<< "$symbolValue"
From man xxd:
-p | -ps | -postscript | -plain
output in postscript continuous hexdump style. Also known as
plain hexdump style.
-r | -revert
reverse operation: convert (or patch) hexdump into binary. If
not writing to stdout, xxd writes into its output file without
truncating it. Use the combination -r -p to read plain hexadecimal
dumps without line number information and without a particular
column layout. Additional Whitespace and line-breaks are
allowed anywhere.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
I have the following strings
$A%^]I"S-)|>J&`_#;!UVPOSM\
IV&|M
This contains special characters and hence Bash can't interpret it correctly unless quoted
The first string is just the substitution of the range A-Z
I tried the following
tr 1
read -r A
tr $A A-Z
tr 2
read -r A
tr "$A" A-Z
Using sed
read -r A
sed "y/${A}/ABCDEFGHIJKLMNOPQRSTUVWXYZ/"
read -r A
sed 'y/'$A'/ABCDEFGHIJKLMNOPQRSTUVWXYZ/'
read -r A
sed 'y/'${A}'/ABCDEFGHIJKLMNOPQRSTUVWXYZ/'
None of them worked. Also, using the options -ps in read also didn't work.
The error for tr is
tr: range-endpoints of 'S-)' are in reverse collating sequence order
Error for sed
sed: -e expression #1, char 56: unterminated `y' command
The output when the first string is translated to A-Z will give "FUNKY"
How to resolve this?
Special characters can be escaped with \ for tr and sed utilities. The programs tr and sed have different special characters sets and parsing rules. However, escaping just [ - and \ seems to be just enough for both programs for the presented input.
IFS= read -r a;
a=$(sed 's/[[\-]/\\&/g' <<<"$a")
tr "$a" 'A-Z'
# or
sed "y/$a/ABCDEFGHIJKLMNOPQRSTUVWXYZ/"
Check your scripts with shellcheck.net . Research quoting in shell and when to use it. Research how to debug scripts and about set -x option. See sed and tr documentation - https://pubs.opengroup.org/onlinepubs/009695299/utilities/tr.html and https://www.gnu.org/software/sed/manual/sed.html . Prefer to use lower case variable names. Also you might be interested in https://mywiki.wooledge.org/BashFAQ/001 and https://en.wikipedia.org/wiki/Here_document#Here_strings .
I think the best full proof way might be to convert first line and convert to octal \NNN numbers and then pass to tr. Like so:
IFS= read -r a;
a=$(printf "%s" "$a" | od -An -b | tr -d ' \n' | sed 's/.../\\&/g');
tr "$a" 'A-Z';
Both tr and sed are filters; they read one or more lines from standard input, execute a transformation (which in sed can be a quite complex script with conditional branches and loops), and print the result to standard output.
The y command in sed is quite similar to tr; they both accept two lists of characters, where the nth character in the first list will be mapped to the nth character in the second.
tr has an additional feature where you can abbreviate a list of adjacent characters; thus tr A-Za-z N-ZA-Mn-za-m implements the familiar rot13 algorithm. This explains why you got an error when you tried to pass in a string with a dash - it was interpreted as a range operator, but regarded as invalid because the range's start had a higher character code than its end.
The sed error comes because the backslash is interpreted as escaping the final delimiter; \/ is how you would put a literal slash in the sbustitution or replacement string.
If I understand your question correctly, you want to map A-Z to the first line of your input, and provide the second as input to this transformation?
#!/bin/bash
exec <"$1"
IFS='' read -r A
sed "y/${A//\\/\\\\}/ABCDEFGHIJKLMNOPQRSTUVWXYZ/"
The exec tells Bash to read standard input from the named file; you would pass the name of your file as the argument to this script. The string substitution ${A//pattern/replacement} takes care to double any literal backslashes in the variable's value.
It's not impossible to come up with something similar for tr. The main challenge is that you have to put the dash first or last, and correspondingly reorder the output mapping. Some tr implementations might allow you to use --- to specify a one-character range where the middle dash is a range operator and the other two are the start and end characters of the range.
Basically I want a "multiline grep that takes binary strings as patterns".
For example:
printf '\x00\x01\n\x02\x03' > big.bin
printf '\x01\n\x02' > small.bin
printf '\x00\n\x02' > small2.bin
Then the following should hold:
small.bin is contained in big.bin
small2.bin is not contained in big.bin
I don't want to have to convert the files to ASCII hex representation with xxd as shown e.g. at: https://unix.stackexchange.com/questions/217936/equivalent-command-to-grep-binary-files because that feels wasteful.
Ideally, the tool should handle large files that don't fit into memory.
Note that the following attempts don't work.
grep -f matches where it shouldn't because it must be splitting newlines:
grep -F -f small.bin big.bin
# Correct: Binary file big.bin matches
grep -F -f small2.bin big.bin
# Wrong: Binary file big.bin matches
Shell substitution as in $(cat) fails because it is impossible to handle null characters in Bash AFAIK, so the string just gets truncated at the first 0 I believe:
grep -F "$(cat small.bin)" big.bin
# Correct: Binary file big.bin matches
grep -F "$(cat small2.bin)" big.bin
# Wrong: Binary file big.bin matches
A C question has been asked at: How can i check if binary file's content is found in other binary file? but is it possible with any widely available CLI (hopefully POSIX, or GNU coreutils) tools?
Notably, implementing an non-naive algorithm such as Boyer-Moore is not entirely trivial.
I can hack up a working Python one liner as follows, but it won't work for files that don't fit into memory:
grepbin() ( python -c 'import sys;sys.exit(not open(sys.argv[1]).read() in open(sys.argv[2]).read())' "$1" "$2" )
grepbin small.bin big.bin && echo 1
grepbin small2.bin big.bin && echo 2
I could also find the following two tools on GitHub:
https://github.com/tmbinc/bgrep in C, installable with (amazing :-)):
curl -L 'https://github.com/tmbinc/bgrep/raw/master/bgrep.c' | gcc -O2 -x c -o /usr/local/bin/bgrep -
https://github.com/gahag/bgrep in Rust, installable with:
cargo install bgrep
but they don't seem so support taking the pattern from a file, you provide the input as hex ASCII on the command line. I could use:
bgrep $(xxd -p small.bin | tr -d '\n') big.bin
since it does not matter as much if the small file gets converted with xxd, but it's not really nice.
In any case, if I were to implement the feature, I'd likely it to the Rust library above.
bgrep is also mentioned at: How does bgrep work?
Tested on Ubuntu 20.10.
How to check if a binary file is contained inside another binary from the Linux command line?
The very POSIX portable way would be to use od to convert to hex and then check for substring with grep, along with some sed scripting in between.
The usual normal portable way, would be to use xxd instead of od:
xxd -p small.bin | tr -d '[ \n]' > small.bin2
xxd -p big.bin | tr -d '[ \n]' > big.bin2
grep -F -f small.bin2 big.bin2
which works fine tested in docker on alpine with busybox.
But:
I don't want to have to convert the files to ASCII hex representation with xxd as shown
then you can't work with binary files in shell. Pick another language. Shell is specifically created to parse nice looking human readable strings - for anything else, it's utterly unpleasant and for files with zero bytes xxd is the first thing you type.
I can hack up a working Python one liner as follows,
awk is also POSIX and available everywhere - I believe someone more skilled in awk may come and write the exact 1:1 of your python script, but:
but it won't work for files that don't fit into memory:
So write a different algorithm, that will not do that.
Overall, when giving the constraint of not using xxd (or od) to convert a binary file with zero bytes to it's hex representation:
is it possible with any widely available CLI (hopefully POSIX, or GNU coreutils) tools?
No. Write your own program for that. You may also write it in perl, it's sometimes available on machines that don't have python.
This question already has answers here:
What's an easy way to read random line from a file?
(13 answers)
Closed 3 years ago.
I have text file which contains numerous lines. I need to do:
shuf txt.txt
From shuf output read first line to a variable $line
How to represent it in one line for bash script?
Now it is like this:
shuf txt.txt -o aaa.txt
n=$(head -n 1 aaa.txt)
rm -rf aaa.txt
As you may notice, it is not very nice
You can easily do this with shuf:
n=$(shuf -n1 file)
↳ https://www.gnu.org/software/coreutils/manual/html_node/shuf-invocation.html
Related question on stack overflow.com: What's an easy way to read random line from a file in Unix command line?
Sounds like homework. Here's some hints.
Assigning a variable to the result of a command:
x=`command`
Assignment with pipe:
x=`command1 | command2`
Assigning the first line of command1's output to a variable:
x=`command1 | head -n1`
I'd like to write non-ASCII characters (0xfe, 0xed, etc) to a program's standard input.
There are a lot of similar questions to this one, but I didn't find an answer, because:
I want to write single-byte characters, not unicode characters
I can't pipe the output of echo or something
On OS X¹ you can test with:
nm - -
I'd like to write object files magic bytes (e.g. 0xfeedface) to nm using standard input so I can see how it does behave and I can recode it.
If I use a pipe then the second argument -, which means stdin, will never match any bytes since all the standard input will go to the first one. When using a terminal instead of a pipe, I can type Ctrl + D so the first one gets 'closed' and the second one start reading.
I tried with Ctrl + Shift + U and the Unicode Hex Input of OS X but it doesn't work -- I can't write the desired characters with it.
I also tried with the clipboard with pbcopy but it fails to read/paste non-ASCII or non-unicode characters.
How can I achieve my goal?
Don't hesitate to edit as this was a difficult question to express.
¹ The nm on linux does not handle stdin.
You can echo your desired hex code into a file.
echo -e -n '\xde\xad\xbe\xef\xde\xad\xbe\xef' >/tmp/yo
or
echo -en \\xde\\xad\\xbe\\xef\\xde\\xad\\xbe\\xef >/tmp/yo
and make your executable to read from this file instead of stdin
./executable </tmp/yo
If you don't wan't to create a file, here's an alternative
python -c 'print("\x61\x62\x63\x64")' | /path/to/exe
If you want stdin control to be transferred back (in case if you're trying to execute an interactive shell --> we need a subshell to keep sending inputs to stdin. Otherwise, after the first input, the executable would exit as it is not going to get anything further on stdin)
( python -c 'print("\x61\x62\x63\x64")' ; cat ) | /path/to/exe
Python does some juggling with the bytes. So, incase of Python3, you'll have to do the following :
( python -c 'import sys; sys.stdout.buffer.write(b"\x61\x62\x63\x64")' ; cat ) | /path/to/exe
This answer helped me :
https://reverseengineering.stackexchange.com/questions/13928/managing-inputs-for-payload-injection
Try a util like xxd:
# echo hex 52 to pipe, convert it to binary, which goes to stdout
echo 52 | xxd -r ; echo
R
Or for a more specialized util try ascii2binary (default input is decimal):
# echo dec 52 to pipe, convert it to binary, which goes to stdout
echo 52 | ascii2binary ; echo
4
# echo base11 52 to pipe, convert it to binary, which goes to stdout
echo 52 | ascii2binary -b 11 ; echo
9
Or dump a series of hex chars, showing what hexdump sees:
echo 7 ef 52 ed 19 | ascii2binary -b h | \
hexdump -v -e '/1 "%_ad# "' -e '/1 " _%_u\_\n"'
0# _bel_
1# _ef_
2# _R_
3# _ed_
4# _em_
See man xxd ascii2binary for the various tricks these utils can do.
Using bash, you can echo the input,
echo -e -n '\x61\x62\x63\x64' | /path/someFile.sh --nox11
or use cat, which might be more comfortable when there are several lines of prompting:
cat $file | /path/someFile.sh --nox11
You can omit the --nox11, but that might help when the script spawns a new instance of terminal
Note: This will not work with /bin/sh!
This question already has answers here:
Base 64 encoding from command line gives different output than other methods
(2 answers)
Closed 7 days ago.
Can anyone explain this?
[vagrant#centos ~]$ echo "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm" | base64
MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQo=
[vagrant#centos ~]$ echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==" | base64 -d
10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm
The first string encodes with o= at the end, but the encoded string with == at the end instead, decodes to the same original string...
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
Compare these
echo "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm" | base64 | od -c
echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==" | base64 -D | od -c
echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQo=" | base64 -D | od -c
If we don't send the newline when using echo the o is missing, have a look at this...
echo -n "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm" | base64
It's the newline that's being encoded that gives the o in o=
The = is padding and it might not always be there. Have a look here..
https://en.wikipedia.org/wiki/Base64#Padding
Different implementations may also use different padding characters. You can see some of the differences here
https://en.wikipedia.org/wiki/Base64#Variants_summary_table
From the RFC
3.2. Padding of Encoded Data
In some circumstances, the use of padding ("=") in base-encoded
data is not required or used. In the general case, when
assumptions about the size of transported data cannot be made,
padding is required to yield correct decoded data.
Implementations MUST include appropriate pad characters at the end
of encoded data unless the specification referring to this document
explicitly states otherwise.
The base64 and base32 alphabets use padding, as described below in
sections 4 and 6, but the base16 alphabet does not need it; see
section 8.
When you use $echo, a newline is appended to the end of the output. This newline character is part of the base64 encoding. When you change the 'o' to a '=', you're changing the encoding of the newline character. In this case, the character it decodes to is still not a printable character.
In my terminal, decoding the two string yields the same output, but the string ending in "o=" has a newline, and the string ending in "==" does not.
$> echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQo=" | base64 -d
10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm
$> echo "MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==" | base64 -d
10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm $>
Using $echo -n would allow you to pass the string into base64 without the trailing newline. The string without the newline encodes to the string ending in "==".
On Macs, I noticed I had to append "\c" to the end of my string to get it to work, like this:
[vagrant#centos ~]$ echo "10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm\c" | base64
The result was:
MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbVxjCg==
PHP also encodes it properly, which leads me to believe there is some issue with the base64 program in bash, as I haven't found any mention of 'o' somehow being used as a padding character.
php > echo base64_encode("10IXydrdsc4DVAgxzrXldNw5GMeVAHKG:TAO04JuWz4PBVWYm");
MTBJWHlkcmRzYzREVkFneHpyWGxkTnc1R01lVkFIS0c6VEFPMDRKdVd6NFBCVldZbQ==