Why is bash swallowing -e in the front of an array [duplicate] - bash

This question already has answers here:
How do I echo "-e"?
(6 answers)
Closed 9 years ago.
Given the following syntax:
x=(-a 2);echo "${x[#]}";x=(-e 2 -e); echo "${x[#]}"
Output:
-a 2
2 -e
Desired output
-a 2
-e 2 -e
Why is this happening? How do I fix?

tl;dr
printf "%s\n" "${x[*]}"
Explanation
echo takes 3 options:
$ help echo
[…]
Options:
-n do not append a newline
-e enable interpretation of the following backslash escapes
-E explicitly suppress interpretation of backslash escapes
So if you run:
$ echo -n
$ echo -n -e
$ echo -n -e -E
You get nothing. Even if you put each option in quotes, it still looks the same to bash:
$ echo "-n"
$ echo "-n" "-e"
The last command runs echo with two arguments: -n and -e. Now contrast that with:
$ echo "-n -e"
-n -e
What we did was run echo with a single argument: -n -e. Since bash does not recognize the (combined) option -n -e, it finally echoes the single argument to the terminal like we want.
Applied to Arrays
In the second case, the array x begins with the element -e. After bash expands the array ${x[#]}, you are effectively running:
$ echo "-e" "2" "-e"
2 -e
Since the first argument is -e, it is interpreted as an option (instead of echoed to the terminal), as we already saw.
Now contrast that with the other style of array expansion ${x[*]}, which effectively does the following:
$ echo "-e 2 -e"
-e 2 -e
bash sees the single argument -e 2 -e — and since it does not recognize that as an option — it echoes the argument to the terminal.
Note that ${x[*]} style expansion is not safe in general. Take the following example:
$ x=(-e)
$ echo "${x[*]}"
Nothing is printed even though we expected -e to be echoed. If you've been paying attention, you already know why this is the case.
Escaping
The solution is to escape any arguments to the echo command. Unfortunately, unlike other commands which offer some way to say, “hey! the following argument is not to be interpreted as an option” (typically a -- argument), bash provides no such escaping mechanism for echo.
Fortunately there is the printf command, which provides a superset of the functionality that echo offers. Hence we arrive at the solution:
printf "%s\n" "${x[*]}"

#MichaelKropat's answer gives sufficient explanation.
As an alternative to echo (and printf), cat and a bash here-string can be used:
$ x=(-a 2);cat <<< "${x[#]}";x=(-e 2 -e); cat <<< "${x[#]}"
-a 2
-e 2 -e
$

Nice one!
What is happening is the first -e is being interpreted as an option for echo (to enable escape sequences'
Usually, you'd do something like echo -- "-e", and it should print simply -e, but echo is happy to behave differently, and simply prints out -- -e as a whole string.
echo does not interpret -- to mean the end of options.
The solution to the problem could also be found in the man pages:
Due to shell aliases and built-in echo command, using an unadorned
echo interactively or in a script may get you different functionality
than that described here. Invoke it via env (i.e., env echo ...)
to avoid interference from the shell.
So something like this should work:
x=(-a 2);echo "${x[#]}";x=(-e 2 -e); env echo "${x[#]}"

Related

-e disappeared in environment variable, how to avoid this?

#!/bin/bash
OPTS='-e EXTERNAL_PORT=443'
echo $OPTS
cat 1.txt
if [ $? -ne 0 ]; then
echo $OPTS
exit 1
fi
in this case, I got output:
EXTERNAL_PORT=443
cat: 1.txt: No such file or directory
EXTERNAL_PORT=443
if I change OPTS to
OPTS='a -e EXTERNAL_PORT=443'
now it worked as normal
a -e EXTERNAL_PORT=443
cat: 1.txt: No such file or directory
a -e EXTERNAL_PORT=443
How can I avoid this? And this is a simplified demo, in my real cases, I have an environment variable OPTS starts with -e.
I echo it, it is correct, but after line if [ $? -ne 0 ]; then, the "-e" disappeared, this causes bug for my scripts.
Thanks.
echo is interpreting the -e as a command option meaning that it should interpret any escape sequences in the string to be printed, rather than as part of the string to be printed. echo has a number of "features" that can cause unexpected trouble (and worse, different versions of echo implement different options). Try this instead:
$ OPTS='-e EXTERNAL_PORT=443'
$ printf '%s\n' "$OPTS"
-e EXTERNAL_PORT=443
BTW, storing command options in a plain string won't work if any of them have spaces (or sometimes any shell wildcards, or...). It's better to use an array for them, and then double-quote the array reference to keep the shell from messing with it. It's also best to use lowercase (or mixed case) variable names to avoid conflicts with the special variables used by the shell and other programs (they're all uppercase):
$ opts=(-e EXTERNAL_PORT=443 -e COMMENT="This is a test")
$ printf '%s\n' "${opts[#]}"
-e
EXTERNAL_PORT=443
-e
COMMENT=This is a test

error because eval command seems to remove backslashes in stored command

I want to be able to add newline characters before every occurences of some tokens appearing in some .tex files that I possess, some of those tokens are '\itemQ', '\pagebreakQ'. I created a procedure that ends up creating a command for sed stored in $sedInpt:
~$ echo "$sedInpt"
-e s/\(\\itemQ\)/\n\1/ -e s/\(\\pagebreakQ\)/\n\1/
I want to use "$sedInpt" as a command for sed:
echo "$inputText" | eval "sed ${sedInpt}"
but if I do the following as a test:
echo 'hello\itemQ' | eval "sed ${sedInpt}"
hello\itemQ
you can see there ain't any newline that has been added before \itemQ.
So I've tried debugging this way of doing thing by calling bash -x to see what's happened in detail:
~$ bash -x
~$ echo "hello\itemQ" | eval "sed ${sedInpt}"
+ echo 'hello\itemQ'
+ eval 'sed -e s/\(\\itemQ\)/\n\1/ -e s/\(\\pagebreakQ\)/\n\1/'
++ sed -e 's/(\itemQ)/n1/' -e 's/(\pagebreakQ)/n1/'
hello\itemQ
you can see that the backslashes of \n and \1 and even the ones before ( and ) that I had placed in "$sedInpt" seem to have disappeared when parsed by eval.
So I am bit lost on what to do next to do what I want.. any ideas?
You could also just combine them into a single command, which in my opinion is more straightforward:
$ cat /tmp/sed.sh
sedInpt='s/\(\\itemQ\)/\n\1/; s/\(\\pagebreakQ\)/\n\1/'
echo "hello\itemQ" | sed "$sedInpt"
$ /tmp/sed.sh
hello
\itemQ
Edit: As #123 rightly points out, storing commands in variables is dangerous and should be avoided if possible. If you have complete control over what is stored, it should be safe, but if it comes from any sort of user input, it is a "Command Injection" vulnerability.
Following #Inian advice I managed to achieve what I wanted to do in this way:
~$ sedInpt=( -e 's/\(\\itemQ\)/\n\1/' -e 's/\(\\pagebreakQ\)/\n\1/' )
~$ echo "hello\itemQ" | sed "${sedInpt[#]}"
hello
\itemQ

Capturing verbatim command line (including quotes!) to call inside script

I'm trying to write a "phone home" script, which will log the exact command line (including any single or double quotes used) into a MySQL database. As a backend, I have a cgi script which wraps the database. The scripts themselves call curl on the cgi script and include as parameters various arguments, including the verbatim command line.
Obviously I have quite a variety of quote escaping to do here and I'm already stuck at the bash stage. At the moment, I can't even get bash to print verbatim the arguments provided:
Desired output:
$ ./caller.sh -f -hello -q "blah"
-f hello -q "blah"
Using echo:
caller.sh:
echo "$#"
gives:
$ ./caller.sh -f -hello -q "blah"
-f hello -q blah
(I also tried echo $# and echo $*)
Using printf %q:
caller.sh:
printf %q $#
printf "\n"
gives:
$ ./caller.sh -f hello -q "blah"
-fhello-qblah
(I also tried print %q "$#")
I would welcome not only help to fix my bash problem, but any more general advice on implementing this "phone home" in a tidier way!
There is no possible way you can write caller.sh to distinguish between these two commands invoked on the shell:
./caller.sh -f -hello -q "blah"
./caller.sh -f -hello -q blah
There are exactly equivalent.
If you want to make sure the command receives special characters, surround the argument with single quotes:
./caller.sh -f -hello -q '"blah"'
Or if you want to pass just one argument to caller.sh:
./caller.sh '-f -hello -q "blah"'
You can get this info from the shell history:
function myhack {
line=$(history 1)
line=${line#* }
echo "You wrote: $line"
}
alias myhack='myhack #'
Which works as you describe:
$ myhack --args="stuff" * {1..10} $PATH
You wrote: myhack --args="stuff" * {1..10} $PATH
However, quoting is just the user's way of telling the shell how to construct the program's argument array. Asking to log how the user quotes their arguments is like asking to log how hard the user punched the keys and what they were wearing at the time.
To log a shell command line which unambiguously captures all of the arguments provided, you don't need any interactive shell hacks:
#!/bin/bash
line=$(printf "%q " "$#")
echo "What you wrote would have been indistinguishable from: $line"
I understand you want to capture the arguments given by the caller.
Firstly, quotes used by the caller are used to protect during the interpretation of the call. But they do not exist as argument.
An example: If someone call your script with one argument "Hello World!" with two spaces between Hello and World. Then you have to protect ALWAYS $1 in your script to not loose this information.
If you want to log all arguments correctly escaped (in the case where they contains, for example, consecutive spaces...) you HAVE to use "$#" with double quotes. "$#" is equivalent to "$1" "$2" "$3" "$4" etc.
So, to log arguments, I suggest the following at the start of the caller:
i=0
for arg in "$#"; do
echo "arg$i=$arg"
let ++i
done
## Example of calls to the previous script
#caller.sh '1' "2" 3 "4 4" "5 5"
#arg1=1
#arg2=2
#arg3=3
#arg4=4 4
#arg5=5 5
#Flimm is correct, there is no way to distinguish between arguments "foo" and foo, simply because the quotes are removed by the shell before the program receives them. What you need is "$#" (with the quotes).

Why do I get a “Can't find string terminator” error only when the command is in a variable?

I wrote a simple shell script to get the version of Perl modules installed
on a server and I keep receiving the following error:
Can't find string terminator "'" anywhere before EOF at -e line 1.
Here is my script:
#!/bin/sh
#
mod_name="Sub::Uplevel"
tmp1="perl -M$mod_name -e 'print \"\$$mod_name::VERSION\"'"
echo $tmp1
$tmp1
If I just directly run the echo'd line (perl -MSub::Uplevel -e 'print "$Sub::Uplevel::VERSION"'), it works. Why doesn't the line work when its run from the variable $tmp1?
In place of just $tmp1, eval works:
eval "$tmp1"
That's because splitting a variable into words (for arguments) is done strictly by splitting on $IFS, not the normal input-parsing. eval forces the normal input parsing.
How did I figure this out?
Change your tmp1= line to put an echo in front, and you get:
perl -MSub::Uplevel -e 'print "$Sub::Uplevel::VERSION"'
Note that the ' are still there, which you wouldn't expect. If you write a quick script:
#!/bin/sh
for a in "$#"; do
echo "arg: $a"
done
and put a call to that in place of echo, you find how the arguments are really split:
arg: perl
arg: -MSub::Uplevel
arg: -e
arg: 'print
arg: "$Sub::Uplevel::VERSION"'
So, you can see that's splitting on spaces, so IFS.
It's always better to construct commands using bash arrays. That will keep arguments with whitespace properly grouped:
#!/bin/bash
mod_name="Sub::Uplevel"
perl_script=$(printf 'print "$%s::VERSION"' $mod_name)
tmp1=(perl -M$mod_name -e "$perl_script")
echo "${tmp1[#]}"
output=$( "${tmp1[#]}" )
Arrays are a bash feature, so the shebang line must reference bash not sh.
I'd usually write what you are doing with backticks, to run the command inside the shell:
#!/bin/sh
#
mod_name="Sub::Uplevel"
tmp1=`perl -M$mod_name -e 'print \"\$$mod_name::VERSION\"'`
echo $tmp1
Then you can work on $tmp1 as needed. It also avoids dealing with escaping.
Try to execute the script the below way(debugging the script):
sh -vx your_script.sh
Then you would be able to see where exactly the problem is.
I donot have the shell to execute it right now.

echo "-e" doesn't print anything

I'm using GNU bash, version 3.00.15(1)-release (x86_64-redhat-linux-gnu). And this command:
echo "-e"
doesn't print anything. I guess this is because "-e" is one of a valid options of echo command because echo "-n" and echo "-E" (the other two options) also produce empty strings.
The question is how to escape the sequence "-e" for echo to get the natural output ("-e").
The one true way to print any arbitrary string:
printf "%s" "$vars"
This is a tough one ;)
Usually you would use double dashes to tell the command that it should stop interpreting options, but echo will only output those:
$ echo -- -e
-- -e
You can use -e itself to get around the problem:
$ echo -e '\055e'
-e
Also, as others have pointed out, if you don't insist on using the bash builtin echo, your /bin/echo binary might be the GNU version of the tool (check the man page) and thus understand the POSIXLY_CORRECT environment variable:
$ POSIXLY_CORRECT=1 /bin/echo -e
-e
There may be a better way, but this works:
printf -- "-e\n"
You could cheat by doing
echo "-e "
That would be dash, e, space.
Alternatively you can use the more complex, but more precise:
echo -e \\\\x2De
[root#scintia mail]# POSIXLY_CORRECT=1; export POSIXLY_CORRECT
[root#scintia mail]# /bin/echo "-e"
-e
[root#scintia mail]#
Another alternative:
echo x-e | sed 's/^x//'
This is the way recommended by the autoconf manual:
[...] It is often possible to avoid this problem using 'echo "x$word"', taking the 'x' into account later in the pipe.
After paying careful attention to the man page :)
SYSV3=1 /usr/bin/echo -e
works, on Solaris at least
I like that one using a herestring:
cat <<<"-e"
Another way:
echo -e' '
echo -e " \b-e"
/bin/echo -e
works, but why?
[resin#nevada ~]$ which echo
/bin/echo

Resources