I have managed to track done a weird problem in an init script I am working on. I have simplified the problem down in the following example:
> set -x # <--- Make Bash show the commands it runs
> cmd="echo \"hello this is a test\""
+ cmd='echo "hello this is a test"'
> $cmd
+ echo '"hello' this is a 'test"' # <--- Where have the single quotes come from?
"hello this is a test"
Why is bash inserting those extra single quotes into the executed command?
The extra quotes don't cause any problems in the above example, but they are really giving me a headache.
For the curious, the actual problem code is:
cmd="start-stop-daemon --start $DAEMON_OPTS \
--quiet \
--oknodo \
--background \
--make-pidfile \
$* \
--pidfile $CELERYD_PID_FILE
--exec /bin/su -- -c \"$CELERYD $CELERYD_OPTS\" - $CELERYD_USER"
Which produces this:
start-stop-daemon --start --chdir /home/continuous/ci --quiet --oknodo --make-pidfile --pidfile /var/run/celeryd.pid --exec /bin/su -- -c '"/home/continuous/ci/manage.py' celeryd -f /var/log/celeryd.log -l 'INFO"' - continuous
And therefore:
/bin/su: invalid option -- 'f'
Note: I am using the su command here as I need to ensure the user's virtualenv is setup before celeryd is run. --chuid will not provide this
Because when you try to execute your command with
$cmd
only one layer of expansion happens. $cmd contains echo "hello this is a test", which is expanded into 6 whitespace-separated tokens:
echo
"hello
this
is
a
test"
and that's what the set -x output is showing you: it's putting single quotes around the tokens that contain double quotes, in order to be clear about what the individual tokens are.
If you want $cmd to be expanded into a string which then has all the bash quoting rules applied again, try executing your command with:
bash -c "$cmd"
or (as #bitmask points out in the comments, and this is probably more efficient)
eval "$cmd"
instead of just
$cmd
Use Bash arrays to achieve the behavior you want, without resorting to the very dangerous (see below) eval and bash -c.
Using arrays:
declare -a CMD=(echo --test-arg \"Hello\ there\ friend\")
set -x
echo "${CMD[#]}"
"${CMD[#]}"
outputs:
+ echo echo --test-arg '"Hello there friend"'
echo --test-arg "Hello there friend"
+ echo --test-arg '"Hello there friend"'
--test-arg "Hello there friend"
Be careful to ensure that your array invocation is wrapped by double-quotes; otherwise, Bash tries to perform the same "bare-minimum safety" escaping of special characters:
declare -a CMD=(echo --test-arg \"Hello\ there\ friend\")
set -x
echo "${CMD[#]}"
${CMD[#]}
outputs:
+ echo echo --test-arg '"Hello there friend"'
echo --test-arg "Hello there friend"
+ echo --test-arg '"Hello' there 'friend"'
--test-arg "Hello there friend"
ASIDE: Why is eval dangerous?
eval is only safe if you can guarantee that every input passed to it will not unexpectedly change the way that the command under eval works.
Example: As a totally contrived example, let's say we have a script that runs as part of our automated code deployment process. The script sorts some input (in this case, three lines of hardcoded text), and outputs the sorted text to a file whose named is based on the current directory name. Similar to the original SO question posed here, we want to dynamically construct the --output= parameter passed to sort, but we must (must? not really) rely on eval because of Bash's auto-quoting "safety" feature.
echo $'3\n2\n1' | eval sort -n --output="$(pwd | sed 's:.*/::')".txt
Running this script in the directory /usr/local/deploy/project1/ results in a new file being created at /usr/local/deploy/project1/project1.txt.
So somehow, if a user were to create a project subdirectory named owned.txt; touch hahaha.txt; echo, the script would actually run the following series of commands:
echo $'3\n2\n1'
sort -n --output=owned.txt; touch hahaha.txt; echo .txt
As you can see, that's totally not what we want. But you may ask, in this contrived example, isn't it unlikely that the user could create a project directory owned.txt; touch hahaha.txt; echo, and if they could, aren't we already in trouble already?
Maybe, but what about a scenario where the script is parsing not the current directory name, but instead the name of a remote git source code repository branch? Unless you plan to be extremely diligent about restricting or sanitizing every user-controlled artifact whose name, identifier, or other data is used by your script, stay well clear of eval.
Related
While debugging a script that runs various commands remotely, I noticed some problems getting output from echo.
I realize that the bash -c isn't necessary here, but it still has me wondering.
In my shell:
> bash -c "echo hello && echo hi"
hello
hi
But, if I bring SSH into the picture:
> ssh ${myhost} bash -c "echo hello && echo hi"
hi
Yet, date outputs, even though that first echo didn't:
> ssh ${myhost} bash -c "date && echo hi"
Thu Jun 3 21:15:26 UTC 2021
hi
What's going on here?
When you run a command via ssh like this, it's parsed twice: first on the local computer (before it's passed to the ssh command as arguments), then again on the remote computer before it's actually executed. Each time it's parsed, the shell will apply and remove quotes and escapes. That means the double-quotes you have around the command get applied and removed by the local shell, before the command is sent to the remote shell. So what looks like this command:
bash -c "echo hello && echo hi"
Turns into this by the time the remote shell sees it:
bash -c echo hello && echo hi
...which is two separate commands, bash -c echo hello and echo hi. The second one, echo hi, works as you expect, but the first may not.
With bash -c, the argument immediately after that is taken as the command string to execute, and any further arguments are assigned to $0, $1, etc as it runs. So bash -c echo hello just runs echo with $0 set to "hello". So it prints a blank line.
If you want the command to be executed as you expect, you need two layers of quotes and/or escapes, one to be applied and removed by the local shell and another to be applied and removed by the remote shell. Any of these will work:
# Single-quotes for local shell, double for remote
ssh ${myhost} 'bash -c "echo hello && echo hi"'
# Double-quotes for local shell, single for remote
ssh ${myhost} "bash -c 'echo hello && echo hi'"
# Double-quotes for local shell, escaped doubles for remote
ssh ${myhost} "bash -c \"echo hello && echo hi\""
# Single-quotes for local shell, escaped characters for remote
ssh ${myhost} 'bash -c echo\ hello\ \&\&\ echo\ hi'
...and many more possibilities. Note that if the command string contains anything like variable references or command substitutions, you need to pay attention to whether you want them to expand on the local or remote computer, and make sure the quoting/escaping method you use accomplishes that.
BTW, in this case since you're running a command with bash -c, there's actually a third layer of parsing done by the shell invoked by bash -c. If that command has anything that needed quoting/escaping, keeping the levels straight will be even more complex.
The command that arrives to the server is : bash -c echo hello && echo hi
ie without quote
and if you run this cde locally, it produces the same output
If you want the good result
ssh mm 'bash -c "echo hello && echo hi"'
Could also try using ";" as a separator
ssh ${myserver} "echo hello; echo hi"
testuser#mymac ~ % ssh ${myserver} "echo hello; echo hi"
hello
hi
testuser#mymac ~ %
I've simplified my example to the following:
file1.sh:
#!/bin/bash
bash -c "./file2.sh $#"
file2.sh:
#!/bin/bash
echo "first $1"
echo "second $2"
I expect that if I call ./file1.sh a b to get:
first a
second b
but instead I get:
first a
second
In other words, my later arguments after the first one are not getting passed through to the command that I'm executing inside a new bash shell. I've tried many variations of removing and moving around the quotation marks in the file1.sh file, but haven't got this to work.
Why is this happening, and how do I get the behavior I want?
(UPDATE - I realize it seems pointless that I'm calling bash -c in this example, my actual file1.sh is a proxy script for a command that gets called locally to run in a docker container so it's actually docker exec -i mycontainer bash -c '')
Change file1.sh to this with different quoting:
#!/bin/bash
bash -c './file2.sh "$#"' - "$#"
- "$#" is passing hyphen to populate $0 and $# is being passed in to populate all other positional parameters in bash -c command line.
You can also make it:
bash -c './file2.sh "$#"' "$0" "$#"
However there is no real need to use bash -c here and you can just use:
./file2.sh "$#"
#!/bin/bash
sudo -u rei_sh -H -s <<-EOM
echo "hi"
for i in $(find -name *.properties)
do
echo "ivalue is \$i"
done
exit
EOM
I am using the above piece of code.
When I run the code only "hi" gets printed.
but if I use "for i in one two" the following gets printed
ivalue is one
ivalue is two.
Why can't I loop through the output of find command inside a here document?
Because the $(...) is run before the sudo kicks in...you need to quote the EOM:
sudo -u rei_sh -H -s <<-'EOM'
echo "hi"
for i in $(find -name *.properties)
do
echo "ivalue is $i"
done
EOM
The exit is redundant. With the quotes around the EOM, you also don't need to escape the $ in the echo command. You could alternatively have escaped the $ before the $(...) to prevent the pre-sudo shell from executing the command.
process substitution & variable expansion happens before heredocs are processed.
Try escaping the $ signs & glob (*).
This must either be really simple or really complex, but I couldn't find anything about it... I am trying to open a new bash instance, then run a few commands inside it, and give the control back to the user inside that same instance.
I tried:
$ bash -lic "some_command"
but this executes some_command inside the new instance, then closes it. I want it to stay open.
One more detail which might affect answers: if I can get this to work I will use it in my .bashrc as alias(es), so bonus points for an alias implementation!
bash --rcfile <(echo '. ~/.bashrc; some_command')
dispenses the creation of temporary files. Question on other sites:
https://serverfault.com/questions/368054/run-an-interactive-bash-subshell-with-initial-commands-without-returning-to-the
https://unix.stackexchange.com/questions/123103/how-to-keep-bash-running-after-command-execution
This is a late answer, but I had the exact same problem and Google sent me to this page, so for completeness here is how I got around the problem.
As far as I can tell, bash does not have an option to do what the original poster wanted to do. The -c option will always return after the commands have been executed.
Broken solution: The simplest and obvious attempt around this is:
bash -c 'XXXX ; bash'
This partly works (albeit with an extra sub-shell layer). However, the problem is that while a sub-shell will inherit the exported environment variables, aliases and functions are not inherited. So this might work for some things but isn't a general solution.
Better: The way around this is to dynamically create a startup file and call bash with this new initialization file, making sure that your new init file calls your regular ~/.bashrc if necessary.
# Create a temporary file
TMPFILE=$(mktemp)
# Add stuff to the temporary file
echo "source ~/.bashrc" > $TMPFILE
echo "<other commands>" >> $TMPFILE
echo "rm -f $TMPFILE" >> $TMPFILE
# Start the new bash shell
bash --rcfile $TMPFILE
The nice thing is that the temporary init file will delete itself as soon as it is used, reducing the risk that it is not cleaned up correctly.
Note: I'm not sure if /etc/bashrc is usually called as part of a normal non-login shell. If so you might want to source /etc/bashrc as well as your ~/.bashrc.
You can pass --rcfile to Bash to cause it to read a file of your choice. This file will be read instead of your .bashrc. (If that's a problem, source ~/.bashrc from the other script.)
Edit: So a function to start a new shell with the stuff from ~/.more.sh would look something like:
more() { bash --rcfile ~/.more.sh ; }
... and in .more.sh you would have the commands you want to execute when the shell starts. (I suppose it would be elegant to avoid a separate startup file -- you cannot use standard input because then the shell will not be interactive, but you could create a startup file from a here document in a temporary location, then read it.)
bash -c '<some command> ; exec /bin/bash'
will avoid additional shell sublayer
You can get the functionality you want by sourcing the script instead of running it. eg:
$cat script
cmd1
cmd2
$ . script
$ at this point cmd1 and cmd2 have been run inside this shell
Append to ~/.bashrc a section like this:
if [ "$subshell" = 'true' ]
then
# commands to execute only on a subshell
date
fi
alias sub='subshell=true bash'
Then you can start the subshell with sub.
The accepted answer is really helpful! Just to add that process substitution (i.e., <(COMMAND)) is not supported in some shells (e.g., dash).
In my case, I was trying to create a custom action (basically a one-line shell script) in Thunar file manager to start a shell and activate the selected Python virtual environment. My first attempt was:
urxvt -e bash --rcfile <(echo ". $HOME/.bashrc; . %f/bin/activate;")
where %f is the path to the virtual environment handled by Thunar.
I got an error (by running Thunar from command line):
/bin/sh: 1: Syntax error: "(" unexpected
Then I realized that my sh (essentially dash) does not support process substitution.
My solution was to invoke bash at the top level to interpret the process substitution, at the expense of an extra level of shell:
bash -c 'urxvt -e bash --rcfile <(echo "source $HOME/.bashrc; source %f/bin/activate;")'
Alternatively, I tried to use here-document for dash but with no success. Something like:
echo -e " <<EOF\n. $HOME/.bashrc; . %f/bin/activate;\nEOF\n" | xargs -0 urxvt -e bash --rcfile
P.S.: I do not have enough reputation to post comments, moderators please feel free to move it to comments or remove it if not helpful with this question.
With accordance with the answer by daveraja, here is a bash script which will solve the purpose.
Consider a situation if you are using C-shell and you want to execute a command
without leaving the C-shell context/window as follows,
Command to be executed: Search exact word 'Testing' in current directory recursively only in *.h, *.c files
grep -nrs --color -w --include="*.{h,c}" Testing ./
Solution 1: Enter into bash from C-shell and execute the command
bash
grep -nrs --color -w --include="*.{h,c}" Testing ./
exit
Solution 2: Write the intended command into a text file and execute it using bash
echo 'grep -nrs --color -w --include="*.{h,c}" Testing ./' > tmp_file.txt
bash tmp_file.txt
Solution 3: Run command on the same line using bash
bash -c 'grep -nrs --color -w --include="*.{h,c}" Testing ./'
Solution 4: Create a sciprt (one-time) and use it for all future commands
alias ebash './execute_command_on_bash.sh'
ebash grep -nrs --color -w --include="*.{h,c}" Testing ./
The script is as follows,
#!/bin/bash
# =========================================================================
# References:
# https://stackoverflow.com/a/13343457/5409274
# https://stackoverflow.com/a/26733366/5409274
# https://stackoverflow.com/a/2853811/5409274
# https://stackoverflow.com/a/2853811/5409274
# https://www.linuxquestions.org/questions/other-%2Anix-55/how-can-i-run-a-command-on-another-shell-without-changing-the-current-shell-794580/
# https://www.tldp.org/LDP/abs/html/internalvariables.html
# https://stackoverflow.com/a/4277753/5409274
# =========================================================================
# Enable following line to see the script commands
# getting printing along with their execution. This will help for debugging.
#set -o verbose
E_BADARGS=85
if [ ! -n "$1" ]
then
echo "Usage: `basename $0` grep -nrs --color -w --include=\"*.{h,c}\" Testing ."
echo "Usage: `basename $0` find . -name \"*.txt\""
exit $E_BADARGS
fi
# Create a temporary file
TMPFILE=$(mktemp)
# Add stuff to the temporary file
#echo "echo Hello World...." >> $TMPFILE
#initialize the variable that will contain the whole argument string
argList=""
#iterate on each argument
for arg in "$#"
do
#if an argument contains a white space, enclose it in double quotes and append to the list
#otherwise simply append the argument to the list
if echo $arg | grep -q " "; then
argList="$argList \"$arg\""
else
argList="$argList $arg"
fi
done
#remove a possible trailing space at the beginning of the list
argList=$(echo $argList | sed 's/^ *//')
# Echoing the command to be executed to tmp file
echo "$argList" >> $TMPFILE
# Note: This should be your last command
# Important last command which deletes the tmp file
last_command="rm -f $TMPFILE"
echo "$last_command" >> $TMPFILE
#echo "---------------------------------------------"
#echo "TMPFILE is $TMPFILE as follows"
#cat $TMPFILE
#echo "---------------------------------------------"
check_for_last_line=$(tail -n 1 $TMPFILE | grep -o "$last_command")
#echo $check_for_last_line
#if tail -n 1 $TMPFILE | grep -o "$last_command"
if [ "$check_for_last_line" == "$last_command" ]
then
#echo "Okay..."
bash $TMPFILE
exit 0
else
echo "Something is wrong"
echo "Last command in your tmp file should be removing itself"
echo "Aborting the process"
exit 1
fi
$ bash --init-file <(echo 'some_command')
$ bash --rcfile <(echo 'some_command')
In case you can't or don't want to use process substitution:
$ cat script
some_command
$ bash --init-file script
Another way:
$ bash -c 'some_command; exec bash'
$ sh -c 'some_command; exec sh'
sh-only way (dash, busybox):
$ ENV=script sh
Here is yet another (working) variant:
This opens a new gnome terminal, then in the new terminal it runs bash. The user's rc file is read first, then a command ls -la is sent for execution to the new shell before it turns interactive.
The last echo adds an extra newline that is needed to finish execution.
gnome-terminal -- bash -c 'bash --rcfile <( cat ~/.bashrc; echo ls -la ; echo)'
I also find it useful sometimes to decorate the terminal, e.g. with colorfor better orientation.
gnome-terminal --profile green -- bash -c 'bash --rcfile <( cat ~/.bashrc; echo ls -la ; echo)'
I have managed to track done a weird problem in an init script I am working on. I have simplified the problem down in the following example:
> set -x # <--- Make Bash show the commands it runs
> cmd="echo \"hello this is a test\""
+ cmd='echo "hello this is a test"'
> $cmd
+ echo '"hello' this is a 'test"' # <--- Where have the single quotes come from?
"hello this is a test"
Why is bash inserting those extra single quotes into the executed command?
The extra quotes don't cause any problems in the above example, but they are really giving me a headache.
For the curious, the actual problem code is:
cmd="start-stop-daemon --start $DAEMON_OPTS \
--quiet \
--oknodo \
--background \
--make-pidfile \
$* \
--pidfile $CELERYD_PID_FILE
--exec /bin/su -- -c \"$CELERYD $CELERYD_OPTS\" - $CELERYD_USER"
Which produces this:
start-stop-daemon --start --chdir /home/continuous/ci --quiet --oknodo --make-pidfile --pidfile /var/run/celeryd.pid --exec /bin/su -- -c '"/home/continuous/ci/manage.py' celeryd -f /var/log/celeryd.log -l 'INFO"' - continuous
And therefore:
/bin/su: invalid option -- 'f'
Note: I am using the su command here as I need to ensure the user's virtualenv is setup before celeryd is run. --chuid will not provide this
Because when you try to execute your command with
$cmd
only one layer of expansion happens. $cmd contains echo "hello this is a test", which is expanded into 6 whitespace-separated tokens:
echo
"hello
this
is
a
test"
and that's what the set -x output is showing you: it's putting single quotes around the tokens that contain double quotes, in order to be clear about what the individual tokens are.
If you want $cmd to be expanded into a string which then has all the bash quoting rules applied again, try executing your command with:
bash -c "$cmd"
or (as #bitmask points out in the comments, and this is probably more efficient)
eval "$cmd"
instead of just
$cmd
Use Bash arrays to achieve the behavior you want, without resorting to the very dangerous (see below) eval and bash -c.
Using arrays:
declare -a CMD=(echo --test-arg \"Hello\ there\ friend\")
set -x
echo "${CMD[#]}"
"${CMD[#]}"
outputs:
+ echo echo --test-arg '"Hello there friend"'
echo --test-arg "Hello there friend"
+ echo --test-arg '"Hello there friend"'
--test-arg "Hello there friend"
Be careful to ensure that your array invocation is wrapped by double-quotes; otherwise, Bash tries to perform the same "bare-minimum safety" escaping of special characters:
declare -a CMD=(echo --test-arg \"Hello\ there\ friend\")
set -x
echo "${CMD[#]}"
${CMD[#]}
outputs:
+ echo echo --test-arg '"Hello there friend"'
echo --test-arg "Hello there friend"
+ echo --test-arg '"Hello' there 'friend"'
--test-arg "Hello there friend"
ASIDE: Why is eval dangerous?
eval is only safe if you can guarantee that every input passed to it will not unexpectedly change the way that the command under eval works.
Example: As a totally contrived example, let's say we have a script that runs as part of our automated code deployment process. The script sorts some input (in this case, three lines of hardcoded text), and outputs the sorted text to a file whose named is based on the current directory name. Similar to the original SO question posed here, we want to dynamically construct the --output= parameter passed to sort, but we must (must? not really) rely on eval because of Bash's auto-quoting "safety" feature.
echo $'3\n2\n1' | eval sort -n --output="$(pwd | sed 's:.*/::')".txt
Running this script in the directory /usr/local/deploy/project1/ results in a new file being created at /usr/local/deploy/project1/project1.txt.
So somehow, if a user were to create a project subdirectory named owned.txt; touch hahaha.txt; echo, the script would actually run the following series of commands:
echo $'3\n2\n1'
sort -n --output=owned.txt; touch hahaha.txt; echo .txt
As you can see, that's totally not what we want. But you may ask, in this contrived example, isn't it unlikely that the user could create a project directory owned.txt; touch hahaha.txt; echo, and if they could, aren't we already in trouble already?
Maybe, but what about a scenario where the script is parsing not the current directory name, but instead the name of a remote git source code repository branch? Unless you plan to be extremely diligent about restricting or sanitizing every user-controlled artifact whose name, identifier, or other data is used by your script, stay well clear of eval.