The POSIX shell standard at
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04
says in Section 2.6:
command substitution (...) shall be performed
(...)
Quote removal (...) shall always be performed last.
It appears to me that quote removal is not performed after command substitution:
$ echo "#"
#
$ echo '"'
"
as expected, but
$ echo $(echo '"')#"
>
What am I not understanding?
Added after reading answer/comments:
From what everybody is saying, the consideration of quotes happens at the very beginning of parsing, for example, to decide if a command is even "acceptable". Then why does the standard bother to emphasise, that the quote removal is performed late in the process??
"then the outer command becomes echo "#" and is balanced"
That is not 'balanced' because the first double-quote does not count. Quotes are only meaningful as quotes if they appear unencumbered on the command line.
To verify, let's look at this:
$ echo $(echo '"')#
"#
That is balanced because the shell does considers that " to be just another character.
By contrast, this is unbalanced because it has one and only one shell-active ":
$ echo $(echo '"')#"
>
Similar example 1
Here we show the same thing but using parameter expansion instead of command substitution:
$ q='"'; echo $q
"
Once the shell has substituted " for $q, one might think that there was an unbalanced double-quote. But, that double-quote was the results of parameter expansion and is therefore not a shell-active quote.
Similar example 2
Let's consider a directory containing file:
$ ls
file
$ ls "file"
file
As you can see above, quote removal is perfomed before ls is run.
But, consider this command:
$ echo ls $(echo '"file"')
ls "file"
As you can see ls $(echo '"file"') expands to ls "file" which is the command which ran successfully above. Now, let's try running that:
$ ls $(echo '"file"')
ls: cannot access '"file"': No such file or directory
As you can see, the shell does not treat the double-quotes that remain after command substitution. This is because those quotes are not considered to be shell-active. As a consequence, they are treated as normal characters and passed on to ls which complains that the file whose name begins and ends with " does not exist in the directory.
The same is happening here:
$ cmd='ls "file"'
$ $cmd
ls: cannot access '"file"': No such file or directory
POSIX standard
From the POSIX standard:
Enclosing characters in single-quotes ( ' ' ) shall preserve the
literal value of each character within the single-quotes
In other words, once the double-quote appears inside single quotes, it has no special powers: it is just another character.
The standard also mentions escaping and double-quotes as methods of preserving "the literal value" of a character.
Practical consequences
People new to shell often want to store a command in a variable as in the cmd='ls "file"' example above. But, because quotes and other shell-active characters cease to be shell active once they are stored in a variable, the complex cases always fail. This leads to a classic essay:
"I'm trying to put a command in a variable, but the complex cases always fail!"
Related
This question already has answers here:
echo "#!" fails -- "event not found"
(5 answers)
Closed 7 years ago.
I am attempting to parse the output of a VNC server startup event and have run into a problem in parsing using sed in a command substitution. Specifically, the remote VNC server is started in a manner such as the following:
address1="user1#lxplus.cern.ch"
VNCServerResponse="$(ssh "${address1}" 'vncserver' 2>&1)"
The standard error output produced in this startup event is then to be parsed in order to extract the server and display information. At this point the content of the variable VNCServerResponse is something such as the following:
New 'lxplus0186.cern.ch:1 (user1)' desktop is lxplus0186.cern.ch:1
Starting applications specified in /afs/cern.ch/user/u/user1/.vnc/xstartup
Log file is /afs/cern.ch/user/u/user1/.vnc/lxplus0186.cern.ch:1.log
This output can be parsed in the following way in order to extract the server and display information:
echo "${VNCServerResponse}" | sed '/New.*desktop.*is/!d' \
| awk -F" desktop is " '{print $2}'
The result is something such as the following:
lxplus0186.cern.ch:1
What I want to do is use this parsing in a command substitution something like the following:
VNCServerAndDisplayNumber="$(echo "${VNCServerResponse}" \
| sed '/New.*desktop.*is/!d' | awk -F" desktop is " '{print $2}')"
On attempting to do this, I am presented with the following error:
bash: !d': event not found
I am not sure how to address this. It appears to be a problem in the way sed is being used in the command substitution. I would appreciate guidance.
Bash history expansion is a very odd corner in the bash command line parser, and you are clearly running into an unexpected history expansion, which is explained below. However, any sort of history expansion in a script is unexpected, because normally history expansion is not enabled in scripts; not even scripts run with the source (or .) builtin.
How history expansion is enabled (or disabled)
There are two shell options which control history expansion:
set -o history: Required for the history to be recorded.
set -H (or set -o histexpand): Additionally required for history expansion to be enabled.
Both of these options must be set for history expansion to be recognized. (I found the manual unclear on this interaction, but it's logical enough.)
According to the bash manual, these options are unset for non-interactive shells, so if you want to enable history expansion in a script (and I cannot imagine a reason you would want this), you would need to set both of them:
set -o history -o histexpand
The situation for scripts run with source is more complicated (and what I'm about to say only applies to bash v4, and since it's undocumented in might change in the future). [Note 3]
History recording (and consequently expansion) is turned off in source'd scripts, but through an internal flag which, as far as I know, is not made visible. It certainly does not appear in $SHELLOPTS. Since a sourced script runs in the current bash context, it shares the current execution environment, including shell options. So in the execution of a sourced script initiated from an interactive session, you'll see both history and histexpand in $SHELLOPTS, but no history expansion will take place. In order to enable it, you need to:
set -o history
which is not a no-op because it has the side-effect of resetting the internal flag which suppresses history recording. Setting the histexpand shell option does not have this side-effect.
In short, I'm not sure how you managed to enable history expansion in a script (if, indeed, the misbehaving command was in a script and not in an interactive shell), but you might want to consider not doing so, unless you have a really good reason.
How history expansion is parsed
The bash implementation of history expansion is designed to work with readline, so that it can be performed during command input. (By default this function is bound to Meta-^; generally Meta is ESC, but you can customize that as well.) However, it is also performed immediately after each line is input, before any bash parsing is performed.
By default, the history expansion character is !, and -- as mostly documented -- that will trigger history expansion except:
when it is followed by whitespace or =
if the shell option extglob is set, and it is followed by ( [Note 1]
if it appears in a single-quoted string
if it is preceded by a \ [Note 2 and see below]
if it is preceded by $ or ${ [Note 1]
if it is preceded by [ [Note 1]
(As of bash v4.3) if it is the last character in a double-quoted string.
The immediate issue here is the precise interpretation of the third case, an ! appearing inside of a single-quoted string. Normally, bash starts a new quoting context for a command substitution ($(...) or the deprecated backtick notation). For example:
$ s=SUBSTITUTED
$ # The interior single quotes are just characters
$ echo "'Echoing $s'"
'Echoing SUBSTITUTED'
$ # The interior single quotes are single quotes
$ echo "$(echo 'Echoing $s')"
Echoing $s
However, the history expansion scanner isn't that intelligent. It keeps track of quotes, but not of command substitution. So as far as it is concerned, both of the single quotes in the above example are double-quoted single quotes, which is to say ordinary characters. So history expansion occurs in both of them:
# A no-op to indicated history expansion
$ HIST() { :; }
# Single-quoted strings inhibit history expansion
$ HIST
$ echo '!!'
!!
# Double-quoted strings allow history expansion
$ HIST
$ echo "'!!'"
echo "'HIST'"
'HIST'
# ... and it applies also to interior command substitution.
$ HIST
$ echo "$(echo '!!')"
echo "$(echo 'HIST')"
HIST
So if you have a perfectly normal command like sed '/foo/!d' file, where you would expect the single-quotes to protect you from history-expansion, and you put it inside a double-quoted command substitution:
result="$(sed '/foo/!d' file)"
you suddenly find that the ! is a history expansion character. Worse, you can't fix this by backslash escaping the exclamation point, because although "\!" inhibits history expansion, it doesn't remove the backslash:
$ echo "\!"
\!
In this particular example -- and the one in the OP -- the double quotes are completely unnecessary, because the right-hand side of a variable assignment does not undergo either filename expansion nor word splitting. However, there are other contexts in which removing the double quotes would change the semantics:
# Undesired history expansion
printf "The answer is '%s'\n" "$(sed '/foo/!d' file)"
# Undesired word splitting
printf "The answer is '%s'\n" $(sed '/foo/!d' file)
In this case, the best solution is probably to put the sed argument in a variable
# Works
sed_prog='/foo/!d'
printf "The answer is '%s'\n" "$(sed "$sed_prog" file)"
(The quotes around $sed_prog were not necessary in this case but usually they would be, and they do no harm.)
Notes:
The inhibition of history expansion when the following character is some form of open parenthesis only works if there is a corresponding close parenthesis in the rest of the string. However, it doesn't have to really match the open parenthesis. For example:
# No matching close parenthesis
$ echo "!("
bash: !: event not found
# The matching close parenthesis has nothing to do with the open
$ echo "!(" ")"
!( )
# An actual extended glob: files whose names don't start with a
$ echo "!(a*)"
b
As indicated in the bash manual, a history-expansion character is treated as an ordinary character if immediately preceded by a backslash. This is literally true; it doesn't matter whether the backslash will later be considered an escape character or not:
$ echo \!
!
$ echo \\!
\!
$ echo \\\!
\!
\ also inhibits history expansion inside double quotes, but \! is not a valid escape sequence inside the double quoted string, so the backslash is not removed:
$ echo "\!"
\!
$ echo "\\!"
\!
$ echo "\\\!"
\\!
I'm referring to the source code for bash v4.2 as I write this, so any undocumented behaviour may be completely different as of v4.3.
The problem is that within double quotes, bash is trying to expand !d before passing it to the subshell. You can get around this problem by removing the double quotes but I would also propose a simplification to your script:
VNCServerAndDisplayNumber=$(echo "$VNCServerResponse" | awk '/desktop/ {print $NF}')
This simply prints the last field on the line containing the word "desktop".
On a newer bash, you can use a herestring rather than piping an echo:
VNCServerAndDisplayNumber=$(awk '/desktop/ {print $NF}' <<<"$VNCServerResponse")
Don't wrap the $(...) command substitution in double quotes. You are asking the shell to perform evaluation on the contents of the quotes and are hitting the history substitution expansion feature. Drop the quotes and you stop telling the shell to do that and you won't hit that problem.
And yes, dropping those quotes is safe on that assignment line even if the output may contain spaces or newlines or whatever. Assignments of that sort are not going to split on those the way command substitution or variable evaluation will on a normal shell execution line.
Alternatively, disable history expansion in your shell/script before you run that. (It should be off when running a script by default I believe anyway.)
This only happens when history expansion is enabled, which it normally isn't and definitely shouldn't be for scripts.
Rather than trying to work around it, figure out why history expansion is enabled and what to do so it isn't.
If you're executing your script with . foo or source foo, use ./foo instead.
If you're writing this as a function in .bashrc or similar, consider making it a separate script.
If your script (or BASH_ENV) explicitly does set -H, don't.
Quote it with '' or \ or disable history expansion with set +H or shopt -u -o histexpand. See History Expansion.
My problem boils down to this:
echo $(echo '*')
That outputs the names of all the files in the current directory.
I do not want that. I want a literal asterisk (*).
How do I do this in a generic way?
My above example is simplified. The asterisk is not literally written in my bash script - it comes from the result of another command.
So this is perhaps closer to my real situation:
echo $(my-special-command)
I just want to get the literal output of my-special-command; I do not want any embedded asterisks (or other special characters) to be expanded.
How do I do this in a general-purpose way?
I suppose I could do set -f before running the command, but how can I be sure that covers everything? That turns off pathname expansion, but what about other kinds? I have zero control over what output might be produced by my-special-command, so must be able to handle everything properly.
Just enclose the Command substitution with double quotes:
echo "$(my-special-command)"
Its called globbing, you have multiply ways to prevent it:
echo * # will expand to files / directories
echo "*" # Will print *
echo '*' # Will also print *
In your example you can simple write:
echo "$(echo '*')"
You can also turn off globbing in your script by calling it with bash -f script.sh or inside your code:
#!/usr/bin/env bash
set -f
echo *
From the "Command Substitution" section of the man page:
If the [command] substitution appears within double quotes, word splitting and
pathname expansion are not performed on the results.
By quoting the command expansion, you prevent its result, *, from undergoing pathname expansion.
$ echo "$(echo "*")"
Running this statement in OS X Terminal
for i in `ls -v *.mkv`; do echo $i; done
will successfully print out all the file names in the directory in name order with each file name on its own line.
Source: This StackOverFlow answer
However, if I run this statement in OS X Terminal
for i in 'ls -v *.mkv'; do echo $i; done
the output is "ls -v fileName1.mkv fileName2.mkv", etc. with all the file names concatenated into one long line (as opposed to each being printed on its own line).
My questions are:
What's the difference between ` and ' in bash?
Why is that difference responsible for the completely different output?
What keyboard combination produces `? (Keyboard combination)
1) Text between backticks is executed and replaced by the output of the enclosed command, so:
echo `echo 42`
Will expand to:
echo 42
This is called Command Substitution and can also be achieved using the syntax $(command). In your case, the following:
for i in `ls -v *.mkv`; do ...
Is replaced by something like (if your directory contains 3 files named a.mkv, b.mkv and c.mkv):
for i in a.mkv b.mkv c.mkv; do ...
Text between quotes or double quotes are just plain Bash strings with characters like space scaped inside them (there are other ways to quote strings in Bash and are described here):
echo "This is just a plain and simple String"
echo 'and this is another string'
A difference between using ' and " is that strings enclosed between " can interpolate variables, for example:
variable=42
echo "Your value is $variable"
Or:
variable=42
echo "Your value is ${variable}"
Prints:
Your value is 42
2) Wildcard expressions like *.mkv are replaced by the expanded filenames in a process known as Globbing. Globbing is activated using wildcards in most of the commands without enclosing the expression inside a string:
echo *.mkv
Will print:
a.mkv b.mkv c.mkv
Meanwhile:
echo "*.mkv"
prints:
*.mkv
The i variable in your for loop takes the value "ls -v *.mkv" but the echo command inside the loop body takes $i without quotes, so Bash applied globbing there, you end up with the following:
for i in 'ls -v *.mkv'; do
# echo $i
#
# which is expanded to:
# echo ls -v *.mkv (no quotes)
#
# and the globbing process transform the above into:
echo ls -v a.mkv b.mkv c.mkv
Which is just a one-line string with the file names after the globbing is applied.
3) It depends on your keyboard layout.
One trick to keep the character around is to use the program ascii, search for the character 96 (Hex 60), copy it and keep it on your clipboard (you can use parcellite or any other clipboard manager that suits your needs).
Update: As suggested by #triplee, you should check useless use of ls as this is considered a bash pitfall and there are better ways to achieve what you're trying to do.
'expression', will output the exact string in expression.
`expression`, will execute the content of the expression and echo outputs it.
For example:
x="ls"
echo "$x" --> $x
echo `$x` --> file1 file2 ... (the content of your current dir)
Backticks mean "run the thing between the backticks as a command, and then act as if I had typed the output of that command here instead". The single quotes mean, as others have said, just a literal string. So in the first case, what happens is this:
bash runs ls -v *.mkv as a command, which outputs something like:
fileName1.mkv
fileName2.mkv
bash then substitutes this back into where the backtick-surrounded command was, i.e. it effectively makes your for statement into this:
for i in fileName1.mkv fileName2.mkv; do echo $i; done
That has two "tokens": "fileName1.mkv" and "fileName2.mkv", so the loop runs its body (echo $i) twice, once for each:
echo fileName1.mkv
echo fileName2.mkv
By default, the echo command will output a newline after it finishes echoing what you told it to echo, so you'll get the output you expect, of each filename on its own line.
When you use single quotes instead of backticks, however, the stuff in between the single quotes doesn't get evaluated; i.e. bash doesn't see it as a command (or as anything special at all; the single quotes are telling bash, "this text is not special; do not try to evaluate it or do anything to it"). So that means what you're running is this:
for i in 'ls -v *.mkv'; do echo $i; done
Which has only one token, the literal string "ls -v *.mkv", so the loop body runs only once:
echo ls -v *.mkv
...but just before bash runs that echo, it expands the "*.mkv".
I glossed over this above, but when you do something like ls *.mkv, it's not actually ls doing the conversion of *.mkv into a list of all the .mkv filenames; it's bash that does that. ls never sees the *.mkv; by the time ls runs, bash has replaced it with "fileName1.mkv fileName2.mkv ...".
Similarly for echo: before running this line, bash expands the *.mkv, so what actually runs is:
echo ls -v fileName1.mkv fileName2.mkv
which outputs this text:
ls -v fileName1.mkv fileName2.mkv
(* Footnote: there's another thing I've glossed over, and that's spaces in filenames. The output of the ls between the backticks is a list of filenames, one per line. The trouble is, bash sees any whitespace -- both spaces and newlines -- as separators, so if your filenames are:
file 1.mkv
file 2.mkv
your loop will run four times ("file", "1.mkv", "file", "2.mkv"). The other form of the loop that someone mentioned, for i in *.mkv; do ... doesn't have this problem. Why? Because when bash is expanding the "*.mkv", it does a clever thing behind the scenes and treats each filename as a unit, as if you'd said "file 1.mkv" "file 2.mkv" in quotes. It can't do that in the case where you use ls because after it passes the expanded list of filenames to ls, bash has no way of knowing that what came back was a list of those same filenames. ls could have been any command.)
This question already has answers here:
echo "#!" fails -- "event not found"
(5 answers)
Closed 7 years ago.
I am attempting to parse the output of a VNC server startup event and have run into a problem in parsing using sed in a command substitution. Specifically, the remote VNC server is started in a manner such as the following:
address1="user1#lxplus.cern.ch"
VNCServerResponse="$(ssh "${address1}" 'vncserver' 2>&1)"
The standard error output produced in this startup event is then to be parsed in order to extract the server and display information. At this point the content of the variable VNCServerResponse is something such as the following:
New 'lxplus0186.cern.ch:1 (user1)' desktop is lxplus0186.cern.ch:1
Starting applications specified in /afs/cern.ch/user/u/user1/.vnc/xstartup
Log file is /afs/cern.ch/user/u/user1/.vnc/lxplus0186.cern.ch:1.log
This output can be parsed in the following way in order to extract the server and display information:
echo "${VNCServerResponse}" | sed '/New.*desktop.*is/!d' \
| awk -F" desktop is " '{print $2}'
The result is something such as the following:
lxplus0186.cern.ch:1
What I want to do is use this parsing in a command substitution something like the following:
VNCServerAndDisplayNumber="$(echo "${VNCServerResponse}" \
| sed '/New.*desktop.*is/!d' | awk -F" desktop is " '{print $2}')"
On attempting to do this, I am presented with the following error:
bash: !d': event not found
I am not sure how to address this. It appears to be a problem in the way sed is being used in the command substitution. I would appreciate guidance.
Bash history expansion is a very odd corner in the bash command line parser, and you are clearly running into an unexpected history expansion, which is explained below. However, any sort of history expansion in a script is unexpected, because normally history expansion is not enabled in scripts; not even scripts run with the source (or .) builtin.
How history expansion is enabled (or disabled)
There are two shell options which control history expansion:
set -o history: Required for the history to be recorded.
set -H (or set -o histexpand): Additionally required for history expansion to be enabled.
Both of these options must be set for history expansion to be recognized. (I found the manual unclear on this interaction, but it's logical enough.)
According to the bash manual, these options are unset for non-interactive shells, so if you want to enable history expansion in a script (and I cannot imagine a reason you would want this), you would need to set both of them:
set -o history -o histexpand
The situation for scripts run with source is more complicated (and what I'm about to say only applies to bash v4, and since it's undocumented in might change in the future). [Note 3]
History recording (and consequently expansion) is turned off in source'd scripts, but through an internal flag which, as far as I know, is not made visible. It certainly does not appear in $SHELLOPTS. Since a sourced script runs in the current bash context, it shares the current execution environment, including shell options. So in the execution of a sourced script initiated from an interactive session, you'll see both history and histexpand in $SHELLOPTS, but no history expansion will take place. In order to enable it, you need to:
set -o history
which is not a no-op because it has the side-effect of resetting the internal flag which suppresses history recording. Setting the histexpand shell option does not have this side-effect.
In short, I'm not sure how you managed to enable history expansion in a script (if, indeed, the misbehaving command was in a script and not in an interactive shell), but you might want to consider not doing so, unless you have a really good reason.
How history expansion is parsed
The bash implementation of history expansion is designed to work with readline, so that it can be performed during command input. (By default this function is bound to Meta-^; generally Meta is ESC, but you can customize that as well.) However, it is also performed immediately after each line is input, before any bash parsing is performed.
By default, the history expansion character is !, and -- as mostly documented -- that will trigger history expansion except:
when it is followed by whitespace or =
if the shell option extglob is set, and it is followed by ( [Note 1]
if it appears in a single-quoted string
if it is preceded by a \ [Note 2 and see below]
if it is preceded by $ or ${ [Note 1]
if it is preceded by [ [Note 1]
(As of bash v4.3) if it is the last character in a double-quoted string.
The immediate issue here is the precise interpretation of the third case, an ! appearing inside of a single-quoted string. Normally, bash starts a new quoting context for a command substitution ($(...) or the deprecated backtick notation). For example:
$ s=SUBSTITUTED
$ # The interior single quotes are just characters
$ echo "'Echoing $s'"
'Echoing SUBSTITUTED'
$ # The interior single quotes are single quotes
$ echo "$(echo 'Echoing $s')"
Echoing $s
However, the history expansion scanner isn't that intelligent. It keeps track of quotes, but not of command substitution. So as far as it is concerned, both of the single quotes in the above example are double-quoted single quotes, which is to say ordinary characters. So history expansion occurs in both of them:
# A no-op to indicated history expansion
$ HIST() { :; }
# Single-quoted strings inhibit history expansion
$ HIST
$ echo '!!'
!!
# Double-quoted strings allow history expansion
$ HIST
$ echo "'!!'"
echo "'HIST'"
'HIST'
# ... and it applies also to interior command substitution.
$ HIST
$ echo "$(echo '!!')"
echo "$(echo 'HIST')"
HIST
So if you have a perfectly normal command like sed '/foo/!d' file, where you would expect the single-quotes to protect you from history-expansion, and you put it inside a double-quoted command substitution:
result="$(sed '/foo/!d' file)"
you suddenly find that the ! is a history expansion character. Worse, you can't fix this by backslash escaping the exclamation point, because although "\!" inhibits history expansion, it doesn't remove the backslash:
$ echo "\!"
\!
In this particular example -- and the one in the OP -- the double quotes are completely unnecessary, because the right-hand side of a variable assignment does not undergo either filename expansion nor word splitting. However, there are other contexts in which removing the double quotes would change the semantics:
# Undesired history expansion
printf "The answer is '%s'\n" "$(sed '/foo/!d' file)"
# Undesired word splitting
printf "The answer is '%s'\n" $(sed '/foo/!d' file)
In this case, the best solution is probably to put the sed argument in a variable
# Works
sed_prog='/foo/!d'
printf "The answer is '%s'\n" "$(sed "$sed_prog" file)"
(The quotes around $sed_prog were not necessary in this case but usually they would be, and they do no harm.)
Notes:
The inhibition of history expansion when the following character is some form of open parenthesis only works if there is a corresponding close parenthesis in the rest of the string. However, it doesn't have to really match the open parenthesis. For example:
# No matching close parenthesis
$ echo "!("
bash: !: event not found
# The matching close parenthesis has nothing to do with the open
$ echo "!(" ")"
!( )
# An actual extended glob: files whose names don't start with a
$ echo "!(a*)"
b
As indicated in the bash manual, a history-expansion character is treated as an ordinary character if immediately preceded by a backslash. This is literally true; it doesn't matter whether the backslash will later be considered an escape character or not:
$ echo \!
!
$ echo \\!
\!
$ echo \\\!
\!
\ also inhibits history expansion inside double quotes, but \! is not a valid escape sequence inside the double quoted string, so the backslash is not removed:
$ echo "\!"
\!
$ echo "\\!"
\!
$ echo "\\\!"
\\!
I'm referring to the source code for bash v4.2 as I write this, so any undocumented behaviour may be completely different as of v4.3.
The problem is that within double quotes, bash is trying to expand !d before passing it to the subshell. You can get around this problem by removing the double quotes but I would also propose a simplification to your script:
VNCServerAndDisplayNumber=$(echo "$VNCServerResponse" | awk '/desktop/ {print $NF}')
This simply prints the last field on the line containing the word "desktop".
On a newer bash, you can use a herestring rather than piping an echo:
VNCServerAndDisplayNumber=$(awk '/desktop/ {print $NF}' <<<"$VNCServerResponse")
Don't wrap the $(...) command substitution in double quotes. You are asking the shell to perform evaluation on the contents of the quotes and are hitting the history substitution expansion feature. Drop the quotes and you stop telling the shell to do that and you won't hit that problem.
And yes, dropping those quotes is safe on that assignment line even if the output may contain spaces or newlines or whatever. Assignments of that sort are not going to split on those the way command substitution or variable evaluation will on a normal shell execution line.
Alternatively, disable history expansion in your shell/script before you run that. (It should be off when running a script by default I believe anyway.)
This only happens when history expansion is enabled, which it normally isn't and definitely shouldn't be for scripts.
Rather than trying to work around it, figure out why history expansion is enabled and what to do so it isn't.
If you're executing your script with . foo or source foo, use ./foo instead.
If you're writing this as a function in .bashrc or similar, consider making it a separate script.
If your script (or BASH_ENV) explicitly does set -H, don't.
Quote it with '' or \ or disable history expansion with set +H or shopt -u -o histexpand. See History Expansion.
Going over the POSIX standard, I came across another rather technical/pointless question. It states:
Within the backquoted style of command substitution, <backslash> shall retain its literal meaning, except when followed by: '$' , '`' , or <backslash>.
It's easy to see why '`' and '\' lose their literal meanings: nested command substitution demands a "different" backquote inside the command substitution, which in turn forces '\' to lose its literal meaning. So, for instance, the following different behavior seems reasonable:
$ echo $(echo \\\\)
\\
$ echo `echo \\\\`
\
But what about '$'? I.e., what's the point or, more concretely, a possible benefit of the following difference?
$ echo $(echo \$\$)
$$
$ echo `echo \$\$`
4735
As '$' by itself is not ruled out inside backquotes, it looks like you would use either '$' or '\\\$' all the time, but never the middle case '\$'.
To recap,
$ echo `echo $$` # PID, OK
4735
$ echo `echo \\\$\\\$` # literal "$$", OK
$$
$ echo `echo \$\$` # What's the point?
4735
PS: I know this question is rather technical... I myself go for the more modern $(...) substitution all the time, but I'm still curious.
By adding a \, you make the inner subshell expand it instead of the outer shell. A good example would be to actually force the starting of a new shell, like this:
$ echo $$
4988
$ echo `sh -c 'echo $$'`
4988
$ echo `sh -c 'echo \$\$'`
4990
$ echo `sh -c 'echo \\\$\\\$'`
$$
Basic Answer
Consider the following command, which finds the base directory where gcc was installed:
gcc_base=$(dirname $(dirname $(which gcc)))
With the $(...) notation, there is no problem with the parsing; it is trivial and is one of the primary reason why the notation is recommended. The equivalent command using back-ticks is:
gcc_base=`dirname \`dirname \\\`which gcc\\\`\``
When the shell first parses this command, it encounters the first backtick, and has to find the matching close backtick. That's when the quoted section comes into effect:
Within the backquoted style of command substitution, shall retain its literal meaning, except when followed by: '$' , '`' , or .
gcc_base=`dirname \`dirname \\\`which gcc\\\`\``
^ ^ ^ ^ ^ ^
1 2 3 4 5 6
backslash-backtick - special rule
backslash-backslash - special rule
backslash-backtick - special rule
backslash-backslash - special rule
backslash-backtick - special rule
backslash-backtick - special rule
So, the unescaped backtick at the end marks the end of the outermost backtick command. The sub-shell that processes that command sees:
dirname `dirname \`which gcc\``
The backslash-back escapes are given the special treatment again, and the sub-sub-shell sees:
dirname `which gcc`
The sub-sub-sub-shell gets to see which gcc and evaluates it (e.g. /usr/gcc/v4.6.1/bin/gcc).
The sub-sub-shell evaluates dirname /usr/gcc/v4.6.1/bin/gcc and produces /usr/gcc/v4.6.1/bin.
The sub-shell evaluates dirname /usr/gcc/v4.6.1/bin and produces /usr/gcc/v4.6.1.
The shell assigns /usr/gcc/v4.6.1 to gcc_base.
In this example, the backslashes were only followed by the special characters - backslash, backtick, dollar. A more complex example would have, for example, \" sequences in the command, and then the special rule would not apply; the \" would simply be copied through unchanged and passed to the relevant sub-shell(s).
Extraordinarily Complex Stuff
For example, suppose you had a command with a blank in its name (heaven forbid; and this shows why!) such as totally amazing (two blanks; it is a more stringent test than a single blank). Then you could write:
$ cmd="totally amazing"
$ echo "$cmd"
totally amazing
$ which "$cmd"
/Users/jleffler/bin/totally amazing
$ dirname $(which "$cmd")
usage: dirname path
$ # Oops!
$ dirname "$(which \"\$cmd\")"
"$cmd": not found
.
$ # Oops!
$ dirname "$(which \"$cmd\")"
"totally: not found
amazing": not found
.
$ dirname "$(eval which \"$cmd\")"
totally amazing: not found
.
$ dirname "$(eval which \"\$cmd\")"
/Users/jleffler/bin
$ # Ouch, but at least that worked!
$ # But how to extend that to the next level?
$ dirname "$(eval dirname \"\$\(eval which \\\"\\\$cmd\\\"\)\")"
/Users/jleffler
$
OK - well, that's the "easy" one! Do you need a better reason to avoid spaces in command names or path names? I've also demonstrated to my own satisfaction that it works correctly with pathnames that contain spaces.
So, can we compress the learning cycle for backticks? Yes...
$ cat x3.sh
cmd="totally amazing"
which "$cmd"
dirname "`which \"$cmd\"`"
dirname "`dirname \"\`which \\"\$cmd\\\"\`\"`"
$ sh -x x3.sh
+ cmd='totally amazing'
+ which 'totally amazing'
/Users/jleffler/bin/totally amazing
++ which 'totally amazing'
+ dirname '/Users/jleffler/bin/totally amazing'
/Users/jleffler/bin
+++ which 'totally amazing'
++ dirname '/Users/jleffler/bin/totally amazing'
+ dirname /Users/jleffler/bin
/Users/jleffler
$
That is still a ghastly, daunting, non-intuitive set of escape sequences. It's actually shorter than the version for $(...) notation, and doesn't use any eval commands (which always complicate things).
This probably has to do with the strange way the Bourne shell parses substitutions (the real Korn shell is slightly similar but most other shells do not exhibit the strange behaviour at all).
Basically, the Bourne shell's parser does not interpret substitutions ($ and `) inside double-quotes, or parameter substitution ($) anywhere. This is only done at expansion time. Also, in many cases unmatched quotes (single-quotes, double-quotes or backquotes) are not an error; the closing quote is assumed at the end.
One consequence is that if a parameter substitution with a word containing spaces like ${v+a b} occurs outside double-quotes, it is not parsed correctly and will cause an expansion error when executed. The space needs to be quoted. Other shells do not have this problem.
Another consequence is that double-quotes inside backquotes inside double-quotes do not work reliably. For example,
v=0; echo "`v=1; echo " $v "`echo b"
will print
1 echo b
in most shells (one command substitution), but
0 b
in the Bourne shell and the real Korn shell (ksh93) (two command substitutions).
(Ways to avoid the above issue are to assign the substitution to a variable first, so double-quotes are not necessary, or to use new-style command substitution.)
The real Korn shell (ksh93) attempts to preserve much of the strange Bourne shell behaviour but does parse substitutions at parse time. Thus, ${v+a b} is accepted but the above example has "strange" behaviour. A further strange thing is that something like
echo "`${v+pwd"
is accepted (the result is like with the missing closing brace). And where does the opening brace in the error message from
echo "`${v+pwd`"
come from?
The below session shows an obscure case where $ and \$ differ in a non-obvious way:
$ echo ${.sh.version}
Version JM 93u 2011-02-08
$ v=0; echo "`v=1; echo "${v+p q}"`echo b"
p qecho b
$ v=0; echo "`v=1; echo "\${v+p q}"`echo b"
p{ q}b
Basically, a backslash is an escape character. You put it before another character to represent something special. An 'n','t','$' and '\'are these special characters.
"\n" --> newline
"\t" --> tab (indent)
"\$" --> $ (because a $ before a word in shell denotes a variable)
"\\" --> \
The backslash before characters is only interpreted the above way when it is inside quotes.
If you want to find more info or other escape chars go here