I have just recently got back into learning bash. Currently working on a project of mine and when using sed I've run into an issue, I've tried looking around the web for help but haven't had any joy. I suspect as I may not be using the correct terminology so I can't find what I'm looking for. ANYHOW.
So in my script I'm trying to assign the output of date to a variable. Here's the line from my script.
origdate=$(date)
When I call it the output looks like this:
Wed Oct 5 19:40:45 BST 2016
Part of my script then generates a file and writes information to it, part of which I am trying to use sed to find lines and replace parts of it. This is the first I've been playing around with sed, I've used it successfully so far for my needs. However I'm getting stuck when I try this:
sed -i '/origdate=empty/c\'$origdate'' $sd/pingcheck-email-$job.txt
When I run the script and it gets to this line, this is the error I'm getting:
sed: can't read Oct: No such file or directory
sed: can't read 5: No such file or directory
sed: can't read 19:52:56: No such file or directory
sed: can't read BST: No such file or directory
sed: can't read 2016: No such file or directory
I suspect it's something to do with the spaces in the date (variable), my question is: how can I work around this? Can I get sed to 'ignore' the spaces? or should I just use cut to cut the field for the date, and set that to a variable and the same thing again to set the time to another variable?
Even if someone could kindly point me in the right direction that'd be great!
Thanks in advance!
double quote the variable
sed -i '/origdate=empty/c\'"$origdate"'' $sd/pingcheck-email-$job.txt
or alternatively, the whole script
sed -i "/origdate=empty/c\$origdate" $sd/pingcheck-email-$job.txt
The problem is not with sed but rather with how bash word splits on your date given your command.
Bash
In bash, word splitting is performed on the command line so that text is broken up into a list of arguments. To illustrate, I'm going to run a simple script that outputs the first argument only.
bash -c 'echo $1' ignored_0 foo bar
Think of bash -c 'echo $1' ignored_0 as the command (sed in your case) and foo bar as the arguments. In this case, foo bar is split into two arguments, foo and bar.
To pass foo bar in as the first parameter, you need to have the text in either single or double quotes. See the GNU manual on quoting.
bash -c 'echo $1' ignored_0 'foo bar'
bash -c 'echo $1' ignored_0 "foo bar"
Parameter expansion does not occur when the variable is inside a single quote.
var="foo bar"
bash -c 'echo $1' ignored_0 '$var'
bash -c 'echo $1' ignored_0 "$var"
NOTE: In the command `bash -c 'echo $1', I do not want $1 to expand before being passed as an argument to bash because that's part of the code I want to execute.
Parameter expansion occurs when variables are outside of quotes, but word splitting will apply after the parameter is expanded. From the bash man page in the Word Splitting section:
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
From the GNU bash manual on Word Splitting:
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
var="foo bar"
bash -c 'echo $1' ignored_0 $var
The last step in Shell Expansions in Quote Removal where unquoted quote characters are removed before being passed to commands. The following command shows that ''"" has no effect on the arguments passed.
bash -c 'echo $1' ignored_0 foo''""
Application
In your example, the trailing '' after $origdate is extraneous. The important part is that $origdate is not quoted so word splitting applies to the expanded variable.
When -e is not passed to the sed command, sed expects the expression to be in one argument, or word from bash. When you run your command, your expression is /origdate=empty/c\Wed and the rest of the date is considered to be files for the expression to be applied to.
The simple fix is to put double quotes around the string for which you want to prevent word splitting. I've modified the command so that anyone can run this example without having the files on their system.
In this example, the \ must be escaped so that it is not considered an escape character for $.
echo "origdate=empty" | sed "/origdate=empty/c\\$origdate"
You can also change the type of quotes you are using without affecting word splitting like so.
echo "origdate=empty" | sed '/origdate=empty/c\'"$origdate"
You need escape by double slash
\ / \%
Related
I am trying to find and replace a specific text content using the sed command and to run it via a shell script.
Below is the sample script that I am using:
fp=/asd/filename.txt
fd="sed -i -E 's ($2).* $2:$3 g' ${fp}"
eval $fd
and executing the same by passing the arguments:
./test.sh update asd asdfgh
But if the argument string contains $ , it breaks the commands and it is replacing with wrong values, like
./test.sh update asd $apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.
How can I make sure that the values inside the variables are not expanded because of the $?
Updated
sh file test.sh
set -xv
fp="/asd/filename.txt"
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
text file filename.txt
hello:world
Outputs
1)
./test.sh update hello WORLD
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:WORLD/g' /asd/filename.txt
2)
./test.sh update hello '$apr1$hosgaxyv$D0KXp5dCyZ2BUYCS9BmHu1'
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:'\''$'\''apr1'\''$'\''hosgaxyv'\''$'\''D0KXp5dCyZ2BUYCS9BmHu1/g' /asd/filename.txt
In both the case , its not replacing the content
You don't need eval here at all:
fp=/asd/filename.txt
sed -i -E "s/(${2//'$'/'\$'}).*/\1:${3//'$'/'\$'}/g" "$fp"
The whole sed command is in double quotes so variables can expand.
I've replaced the blank as the s separator with / (doesn't really matter in the example).
I've used \1 to reference the first capture group instead of repeating the variable in the substitution.
Most importantly, I've used ${2//'$'/'\$'} instead of $2 (and similar for $3). This escapes every $ sign as \$; this is required because of the double quoting, or the $ get eaten by the shell before sed gets to see them.
When you call your script, you must escape any $ in the input, or the shell tries to expand them as variable names:
./test.sh update asd '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.'
Put the command-line arguments that are filenames in single quotes:
./test.sh update 'asd' '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1'
must protect all the script arguments with quotes if having space and special shell char, and escape it if it's a dollar $, and -Ei instead of -iE even better drop it first for test, may add it later if being really sure
I admit i won't understant your regex so let's just get in the gist of solution, no need eval;
fp=/asd/filename.txt
sed -Ei "s/($2).*/$2:$3/g" $fp
./test.sh update asd '\$apr1\$HnIF6bOt\$9m3NzAwr.aG1Yp.t.bpIS1.'
This question already has answers here:
echo "#!" fails -- "event not found"
(5 answers)
Closed 7 years ago.
I am attempting to parse the output of a VNC server startup event and have run into a problem in parsing using sed in a command substitution. Specifically, the remote VNC server is started in a manner such as the following:
address1="user1#lxplus.cern.ch"
VNCServerResponse="$(ssh "${address1}" 'vncserver' 2>&1)"
The standard error output produced in this startup event is then to be parsed in order to extract the server and display information. At this point the content of the variable VNCServerResponse is something such as the following:
New 'lxplus0186.cern.ch:1 (user1)' desktop is lxplus0186.cern.ch:1
Starting applications specified in /afs/cern.ch/user/u/user1/.vnc/xstartup
Log file is /afs/cern.ch/user/u/user1/.vnc/lxplus0186.cern.ch:1.log
This output can be parsed in the following way in order to extract the server and display information:
echo "${VNCServerResponse}" | sed '/New.*desktop.*is/!d' \
| awk -F" desktop is " '{print $2}'
The result is something such as the following:
lxplus0186.cern.ch:1
What I want to do is use this parsing in a command substitution something like the following:
VNCServerAndDisplayNumber="$(echo "${VNCServerResponse}" \
| sed '/New.*desktop.*is/!d' | awk -F" desktop is " '{print $2}')"
On attempting to do this, I am presented with the following error:
bash: !d': event not found
I am not sure how to address this. It appears to be a problem in the way sed is being used in the command substitution. I would appreciate guidance.
Bash history expansion is a very odd corner in the bash command line parser, and you are clearly running into an unexpected history expansion, which is explained below. However, any sort of history expansion in a script is unexpected, because normally history expansion is not enabled in scripts; not even scripts run with the source (or .) builtin.
How history expansion is enabled (or disabled)
There are two shell options which control history expansion:
set -o history: Required for the history to be recorded.
set -H (or set -o histexpand): Additionally required for history expansion to be enabled.
Both of these options must be set for history expansion to be recognized. (I found the manual unclear on this interaction, but it's logical enough.)
According to the bash manual, these options are unset for non-interactive shells, so if you want to enable history expansion in a script (and I cannot imagine a reason you would want this), you would need to set both of them:
set -o history -o histexpand
The situation for scripts run with source is more complicated (and what I'm about to say only applies to bash v4, and since it's undocumented in might change in the future). [Note 3]
History recording (and consequently expansion) is turned off in source'd scripts, but through an internal flag which, as far as I know, is not made visible. It certainly does not appear in $SHELLOPTS. Since a sourced script runs in the current bash context, it shares the current execution environment, including shell options. So in the execution of a sourced script initiated from an interactive session, you'll see both history and histexpand in $SHELLOPTS, but no history expansion will take place. In order to enable it, you need to:
set -o history
which is not a no-op because it has the side-effect of resetting the internal flag which suppresses history recording. Setting the histexpand shell option does not have this side-effect.
In short, I'm not sure how you managed to enable history expansion in a script (if, indeed, the misbehaving command was in a script and not in an interactive shell), but you might want to consider not doing so, unless you have a really good reason.
How history expansion is parsed
The bash implementation of history expansion is designed to work with readline, so that it can be performed during command input. (By default this function is bound to Meta-^; generally Meta is ESC, but you can customize that as well.) However, it is also performed immediately after each line is input, before any bash parsing is performed.
By default, the history expansion character is !, and -- as mostly documented -- that will trigger history expansion except:
when it is followed by whitespace or =
if the shell option extglob is set, and it is followed by ( [Note 1]
if it appears in a single-quoted string
if it is preceded by a \ [Note 2 and see below]
if it is preceded by $ or ${ [Note 1]
if it is preceded by [ [Note 1]
(As of bash v4.3) if it is the last character in a double-quoted string.
The immediate issue here is the precise interpretation of the third case, an ! appearing inside of a single-quoted string. Normally, bash starts a new quoting context for a command substitution ($(...) or the deprecated backtick notation). For example:
$ s=SUBSTITUTED
$ # The interior single quotes are just characters
$ echo "'Echoing $s'"
'Echoing SUBSTITUTED'
$ # The interior single quotes are single quotes
$ echo "$(echo 'Echoing $s')"
Echoing $s
However, the history expansion scanner isn't that intelligent. It keeps track of quotes, but not of command substitution. So as far as it is concerned, both of the single quotes in the above example are double-quoted single quotes, which is to say ordinary characters. So history expansion occurs in both of them:
# A no-op to indicated history expansion
$ HIST() { :; }
# Single-quoted strings inhibit history expansion
$ HIST
$ echo '!!'
!!
# Double-quoted strings allow history expansion
$ HIST
$ echo "'!!'"
echo "'HIST'"
'HIST'
# ... and it applies also to interior command substitution.
$ HIST
$ echo "$(echo '!!')"
echo "$(echo 'HIST')"
HIST
So if you have a perfectly normal command like sed '/foo/!d' file, where you would expect the single-quotes to protect you from history-expansion, and you put it inside a double-quoted command substitution:
result="$(sed '/foo/!d' file)"
you suddenly find that the ! is a history expansion character. Worse, you can't fix this by backslash escaping the exclamation point, because although "\!" inhibits history expansion, it doesn't remove the backslash:
$ echo "\!"
\!
In this particular example -- and the one in the OP -- the double quotes are completely unnecessary, because the right-hand side of a variable assignment does not undergo either filename expansion nor word splitting. However, there are other contexts in which removing the double quotes would change the semantics:
# Undesired history expansion
printf "The answer is '%s'\n" "$(sed '/foo/!d' file)"
# Undesired word splitting
printf "The answer is '%s'\n" $(sed '/foo/!d' file)
In this case, the best solution is probably to put the sed argument in a variable
# Works
sed_prog='/foo/!d'
printf "The answer is '%s'\n" "$(sed "$sed_prog" file)"
(The quotes around $sed_prog were not necessary in this case but usually they would be, and they do no harm.)
Notes:
The inhibition of history expansion when the following character is some form of open parenthesis only works if there is a corresponding close parenthesis in the rest of the string. However, it doesn't have to really match the open parenthesis. For example:
# No matching close parenthesis
$ echo "!("
bash: !: event not found
# The matching close parenthesis has nothing to do with the open
$ echo "!(" ")"
!( )
# An actual extended glob: files whose names don't start with a
$ echo "!(a*)"
b
As indicated in the bash manual, a history-expansion character is treated as an ordinary character if immediately preceded by a backslash. This is literally true; it doesn't matter whether the backslash will later be considered an escape character or not:
$ echo \!
!
$ echo \\!
\!
$ echo \\\!
\!
\ also inhibits history expansion inside double quotes, but \! is not a valid escape sequence inside the double quoted string, so the backslash is not removed:
$ echo "\!"
\!
$ echo "\\!"
\!
$ echo "\\\!"
\\!
I'm referring to the source code for bash v4.2 as I write this, so any undocumented behaviour may be completely different as of v4.3.
The problem is that within double quotes, bash is trying to expand !d before passing it to the subshell. You can get around this problem by removing the double quotes but I would also propose a simplification to your script:
VNCServerAndDisplayNumber=$(echo "$VNCServerResponse" | awk '/desktop/ {print $NF}')
This simply prints the last field on the line containing the word "desktop".
On a newer bash, you can use a herestring rather than piping an echo:
VNCServerAndDisplayNumber=$(awk '/desktop/ {print $NF}' <<<"$VNCServerResponse")
Don't wrap the $(...) command substitution in double quotes. You are asking the shell to perform evaluation on the contents of the quotes and are hitting the history substitution expansion feature. Drop the quotes and you stop telling the shell to do that and you won't hit that problem.
And yes, dropping those quotes is safe on that assignment line even if the output may contain spaces or newlines or whatever. Assignments of that sort are not going to split on those the way command substitution or variable evaluation will on a normal shell execution line.
Alternatively, disable history expansion in your shell/script before you run that. (It should be off when running a script by default I believe anyway.)
This only happens when history expansion is enabled, which it normally isn't and definitely shouldn't be for scripts.
Rather than trying to work around it, figure out why history expansion is enabled and what to do so it isn't.
If you're executing your script with . foo or source foo, use ./foo instead.
If you're writing this as a function in .bashrc or similar, consider making it a separate script.
If your script (or BASH_ENV) explicitly does set -H, don't.
Quote it with '' or \ or disable history expansion with set +H or shopt -u -o histexpand. See History Expansion.
This is a simple question but i am unable to find it in tutorials. Could anybody please explain what this statement does when executed in a bash shell within a folder containing .sh scripts. I know -i does in place editing, i understand that it will run sed on all scripts within the current directory. And i know that it does some sort of substitution. But what does this \(.*\) mean?
sed -i 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
Thanks in advance.
You have an expression like:
sed -i 's/XXX=\(YYY\)/XXX=ZZZ/' file
This looks for a string XXX= in a file and captures what goes after. Then, it replaces this captured content with ZZZ. Since there is a captured group, it is accessed with \1. Finally, using the -i flag in sed makes the edition to be in-place.
For the replacement, it uses the following syntax described in Shell parameter expansion:
${parameter:-word}
If parameter is unset or null, the expansion of word is substituted.
Otherwise, the value of parameter is substituted.
Example:
$ d=5
$ echo ${d-3}
5
$ echo ${a-3}
3
So with ${MY_BASE_DIR-SOMETHING-\1} you are saying: print $MY_BAS_DIR. And if this variable is unset or null, print what is stored in \1.
All together, this is resetting MY_BASE_DIR to the value in the variable $MY_BASE_DIR unless this is not set; in such case, the value remains the same.
Note though that the variable won't be expanded unless you use double quotes.
Test:
$ d=5
$ cat a
d=23
blabla
$ sed "s/d=\(.*\)/d=${d-\1}/" a # double quotes -> value is replaced
d=5
blabla
$ sed 's/d=\(.*\)/d=${d-\1}/' a # single quotes -> variable is not expanded
d=${d-23}
blabla
Andd see how the value remains the same if $d is not set:
$ unset d
$ sed "s/d=\(.*\)/d=${d-\1}/" a
d=23
The scripts contain lines like this:
MY_BASE_DIR=/usr/local
The sed expression changes them to:
MY_BASE_DIR=${MY_BASE_DIR-/usr/local}
The effect is that /usr/local is not used as a fixed value, but only as the default value. You can override it by setting the environment variable MY_BASE_DIR.
For future reference, I would take a look at the ExplainShell website:
http://explainshell
that will give you a breakdown of the command structure etc. In this instance, let step through the details...Let's start with a simple example, let's assume that we were going to make the simple change - commenting out all lines by adding a "#" before each line. We can do this for all *.sh files in a directory with the ".sh" extension in the current directory:
sed 's/^/\#/' *.sh
i.e. Substitute beginning of line ^, with a # ...
Caveat: You did not specify the OS you are using. You may get different results with different versions of sed and OS...
ok, now we can drill into the substitution in the script. An example is probably easier to explain:
File: t.sh
MY_BASE_DIR="/important data/data/bin"
the command 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
will search for "MY_BASE_DIR" in each .sh file in the directory.
When it encounters the string "MY_BASE_DIR=.*", in the file, it expands it to be MY_BASE_DIR="/important data/data/bin", this is now replaced on the right side of the expression /MY_BASE_DIR=${MY_BASE_DIR-\1}/ which becomes
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
essentially what happens is that the substitute operation takes
MY_BASE_DIR="/important data/data/bin"
and inserts
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
now if we run the script with the variable MY_BASE_DIR set
export MY_BASE_DIR="/new/import/dir"
the scripts modified by the sed script referenced will now substitute /important data/data/bin with /new/import/dir...
I want to issue this command from the bash script
sed -e $beginning,$s/pattern/$variable/ file
but any possible combination of quotes gives me an error, only one that works:
sed -e "$beginning,$"'s/pattern/$variable/' file
also not good, because it do not dereferences the variable.
Does my approach can be implemented with sed?
Feel free to switch the quotes up. The shell can keep things straight.
sed -e "$beginning"',$s/pattern/'"$variable"'/' file
You can try this:
$ sed -e "$beginning,$ s/pattern/$variable/" file
Example
file.txt:
one
two
three
Try:
$ beginning=1
$ variable=ONE
$ sed -e "$beginning,$ s/one/$variable/" file.txt
Output:
ONE
two
three
There are two types of quotes:
Single quotes preserve their contents (> is the prompt):
> var=blah
> echo '$var'
$var
Double quotes allow for parameter expansion:
> var=blah
> echo "$var"
blah
And two types of $ sign:
One to tell the shell that what follows is the name of a parameter to be expanded
One that stands for "last line" in sed.
You have to combine these so
The shell doesn't think sed's $ has anything to do with a parameter
The shell parameters still get expanded (can't be within single quotes)
The whole sed command is quoted.
One possibility would be
sed "$beginning,\$s/pattern/$variable/" file
The whole command is in double quotes, i.e., parameters get expanded ($beginning and $variable). To make sure the shell doesn't try to expand $s, which doesn't exist, the "end of line" $ is escaped so the shell doesn't try anything funny.
Other options are
Double quoting everything but adding a space between $ and s (see Ren's answer)
Mixing quoting types as needed (see Ignacio's answer)
Methods that don't work
sed '$beginning,$s/pattern/$variable/' file
Everything in single quotes: the shell parameters are not expanded (doesn't follow rule 2 above). $beginning is not a valid address, and pattern would be literally replaced by $variable.
sed "$beginning,$s/pattern/$variable/" file
Everything in double qoutes: the parameters are expanded, including $s, which isn't supposed to (doesn't follow rule 1 above).
the following form worked for me from within script
sed $beg,$ -e s/pattern/$variable/ file
the same form will also work if executed from the shell
This question already has answers here:
echo "#!" fails -- "event not found"
(5 answers)
Closed 7 years ago.
I am attempting to parse the output of a VNC server startup event and have run into a problem in parsing using sed in a command substitution. Specifically, the remote VNC server is started in a manner such as the following:
address1="user1#lxplus.cern.ch"
VNCServerResponse="$(ssh "${address1}" 'vncserver' 2>&1)"
The standard error output produced in this startup event is then to be parsed in order to extract the server and display information. At this point the content of the variable VNCServerResponse is something such as the following:
New 'lxplus0186.cern.ch:1 (user1)' desktop is lxplus0186.cern.ch:1
Starting applications specified in /afs/cern.ch/user/u/user1/.vnc/xstartup
Log file is /afs/cern.ch/user/u/user1/.vnc/lxplus0186.cern.ch:1.log
This output can be parsed in the following way in order to extract the server and display information:
echo "${VNCServerResponse}" | sed '/New.*desktop.*is/!d' \
| awk -F" desktop is " '{print $2}'
The result is something such as the following:
lxplus0186.cern.ch:1
What I want to do is use this parsing in a command substitution something like the following:
VNCServerAndDisplayNumber="$(echo "${VNCServerResponse}" \
| sed '/New.*desktop.*is/!d' | awk -F" desktop is " '{print $2}')"
On attempting to do this, I am presented with the following error:
bash: !d': event not found
I am not sure how to address this. It appears to be a problem in the way sed is being used in the command substitution. I would appreciate guidance.
Bash history expansion is a very odd corner in the bash command line parser, and you are clearly running into an unexpected history expansion, which is explained below. However, any sort of history expansion in a script is unexpected, because normally history expansion is not enabled in scripts; not even scripts run with the source (or .) builtin.
How history expansion is enabled (or disabled)
There are two shell options which control history expansion:
set -o history: Required for the history to be recorded.
set -H (or set -o histexpand): Additionally required for history expansion to be enabled.
Both of these options must be set for history expansion to be recognized. (I found the manual unclear on this interaction, but it's logical enough.)
According to the bash manual, these options are unset for non-interactive shells, so if you want to enable history expansion in a script (and I cannot imagine a reason you would want this), you would need to set both of them:
set -o history -o histexpand
The situation for scripts run with source is more complicated (and what I'm about to say only applies to bash v4, and since it's undocumented in might change in the future). [Note 3]
History recording (and consequently expansion) is turned off in source'd scripts, but through an internal flag which, as far as I know, is not made visible. It certainly does not appear in $SHELLOPTS. Since a sourced script runs in the current bash context, it shares the current execution environment, including shell options. So in the execution of a sourced script initiated from an interactive session, you'll see both history and histexpand in $SHELLOPTS, but no history expansion will take place. In order to enable it, you need to:
set -o history
which is not a no-op because it has the side-effect of resetting the internal flag which suppresses history recording. Setting the histexpand shell option does not have this side-effect.
In short, I'm not sure how you managed to enable history expansion in a script (if, indeed, the misbehaving command was in a script and not in an interactive shell), but you might want to consider not doing so, unless you have a really good reason.
How history expansion is parsed
The bash implementation of history expansion is designed to work with readline, so that it can be performed during command input. (By default this function is bound to Meta-^; generally Meta is ESC, but you can customize that as well.) However, it is also performed immediately after each line is input, before any bash parsing is performed.
By default, the history expansion character is !, and -- as mostly documented -- that will trigger history expansion except:
when it is followed by whitespace or =
if the shell option extglob is set, and it is followed by ( [Note 1]
if it appears in a single-quoted string
if it is preceded by a \ [Note 2 and see below]
if it is preceded by $ or ${ [Note 1]
if it is preceded by [ [Note 1]
(As of bash v4.3) if it is the last character in a double-quoted string.
The immediate issue here is the precise interpretation of the third case, an ! appearing inside of a single-quoted string. Normally, bash starts a new quoting context for a command substitution ($(...) or the deprecated backtick notation). For example:
$ s=SUBSTITUTED
$ # The interior single quotes are just characters
$ echo "'Echoing $s'"
'Echoing SUBSTITUTED'
$ # The interior single quotes are single quotes
$ echo "$(echo 'Echoing $s')"
Echoing $s
However, the history expansion scanner isn't that intelligent. It keeps track of quotes, but not of command substitution. So as far as it is concerned, both of the single quotes in the above example are double-quoted single quotes, which is to say ordinary characters. So history expansion occurs in both of them:
# A no-op to indicated history expansion
$ HIST() { :; }
# Single-quoted strings inhibit history expansion
$ HIST
$ echo '!!'
!!
# Double-quoted strings allow history expansion
$ HIST
$ echo "'!!'"
echo "'HIST'"
'HIST'
# ... and it applies also to interior command substitution.
$ HIST
$ echo "$(echo '!!')"
echo "$(echo 'HIST')"
HIST
So if you have a perfectly normal command like sed '/foo/!d' file, where you would expect the single-quotes to protect you from history-expansion, and you put it inside a double-quoted command substitution:
result="$(sed '/foo/!d' file)"
you suddenly find that the ! is a history expansion character. Worse, you can't fix this by backslash escaping the exclamation point, because although "\!" inhibits history expansion, it doesn't remove the backslash:
$ echo "\!"
\!
In this particular example -- and the one in the OP -- the double quotes are completely unnecessary, because the right-hand side of a variable assignment does not undergo either filename expansion nor word splitting. However, there are other contexts in which removing the double quotes would change the semantics:
# Undesired history expansion
printf "The answer is '%s'\n" "$(sed '/foo/!d' file)"
# Undesired word splitting
printf "The answer is '%s'\n" $(sed '/foo/!d' file)
In this case, the best solution is probably to put the sed argument in a variable
# Works
sed_prog='/foo/!d'
printf "The answer is '%s'\n" "$(sed "$sed_prog" file)"
(The quotes around $sed_prog were not necessary in this case but usually they would be, and they do no harm.)
Notes:
The inhibition of history expansion when the following character is some form of open parenthesis only works if there is a corresponding close parenthesis in the rest of the string. However, it doesn't have to really match the open parenthesis. For example:
# No matching close parenthesis
$ echo "!("
bash: !: event not found
# The matching close parenthesis has nothing to do with the open
$ echo "!(" ")"
!( )
# An actual extended glob: files whose names don't start with a
$ echo "!(a*)"
b
As indicated in the bash manual, a history-expansion character is treated as an ordinary character if immediately preceded by a backslash. This is literally true; it doesn't matter whether the backslash will later be considered an escape character or not:
$ echo \!
!
$ echo \\!
\!
$ echo \\\!
\!
\ also inhibits history expansion inside double quotes, but \! is not a valid escape sequence inside the double quoted string, so the backslash is not removed:
$ echo "\!"
\!
$ echo "\\!"
\!
$ echo "\\\!"
\\!
I'm referring to the source code for bash v4.2 as I write this, so any undocumented behaviour may be completely different as of v4.3.
The problem is that within double quotes, bash is trying to expand !d before passing it to the subshell. You can get around this problem by removing the double quotes but I would also propose a simplification to your script:
VNCServerAndDisplayNumber=$(echo "$VNCServerResponse" | awk '/desktop/ {print $NF}')
This simply prints the last field on the line containing the word "desktop".
On a newer bash, you can use a herestring rather than piping an echo:
VNCServerAndDisplayNumber=$(awk '/desktop/ {print $NF}' <<<"$VNCServerResponse")
Don't wrap the $(...) command substitution in double quotes. You are asking the shell to perform evaluation on the contents of the quotes and are hitting the history substitution expansion feature. Drop the quotes and you stop telling the shell to do that and you won't hit that problem.
And yes, dropping those quotes is safe on that assignment line even if the output may contain spaces or newlines or whatever. Assignments of that sort are not going to split on those the way command substitution or variable evaluation will on a normal shell execution line.
Alternatively, disable history expansion in your shell/script before you run that. (It should be off when running a script by default I believe anyway.)
This only happens when history expansion is enabled, which it normally isn't and definitely shouldn't be for scripts.
Rather than trying to work around it, figure out why history expansion is enabled and what to do so it isn't.
If you're executing your script with . foo or source foo, use ./foo instead.
If you're writing this as a function in .bashrc or similar, consider making it a separate script.
If your script (or BASH_ENV) explicitly does set -H, don't.
Quote it with '' or \ or disable history expansion with set +H or shopt -u -o histexpand. See History Expansion.