I have a job in Jenkins (under Linux) with build parameter of type "Text". I use the parameter to form contents of a file which is used in build process, with bash "shell execute" step like echo "$TEXTPARAM" > file.
It works quite well if there is general text. But when characters like "$" appears - it behaves strangely.
E. g. text
Some $one and $$two and $$$more bucks and $ bucks $$ that $$$ stand alone and$ after$$ words$$$
is transformed into
Some $one and $two and $-sl bucks and $ bucks $ that $$ stand alone and$ after$ words$$
though I want the text to appear in file just as it appears in input textbox.
Is it a bug in jenkins (so I should post an issue to their tracker) or am I doing something wrong?
UPDATE
I suppose that is due to variable substitution done by Jenkins. I.e all $VARNAMEs are substituted by VARNAME values prior to any "shell execute" steps are executed. And this substitution cannot be turned off as for now.
According to a comment in this ticket https://issues.jenkins-ci.org/browse/JENKINS-16143
This appears to not be a bug. Compare the parameter values $JENKINS_URL and $$JENKINS_URL.
Jenkins internally resolves placeholders in variables,
and dollars are used for that. $$ is an escaped $.
I am observing the same behavior for string and text fields on Jenkins ver. 1.562
Expansion with jenkins-cli.jar
It's true, Jenkins expands build variable names like JENKINS_URL or BUILD_NUMBER when they are prefixed by $. However, there are additional transformations—rather unexpected ones—if you use jenkins-cli.jar.
carriage return becomes \r (backslash + "r")
new line becomes \n (backslash plus + "n")
tab becomes \t (backslash plus "t")
Here is the corresponding part of the source code of jenkins-cli.jar.
I do not know any way of escaping or quoting to keep a white space character that is part of the value of a parameter for a Jenkins job when using jenkins-cli.jar
Expansion when using "raw" ssh (without jenkins-cli.jar)
The Jenkins master handles white space, backslashes and quotes on the command line somewhat like a shell:
a remains a
'a' becomes a
"a b" becomes a b
a b is an error, because the command line parser of Jenkins will see b
a" " becomes a ("a" plus space)
My idea was to re-implement the code that does the quoting in jenkins-cli.jar (minus the bugs when handling tab characters and the like). So here's my recipe:
For each argument, escape each backslash and
each single quote with a backslash.
Then surround it with single quotes.
Example: Instead of a'"b, send 'a\'"b'.
This has proven to be protect white space and quotes. And instead of using single quotes, you can use double quotes instead.
Pipeline for Testing
This is how I tested: I created the pipeline test-quoting with the string parameter "PARAM" and the following script:
import groovy.json.JsonOutput
node {
println JsonOutput.toJson(env.PARAM)
}
Then I started the pipeline (in bash), adding an additional layer of quoting that the local shell will remove:
# for testing 'a b'
$ ssh -x -p 50022 <jenkins-server> -l <user-name> build -s -v test-quoting -p PARAM="'a b'"
# for testing the behaviour of the cli with tab character
$ java -jar jenkins-cli.jar -ssh -s <jenkins-url> -user <user-name> build -s -v test-quoting -p PARAM="$(printf '\t')"
If you are unsure about what your local shell really passes to ssh (or any other command), prefix the command with strace -e execve.
This has nothing to do with Jenkins.
Write a bash script with echo "some $$$more" > file and execute that on a linux command prompt, and you will get the same gibberish.
Special characters, like $ must be escaped, since this is a linux environment and $ means a variable. There are several ways to do it.
Option 1.
Use per-character escape, i.e. for every $ that you want to appear literally, use \$ instead. So it becomes:
echo "some \$\$\$more" > file
Option 2.
Use strong-quoting, i.e. single quotes '. Nothing within single quotes has any special meaning, except for a second single quote to close the string:
echo 'some $$$more' > file
Of course with this method, you have to make sure your $TEXTPARAM string does not have any single quotes of it's own.
In either case, you will have to sanitize your input. Before you output it to file, you will need to parse the content of $TEXTPARAM and either replace all $ with \$ and use Option 1. Or parse your $TEXTPARAM and remove all single quotes ' before outputting that to file using Option 2.
Edit
In your case, I think you just want:
echo $TEXTPARAM > file without any extra quotes
root# ~ $ cat test.sh
#!/bin/bash
TEXTPARAM='Some $one and $$two and $$$more bucks and $ bucks $$ that $$$ stand alone and$ after$$ words$$$'
echo $TEXTPARAM
echo $TEXTPARAM > file
cat file
root# ~ $ ./test.sh
Some $one and $$two and $$$more bucks and $ bucks $$ that $$$ stand alone and$ after$$ words$$$
Some $one and $$two and $$$more bucks and $ bucks $$ that $$$ stand alone and$ after$$ words$$$
Related
This question already has answers here:
echo "#!" fails -- "event not found"
(5 answers)
Closed 7 years ago.
I am attempting to parse the output of a VNC server startup event and have run into a problem in parsing using sed in a command substitution. Specifically, the remote VNC server is started in a manner such as the following:
address1="user1#lxplus.cern.ch"
VNCServerResponse="$(ssh "${address1}" 'vncserver' 2>&1)"
The standard error output produced in this startup event is then to be parsed in order to extract the server and display information. At this point the content of the variable VNCServerResponse is something such as the following:
New 'lxplus0186.cern.ch:1 (user1)' desktop is lxplus0186.cern.ch:1
Starting applications specified in /afs/cern.ch/user/u/user1/.vnc/xstartup
Log file is /afs/cern.ch/user/u/user1/.vnc/lxplus0186.cern.ch:1.log
This output can be parsed in the following way in order to extract the server and display information:
echo "${VNCServerResponse}" | sed '/New.*desktop.*is/!d' \
| awk -F" desktop is " '{print $2}'
The result is something such as the following:
lxplus0186.cern.ch:1
What I want to do is use this parsing in a command substitution something like the following:
VNCServerAndDisplayNumber="$(echo "${VNCServerResponse}" \
| sed '/New.*desktop.*is/!d' | awk -F" desktop is " '{print $2}')"
On attempting to do this, I am presented with the following error:
bash: !d': event not found
I am not sure how to address this. It appears to be a problem in the way sed is being used in the command substitution. I would appreciate guidance.
Bash history expansion is a very odd corner in the bash command line parser, and you are clearly running into an unexpected history expansion, which is explained below. However, any sort of history expansion in a script is unexpected, because normally history expansion is not enabled in scripts; not even scripts run with the source (or .) builtin.
How history expansion is enabled (or disabled)
There are two shell options which control history expansion:
set -o history: Required for the history to be recorded.
set -H (or set -o histexpand): Additionally required for history expansion to be enabled.
Both of these options must be set for history expansion to be recognized. (I found the manual unclear on this interaction, but it's logical enough.)
According to the bash manual, these options are unset for non-interactive shells, so if you want to enable history expansion in a script (and I cannot imagine a reason you would want this), you would need to set both of them:
set -o history -o histexpand
The situation for scripts run with source is more complicated (and what I'm about to say only applies to bash v4, and since it's undocumented in might change in the future). [Note 3]
History recording (and consequently expansion) is turned off in source'd scripts, but through an internal flag which, as far as I know, is not made visible. It certainly does not appear in $SHELLOPTS. Since a sourced script runs in the current bash context, it shares the current execution environment, including shell options. So in the execution of a sourced script initiated from an interactive session, you'll see both history and histexpand in $SHELLOPTS, but no history expansion will take place. In order to enable it, you need to:
set -o history
which is not a no-op because it has the side-effect of resetting the internal flag which suppresses history recording. Setting the histexpand shell option does not have this side-effect.
In short, I'm not sure how you managed to enable history expansion in a script (if, indeed, the misbehaving command was in a script and not in an interactive shell), but you might want to consider not doing so, unless you have a really good reason.
How history expansion is parsed
The bash implementation of history expansion is designed to work with readline, so that it can be performed during command input. (By default this function is bound to Meta-^; generally Meta is ESC, but you can customize that as well.) However, it is also performed immediately after each line is input, before any bash parsing is performed.
By default, the history expansion character is !, and -- as mostly documented -- that will trigger history expansion except:
when it is followed by whitespace or =
if the shell option extglob is set, and it is followed by ( [Note 1]
if it appears in a single-quoted string
if it is preceded by a \ [Note 2 and see below]
if it is preceded by $ or ${ [Note 1]
if it is preceded by [ [Note 1]
(As of bash v4.3) if it is the last character in a double-quoted string.
The immediate issue here is the precise interpretation of the third case, an ! appearing inside of a single-quoted string. Normally, bash starts a new quoting context for a command substitution ($(...) or the deprecated backtick notation). For example:
$ s=SUBSTITUTED
$ # The interior single quotes are just characters
$ echo "'Echoing $s'"
'Echoing SUBSTITUTED'
$ # The interior single quotes are single quotes
$ echo "$(echo 'Echoing $s')"
Echoing $s
However, the history expansion scanner isn't that intelligent. It keeps track of quotes, but not of command substitution. So as far as it is concerned, both of the single quotes in the above example are double-quoted single quotes, which is to say ordinary characters. So history expansion occurs in both of them:
# A no-op to indicated history expansion
$ HIST() { :; }
# Single-quoted strings inhibit history expansion
$ HIST
$ echo '!!'
!!
# Double-quoted strings allow history expansion
$ HIST
$ echo "'!!'"
echo "'HIST'"
'HIST'
# ... and it applies also to interior command substitution.
$ HIST
$ echo "$(echo '!!')"
echo "$(echo 'HIST')"
HIST
So if you have a perfectly normal command like sed '/foo/!d' file, where you would expect the single-quotes to protect you from history-expansion, and you put it inside a double-quoted command substitution:
result="$(sed '/foo/!d' file)"
you suddenly find that the ! is a history expansion character. Worse, you can't fix this by backslash escaping the exclamation point, because although "\!" inhibits history expansion, it doesn't remove the backslash:
$ echo "\!"
\!
In this particular example -- and the one in the OP -- the double quotes are completely unnecessary, because the right-hand side of a variable assignment does not undergo either filename expansion nor word splitting. However, there are other contexts in which removing the double quotes would change the semantics:
# Undesired history expansion
printf "The answer is '%s'\n" "$(sed '/foo/!d' file)"
# Undesired word splitting
printf "The answer is '%s'\n" $(sed '/foo/!d' file)
In this case, the best solution is probably to put the sed argument in a variable
# Works
sed_prog='/foo/!d'
printf "The answer is '%s'\n" "$(sed "$sed_prog" file)"
(The quotes around $sed_prog were not necessary in this case but usually they would be, and they do no harm.)
Notes:
The inhibition of history expansion when the following character is some form of open parenthesis only works if there is a corresponding close parenthesis in the rest of the string. However, it doesn't have to really match the open parenthesis. For example:
# No matching close parenthesis
$ echo "!("
bash: !: event not found
# The matching close parenthesis has nothing to do with the open
$ echo "!(" ")"
!( )
# An actual extended glob: files whose names don't start with a
$ echo "!(a*)"
b
As indicated in the bash manual, a history-expansion character is treated as an ordinary character if immediately preceded by a backslash. This is literally true; it doesn't matter whether the backslash will later be considered an escape character or not:
$ echo \!
!
$ echo \\!
\!
$ echo \\\!
\!
\ also inhibits history expansion inside double quotes, but \! is not a valid escape sequence inside the double quoted string, so the backslash is not removed:
$ echo "\!"
\!
$ echo "\\!"
\!
$ echo "\\\!"
\\!
I'm referring to the source code for bash v4.2 as I write this, so any undocumented behaviour may be completely different as of v4.3.
The problem is that within double quotes, bash is trying to expand !d before passing it to the subshell. You can get around this problem by removing the double quotes but I would also propose a simplification to your script:
VNCServerAndDisplayNumber=$(echo "$VNCServerResponse" | awk '/desktop/ {print $NF}')
This simply prints the last field on the line containing the word "desktop".
On a newer bash, you can use a herestring rather than piping an echo:
VNCServerAndDisplayNumber=$(awk '/desktop/ {print $NF}' <<<"$VNCServerResponse")
Don't wrap the $(...) command substitution in double quotes. You are asking the shell to perform evaluation on the contents of the quotes and are hitting the history substitution expansion feature. Drop the quotes and you stop telling the shell to do that and you won't hit that problem.
And yes, dropping those quotes is safe on that assignment line even if the output may contain spaces or newlines or whatever. Assignments of that sort are not going to split on those the way command substitution or variable evaluation will on a normal shell execution line.
Alternatively, disable history expansion in your shell/script before you run that. (It should be off when running a script by default I believe anyway.)
This only happens when history expansion is enabled, which it normally isn't and definitely shouldn't be for scripts.
Rather than trying to work around it, figure out why history expansion is enabled and what to do so it isn't.
If you're executing your script with . foo or source foo, use ./foo instead.
If you're writing this as a function in .bashrc or similar, consider making it a separate script.
If your script (or BASH_ENV) explicitly does set -H, don't.
Quote it with '' or \ or disable history expansion with set +H or shopt -u -o histexpand. See History Expansion.
The POSIX shell standard at
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04
says in Section 2.6:
command substitution (...) shall be performed
(...)
Quote removal (...) shall always be performed last.
It appears to me that quote removal is not performed after command substitution:
$ echo "#"
#
$ echo '"'
"
as expected, but
$ echo $(echo '"')#"
>
What am I not understanding?
Added after reading answer/comments:
From what everybody is saying, the consideration of quotes happens at the very beginning of parsing, for example, to decide if a command is even "acceptable". Then why does the standard bother to emphasise, that the quote removal is performed late in the process??
"then the outer command becomes echo "#" and is balanced"
That is not 'balanced' because the first double-quote does not count. Quotes are only meaningful as quotes if they appear unencumbered on the command line.
To verify, let's look at this:
$ echo $(echo '"')#
"#
That is balanced because the shell does considers that " to be just another character.
By contrast, this is unbalanced because it has one and only one shell-active ":
$ echo $(echo '"')#"
>
Similar example 1
Here we show the same thing but using parameter expansion instead of command substitution:
$ q='"'; echo $q
"
Once the shell has substituted " for $q, one might think that there was an unbalanced double-quote. But, that double-quote was the results of parameter expansion and is therefore not a shell-active quote.
Similar example 2
Let's consider a directory containing file:
$ ls
file
$ ls "file"
file
As you can see above, quote removal is perfomed before ls is run.
But, consider this command:
$ echo ls $(echo '"file"')
ls "file"
As you can see ls $(echo '"file"') expands to ls "file" which is the command which ran successfully above. Now, let's try running that:
$ ls $(echo '"file"')
ls: cannot access '"file"': No such file or directory
As you can see, the shell does not treat the double-quotes that remain after command substitution. This is because those quotes are not considered to be shell-active. As a consequence, they are treated as normal characters and passed on to ls which complains that the file whose name begins and ends with " does not exist in the directory.
The same is happening here:
$ cmd='ls "file"'
$ $cmd
ls: cannot access '"file"': No such file or directory
POSIX standard
From the POSIX standard:
Enclosing characters in single-quotes ( ' ' ) shall preserve the
literal value of each character within the single-quotes
In other words, once the double-quote appears inside single quotes, it has no special powers: it is just another character.
The standard also mentions escaping and double-quotes as methods of preserving "the literal value" of a character.
Practical consequences
People new to shell often want to store a command in a variable as in the cmd='ls "file"' example above. But, because quotes and other shell-active characters cease to be shell active once they are stored in a variable, the complex cases always fail. This leads to a classic essay:
"I'm trying to put a command in a variable, but the complex cases always fail!"
This question already has answers here:
echo "#!" fails -- "event not found"
(5 answers)
Closed 7 years ago.
I am attempting to parse the output of a VNC server startup event and have run into a problem in parsing using sed in a command substitution. Specifically, the remote VNC server is started in a manner such as the following:
address1="user1#lxplus.cern.ch"
VNCServerResponse="$(ssh "${address1}" 'vncserver' 2>&1)"
The standard error output produced in this startup event is then to be parsed in order to extract the server and display information. At this point the content of the variable VNCServerResponse is something such as the following:
New 'lxplus0186.cern.ch:1 (user1)' desktop is lxplus0186.cern.ch:1
Starting applications specified in /afs/cern.ch/user/u/user1/.vnc/xstartup
Log file is /afs/cern.ch/user/u/user1/.vnc/lxplus0186.cern.ch:1.log
This output can be parsed in the following way in order to extract the server and display information:
echo "${VNCServerResponse}" | sed '/New.*desktop.*is/!d' \
| awk -F" desktop is " '{print $2}'
The result is something such as the following:
lxplus0186.cern.ch:1
What I want to do is use this parsing in a command substitution something like the following:
VNCServerAndDisplayNumber="$(echo "${VNCServerResponse}" \
| sed '/New.*desktop.*is/!d' | awk -F" desktop is " '{print $2}')"
On attempting to do this, I am presented with the following error:
bash: !d': event not found
I am not sure how to address this. It appears to be a problem in the way sed is being used in the command substitution. I would appreciate guidance.
Bash history expansion is a very odd corner in the bash command line parser, and you are clearly running into an unexpected history expansion, which is explained below. However, any sort of history expansion in a script is unexpected, because normally history expansion is not enabled in scripts; not even scripts run with the source (or .) builtin.
How history expansion is enabled (or disabled)
There are two shell options which control history expansion:
set -o history: Required for the history to be recorded.
set -H (or set -o histexpand): Additionally required for history expansion to be enabled.
Both of these options must be set for history expansion to be recognized. (I found the manual unclear on this interaction, but it's logical enough.)
According to the bash manual, these options are unset for non-interactive shells, so if you want to enable history expansion in a script (and I cannot imagine a reason you would want this), you would need to set both of them:
set -o history -o histexpand
The situation for scripts run with source is more complicated (and what I'm about to say only applies to bash v4, and since it's undocumented in might change in the future). [Note 3]
History recording (and consequently expansion) is turned off in source'd scripts, but through an internal flag which, as far as I know, is not made visible. It certainly does not appear in $SHELLOPTS. Since a sourced script runs in the current bash context, it shares the current execution environment, including shell options. So in the execution of a sourced script initiated from an interactive session, you'll see both history and histexpand in $SHELLOPTS, but no history expansion will take place. In order to enable it, you need to:
set -o history
which is not a no-op because it has the side-effect of resetting the internal flag which suppresses history recording. Setting the histexpand shell option does not have this side-effect.
In short, I'm not sure how you managed to enable history expansion in a script (if, indeed, the misbehaving command was in a script and not in an interactive shell), but you might want to consider not doing so, unless you have a really good reason.
How history expansion is parsed
The bash implementation of history expansion is designed to work with readline, so that it can be performed during command input. (By default this function is bound to Meta-^; generally Meta is ESC, but you can customize that as well.) However, it is also performed immediately after each line is input, before any bash parsing is performed.
By default, the history expansion character is !, and -- as mostly documented -- that will trigger history expansion except:
when it is followed by whitespace or =
if the shell option extglob is set, and it is followed by ( [Note 1]
if it appears in a single-quoted string
if it is preceded by a \ [Note 2 and see below]
if it is preceded by $ or ${ [Note 1]
if it is preceded by [ [Note 1]
(As of bash v4.3) if it is the last character in a double-quoted string.
The immediate issue here is the precise interpretation of the third case, an ! appearing inside of a single-quoted string. Normally, bash starts a new quoting context for a command substitution ($(...) or the deprecated backtick notation). For example:
$ s=SUBSTITUTED
$ # The interior single quotes are just characters
$ echo "'Echoing $s'"
'Echoing SUBSTITUTED'
$ # The interior single quotes are single quotes
$ echo "$(echo 'Echoing $s')"
Echoing $s
However, the history expansion scanner isn't that intelligent. It keeps track of quotes, but not of command substitution. So as far as it is concerned, both of the single quotes in the above example are double-quoted single quotes, which is to say ordinary characters. So history expansion occurs in both of them:
# A no-op to indicated history expansion
$ HIST() { :; }
# Single-quoted strings inhibit history expansion
$ HIST
$ echo '!!'
!!
# Double-quoted strings allow history expansion
$ HIST
$ echo "'!!'"
echo "'HIST'"
'HIST'
# ... and it applies also to interior command substitution.
$ HIST
$ echo "$(echo '!!')"
echo "$(echo 'HIST')"
HIST
So if you have a perfectly normal command like sed '/foo/!d' file, where you would expect the single-quotes to protect you from history-expansion, and you put it inside a double-quoted command substitution:
result="$(sed '/foo/!d' file)"
you suddenly find that the ! is a history expansion character. Worse, you can't fix this by backslash escaping the exclamation point, because although "\!" inhibits history expansion, it doesn't remove the backslash:
$ echo "\!"
\!
In this particular example -- and the one in the OP -- the double quotes are completely unnecessary, because the right-hand side of a variable assignment does not undergo either filename expansion nor word splitting. However, there are other contexts in which removing the double quotes would change the semantics:
# Undesired history expansion
printf "The answer is '%s'\n" "$(sed '/foo/!d' file)"
# Undesired word splitting
printf "The answer is '%s'\n" $(sed '/foo/!d' file)
In this case, the best solution is probably to put the sed argument in a variable
# Works
sed_prog='/foo/!d'
printf "The answer is '%s'\n" "$(sed "$sed_prog" file)"
(The quotes around $sed_prog were not necessary in this case but usually they would be, and they do no harm.)
Notes:
The inhibition of history expansion when the following character is some form of open parenthesis only works if there is a corresponding close parenthesis in the rest of the string. However, it doesn't have to really match the open parenthesis. For example:
# No matching close parenthesis
$ echo "!("
bash: !: event not found
# The matching close parenthesis has nothing to do with the open
$ echo "!(" ")"
!( )
# An actual extended glob: files whose names don't start with a
$ echo "!(a*)"
b
As indicated in the bash manual, a history-expansion character is treated as an ordinary character if immediately preceded by a backslash. This is literally true; it doesn't matter whether the backslash will later be considered an escape character or not:
$ echo \!
!
$ echo \\!
\!
$ echo \\\!
\!
\ also inhibits history expansion inside double quotes, but \! is not a valid escape sequence inside the double quoted string, so the backslash is not removed:
$ echo "\!"
\!
$ echo "\\!"
\!
$ echo "\\\!"
\\!
I'm referring to the source code for bash v4.2 as I write this, so any undocumented behaviour may be completely different as of v4.3.
The problem is that within double quotes, bash is trying to expand !d before passing it to the subshell. You can get around this problem by removing the double quotes but I would also propose a simplification to your script:
VNCServerAndDisplayNumber=$(echo "$VNCServerResponse" | awk '/desktop/ {print $NF}')
This simply prints the last field on the line containing the word "desktop".
On a newer bash, you can use a herestring rather than piping an echo:
VNCServerAndDisplayNumber=$(awk '/desktop/ {print $NF}' <<<"$VNCServerResponse")
Don't wrap the $(...) command substitution in double quotes. You are asking the shell to perform evaluation on the contents of the quotes and are hitting the history substitution expansion feature. Drop the quotes and you stop telling the shell to do that and you won't hit that problem.
And yes, dropping those quotes is safe on that assignment line even if the output may contain spaces or newlines or whatever. Assignments of that sort are not going to split on those the way command substitution or variable evaluation will on a normal shell execution line.
Alternatively, disable history expansion in your shell/script before you run that. (It should be off when running a script by default I believe anyway.)
This only happens when history expansion is enabled, which it normally isn't and definitely shouldn't be for scripts.
Rather than trying to work around it, figure out why history expansion is enabled and what to do so it isn't.
If you're executing your script with . foo or source foo, use ./foo instead.
If you're writing this as a function in .bashrc or similar, consider making it a separate script.
If your script (or BASH_ENV) explicitly does set -H, don't.
Quote it with '' or \ or disable history expansion with set +H or shopt -u -o histexpand. See History Expansion.
I have a simple bash script to run a remote command on a given set of servers.
#!/bin/bash
echo "Command to be run:"
echo "$*"
read nothing
servers="server1 server2 server3"
for server in `echo $servers`
do
echo $server
ssh $server "$*"
echo ""
done
The problem is that the command could contain any number of arguments, hence the use of $* and could also have many different characters including quotes and regular expressions. The basic need here is for the shell to take the arguments, whatever they are, literally so they are passed to the remote server intact without removing quotes or interpreting parenthesis etc.
There are a number of variations I have seen but most deal with a specific character problem or overcomplicate the script or arguments required, and I'm looking to keep at least the arguments free of escape characters etc.
An example with using "#":
./cmd tw_query --no-headings "search Host where name matches '(?i)peter' show summary, nodecount(traverse :::Detail where name matches 'bob')"
Gives:
Command to be run:
tw_query --no-headings search Host where name matches '(?i)peter' show summary, nodecount(traverse :::Detail where name matches 'bob')
You seem to be looking for $#. Say:
ssh $server "$#"
instead. From the manual:
*
Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, it expands to a single word
with the value of each parameter separated by the first character of
the IFS special variable. That is, "$*" is equivalent to "$1c$2c…",
where c is the first character of the value of the IFS variable. If
IFS is unset, the parameters are separated by spaces. If IFS is null,
the parameters are joined without intervening separators.
#
Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a
separate word. That is, "$#" is equivalent to "$1" "$2" …. If the
double-quoted expansion occurs within a word, the expansion of the
first parameter is joined with the beginning part of the original
word, and the expansion of the last parameter is joined with the last
part of the original word. When there are no positional parameters,
"$#" and $# expand to nothing (i.e., they are removed).
You actually don't want the arguments passed to the remote server intact, you want them passed to the remote command intact. But that means they need to be wrapped in an extra layer of quotes/escapes/etc so that so that they will come out intact after the remote shell has parsed them.
bash actually has a feature in its printf builtin to add quoting/escaping to a string, but it quotes suitably for interpretation by bash itself -- if the remote shell were something else, it might not understand the quoting mode that it chooses. So in this case I'd recommend a simple-and-dumb quoting style: just add single-quotes around each argument, and replace each single-quote within the argument with '\'' (that'll end the current quoted string, add an escaped (literal) quote, then start another quoted string). It'll look a bit weird, but should decode properly under any POSIX-compliant shell.
Converting to this format is a bit tricky, since bash does inconsistent things with quotes in its search-and-replace patterns. Here's what I came up with:
#!/bin/bash
quotedquote="'\''"
printf -v quotedcommand "'%s' " "${#//\'/$quotedquote}"
echo "Command to be run:"
echo "$quotedcommand"
read nothing
servers="server1 server2 server3"
for server in $servers
do
echo $server
ssh $server "$quotedcommand"
echo ""
done
And here's how it quotes your example command:
'tw_query' '--no-headings' 'search Host where name matches '\''(?i)peter'\'' show summary, nodecount(traverse :::Detail where name matches '\''bob'\'')'
It looks strange to have the command itself quoted, but as long as you aren't trying to use an alias this doesn't cause any actual trouble. There is one significant limitation, though: there's no way to pass shell metacharacters (like > for output redirection) to the remote shell:
./cmd somecommand >outfile # redirect is done on local computer
./cmd somecommand '>outfile' # ">outfile" is passed to somecommand as an argument
If you need to do things like remote redirects, things get a good deal more complicated.
Besides the issue with $* versus $#, if this is for use in a production environment, you might want to consider a tool such as pdsh.
Otherwise, you can try feeding the commands to your script through stdin rather than putting them in argument so you avoid one level of parsing.
#!/bin/bash
read cmd
echo "Command to be run:"
echo $cmd
servers="server1 server2 server3"
for server in `echo $servers` do
echo $server
ssh $server "$cmd"
echo ""
done
and use it like this
$ ./cmd <<'EOT'
> tw_query --no-headings "search Host where name matches '(?i)peter' show summary, nodecount(traverse :::Detail where name matches 'bob')"
> EOT
Command to be run:
tw_query --no-headings "search Host where name matches '(?i)peter' show summary, nodecount(traverse :::Detail where name matches 'bob')"
Maybe a little far-fetched, but it could work.
How do I echo one or more tab characters using a bash script?
When I run this code
res=' 'x # res = "\t\tx"
echo '['$res']' # expect [\t\tx]
I get this
res=[ x] # that is [<space>x]
echo -e ' \t '
will echo 'space tab space newline' (-e means 'enable interpretation of backslash escapes'):
$ echo -e ' \t ' | hexdump -C
00000000 20 09 20 0a | . .|
Use printf, not echo.
There are multiple different versions of the echo command. There's /bin/echo (which may or may not be the GNU Coreutils version, depending on the system), and the echo command is built into most shells. Different versions have different ways (or no way) to specify or disable escapes for control characters.
printf, on the other hand, has much less variation. It can exist as a command, typically /bin/printf, and it's built into some shells (bash and zsh have it, tcsh and ksh don't), but the various versions are much more similar to each other than the different versions of echo are. And you don't have to remember command-line options (with a few exceptions; GNU Coreutils printf accepts --version and --help, and the built-in bash printf accepts -v var to store the output in a variable).
For your example:
res=' 'x # res = "\t\tx"
printf '%s\n' "[$res]"
And now it's time for me to admit that echo will work just as well for the example you're asking about; you just need to put double quotes around the argument:
echo "[$res]"
as kmkaplan wrote (two and a half years ago, I just noticed!). The problem with your original commands:
res=' 'x # res = "\t\tx"
echo '['$res']' # expect [\t\tx]
isn't with echo; it's that the shell replaced the tab with a space before echo ever saw it.
echo is fine for simple output, like echo hello world, but you should use printf whenever you want to do something more complex. You can get echo to work, but the resulting code is likely to fail when you run it with a different echo implementation or a different shell.
You can also try:
echo Hello$'\t'world.
Put your string between double quotes:
echo "[$res]"
you need to use -e flag for echo then you can
echo -e "\t\t x"
From the bash man page:
Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
So you can do this:
echo $'hello\tworld'
Use the verbatim keystroke, ^V (CTRL+V, C-v, whatever).
When you type ^V into the terminal (or in most Unix editors), the following character is taken verbatim. You can use this to type a literal tab character inside a string you are echoing.
Something like the following works:
echo "^V<tab>" # CTRL+V, TAB
Bash docs (q.v., "quoted-insert")
quoted-insert (C-q, C-v)
Add the next character that you type to the line verbatim. This is how to insert key sequences like C-q, for example.
side note: according to this, ALT+TAB should do the same thing, but we've all bound that sequence to window switching so we can't use it
tab-insert (M-TAB)
Insert a tab character.
--
Note: you can use this strategy with all sorts of unusual characters. Like a carriage return:
echo "^V^M" # CTRL+V, CTRL+M
This is because carriage return is ASCII 13, and M is the 13th letter of the alphabet, so when you type ^M, you get the 13th ASCII character. You can see it in action using ls^M, at an empty prompt, which will insert a carriage return, causing the prompt to act just like you hit return. When these characters are normally interpreted, verbatim gets you get the literal character.
Using echo to print values of variables is a common Bash pitfall.
Reference link:
http://mywiki.wooledge.org/BashPitfalls#echo_.24foo
If you want to use echo "a\tb" in a script, you run the script as:
# sh -e myscript.sh
Alternatively, you can give to myscript.sh the execution permission, and then run the script.
# chmod +x myscript.sh
# ./myscript.sh
res="\t\tx"
echo -e "[${res}]"