Let's say I have a really simple shell script 'foo':
#!/bin/sh
echo $#
If I invoke it like so:
foo 1 2 3
It happily prints:
1 2 3
However, let's say one of my arguments is double-quote enclosed and contains whitespace:
foo 1 "this arg has whitespace" 3
foo happily prints:
1 this arg has whitespace 3
The double-quotes have been stripped! I know shell thinks its doing me a favor, but... I would like to get at the original version of the arguments, unmolested by shell's interpretation. Is there any way to do so?
First, you probably want quoted version of $#, i.e. "$#". To feel the difference, try putting more than one space inside the string.
Second, quotes are element of shell's syntax -- it doesn't do you a favor. To preserve them, you need to escape them. Examples:
foo 1 "\"this arg has whitespace\"" 3
foo 1 '"this arg has whitespace"' 3
Double quote $#:
#!/bin/sh
for ARG in "$#"
do
echo $ARG
done
Then:
foo 1 "this arg has whitespace" 3
will give you:
1
this arg has whitespace
3
What i'd do is to quote all the arguments received with spaces that might help your case.
for x in "${#}" ; do
# try to figure out if quoting was required for the $x
if [[ "$x" != "${x%[[:space:]]*}" ]]; then
x="\""$x"\""
fi
echo $x
_args=$_args" "$x
done
echo "All Cmd Args are: $_args"
You need to quote the quotes:
foo 1 "\"this arg has whitespace\"" 3
or (more simply)
foo 1 '"this arg has whitespace"' 3
You need to quote the double quotes to make sure that the shell doesn't remove them when parsing word arguments.
Let's suppose you are in a more rigid set-up and you CANNOT change your command line, and make it more "friendly" by escaping the double quotes. For example:
example_script.sh argument_without_quotes "argument with quotes i cannot escape"
First consider that inside your script you can't tell if an argument is passed with or without quotes, because the shell strips them.
So what you can possibly do is rebuilding double quotes for arguments containing whitespaces
This example rebuilds the whole command line, double-quoting arguments that have white spaces
#!/bin/sh
#initialize the variable that will contain the whole argument string
argList=""
#iterate on each argument
for arg in "$#"
do
#if an argument contains a white space, enclose it in double quotes and append to the list
#otherwise simply append the argument to the list
if echo $arg | grep -q " "; then
argList="$argList \"$arg\""
else
argList="$argList $arg"
fi
done
#remove a possible trailing space at the beginning of the list
argList=$(echo $argList | sed 's/^ *//')
#pass your argument list WITH QUOTES
echo "my_executable" $argList
#my_executable $argList
Note this limitation. If you run this example
example_script.sh "argument with spaces" argument_without_spaces "argument_doublequoted_but_without_spaces"
you will get this output
my_executable "argument with spaces" argument_without_spaces argument_doublequoted_but_without_spaces
Note the last argument: since it had no spaces, it has not been enclosed again in double quotes, but this shouldn't be an issue.
The most reliable method that I've found to do this is to leverage the logic that's built into the shell for "xtrace". This is a subsystem that's turned on via set -x which will print every command that's run by the shell. If we turn on xtrace then run a command that does nothing (true) with the same arguments, the shell (most shells...) will quote the arguments for us.
show_quoted() {
(
# use > as the xtrace prefix (this is the default)
PS4='+'
exec 2>&1 # send the xtrace (on stderr, 2) to stdout (1) for processing
# turn on xtrace -- all commands are printed, with $PS4 as a prefix
set -x
# `true` is the command that does nothing, successfully
true "$#"
) |
sed '
# remove the xtrace prefix -- any number of + characters
s/^+*//
# hide the "true" command
s/^true //
'
}
Same thing, but with a normal amount of comments:
show_quoted() {
(
PS4='+' # reset to default
exec 2>&1 # send xtrace to stdout
set -x
true "$#"
) | sed 's/^+*true //' # remove the xtrace prefix
}
Here's my test results: (these vary a bit but are all valid)
$ sh show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
1 '2 3' 'he said: "hi"' 'she said: '\''bye'\'''
$ bash show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
1 '2 3' 'he said: "hi"' 'she said: '\''bye'\'''
$ zsh show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
1 '2 3' 'he said: "hi"' 'she said: '\''bye'\'
$ busybox sh show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
1 '2 3' 'he said: "hi"' 'she said: '"'"'bye'"'"
These shells (below) weren't up to the task however. To be fair, the toybox shell is "80% done".
$ dash show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
1 2 3 he said: "hi" she said: 'bye'
$ ./toybox sh show_quoted.sh 1 '2 3' 'he said: "hi"' "she said: 'bye'"
"$#"
Related
I've got a program that I want to call by passing parameters from a shell variable. Throughout this question, I am going to assume that it is given by
#!/bin/sh
echo $#
i.e. that it prints out the number of arguments that are passed to it. Let's call it count-args.
I call my program like this:
X="arg1 arg2"
count-args $X
This works quite well. But now one of my arguments has a whitespace in it and I can't find a way to escape it, e.g. the following things do not work:
X="Hello\ World"
X="Hello\\ World"
X="'Hello World'"
In all of the cases, my program count-args prints out 2. I want to find a way so I can pass the string Hello World and that it returns 1 instead. How?
Just for clarification: I do not want to pass all parameters as a single string, e.g.
X="Hello World"
count-args $X
should print out 2. I want a way to pass parameters that contain whitespaces.
Use an array to store multiple, space-containing arguments.
$ args=("first one" "second one")
$ count-args "${args[#]}"
2
This can be solved with xargs. By replacing
count-args $X
with
echo $X | xargs count-args
I can use backslashes to escape whitespaces in $X, e.g.
X="Hello\\ World"
echo $X | xargs count-args
prints out 1 and
X="Hello World"
echo $X | xargs count-args
prints out 2.
count-args "$X"
The quotes ensure in bash, that the whole content of variable X is passed as a single parameter.
Your Counting script:
$ cat ./params.sh
#!/bin/sh
echo $#
For completeness here is what happens with various arguments:
$ ./params.sh
0
$ ./params.sh 1 2
2
$ ./params.sh
0
$ ./params.sh 1
1
$ ./params.sh 1 2
2
$ ./params.sh "1 2"
1
And here is what you get with variables:
$ XYZ="1 2" sh -c './params.sh $XYZ'
2
$ XYZ="1 2" sh -c './params.sh "$XYZ"'
1
Taking this a bit further:
$ cat params-printer.sh
#!/bin/sh
echo "Count: $#"
echo "1 : '$1'"
echo "2 : '$2'"
We get:
$ XYZ="1 2" sh -c './params-printer.sh "$XYZ"'
Count: 1
1 : '1 2'
2 : ''
This looks like what you wanted to do.
Now: If you have a script you cannot control and neither can you control the way the script is invoked. Then there is very little you can do to prevent a variable with spaces turning into multiple arguments.
There are quite a few questions around this on StackOverflow which indicate that you need the ability to control how the command is invoked else there is little you can do.
Passing arguments with spaces between (bash) script
Passing a string with spaces as a function argument in bash
Passing arguments to a command in Bash script with spaces
And wow! this has been asked so many times before:
How to pass argument with spaces to a shell script function
I'd like to store a command line in a variable, and then execute that command line. The problem is, the command line has arguments with spaces in them. If I do
$ x='command "complex argument"'
$ $x
it calls command with "complex and argument". I tried using "$x" thinking it would preserve the argument splittings, but it only tries to execute a program with the file name command "complex argument". I also tried variations of the quotes (' vs ") and using exec, but it didn't help. Any ideas?
Edit: eval "$x" almost works, but if the whitespaces separating arguments are newlines and not spaces, then it treats the lines as separate commands.
Edit2: The extra " quotes were too much, and made eval interpret the newlines not as spaces, but as command delimiters. The solutions in the answers all work.
For testing purposes, I define:
$ function args() { while [[ "$1" != "" ]]; do echo arg: $1; shift; done }
This works as expected:
$ args "1 2" 3
arg: 1 2
arg: 3
$ x="arg 4 5 6"
$ $x
arg: 4
arg: 5
arg: 6
This doesnt:
$ x="args \"3 4\" 5"
$ $x
arg: "3
arg: 4"
arg: 5
A safe approach is to store your command line in a BASH array:
arr=( command "complex argument" )
Then execute it:
"${arr[#]}"
OR else another approach is to use BASH function:
cmdfunc() {
command "complex argument"
}
cmdfunc
Another solution is
x='command "complex argument"'
eval $x
In many SO questions and bash tutorials I see that I can access command line args in bash scripts in two ways:
$ ~ >cat testargs.sh
#!/bin/bash
echo "you passed me" $*
echo "you passed me" $#
Which results in:
$ ~> bash testargs.sh arg1 arg2
you passed me arg1 arg2
you passed me arg1 arg2
What is the difference between $* and $#?
When should one use the former and when shall one use the latter?
The difference appears when the special parameters are quoted. Let me illustrate the differences:
$ set -- "arg 1" "arg 2" "arg 3"
$ for word in $*; do echo "$word"; done
arg
1
arg
2
arg
3
$ for word in $#; do echo "$word"; done
arg
1
arg
2
arg
3
$ for word in "$*"; do echo "$word"; done
arg 1 arg 2 arg 3
$ for word in "$#"; do echo "$word"; done
arg 1
arg 2
arg 3
one further example on the importance of quoting: note there are 2 spaces between "arg" and the number, but if I fail to quote $word:
$ for word in "$#"; do echo $word; done
arg 1
arg 2
arg 3
and in bash, "$#" is the "default" list to iterate over:
$ for word; do echo "$word"; done
arg 1
arg 2
arg 3
A nice handy overview table from the Bash Hackers Wiki:
Syntax
Effective result
$*
$1 $2 $3 … ${N}
$#
$1 $2 $3 … ${N}
"$*"
"$1c$2c$3c…c${N}"
"$#"
"$1" "$2" "$3" … "${N}"
where c in the third row is the first character of $IFS, the Input Field Separator, a shell variable.
If the arguments are to be stored, load them in an array variable.
$*
Expands to the positional parameters, starting from one. When the
expansion occurs within double quotes, it expands to a single word
with the value of each parameter separated by the first character of
the IFS special variable. That is, "$*" is equivalent to "$1c$2c...",
where c is the first character of the value of the IFS variable. If
IFS is unset, the parameters are separated by spaces. If IFS is null,
the parameters are joined without intervening separators.
$#
Expands to the positional parameters, starting from one. When the
expansion occurs within double quotes, each parameter expands to a
separate word. That is, "$#" is equivalent to "$1" "$2" ... If the
double-quoted expansion occurs within a word, the expansion of the
first parameter is joined with the beginning part of the original
word, and the expansion of the last parameter is joined with the last
part of the original word. When there are no positional parameters,
"$#" and $# expand to nothing (i.e., they are removed).
Source: Bash man
$# is same as $*, but each parameter is a quoted string, that is, the parameters are passed on intact, without interpretation or expansion. This means, among other things, that each parameter in the argument list is seen as a separate word.
Of course, "$#" should be quoted.
http://tldp.org/LDP/abs/html/internalvariables.html#ARGLIST
This example let may highlight the differ between "at" and "asterix" while we using them.
I declared two arrays "fruits" and "vegetables"
fruits=(apple pear plumm peach melon)
vegetables=(carrot tomato cucumber potatoe onion)
printf "Fruits:\t%s\n" "${fruits[*]}"
printf "Fruits:\t%s\n" "${fruits[#]}"
echo + --------------------------------------------- +
printf "Vegetables:\t%s\n" "${vegetables[*]}"
printf "Vegetables:\t%s\n" "${vegetables[#]}"
See the following result the code above:
Fruits: apple pear plumm peach melon
Fruits: apple
Fruits: pear
Fruits: plumm
Fruits: peach
Fruits: melon
+ --------------------------------------------- +
Vegetables: carrot tomato cucumber potatoe onion
Vegetables: carrot
Vegetables: tomato
Vegetables: cucumber
Vegetables: potatoe
Vegetables: onion
I'm trying to pass 2 arguments to a command and each argument contains spaces, I've tried escaping the spaces in the args, I've tried wrapping in single quotes, I've tried escaping \" but nothing will work.
Here's a simple example.
#!/bin/bash -xv
ARG="/tmp/a b/1.txt"
ARG2="/tmp/a b/2.txt"
ARG_BOTH="\"$ARG\" \"$ARG2\""
cat $ARG_BOTH
I'm getting the following when it runs:
ARG_BOTH="$ARG $ARG2"
+ ARG_BOTH='/tmp/a\ b/1.txt /tmp/a\ b/2.txt'
cat $ARG_BOTH
+ cat '/tmp/a\' b/1.txt '/tmp/a\' b/2.txt
cat: /tmp/a\: No such file or directory
cat: b/1.txt: No such file or directory
cat: /tmp/a\: No such file or directory
cat: b/2.txt: No such file or directory
See http://mywiki.wooledge.org/BashFAQ/050
TLDR
Put your args in an array and call your program as myutil "${arr[#]}"
#!/bin/bash -xv
file1="file with spaces 1"
file2="file with spaces 2"
echo "foo" > "$file1"
echo "bar" > "$file2"
arr=("$file1" "$file2")
cat "${arr[#]}"
Output
file1="file with spaces 1"
+ file1='file with spaces 1'
file2="file with spaces 2"
+ file2='file with spaces 2'
echo "foo" > "$file1"
+ echo foo
echo "bar" > "$file2"
+ echo bar
arr=("$file1" "$file2")
+ arr=("$file1" "$file2")
cat "${arr[#]}"
+ cat 'file with spaces 1' 'file with spaces 2'
foo
bar
This might be a good use-case for the generic "set" command, which sets the top-level shell parameters to a word list. That is, $1, $2, ... and so also $* and $# get reset.
This gives you some of the advantages of arrays while also staying all-Posix-shell-compatible.
So:
set "arg with spaces" "another thing with spaces"
cat "$#"
The most straightforward revision of your example shell script that will work correctly is:
#! /bin/sh
ARG="/tmp/a b/1.txt"
ARG2="/tmp/a b/2.txt"
cat "$ARG" "$ARG2"
However, if you need to wrap up a whole bunch of arguments in one shell variable, you're up a creek; there is no portable, reliable way to do it. (Arrays are Bash-specific; the only portable options are set and eval, both of which are asking for grief.) I would consider a need for this as an indication that it was time to rewrite in a more powerful scripting language, e.g. Perl or Python.
What I'd like to do is take, as an input to a function, a line that may include quotes (single or double) and echo that line exactly as it was provided to the function. For instance:
function doit {
printf "%s " ${#}
eval "${#}"
printf " # [%3d]\n" ${?}
}
Which, given the following input
doit VAR=42
doit echo 'single quote $VAR'
doit echo "double quote $VAR"
Yields the following:
VAR=42 # [ 0]
echo single quote $VAR # [ 0]
echo double quote 42 # [ 0]
So the semantics of the variable expansion are preserved as I'd expect, but I can not get the exact format of the line as it was provided to the function. What I'd like is to have doit echo 'single quote $VAR' result in echo 'single quote $VAR'.
I'm sure this has to do with bash processing the arguments before they are passed to the function; I'm just looking for a way around that (if possible).
Edit
So what I had intended was to shadow the execution of a script while providing an exact replica of the execution that could be used as a diagnostic tool including exit status of each step.
While I can get the desired behavior described above by doing something like
while read line ; do
doit ${line}
done < ${INPUT}
That approach fails in the face of control structures (i.e. if, while, etc). I thought about using set -x but that has it's limitations as well: " becomes ' and exit status is not visible for commands that fail.
I was in a similar position to you in that I needed a script to wrap around an existing command and pass arguments preserving quoting.
I came up with something that doesn't preserve the command line exactly as typed but does pass the arguments correctly and show you what they were.
Here's my script set up to shadow ls:
CMD=ls
PARAMS=""
for PARAM in "$#"
do
PARAMS="${PARAMS} \"${PARAM}\""
done
echo Running: ${CMD} ${PARAMS}
bash -c "${CMD} ${PARAMS}"
echo Exit Code: $?
And this is some sample output:
$ ./shadow.sh missing-file "not a file"
Running: ls "missing-file" "not a file"
ls: missing-file: No such file or directory
ls: not a file: No such file or directory
Exit Code: 1
So as you can see it adds quotes which weren't originally there but it does preserve arguments with spaces in which is what I needed.
The reason this happens is because bash interprets the arguments, as you thought. The quotes simply aren't there any more when it calls the function, so this isn't possible. It worked in DOS because programs could interpret the command line themselves, not that it helps you!
Although #Peter Westlake's answer is correct, and there are no quotes to preserve one can try to deduce if the quotes where required and thus passed in originally. Personally I used this requote function when I needed a proof in my logs that a command ran with the correct quoting:
function requote() {
local res=""
for x in "${#}" ; do
# try to figure out if quoting was required for the $x:
grep -q "[[:space:]]" <<< "$x" && res="${res} '${x}'" || res="${res} ${x}"
done
# remove first space and print:
sed -e 's/^ //' <<< "${res}"
}
And here is how I use it:
CMD=$(requote "${#}")
# ...
echo "${CMD}"
doit echo "'single quote $VAR'"
doit echo '"double quote $VAR"'
Both will work.
bash will only strip the outside set of quotes when entering the function.
Bash will remove the quote when you pass a string with quote in as command line argument. The quote is simply not there anymore when the string is pass to your script. You have no way to know there is a single quote or double quote.
What you probably can do is sth like this:
doit VAR=42
doit echo \'single quote $VAR\'
doit echo \"double quote $VAR\"
In your script you get
echo 'single quote $VAR'
echo "double quote $VAR"
Or do this
doit VAR=42
doit echo 'single quote $VAR'
doit echo '"double quote $VAR"'
In your script you get
echo single quote $VAR
echo "double quote $VAR"
This:
ponerApostrofes1 ()
{
for (( i=1; i<=$#; i++ ));
do
eval VAR="\${$i}";
echo \'"${VAR}"\';
done;
return;
}
As an example has problems when the parameters have apostrophes.
This function:
ponerApostrofes2 ()
{
for ((i=1; i<=$#; i++ ))
do
eval PARAM="\${$i}";
echo -n \'${PARAM//\'/\'\\\'\'}\'' ';
done;
return
}
solves the mentioned problem and you can use parameters including apostrophes inside, like "Porky's", and returns, apparently(?), the same string of parameters when each parameter is quoted; if not, it quotes it. Surprisingly, I don't understand why, if you use it recursively, it doesn't return the same list but each parameter is quoted again. But if you do echo of each one you recover the original parameter.
Example:
$ ponerApostrofes2 'aa aaa' 'bbbb b' 'c'
'aa aaa' 'bbbb b' 'c'
$ ponerApostrofes2 $(ponerApostrofes2 'aa aaa' 'bbbb b' 'c' )
''\''aa' 'aaa'\''' ''\''bbbb' 'b'\''' ''\''c'\'''
And:
$ echo ''\''bbbb' 'b'\'''
'bbbb b'
$ echo ''\''aa' 'aaa'\'''
'aa aaa'
$ echo ''\''c'\'''
'c'
And this one:
ponerApostrofes3 ()
{
for ((i=1; i<=$#; i++ ))
do
eval PARAM="\${$i}";
echo -n ${PARAM//\'/\'\\\'\'} ' ';
done;
return
}
returning one level of quotation less,
doesn't work either, neither alternating both recursively.
If one's shell does not support pattern substitution, i.e. ${param/pattern/string} then the following sed expression can be used to safely quote any string such that it will eval into a single parameter again:
sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/'/"
Combining this with printf it is possible to write a little function that will take any list of strings produced by filename expansion or "$#" and turn it into something that can be safely passed to eval to expand it into arguments for another command while safely preserving parameter separation.
# Usage: quotedlist=$(shell_quote args...)
#
# e.g.: quotedlist=$(shell_quote *.pdf) # filenames with spaces
#
# or: quotedlist=$(shell_quote "$#")
#
# After building up a quoted list, use it by evaling it inside
# double quotes, like this:
#
# eval "set -- $quotedlist"
# for str in "$#"; do
# # fiddle "${str}"
# done
#
# or like this:
#
# eval "\$a_command $quotedlist \$another_parameter"
#
shell_quote()
{
local result=''
local arg
for arg in "$#" ; do
# Append a space to our result, if necessary
#
result=${result}${result:+ }
# Convert each embedded ' to \' , then insert ' at the
# beginning of the line, and append ' at the end of
# the line.
#
result=${result}$(printf "%s\n" "$arg" | \
sed -e "s/'/'\\\\''/g" -e "1s/^/'/" -e "\$s/\$/'/")
done
# use printf(1) instead of echo to avoid weird "echo"
# implementations.
#
printf "%s\n" "$result"
}
It may be easier (and maybe safer, i.e. avoid eval) in some situations to use an "impossible" character as the field separator and then use IFS to control expansion of the value again.
The shell is going to interpret the quotes and the $ before it passes it to your function. There's not a lot your function can do to get the special characters back, because it has no way of knowing (in the double-quote example) whether 42 was hard-coded or if it came from a variable. You will have to escape the special characters if you want them to survive long enough to make it to your function.