I have a string:
{2013/05/01},{2013/05/02},{2013/05/03}
I want to append a { at the beginning and a } at the end. The output should be:
{{2013/05/01},{2013/05/02},{2013/05/03}}
However, in my shell script when I concatenate the curly braces to the beginning and end of the string, the output is as follows:
{2013/05/01} {2013/05/02} {2013/05/03}
Why does this happen? How can I achieve my result? Am sure there is a simple solution to this but I am a unix newbie, thus would appreciate some help.
Test script:
#!/usr/bin/ksh
valid_data_range="{2013/05/01},{2013/05/02},{2013/05/03}"
finalDates="{"$valid_data_range"}"
print $finalDates
The problem is that when you have a list in braces outside quotes, the shell performs Brace Expansion (bash manual, but ksh will be similar). Since the 'outside quotes' bit is important, it also tells you how to avoid the problem — enclose the string in quotes when printing:
#!/usr/bin/ksh
valid_data_range="{2013/05/01},{2013/05/02},{2013/05/03}"
finalDates="{$valid_data_range}"
print "$finalDates"
(The print command is specific to ksh and is not present in bash. The change in the assignment line is more cosmetic than functional.)
Also, the brace expansion would not occur in bash; it only occurs when the braces are written directly. This bilingual script (ksh and bash):
valid_data_range="{2013/05/01},{2013/05/02},{2013/05/03}"
finalDates="{$valid_data_range}"
printf "%s\n" "$finalDates"
printf "%s\n" $finalDates
produces:
ksh
{{2013/05/01},{2013/05/02},{2013/05/03}}
{2013/05/01}
{2013/05/02}
{2013/05/03}
bash (also zsh)
{{2013/05/01},{2013/05/02},{2013/05/03}}
{{2013/05/01},{2013/05/02},{2013/05/03}}
Thus, when you need to use the variable $finalDates, ensure it is inside double quotes:
other_command "$finalDates"
if [ "$finalDates" = "$otherString" ]
then : whatever
else : something
fi
Etc — using your preferred layout for whatever you don't like about mine.
You can say:
finalDates=$'{'"$valid_data_range"$'}'
The problem is that the shell is performing brace expansion. This allows you to generate a series of similar strings:
$ echo {a,b,c}
a b c
That's not very impressive, but consider
$ echo a{b,c,d}e
abc ace ade
In order to suppress brace expansion, you can use the set command to turn it off temporarily
$ set +B
$ echo a{b,c,d}e
a{b,c,d}e
$ set -B
$ echo a{b,c,d}e
abe ace ade
Related
$ a=33
$ echo "$(echo '$a')" # (1)
$a
$ echo "$(echo "$a")" # (2)
33
I can't understand the shell command-line parsing result above.
For command line (1), according to man bash, single quote inside the double-quotes will be parsed as literal, and $a inside the double quotes will be interpolated, so I think the result of command line (1) should be '33'.
You don't have single-quotes in double-quotes. $(....) is a command substitution. Essentially what occurs within (...) takes place within its own subshell. You must employ quoting rules within that subshell. The subshell represents its own environment.
For example:
$ a=33
$ echo $(echo "'$a'") # single-quotes within double-quotes
The quoting rules within the command substitution are applied correctly resulting in the output:
$ '33'
Let me know if you have further questions.
If I do this:
echo{,}
The result is:
echo
I don't really understand the {,} at the end and the result
Thanks to clarify this.
I would start with something simpler to see how {} works: As #anubhava linked, it generates strings. Essentially, it expands all the elements in it and combines them with whatever is before and after it (space is separator if you don't quote).
Example:
$ bash -xc 'echo before{1,2}after and_sth_else'
+ echo before1after before2after and_sth_else
before1after before2after and_sth_else
Note that there is a space between echo and the arguments. This is not the case on what you have posted. So what happened there? Check the following:
$ bash -xc 'man{1,2}'
+ man1 man2
bash: man1: command not found
The result of the expansion is fed to bash and bash tries to execute it. In the above case, the command that is looking for is man1 (which does not exist).
Finally, combine the above to your question:
echo{,}
{,} expands to two empty elements/strings
These are then prefixed/concatenated with "echo" so we now have echo echo
Expansion finished and this is given to bash to execute
Command is echo and its first argument is "echo"... so it echoes echo!
echo{,}
is printing just echo because it is equivalent of echo echo.
More examples to clarify:
bash -xc 'echo{,}'
+ echo echo
echo
echo foo{,}
foo foo
echo foo{,,}
foo foo foo
More about Brace Expansion
Brace expansion is a mechanism by which arbitrary strings may be generated. This mechanism is similar to pathname expansion, but the filenames generated
need not exist. Patterns to be brace expanded take the form of an optional preamble, followed by either a series of comma-separated strings or a sequence
expression between a pair of braces, followed by an optional postscript. The preamble is prefixed to each string contained within the braces, and the
postscript is then appended to each resulting string, expanding left to right.
The {item1,item2,...} is a brace expansion.
So echo{,} is expanded as echo echo because {,} has two (empty) elements, then echo prints it argument.
Try this :
$ set -x
$ echo{,}
+ echo echo
echo
$ set +x
+ set +x
$
It's also handy to generate "cross products" without nested loops:
$ ary=( {1,2,3}{a,b,c} )
$ declare -p ary
declare -a ary=([0]="1a" [1]="1b" [2]="1c" [3]="2a" [4]="2b" [5]="2c" [6]="3a" [7]="3b" [8]="3c")
I am writing a BASH script and I am using the bash command. Which one of the following is correct (or are both incorrect)?
bash $pbs_dir/${module_name}.${target_ID}.${instance_ID}.pbs
or
bash \$pbs_dir/\${module_name}.\${target_ID}.\${instance_ID}.pbs
\$ will be expanded to literal $, so there is a big difference:
$ a="hello"
$ echo $a
hello
$ echo \$a
$a
Also note that you almost always want to double quote your parameter expansions to avoid word splitting and pathname expansion:
echo "$a"
So you properly want to use the following:
bash "$pbs_dir/${module_name}.${target_ID}.${instance_ID}.pbs"
Going over the POSIX standard, I came across another rather technical/pointless question. It states:
Within the backquoted style of command substitution, <backslash> shall retain its literal meaning, except when followed by: '$' , '`' , or <backslash>.
It's easy to see why '`' and '\' lose their literal meanings: nested command substitution demands a "different" backquote inside the command substitution, which in turn forces '\' to lose its literal meaning. So, for instance, the following different behavior seems reasonable:
$ echo $(echo \\\\)
\\
$ echo `echo \\\\`
\
But what about '$'? I.e., what's the point or, more concretely, a possible benefit of the following difference?
$ echo $(echo \$\$)
$$
$ echo `echo \$\$`
4735
As '$' by itself is not ruled out inside backquotes, it looks like you would use either '$' or '\\\$' all the time, but never the middle case '\$'.
To recap,
$ echo `echo $$` # PID, OK
4735
$ echo `echo \\\$\\\$` # literal "$$", OK
$$
$ echo `echo \$\$` # What's the point?
4735
PS: I know this question is rather technical... I myself go for the more modern $(...) substitution all the time, but I'm still curious.
By adding a \, you make the inner subshell expand it instead of the outer shell. A good example would be to actually force the starting of a new shell, like this:
$ echo $$
4988
$ echo `sh -c 'echo $$'`
4988
$ echo `sh -c 'echo \$\$'`
4990
$ echo `sh -c 'echo \\\$\\\$'`
$$
Basic Answer
Consider the following command, which finds the base directory where gcc was installed:
gcc_base=$(dirname $(dirname $(which gcc)))
With the $(...) notation, there is no problem with the parsing; it is trivial and is one of the primary reason why the notation is recommended. The equivalent command using back-ticks is:
gcc_base=`dirname \`dirname \\\`which gcc\\\`\``
When the shell first parses this command, it encounters the first backtick, and has to find the matching close backtick. That's when the quoted section comes into effect:
Within the backquoted style of command substitution, shall retain its literal meaning, except when followed by: '$' , '`' , or .
gcc_base=`dirname \`dirname \\\`which gcc\\\`\``
^ ^ ^ ^ ^ ^
1 2 3 4 5 6
backslash-backtick - special rule
backslash-backslash - special rule
backslash-backtick - special rule
backslash-backslash - special rule
backslash-backtick - special rule
backslash-backtick - special rule
So, the unescaped backtick at the end marks the end of the outermost backtick command. The sub-shell that processes that command sees:
dirname `dirname \`which gcc\``
The backslash-back escapes are given the special treatment again, and the sub-sub-shell sees:
dirname `which gcc`
The sub-sub-sub-shell gets to see which gcc and evaluates it (e.g. /usr/gcc/v4.6.1/bin/gcc).
The sub-sub-shell evaluates dirname /usr/gcc/v4.6.1/bin/gcc and produces /usr/gcc/v4.6.1/bin.
The sub-shell evaluates dirname /usr/gcc/v4.6.1/bin and produces /usr/gcc/v4.6.1.
The shell assigns /usr/gcc/v4.6.1 to gcc_base.
In this example, the backslashes were only followed by the special characters - backslash, backtick, dollar. A more complex example would have, for example, \" sequences in the command, and then the special rule would not apply; the \" would simply be copied through unchanged and passed to the relevant sub-shell(s).
Extraordinarily Complex Stuff
For example, suppose you had a command with a blank in its name (heaven forbid; and this shows why!) such as totally amazing (two blanks; it is a more stringent test than a single blank). Then you could write:
$ cmd="totally amazing"
$ echo "$cmd"
totally amazing
$ which "$cmd"
/Users/jleffler/bin/totally amazing
$ dirname $(which "$cmd")
usage: dirname path
$ # Oops!
$ dirname "$(which \"\$cmd\")"
"$cmd": not found
.
$ # Oops!
$ dirname "$(which \"$cmd\")"
"totally: not found
amazing": not found
.
$ dirname "$(eval which \"$cmd\")"
totally amazing: not found
.
$ dirname "$(eval which \"\$cmd\")"
/Users/jleffler/bin
$ # Ouch, but at least that worked!
$ # But how to extend that to the next level?
$ dirname "$(eval dirname \"\$\(eval which \\\"\\\$cmd\\\"\)\")"
/Users/jleffler
$
OK - well, that's the "easy" one! Do you need a better reason to avoid spaces in command names or path names? I've also demonstrated to my own satisfaction that it works correctly with pathnames that contain spaces.
So, can we compress the learning cycle for backticks? Yes...
$ cat x3.sh
cmd="totally amazing"
which "$cmd"
dirname "`which \"$cmd\"`"
dirname "`dirname \"\`which \\"\$cmd\\\"\`\"`"
$ sh -x x3.sh
+ cmd='totally amazing'
+ which 'totally amazing'
/Users/jleffler/bin/totally amazing
++ which 'totally amazing'
+ dirname '/Users/jleffler/bin/totally amazing'
/Users/jleffler/bin
+++ which 'totally amazing'
++ dirname '/Users/jleffler/bin/totally amazing'
+ dirname /Users/jleffler/bin
/Users/jleffler
$
That is still a ghastly, daunting, non-intuitive set of escape sequences. It's actually shorter than the version for $(...) notation, and doesn't use any eval commands (which always complicate things).
This probably has to do with the strange way the Bourne shell parses substitutions (the real Korn shell is slightly similar but most other shells do not exhibit the strange behaviour at all).
Basically, the Bourne shell's parser does not interpret substitutions ($ and `) inside double-quotes, or parameter substitution ($) anywhere. This is only done at expansion time. Also, in many cases unmatched quotes (single-quotes, double-quotes or backquotes) are not an error; the closing quote is assumed at the end.
One consequence is that if a parameter substitution with a word containing spaces like ${v+a b} occurs outside double-quotes, it is not parsed correctly and will cause an expansion error when executed. The space needs to be quoted. Other shells do not have this problem.
Another consequence is that double-quotes inside backquotes inside double-quotes do not work reliably. For example,
v=0; echo "`v=1; echo " $v "`echo b"
will print
1 echo b
in most shells (one command substitution), but
0 b
in the Bourne shell and the real Korn shell (ksh93) (two command substitutions).
(Ways to avoid the above issue are to assign the substitution to a variable first, so double-quotes are not necessary, or to use new-style command substitution.)
The real Korn shell (ksh93) attempts to preserve much of the strange Bourne shell behaviour but does parse substitutions at parse time. Thus, ${v+a b} is accepted but the above example has "strange" behaviour. A further strange thing is that something like
echo "`${v+pwd"
is accepted (the result is like with the missing closing brace). And where does the opening brace in the error message from
echo "`${v+pwd`"
come from?
The below session shows an obscure case where $ and \$ differ in a non-obvious way:
$ echo ${.sh.version}
Version JM 93u 2011-02-08
$ v=0; echo "`v=1; echo "${v+p q}"`echo b"
p qecho b
$ v=0; echo "`v=1; echo "\${v+p q}"`echo b"
p{ q}b
Basically, a backslash is an escape character. You put it before another character to represent something special. An 'n','t','$' and '\'are these special characters.
"\n" --> newline
"\t" --> tab (indent)
"\$" --> $ (because a $ before a word in shell denotes a variable)
"\\" --> \
The backslash before characters is only interpreted the above way when it is inside quotes.
If you want to find more info or other escape chars go here
This question already has answers here:
How can I preserve quotes in printing a bash script's arguments
(7 answers)
Closed 3 years ago.
I have a Bash script where I want to keep quotes in the arguments passed.
Example:
./test.sh this is "some test"
then I want to use those arguments, and re-use them, including quotes and quotes around the whole argument list.
I tried using \"$#\", but that removes the quotes inside the list.
How do I accomplish this?
using "$#" will substitute the arguments as a list, without re-splitting them on whitespace (they were split once when the shell script was invoked), which is generally exactly what you want if you just want to re-pass the arguments to another program.
Note that this is a special form and is only recognized as such if it appears exactly this way. If you add anything else in the quotes the result will get combined into a single argument.
What are you trying to do and in what way is it not working?
There are two safe ways to do this:
1. Shell parameter expansion: ${variable#Q}:
When expanding a variable via ${variable#Q}:
The expansion is a string that is the value of parameter quoted in a format that can be reused as input.
Example:
$ expand-q() { for i; do echo ${i#Q}; done; } # Same as for `i in "$#"`...
$ expand-q word "two words" 'new
> line' "single'quote" 'double"quote'
word
'two words'
$'new\nline'
'single'\''quote'
'double"quote'
2. printf %q "$quote-me"
printf supports quoting internally. The manual's entry for printf says:
%q Causes printf to output the corresponding argument in a format that can be reused as shell input.
Example:
$ cat test.sh
#!/bin/bash
printf "%q\n" "$#"
$
$ ./test.sh this is "some test" 'new
>line' "single'quote" 'double"quote'
this
is
some\ test
$'new\nline'
single\'quote
double\"quote
$
Note the 2nd way is a bit cleaner if displaying the quoted text to a human.
Related: For bash, POSIX sh and zsh: Quote string with single quotes rather than backslashes
Yuku's answer only works if you're the only user of your script, while Dennis Williamson's is great if you're mainly interested in printing the strings, and expect them to have no quotes-in-quotes.
Here's a version that can be used if you want to pass all arguments as one big quoted-string argument to the -c parameter of bash or su:
#!/bin/bash
C=''
for i in "$#"; do
i="${i//\\/\\\\}"
C="$C \"${i//\"/\\\"}\""
done
bash -c "$C"
So, all the arguments get a quote around them (harmless if it wasn't there before, for this purpose), but we also escape any escapes and then escape any quotes that were already in an argument (the syntax ${var//from/to} does global substring substitution).
You could of course only quote stuff which already had whitespace in it, but it won't matter here. One utility of a script like this is to be able to have a certain predefined set of environment variables (or, with su, to run stuff as a certain user, without that mess of double-quoting everything).
Update: I recently had reason to do this in a POSIX way with minimal forking, which lead to this script (the last printf there outputs the command line used to invoke the script, which you should be able to copy-paste in order to invoke it with equivalent arguments):
#!/bin/sh
C=''
for i in "$#"; do
case "$i" in
*\'*)
i=`printf "%s" "$i" | sed "s/'/'\"'\"'/g"`
;;
*) : ;;
esac
C="$C '$i'"
done
printf "$0%s\n" "$C"
I switched to '' since shells also interpret things like $ and !! in ""-quotes.
If it's safe to make the assumption that an argument that contains white space must have been (and should be) quoted, then you can add them like this:
#!/bin/bash
whitespace="[[:space:]]"
for i in "$#"
do
if [[ $i =~ $whitespace ]]
then
i=\"$i\"
fi
echo "$i"
done
Here is a sample run:
$ ./argtest abc def "ghi jkl" $'mno\tpqr' $'stu\nvwx'
abc
def
"ghi jkl"
"mno pqr"
"stu
vwx"
You can also insert literal tabs and newlines using Ctrl-V Tab and Ctrl-V Ctrl-J within double or single quotes instead of using escapes within $'...'.
A note on inserting characters in Bash: If you're using Vi key bindings (set -o vi) in Bash (Emacs is the default - set -o emacs), you'll need to be in insert mode in order to insert characters. In Emacs mode, you're always in insert mode.
I needed this for forwarding all arguments to another interpreter.
What ended up right for me is:
bash -c "$(printf ' %q' "$#")"
Example (when named as forward.sh):
$ ./forward.sh echo "3 4"
3 4
$ ./forward.sh bash -c "bash -c 'echo 3'"
3
(Of course the actual script I use is more complex, involving in my case nohup and redirections etc., but this is the key part.)
Like Tom Hale said, one way to do this is with printf using %q to quote-escape.
For example:
send_all_args.sh
#!/bin/bash
if [ "$#" -lt 1 ]; then
quoted_args=""
else
quoted_args="$(printf " %q" "${#}")"
fi
bash -c "$( dirname "${BASH_SOURCE[0]}" )/receiver.sh${quoted_args}"
send_fewer_args.sh
#!/bin/bash
if [ "$#" -lt 2 ]; then
quoted_last_args=""
else
quoted_last_args="$(printf " %q" "${#:2}")"
fi
bash -c "$( dirname "${BASH_SOURCE[0]}" )/receiver.sh${quoted_last_args}"
receiver.sh
#!/bin/bash
for arg in "$#"; do
echo "$arg"
done
Example usage:
$ ./send_all_args.sh
$ ./send_all_args.sh a b
a
b
$ ./send_all_args.sh "a' b" 'c "e '
a' b
c "e
$ ./send_fewer_args.sh
$ ./send_fewer_args.sh a
$ ./send_fewer_args.sh a b
b
$ ./send_fewer_args.sh "a' b" 'c "e '
c "e
$ ./send_fewer_args.sh "a' b" 'c "e ' 'f " g'
c "e
f " g
Just use:
"${#}"
For example:
# cat t2.sh
for I in "${#}"
do
echo "Param: $I"
done
# cat t1.sh
./t2.sh "${#}"
# ./t1.sh "This is a test" "This is another line" a b "and also c"
Param: This is a test
Param: This is another line
Param: a
Param: b
Param: and also c
Changed unhammer's example to use array.
printargs() { printf "'%s' " "$#"; echo; }; # http://superuser.com/a/361133/126847
C=()
for i in "$#"; do
C+=("$i") # Need quotes here to append as a single array element.
done
printargs "${C[#]}" # Pass array to a program as a list of arguments.
My problem was similar and I used mixed ideas posted here.
We have a server with a PHP script that sends e-mails. And then we have a second server that connects to the 1st server via SSH and executes it.
The script name is the same on both servers and both are actually executed via a bash script.
On server 1 (local) bash script we have just:
/usr/bin/php /usr/local/myscript/myscript.php "$#"
This resides on /usr/local/bin/myscript and is called by the remote server. It works fine even for arguments with spaces.
But then at the remote server we can't use the same logic because the 1st server will not receive the quotes from "$#". I used the ideas from JohnMudd and Dennis Williamson to recreate the options and parameters array with the quotations. I like the idea of adding escaped quotations only when the item has spaces in it.
So the remote script runs with:
CSMOPTS=()
whitespace="[[:space:]]"
for i in "$#"
do
if [[ $i =~ $whitespace ]]
then
CSMOPTS+=(\"$i\")
else
CSMOPTS+=($i)
fi
done
/usr/bin/ssh "$USER#$SERVER" "/usr/local/bin/myscript ${CSMOPTS[#]}"
Note that I use "${CSMOPTS[#]}" to pass the options array to the remote server.
Thanks for eveyone that posted here! It really helped me! :)
Quotes are interpreted by bash and are not stored in command line arguments or variable values.
If you want to use quoted arguments, you have to quote them each time you use them:
val="$3"
echo "Hello World" > "$val"
As Gary S. Weaver shown in his source code tips, the trick is to call bash with parameter '-c' and then quote the next.
e.g.
bash -c "<your program> <parameters>"
or
docker exec -it <my docker> bash -c "$SCRIPT $quoted_args"
If you need to pass all arguments to bash from another programming language (for example, if you'd want to execute bash -c or emit_bash_code | bash), use this:
escape all single quote characters you have with '\''.
then, surround the result with singular quotes
The argument of abc'def will thus be converted to 'abc'\''def'. The characters '\'' are interpreted as following: the already existing quoting is terminated with the first first quote, then the escaped singular single quote \' comes, then the new quoting starts.
Yes, seems that it is not possible to ever preserve the quotes, but for the issue I was dealing with it wasn't necessary.
I have a bash function that will search down folder recursively and grep for a string, the problem is passing a string that has spaces, such as "find this string". Passing this to the bash script will then take the base argument $n and pass it to grep, this has grep believing these are different arguments. The way I solved this by using the fact that when you quote bash to call the function it groups the items in the quotes into a single argument. I just needed to decorate that argument with quotes and pass it to the grep command.
If you know what argument you are receiving in bash that needs quotes for its next step you can just decorate with with quotes.
Just use single quotes around the string with the double quotes:
./test.sh this is '"some test"'
So the double quotes of inside the single quotes were also interpreted as string.
But I would recommend to put the whole string between single quotes:
./test.sh 'this is "some test" '
In order to understand what the shell is doing or rather interpreting arguments in scripts, you can write a little script like this:
#!/bin/bash
echo $#
echo "$#"
Then you'll see and test, what's going on when calling a script with different strings