Bash: Brace expansion in scripts not working due to unwanted escaping - bash

I want to do something like this in a bash script. I'm using bash 4.1.10.
# rm -rf /some/path/{folder1,folder2,folder3}
Works nicely (and as expected) from the shell itself. It deletes the 3 desired folders leaving all others untouched.
When I put it into script something unwanted happens. For example, my script:
#!/bin/bash
set -x
VAR="folder1,folder2,folder3"
rm -rf /some/path/{$VAR}
When I execute this script, the folders are not deleted.
I think this is due to the fact that some unwanted quoting is occurring. Output from the script using #!/bin/bash -x:
rm -rf '/some/path/{folder1,folder2,folder3}'
which of course cannot succeed due to the ' marks.
How can I get this working within my script?

According to the man page:
The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.
So to get around this, add another level of expansion:
eval "rm -rf /some/path/{$VAR}"

Since you're writing a script, there's no reason to write hard-to-maintain code using eval tricks
VAR="f1,f2,f3"
IFS=,
set -- $VAR
for f; do
rm -r "/path/to/$f"
done
or
VAR=( f1 f2 f3 )
for f in "${VAR[#]}"; do
rm -r "/path/to/$f"
done

No, it's due to the fact that brace expansion happens before parameter expansion. Find another way of doing this, such as with xargs.
xargs -d , -I {} rm -rf /some/path/{} <<< "$VAR"

If your code can be rearranged, you can use echo and command substitution in bash.
Something like this:
#!/bin/bash
set -x
VAR=`echo /some/path/{folder1,folder2,folder3}`
rm -rf $VAR

You need to enable braceexpand flag:
#!/bin/bash
set -B
for i in /some/path/{folder1,folder2,folder3}
do
rm -rf "$i"
done

The problem is not that in script mode some unwanted quoting is happening but that you put the folder names into a variable and the variable content is expanded after the brace expansion is done.
If you really want to do it like this you have to use eval:
eval "rm -rf /some/path/{$VAR}"

#!/bin/bash
set -x
VAR="folder1,folder2,folder3"
eval "rm -rf /some/path/{$VAR}"
Edit The remainder is just for info. I try to be informative, but not wordy :_)
Recent bashes have the globstar option. This might perhaps come in handy in the future
shopt -s globstar
rm -rfvi some/**/folder?

Another trick you can use (instead of the dangerous eval) is just plain echo inside a subshell. This works, for instance:
paths=`echo /some/path/{folder1,folder2,folder3}`
echo rm -rf $paths
outputs:
rm -rf /some/path/folder1 /some/path/folder2 /some/path/folder3
as desired. (Remove the "echo" in the second line, to make it actually do the rm.)
The crucial point is that bash does brace expansion before parameter expansion, so you never want to put a comma-separated list (surrounded by braces or not) into a variable -- if you do, then you'll have to resort to eval. You can however put a list of space-separated strings into a variable, by having the brace expansion happen in a subshell before assignment.

replace {$VAR} by ${VAR} :-)

Related

pushd "no such file or directory" only sometimes [duplicate]

Say I have a folder called Foo located in /home/user/ (my /home/user also being represented by ~).
I want to have a variable
a="~/Foo" and then do
cd $a
I get
-bash: cd: ~/Foo: No such file or directory
However if I just do cd ~/Foo it works fine. Any clue on how to get this to work?
You can do (without quotes during variable assignment):
a=~/Foo
cd "$a"
But in this case the variable $a will not store ~/Foo but the expanded form /home/user/Foo. Or you could use eval:
a="~/Foo"
eval cd "$a"
You can use $HOME instead of the tilde (the tilde is expanded by the shell to the contents of $HOME).
Example:
dir="$HOME/Foo";
cd "$dir";
Although this question is merely asking for a workaround, this is listed as the duplicate of many questions that are asking why this happens, so I think it's worth giving an explanation. According to https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06:
The order of word expansion shall be as follows:
Tilde expansion, parameter expansion, command substitution, and arithmetic expansion shall be performed, beginning to end.
When the shell evaluates the string cd $a, it first performs tilde expansion (which is a no-op, since $a does not contain a tilde), then it expands $a to the string ~/Foo, which is the string that is finally passed as the argument to cd.
A much more robust solution would be to use something like sed or even better, bash parameter expansion:
somedir="~/Foo/test~/ing";
cd "${somedir/#\~/$HOME}"
or if you must use sed,
cd $(echo "$somedir" | sed "s#^~#$HOME#")
If you use double quotes the ~ will be kept as that character in $a.
cd $a will not expand the ~ since variable values are not expanded by the shell.
The solution is:
eval "cd $a"

Why does "basename <(readlink -f filename)" return "63", not the actual basename?

Is it possible to use basename and readlink in a one line? Something like:
ln -s /usr/local/src symlink
echo `basename <(readlink -f "./symlink")`
except that script above prints 63 instead of src.
Use the command substitution instead of the process substitution:
echo "$(basename "$(readlink -f "./symlink")")"
or, if that's your complete line, echo is redundant:
basename "$(readlink -f "./symlink")"
Multiple $(..) command substitutions can be nested without any escaping or quoting needed (unlike with the old-style backquote version). Also note that if the substitution appears within double quotes, word splitting and filename expansion are not performed on the results.
To clarify the difference: when you say <(cmd), the cmd is executed and the result is made available in a file, handle to which is returned, something like /dev/fd/63. Since the basename acts on a filename given, not its contents, it returns 63.
Unlike process substitution, the $(cmd) will execute the cmd and return the result of command (its standard output). You can then store it in a variable, like res=$(cmd), or reuse it in-place, like cmd "$(cmd)".

How to store curly brackets in a Bash variable

I am trying to write a bash script. I am not sure why in my script:
ls {*.xml,*.txt}
works okay, but
name="{*.xml,*.txt}"
ls $name
doesn't work. I get
ls: cannot access {*.xml,*.txt}: No such file or directory
The expression
ls {*.xml,*.txt}
results in Brace expansion and shell passes the expansion (if any) to ls as arguments. Setting shopt -s nullglob makes this expression evaluate to nothing when there are no matching files.
Double quoting the string suppresses the expansion and shell stores the literal contents in your variable name (not sure if that is what you wanted). When you invoke ls with $name as the argument, shell does the variable expansion but no brace expansion is done.
As #Cyrus has mentioned, eval ls $name will force brace expansion and you get the same result as that of ls {\*.xml,\*.txt}.
The reason your expansion doesn't work is that brace expansion is performed before variable expansion, see Shell expansions in the manual.
I'm not sure what it is you're trying to do, but if you want to store a list of file names, use an array:
files=( {*.txt,*.xml} ) # these two are the same
files=(*.txt *.xml)
ls -l "${files[#]}" # give them to a command
for file in "${files[#]}" ; do # or loop over them
dosomething "$file"
done
"${array[#]}" expands to all elements of the array, as separate words. (remember the quotes!)

How to copy multiple files from a different directory using cp, variable and brackets?

My question is very similar to How to copy multiple files from a different directory using cp?
I don't want to use an explicit loop. Here is what I do:
$ FILES_TOOLS="fastboot,fastboot-HW.sh"
$ cp $HOME/tools/{$FILES_TOOLS} $TOP_DIR/removeme
cp: cannot stat `/home/johndoe/tools/{fastboot,fastboot-HW.sh}': No such file or directory
The files are present and destination is valid, because:
$ cp $HOME/tools/{fastboot,fastboot-HW.sh} $TOP_DIR/removeme
$ echo $?
0
I tried to remove the double quote from FILES_TOOLS, no luck.
I tried to quote and double quote {...}, no luck
I tried to backslash the brackets, no luck
I guess this is a problem of when the shell expansion actually occurs.
This answer is limited to the bash.
Prepend an echo to see what your cp command turns into:
echo cp $HOME/tools/{$FILES_TOOLS} $TOP_DIR/removeme
You have to insert an eval inside a sub-shell to make it work:
cp $( eval echo $HOME/tools/{$FILES_TOOLS} ) $TOP_DIR/removeme
I guess this is a problem of when the shell expansion actually occurs.
Yes. Different shells have different rules about brace expansion in relation to variable expansion. Your way works in ksh, but not in zsh or bash. {1..$n} works in ksh and zsh but not in bash. In bash, variable expansion always happens after brace expansion.
The closest you'll get to this in bash is with eval.
As long as the contents of the braces are literals, you can use brace expansion to populate an array with the full path names of the files to copy, then expand the contents of the array in your cp command.
$ FILES_TOOLS=( $HOME/tools/{fastboot,fastboot-HW.sh} )
$ cp "${FILES_TOOLS[#]}" $TOP_DIR/removeme
Update: I realized you might have a reason for having the base names alone in the variable. Here's another array-based solution that lets you prefix each element of the array with a path, again without an explicit loop:
$ FILES_TOOLS=( fastboot fastboot-HW.sh )
$ cp "${FILES_TOOLS[#]/#/$HOME/tools/}" $TOP_DIR/removeme
In this case, you use the pattern substitution operator to replace the empty string at the beginning of each array element with the directory name.

how to make bash expand wildcards in variables?

I am trying achieve the same effect as typing
mv ./images/*.{pdf,eps,jpg,svg} ./images/junk/
at the command line, from inside a bash script. I have:
MYDIR="./images"
OTHERDIR="./images/junk"
SUFFIXES='{pdf,eps,jpg,svg}'
mv "$MYDIR/"*.$SUFFIXES "$OTHERDIR/"
which, when run, gives the not unexpected error:
mv: rename ./images/*.{pdf,eps,jpg,svg} to ./images/junk/*.{pdf,eps,jpg,svg}:
No such file or directory
What is the correct way to quote all this so that mv will actually do the desired expansion? (Yes, there are plenty of files that match the pattern in ./images/.)
A deleted answer was on the right track. A slight modification to your attempt:
shopt -s extglob
MYDIR="./images"
OTHERDIR="./images/junk"
SUFFIXES='#(pdf|eps|jpg|svg)'
mv "$MYDIR/"*.$SUFFIXES "$OTHERDIR/"
Brace expansion is done before variable expansion, but variable expansion is done before pathname expansion. So the braces are still braces when the variable is expanded in your original, but when the variable instead contains pathname elements, they have already been expanded when the pathname expansion gets done.
You'll need to eval that line in order for it to work, like so:
MYDIR="./images"
OTHERDIR="./images/junk"
SUFFIXES='{pdf,eps,jpg,svg}'
eval "mv \"$MYDIR\"/*.$SUFFIXES \"$OTHERDIR/\""
Now, this has problems, in particular, if you don't trust $SUFFIXES, it might contain an injection attack, but for this simple case it should be alright.
If you are open to other solutions, you might want to experiment with find and xargs.
You can write a function:
function expand { for arg in "$#"; do [[ -f $arg ]] && echo $arg; done }
then call it with what you want to expand:
expand "$MYDIR/"*.$SUFFIXES
You can also make it a script expand.sh if you like.

Resources