I am trying achieve the same effect as typing
mv ./images/*.{pdf,eps,jpg,svg} ./images/junk/
at the command line, from inside a bash script. I have:
MYDIR="./images"
OTHERDIR="./images/junk"
SUFFIXES='{pdf,eps,jpg,svg}'
mv "$MYDIR/"*.$SUFFIXES "$OTHERDIR/"
which, when run, gives the not unexpected error:
mv: rename ./images/*.{pdf,eps,jpg,svg} to ./images/junk/*.{pdf,eps,jpg,svg}:
No such file or directory
What is the correct way to quote all this so that mv will actually do the desired expansion? (Yes, there are plenty of files that match the pattern in ./images/.)
A deleted answer was on the right track. A slight modification to your attempt:
shopt -s extglob
MYDIR="./images"
OTHERDIR="./images/junk"
SUFFIXES='#(pdf|eps|jpg|svg)'
mv "$MYDIR/"*.$SUFFIXES "$OTHERDIR/"
Brace expansion is done before variable expansion, but variable expansion is done before pathname expansion. So the braces are still braces when the variable is expanded in your original, but when the variable instead contains pathname elements, they have already been expanded when the pathname expansion gets done.
You'll need to eval that line in order for it to work, like so:
MYDIR="./images"
OTHERDIR="./images/junk"
SUFFIXES='{pdf,eps,jpg,svg}'
eval "mv \"$MYDIR\"/*.$SUFFIXES \"$OTHERDIR/\""
Now, this has problems, in particular, if you don't trust $SUFFIXES, it might contain an injection attack, but for this simple case it should be alright.
If you are open to other solutions, you might want to experiment with find and xargs.
You can write a function:
function expand { for arg in "$#"; do [[ -f $arg ]] && echo $arg; done }
then call it with what you want to expand:
expand "$MYDIR/"*.$SUFFIXES
You can also make it a script expand.sh if you like.
Related
I want to run this cmd line script
$ script.sh lib/* ../test_git_thing
I want it to process all the files in the /lib folder.
FILES=$1
for f in $FILES
do
echo "Processing $f file..."
done
Currently it only prints the first file. If I use $#, it gives me all the files, but also the last param which I don't want. Any thoughts?
The argument list is being expanded at the command line when you invoke "script.sh lib/*" your script is being called with all the files in lib/ as args. Since you only reference $1 in your script, it's only printing the first file. You need to escape the wildcard on the command line so it's passed to your script to perform the globbing.
As correctly noted, lib/* on the command line is being expanded into all files in lib. To prevent expansion, you have 2 options. (1) quote your input:
$ script.sh 'lib/*' ../test_git_thing
Or (2), turn file globbing off. However, the option set -f will disable pathname expansion within the shell, but it will disable all pathname expansion (setting it within the script doesn't help as expansion is done by the shell before passing arguments to your script). In your case, it is probably better to quote the input or pass the first arguments as a directory name, and add the expansion in the script:
DIR=$1
for f in "$DIR"/*
In bash and ksh you can iterate through all arguments except the last like this:
for f in "${#:1:$#-1}"; do
echo "$f"
done
In zsh, you can do something similar:
for f in $#[1,${#}-1]; do
echo "$f"
done
$# is the number of arguments and ${#:start:length} is substring/subsequence notation in bash and ksh, while $#[start,end] is subsequence in zsh. In all cases, the subscript expressions are evaluated as arithmetic expressions, which is why $#-1 works. (In zsh, you need ${#}-1 because $#- is interpreted as "the length of $-".)
In all three shells, you can use the ${x:start:length} syntax with a scalar variable, to extract a substring; in bash and ksh, you can use ${a[#]:start:length} with an array to extract a subsequence of values.
This answers the question as given, without using non-POSIX features, and without workarounds such as disabling globbing.
You can find the last argument using a loop, and then exclude that when processing the list of files. In this example, $d is the directory name, while $f has the same meaning as in the original answer:
#!/bin/sh
if [ $# != 0 ]
then
for d in "$#"; do :; done
if [ -d "$d" ]
then
for f in "$#"
do
if [ "x$f" != "x$d" ]
then
echo "Processing $f file..."
fi
done
fi
fi
Additionally, it would be a good idea to also test if "$f" is a file, since it is common for shells to pass the wildcard character through the argument list if no match is found.
Say I have a folder called Foo located in /home/user/ (my /home/user also being represented by ~).
I want to have a variable
a="~/Foo" and then do
cd $a
I get
-bash: cd: ~/Foo: No such file or directory
However if I just do cd ~/Foo it works fine. Any clue on how to get this to work?
You can do (without quotes during variable assignment):
a=~/Foo
cd "$a"
But in this case the variable $a will not store ~/Foo but the expanded form /home/user/Foo. Or you could use eval:
a="~/Foo"
eval cd "$a"
You can use $HOME instead of the tilde (the tilde is expanded by the shell to the contents of $HOME).
Example:
dir="$HOME/Foo";
cd "$dir";
Although this question is merely asking for a workaround, this is listed as the duplicate of many questions that are asking why this happens, so I think it's worth giving an explanation. According to https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06:
The order of word expansion shall be as follows:
Tilde expansion, parameter expansion, command substitution, and arithmetic expansion shall be performed, beginning to end.
When the shell evaluates the string cd $a, it first performs tilde expansion (which is a no-op, since $a does not contain a tilde), then it expands $a to the string ~/Foo, which is the string that is finally passed as the argument to cd.
A much more robust solution would be to use something like sed or even better, bash parameter expansion:
somedir="~/Foo/test~/ing";
cd "${somedir/#\~/$HOME}"
or if you must use sed,
cd $(echo "$somedir" | sed "s#^~#$HOME#")
If you use double quotes the ~ will be kept as that character in $a.
cd $a will not expand the ~ since variable values are not expanded by the shell.
The solution is:
eval "cd $a"
Say I have a folder called Foo located in /home/user/ (my /home/user also being represented by ~).
I want to have a variable
a="~/Foo" and then do
cd $a
I get
-bash: cd: ~/Foo: No such file or directory
However if I just do cd ~/Foo it works fine. Any clue on how to get this to work?
You can do (without quotes during variable assignment):
a=~/Foo
cd "$a"
But in this case the variable $a will not store ~/Foo but the expanded form /home/user/Foo. Or you could use eval:
a="~/Foo"
eval cd "$a"
You can use $HOME instead of the tilde (the tilde is expanded by the shell to the contents of $HOME).
Example:
dir="$HOME/Foo";
cd "$dir";
Although this question is merely asking for a workaround, this is listed as the duplicate of many questions that are asking why this happens, so I think it's worth giving an explanation. According to https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06:
The order of word expansion shall be as follows:
Tilde expansion, parameter expansion, command substitution, and arithmetic expansion shall be performed, beginning to end.
When the shell evaluates the string cd $a, it first performs tilde expansion (which is a no-op, since $a does not contain a tilde), then it expands $a to the string ~/Foo, which is the string that is finally passed as the argument to cd.
A much more robust solution would be to use something like sed or even better, bash parameter expansion:
somedir="~/Foo/test~/ing";
cd "${somedir/#\~/$HOME}"
or if you must use sed,
cd $(echo "$somedir" | sed "s#^~#$HOME#")
If you use double quotes the ~ will be kept as that character in $a.
cd $a will not expand the ~ since variable values are not expanded by the shell.
The solution is:
eval "cd $a"
I want to run this cmd line script
$ script.sh lib/* ../test_git_thing
I want it to process all the files in the /lib folder.
FILES=$1
for f in $FILES
do
echo "Processing $f file..."
done
Currently it only prints the first file. If I use $#, it gives me all the files, but also the last param which I don't want. Any thoughts?
The argument list is being expanded at the command line when you invoke "script.sh lib/*" your script is being called with all the files in lib/ as args. Since you only reference $1 in your script, it's only printing the first file. You need to escape the wildcard on the command line so it's passed to your script to perform the globbing.
As correctly noted, lib/* on the command line is being expanded into all files in lib. To prevent expansion, you have 2 options. (1) quote your input:
$ script.sh 'lib/*' ../test_git_thing
Or (2), turn file globbing off. However, the option set -f will disable pathname expansion within the shell, but it will disable all pathname expansion (setting it within the script doesn't help as expansion is done by the shell before passing arguments to your script). In your case, it is probably better to quote the input or pass the first arguments as a directory name, and add the expansion in the script:
DIR=$1
for f in "$DIR"/*
In bash and ksh you can iterate through all arguments except the last like this:
for f in "${#:1:$#-1}"; do
echo "$f"
done
In zsh, you can do something similar:
for f in $#[1,${#}-1]; do
echo "$f"
done
$# is the number of arguments and ${#:start:length} is substring/subsequence notation in bash and ksh, while $#[start,end] is subsequence in zsh. In all cases, the subscript expressions are evaluated as arithmetic expressions, which is why $#-1 works. (In zsh, you need ${#}-1 because $#- is interpreted as "the length of $-".)
In all three shells, you can use the ${x:start:length} syntax with a scalar variable, to extract a substring; in bash and ksh, you can use ${a[#]:start:length} with an array to extract a subsequence of values.
This answers the question as given, without using non-POSIX features, and without workarounds such as disabling globbing.
You can find the last argument using a loop, and then exclude that when processing the list of files. In this example, $d is the directory name, while $f has the same meaning as in the original answer:
#!/bin/sh
if [ $# != 0 ]
then
for d in "$#"; do :; done
if [ -d "$d" ]
then
for f in "$#"
do
if [ "x$f" != "x$d" ]
then
echo "Processing $f file..."
fi
done
fi
fi
Additionally, it would be a good idea to also test if "$f" is a file, since it is common for shells to pass the wildcard character through the argument list if no match is found.
My question is very similar to How to copy multiple files from a different directory using cp?
I don't want to use an explicit loop. Here is what I do:
$ FILES_TOOLS="fastboot,fastboot-HW.sh"
$ cp $HOME/tools/{$FILES_TOOLS} $TOP_DIR/removeme
cp: cannot stat `/home/johndoe/tools/{fastboot,fastboot-HW.sh}': No such file or directory
The files are present and destination is valid, because:
$ cp $HOME/tools/{fastboot,fastboot-HW.sh} $TOP_DIR/removeme
$ echo $?
0
I tried to remove the double quote from FILES_TOOLS, no luck.
I tried to quote and double quote {...}, no luck
I tried to backslash the brackets, no luck
I guess this is a problem of when the shell expansion actually occurs.
This answer is limited to the bash.
Prepend an echo to see what your cp command turns into:
echo cp $HOME/tools/{$FILES_TOOLS} $TOP_DIR/removeme
You have to insert an eval inside a sub-shell to make it work:
cp $( eval echo $HOME/tools/{$FILES_TOOLS} ) $TOP_DIR/removeme
I guess this is a problem of when the shell expansion actually occurs.
Yes. Different shells have different rules about brace expansion in relation to variable expansion. Your way works in ksh, but not in zsh or bash. {1..$n} works in ksh and zsh but not in bash. In bash, variable expansion always happens after brace expansion.
The closest you'll get to this in bash is with eval.
As long as the contents of the braces are literals, you can use brace expansion to populate an array with the full path names of the files to copy, then expand the contents of the array in your cp command.
$ FILES_TOOLS=( $HOME/tools/{fastboot,fastboot-HW.sh} )
$ cp "${FILES_TOOLS[#]}" $TOP_DIR/removeme
Update: I realized you might have a reason for having the base names alone in the variable. Here's another array-based solution that lets you prefix each element of the array with a path, again without an explicit loop:
$ FILES_TOOLS=( fastboot fastboot-HW.sh )
$ cp "${FILES_TOOLS[#]/#/$HOME/tools/}" $TOP_DIR/removeme
In this case, you use the pattern substitution operator to replace the empty string at the beginning of each array element with the directory name.