I want to run this cmd line script
$ script.sh lib/* ../test_git_thing
I want it to process all the files in the /lib folder.
FILES=$1
for f in $FILES
do
echo "Processing $f file..."
done
Currently it only prints the first file. If I use $#, it gives me all the files, but also the last param which I don't want. Any thoughts?
The argument list is being expanded at the command line when you invoke "script.sh lib/*" your script is being called with all the files in lib/ as args. Since you only reference $1 in your script, it's only printing the first file. You need to escape the wildcard on the command line so it's passed to your script to perform the globbing.
As correctly noted, lib/* on the command line is being expanded into all files in lib. To prevent expansion, you have 2 options. (1) quote your input:
$ script.sh 'lib/*' ../test_git_thing
Or (2), turn file globbing off. However, the option set -f will disable pathname expansion within the shell, but it will disable all pathname expansion (setting it within the script doesn't help as expansion is done by the shell before passing arguments to your script). In your case, it is probably better to quote the input or pass the first arguments as a directory name, and add the expansion in the script:
DIR=$1
for f in "$DIR"/*
In bash and ksh you can iterate through all arguments except the last like this:
for f in "${#:1:$#-1}"; do
echo "$f"
done
In zsh, you can do something similar:
for f in $#[1,${#}-1]; do
echo "$f"
done
$# is the number of arguments and ${#:start:length} is substring/subsequence notation in bash and ksh, while $#[start,end] is subsequence in zsh. In all cases, the subscript expressions are evaluated as arithmetic expressions, which is why $#-1 works. (In zsh, you need ${#}-1 because $#- is interpreted as "the length of $-".)
In all three shells, you can use the ${x:start:length} syntax with a scalar variable, to extract a substring; in bash and ksh, you can use ${a[#]:start:length} with an array to extract a subsequence of values.
This answers the question as given, without using non-POSIX features, and without workarounds such as disabling globbing.
You can find the last argument using a loop, and then exclude that when processing the list of files. In this example, $d is the directory name, while $f has the same meaning as in the original answer:
#!/bin/sh
if [ $# != 0 ]
then
for d in "$#"; do :; done
if [ -d "$d" ]
then
for f in "$#"
do
if [ "x$f" != "x$d" ]
then
echo "Processing $f file..."
fi
done
fi
fi
Additionally, it would be a good idea to also test if "$f" is a file, since it is common for shells to pass the wildcard character through the argument list if no match is found.
Related
I want to run this cmd line script
$ script.sh lib/* ../test_git_thing
I want it to process all the files in the /lib folder.
FILES=$1
for f in $FILES
do
echo "Processing $f file..."
done
Currently it only prints the first file. If I use $#, it gives me all the files, but also the last param which I don't want. Any thoughts?
The argument list is being expanded at the command line when you invoke "script.sh lib/*" your script is being called with all the files in lib/ as args. Since you only reference $1 in your script, it's only printing the first file. You need to escape the wildcard on the command line so it's passed to your script to perform the globbing.
As correctly noted, lib/* on the command line is being expanded into all files in lib. To prevent expansion, you have 2 options. (1) quote your input:
$ script.sh 'lib/*' ../test_git_thing
Or (2), turn file globbing off. However, the option set -f will disable pathname expansion within the shell, but it will disable all pathname expansion (setting it within the script doesn't help as expansion is done by the shell before passing arguments to your script). In your case, it is probably better to quote the input or pass the first arguments as a directory name, and add the expansion in the script:
DIR=$1
for f in "$DIR"/*
In bash and ksh you can iterate through all arguments except the last like this:
for f in "${#:1:$#-1}"; do
echo "$f"
done
In zsh, you can do something similar:
for f in $#[1,${#}-1]; do
echo "$f"
done
$# is the number of arguments and ${#:start:length} is substring/subsequence notation in bash and ksh, while $#[start,end] is subsequence in zsh. In all cases, the subscript expressions are evaluated as arithmetic expressions, which is why $#-1 works. (In zsh, you need ${#}-1 because $#- is interpreted as "the length of $-".)
In all three shells, you can use the ${x:start:length} syntax with a scalar variable, to extract a substring; in bash and ksh, you can use ${a[#]:start:length} with an array to extract a subsequence of values.
This answers the question as given, without using non-POSIX features, and without workarounds such as disabling globbing.
You can find the last argument using a loop, and then exclude that when processing the list of files. In this example, $d is the directory name, while $f has the same meaning as in the original answer:
#!/bin/sh
if [ $# != 0 ]
then
for d in "$#"; do :; done
if [ -d "$d" ]
then
for f in "$#"
do
if [ "x$f" != "x$d" ]
then
echo "Processing $f file..."
fi
done
fi
fi
Additionally, it would be a good idea to also test if "$f" is a file, since it is common for shells to pass the wildcard character through the argument list if no match is found.
So, lets say I have 6 files that are all the same type. In my specific case all of them are zip files and I want to select all of them and "pass them through" a shell script that "unzips" all of them.
I can already do it selecting one by one as the script simply does:
#!/bin/bash
DIR=$(dirname "$#")
exec unzip "$#" -d "${DIR}"
So it unzips the "zip file" exactly where I have it.
Now, when I select multiple files (aka more than one file). I don't know what happens as I don't fully understand what is "parsed" into the script.
I found this What does $# mean in a shell script?
So I would like to know how to do it right.
Thanks a lot.
Fixing Your Script: Iterating Over Arguments
If you're calling commands (like unzip) that only take one argument (of the type you want to pass) at a time, then you need to iterate over them. That is:
#!/bin/bash
for arg in "$#"; do # or just "for arg do"
dir=$(dirname "$arg")
unzip "$arg" -d "$dir"
done
Literal Answer (What The Original Syntax Did)
"$#" expands to the complete list of positional arguments. What does that mean in practice?
Let's say your code were called with:
./yourscript "Directory One/file1.zip" "Directory Two/file2.zip"
In this case, you would have:
# this is what your code would try to do (it's an error!)
DIR=$(dirname "Directory One/file1.zip" "Directory Two/file2.zip")
...followed by:
# also doesn't work, since "unzip" only takes a single zipfile argument
# ...and because the above dirname fails, DIR is empty here
unzip "Directory One/file1.zip" "Directory Two/file2.zip" -d "$DIR"
Quoting from the official manual:
# - Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a separate word. That is, "$#" is equivalent to "$1" "$2" .... If the double-quoted expansion occurs within a word, the expansion of the first parameter is joined with the beginning part of the original word, and the expansion of the last parameter is joined with the last part of the original word. When there are no positional parameters, "$#" and $# expand to nothing (i.e., they are removed).
I have been following the answers given in these questions
Shellscript Looping Through All Files in a Folder
How to iterate over files in a directory with Bash?
to write a bash script which goes over files inside a folder and processes them. So, here is the code I have:
#!/bin/bash
YEAR="2002/"
INFOLDER="/local/data/datasets/Convergence/"
for f in "$INFOLDER$YEAR*.mdb";
do
echo $f
absname=$INFOLDER$YEAR$(basename $f)
# ... the rest of the script ...
done
I am receiving this error: basename: extra operand.
I added echo $f and I realized that f contains all the filenames separated by space. But I expected to get one at a time. What could be the problem here?
You're running into problems with quoting. In the shell, double-quotes prevent word splitting and wildcard expansion; generally, you don't want these things to happen to variable's values, so you should double-quote variable references. But when you have something that should be word-split or wildcard-expanded, it cannot be double-quoted. In your for statement, you have the entire file pattern in double-quotes:
for f in "$INFOLDER$YEAR*.mdb";
...which prevents word-splitting and wildcard expansion on the variables' values (good) but also prevents it on the * which you need expanded (that's the point of the loop). So you need to quote selectively, with the variables inside quotes and the wildcard outside them:
for f in "$INFOLDER$YEAR"*.mdb;
And then inside the loop, you should double-quote the references to $f in case any filenames contain whitespace or wildcards (which are completely legal in filenames):
echo "$f"
absname="$INFOLDER$YEAR$(basename "$f")"
(Note: the double-quotes around the assignment to absname aren't actually needed -- the right side of an assignment is one of the few places in the shell where it's safe to skip them -- but IMO it's easier and safer to just double-quote all variable references and $( ) expressions than to try to keep track of where it's safe and where it's not.)
Just quote your shell variables if they are supposed to contain strings with spaces in between.
basename "$f"
Not doing so will lead to splitting of the string into separate characters (see WordSplitting in bash), thereby messing up the basename command which expects one string argument rather than multiple.
Also it would be a wise to include the * outside the double-quotes as shell globbing wouldn't work inside them (single or double-quote).
#!/bin/bash
# good practice to lower-case variable names to distinguish them from
# shell environment variables
year="2002/"
in_folder="/local/data/datasets/Convergence/"
for file in "${in_folder}${year}"*.mdb; do
# break the loop gracefully if no files are found
[ -e "$file" ] || continue
echo "$file"
# Worth noting here, the $file returns the name of the file
# with absolute path just as below. You don't need to
# construct in manually
absname=${in_folder}${year}$(basename "$file")
done
just remove "" from this line
for f in "$INFOLDER$YEAR*.mdb";
so it looks like this
#!/bin/bash
YEAR="2002/"
INFOLDER="/local/data/datasets/Convergence/"
for f in $INFOLDER$YEAR*.mdb;
do
echo $f
absname=$INFOLDER$YEAR$(basename $f)
# ... the rest of the script ...
done
I would like to run recursively myscript.sh, to execute all files in the directory:
It has been discussed here that I could do like this:
#!/bin/bash
for file in * ; do
echo $file
done
But I would like myscript.sh to execute with this syntax, so that I could select only certain filetypes to be executed:
./myscript.sh *.dat
Thus I modify the script above:
#!/bin/bash
for file in $1 ; do
echo $file
done
In which when executing, it only executes first occurrence, not all files with *.dat extensions.
What is wrong here?
The wildcard *.dat is expanded by the shell before your script ever sees it. So the filenames show up in your script as $1, $2, $3, etc.
You can work with them all at once by using the special $# variable:
for file in "$#"; do
echo $file
done
Note that the double quotes around "$#" is special. From man bash:
# Expands to the positional parameters, starting from one. When the expansion
occurs within double quotes, each parameter expands to a separate word. That
is, "$#" is equivalent to "$1" "$2" ... If the double-quoted expansion occurs
within a word, the expansion of the first parameter is joined with the begin-
ning part of the original word, and the expansion of the last parameter is
joined with the last part of the original word. When there are no positional
parameters, "$#" and $# expand to nothing (i.e., they are removed).
You could do the same thing with just: ls *.dat | xargs echo
You need to get the actual contents of *, like so:
for file in $* ; do
echo $file
done
I am trying achieve the same effect as typing
mv ./images/*.{pdf,eps,jpg,svg} ./images/junk/
at the command line, from inside a bash script. I have:
MYDIR="./images"
OTHERDIR="./images/junk"
SUFFIXES='{pdf,eps,jpg,svg}'
mv "$MYDIR/"*.$SUFFIXES "$OTHERDIR/"
which, when run, gives the not unexpected error:
mv: rename ./images/*.{pdf,eps,jpg,svg} to ./images/junk/*.{pdf,eps,jpg,svg}:
No such file or directory
What is the correct way to quote all this so that mv will actually do the desired expansion? (Yes, there are plenty of files that match the pattern in ./images/.)
A deleted answer was on the right track. A slight modification to your attempt:
shopt -s extglob
MYDIR="./images"
OTHERDIR="./images/junk"
SUFFIXES='#(pdf|eps|jpg|svg)'
mv "$MYDIR/"*.$SUFFIXES "$OTHERDIR/"
Brace expansion is done before variable expansion, but variable expansion is done before pathname expansion. So the braces are still braces when the variable is expanded in your original, but when the variable instead contains pathname elements, they have already been expanded when the pathname expansion gets done.
You'll need to eval that line in order for it to work, like so:
MYDIR="./images"
OTHERDIR="./images/junk"
SUFFIXES='{pdf,eps,jpg,svg}'
eval "mv \"$MYDIR\"/*.$SUFFIXES \"$OTHERDIR/\""
Now, this has problems, in particular, if you don't trust $SUFFIXES, it might contain an injection attack, but for this simple case it should be alright.
If you are open to other solutions, you might want to experiment with find and xargs.
You can write a function:
function expand { for arg in "$#"; do [[ -f $arg ]] && echo $arg; done }
then call it with what you want to expand:
expand "$MYDIR/"*.$SUFFIXES
You can also make it a script expand.sh if you like.