Script is not glob-expanding, but works fine when running the culprit as a minimalistic example - bash

I've been trying for hours on this problem, and cannot set it straight.
This minimal script works as it should:
#!/bin/bash
wipe_thumbs=1
if (( wipe_thumbs )); then
src_dir=$1
thumbs="$src_dir"/*/t1*.jpg
echo $thumbs
fi
Invoke with ./script workdir and a lot of filenames starting with t1* in all the sub-dirs of workdir are shown.
When putting the above if-case in the bigger script, the globbing is not executed:
SRC: -- workdir/ --
THUMBS: -- workdir//*/t1*.jpg --
ls: cannot access workdir//*/t1*.jpg: No such file or directory
The only difference with the big script and the minimal script is that the big script has a path-validator and getopts-extractor. This code is immediately above the if-case:
#!/bin/bash
OPTIONS=":ts:d:"
src_dir=""
dest_dir=""
wipe_thumbs=0
while getopts $OPTIONS opt ; do
case "$opt" in
t) wipe_thumbs=1
;;
esac
done
shift $((OPTIND - 1))
src_dir="$1"
dest_dir="${2:-${src_dir%/*}.WORK}"
# Validate source
echo -n "Validating source..."
if [[ -z "$src_dir" ]]; then
echo "Can't do anything without a source-dir."
exit
else
if [[ ! -d "$src_dir" ]]; then
echo "\"$src_dir\" is really not a directory."
exit
fi
fi
echo "done"
# Validate dest
echo -n "Validating destination..."
if [[ ! -d "$dest_dir" ]]; then
mkdir "$dest_dir"
(( $? > 0 )) && exit
else
if [[ ! -w "$dest_dir" ]]; then
echo "Can't write into the specified destination-dir."
exit
fi
fi
echo "done"
# Move out the files into extension-named directories
echo -n "Moving files..."
if (( wipe_thumbs )); then
thumbs="$src_dir"/*/t1*.jpg # not expanded
echo DEBUG THUMBS: -- "$thumbs" --
n_thumbs=$(ls "$thumbs" | wc -l)
rm "$thumbs"
fi
...rest of script, never reached due to error...
Can anyone shed some lights on this? Why is the glob not expanded in the big script, but working fine in the minimalistic test script?
EDIT: Added the complete if-case.

The problem is that wildcards aren't expanded in assignment statements (e.g. thumbs="$src_dir"/*/t1*.jpg), but are expanded when variables are used without double-quotes. Here's an interactive example:
$ src_dir=workdir
$ thumbs="$src_dir"/*/t1*.jpg
$ echo $thumbs # No double-quotes, wildcards will be expanded
workdir/sub1/t1-1.jpg workdir/sub1/t1-2.jpg workdir/sub2/t1-1.jpg workdir/sub2/t1-2.jpg
$ echo "$thumbs" # Double-quotes, wildcards printed literally
workdir/*/t1*.jpg
$ ls $thumbs # No double-quotes, wildcards will be expanded
workdir/sub1/t1-1.jpg workdir/sub2/t1-1.jpg
workdir/sub1/t1-2.jpg workdir/sub2/t1-2.jpg
$ ls "$thumbs" # Double-quotes, wildcards treated as literal parts of filename
ls: workdir/*/t1*.jpg: No such file or directory
...so the quick-n-easy fix is to remove the double-quotes from the ls and rm commands. But this isn't safe, as it'll also cause parsing problems if $src_dir contains any whitespace or wildcard characters (this may not be an issue for you, but I'm used to OS X where spaces in filenames are everywhere, and I've learned to be careful about these things). The best way to do this is to store the list of thumb files as an array:
$ src="work dir"
$ thumbs=("$src_dir"/*/t1*.jpg) # No double-quotes protect $src_dir, but not the wildcard portions
$ echo "${thumbs[#]}" # The "${array[#]}" idiom expands each array element as a separate word
work dir/sub1/t1-1.jpg work dir/sub1/t1-2.jpg work dir/sub2/t1-1.jpg work dir/sub2/t1-2.jpg
$ ls "${thumbs[#]}"
work dir/sub1/t1-1.jpg work dir/sub2/t1-1.jpg
work dir/sub1/t1-2.jpg work dir/sub2/t1-2.jpg
You might also want to set nullglob in case there aren't any matches (so it'll expand to a zero-length array).
In your script, this'd come out something like this:
if (( wipe_thumbs )); then
shopt -s nullglob
thumbs=("$src_dir"/*/t1*.jpg) # expanded as array elements
shopt -u nullglob # back to "normal" to avoid unexpected behavior later
printf 'DEBUG THUMBS: --'
printf ' "%s"' "${thumbs[#]}"
printf ' --\n'
# n_thumbs=$(ls "${thumbs[#]}" | wc -l) # wrong way to do this...
n_thumbs=${#thumbs[#]} # better...
if (( n_thumbs == 0 )); then
echo "No thumb files found" >&2
exit
fi
rm "${thumbs[#]}"
fi

Related

List the files in Directory and Copy-Replace them into another Directory in Linux

I am trying to automate the below: Any help, please.
We have 2 directories as mentioned below, whenever we get new files in Directory-1, only they should be copied and replaced into Directory-2. How to achieve this in Linux scripting. Filename remains the same but the version will be different.
Directory-1:
FileOne_2.0.0.txt
FileTwo_3.0.0.txt
Directory-2:
FileOne_1.0.0.txt
FileTwo_2.0.0.txt
FileThree_3.0.0.txt
FileFive_5.0.0.txt
Try this code (on a test setup before you trust your real directories and files with it):
#! /bin/bash -p
shopt -s extglob # Enable extended globbing ( +([0-9]) ... )
shopt -s nullglob # Globs that match nothing expand to nothing
shopt -s dotglob # Globs match files with names starting with '.'
srcdir='Directory-1'
destdir='Directory-2'
# A(n extended) glob pattern to match a version string (e.g. '543.21.0')
readonly kVERGLOB='+([0-9]).+([0-9]).+([0-9])'
# shellcheck disable=SC2231 # (Bad warning re. unquoted ${kVERGLOB})
for srcpath in "$srcdir"/*_${kVERGLOB}.txt; do
srcfile=${srcpath##*/} # E.g. 'FileOne_2.0.0.txt'
srcbase=${srcfile%_*} # E.g. 'FileOne'
# Set and check the path that the file will be moved to
destpath=$destdir/$srcfile
if [[ -e $destpath ]]; then
printf "Warning: '%s' already exists. Skipping '%s'.\\n" \
"$destpath" "$srcpath" >&2
continue
fi
# Make a list of the old versions of the file
# shellcheck disable=SC2206 # (Bad warning re. unquoted ${kVERGLOB})
old_destpaths=( "$destdir/$srcbase"_${kVERGLOB}.txt )
# TODO: Add checks that the number of old files (${#old_destpaths[*]})
# is what is expected (exactly one?)
# Move the file
if mv -i -- "$srcpath" "$destpath"; then
printf "Moved '%s' to '%s'\\n" "$srcpath" "$destpath" >&2
else
printf "Warning: Failed to move '%s' to '%s'. Skipping '%s'.\\n" \
"$srcpath" "$destpath" "$srcpath" >&2
continue
fi
# Remove the old version(s) of the file (if any)
for oldpath in "${old_destpaths[#]}"; do
if rm -- "$oldpath"; then
printf "Removed '%s'\\n" "$oldpath" >&2
else
printf "Warning: Failed to remove '%s'.\\n" "$oldpath" >&2
fi
done
done
The code is Shellcheck-clean. Two Shellcheck suppression comments are used because the unquoted expansions are necessary here.
srcdir and destdir are set to constant values. You might want to take them from command line parameters, or set them to different constant values.
The code could be made shorter by removing checks. However, moves and removes are destructive operations that can do a lot of damage if they are done incorrectly. I'd add even more checks if it was my own data.
See glob - Greg's Wiki for an explanation of the "extended globbing" used in the code.
See Parameter expansion [Bash Hackers Wiki] for an explanation of ${srcpath##*/} and ${srcfile%_*}.
mv -i is used as a double protection against overwriting an existing file.
All external commands are invoked with -- to explicitly end options, in case they are ever used with paths that begin with -.
Make sure that you understand the code and test it VERY carefully before using it for real.
source_dir=./files/0
dest_dir=./files/1/
for file in $source_dir/*
do
echo $file
echo "processing"
if [[ "1" == "1" ]]; then
mv $file $dest_dir
fi
done
Where processing and the 1 == 1 is whatever your 'prechecks' are (which you haven't told us)
If your coreutils sort is newer than or equal to v7.0 (2008-10-5) after which sort command
supports -V option (version-sort), would you please try:
declare -A base2ver base2file
# compare versions
# returns 0 if $1 equals to $2
# 1 if $1 is newer than $2
# -1 if $1 is older than $2
vercomp() {
if [[ $1 = $2 ]]; then
echo 0
else
newer=$(echo -e "$1\n$2" | sort -Vr | head -n 1)
if [[ $newer = $1 ]]; then
echo 1
else
echo -1
fi
fi
}
for f in Directory-1/*.txt; do
basename=${f##*/}
version=${basename##*_}
version=${version%.txt} # version number such as "2.0.0"
basename=${basename%_*} # basename such as "FileOne"
base2ver[$basename]=$version # associates basename with version number
base2file[$basename]=$f # associates basename with full filename
done
for f in Directory-2/*.txt; do
basename=${f##*/}
version=${basename##*_}
version=${version%.txt}
basename=${basename%_*}
if [[ -n ${base2ver[$basename]} ]] && (( $(vercomp "${base2ver[$basename]}" "$version") > 0 )); then
# echo "${base2file[$basename]} is newer than $f"
rm -- "$f"
cp -p -- "${base2file[$basename]}" Directory-2
fi
done

Looping over shell script arguments and passing quoted arguments to function

I have a script below that sources a directory of bash scripts and then parses the flags of the command to run a specific function from the sourced files.
Given this function within the scripts dir:
function reggiEcho () {
echo $1
}
Here are some examples of current output
$ reggi --echo hello
hello
$ reggi --echo hello world
hello
$ reggi --echo "hello world"
hello
$ reggi --echo "hello" --echo "world"
hello
world
As you can see quoted parameters are not honored as they should be `"hello world" should echo properly.
This is the script, the issue is within the while loop.
How do I parse these flags, and maintain passing in quoted parameters into the function?
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
STR="$(find $DIR/scripts -type f -name '*.sh' -print)"
ARR=( $STR )
TUSAGE="\n"
for f in "${ARR[#]}"; do
if [ -f $f ]
then
. $f --source-only
if [ -z "$USAGE" ]
then
:
else
TUSAGE="$TUSAGE \t$USAGE\n"
fi
USAGE=""
else
echo "$f not found"
fi
done
TUSAGE="$TUSAGE \t--help (shows this help output)\n"
function usage() {
echo "Usage: --function <args> [--function <args>]"
echo $TUSAGE
exit 1
}
HELP=false
cmd=()
while [ $# -gt 0 ]; do # loop until no args left
if [[ $1 = '--help' ]] || [[ $1 = '-h' ]] || [[ $1 = '--h' ]] || [[ $1 = '-help' ]]; then
HELP=true
fi
if [[ $1 = --* ]] || [[ $1 = -* ]]; then # arg starts with --
if [[ ${#cmd[#]} -gt 0 ]]; then
"${cmd[#]}"
fi
top=`echo $1 | tr -d -` # remove all flags
top=`echo ${top:0:1} | tr '[a-z]' '[A-Z]'`${top:1} # make sure first letter is uppercase
top=reggi$top # prepend reggi
cmd=( "$top" ) # start new array
else
echo $1
cmd+=( "$1" )
fi
shift
done
if [[ "$HELP" = true ]]; then
usage
elif [[ ${#cmd[#]} -gt 0 ]]; then
${cmd[#]}
else
usage
fi
There are many places in this script where you have variable references without double-quotes around them. This means the variables' values will be subject to word spitting and wildcard expansion, which can have various weird effects.
The specific problem you're seeing is due to an unquoted variable reference on the fourth-from-last line, ${cmd[#]}. With cmd=( echo "hello world" ), word splitting makes this equivalent to echo hello world rather than echo "hello world".
Fixing that one line will fix your current problem, but there are a number of other unquoted variable references that may cause other problems later. I recommend fixing all of them. Cyrus' recommendation of shellcheck.net is good at pointing them out, and will also note some other issues I won't cover here. One thing it won't mention is that you should avoid all-caps variable names (DIR, TUSAGE, etc) -- there are a bunch of all-caps variables with special meanings, and it's easy to accidentally reuse one of them and wind up with weird effects. Lowercase and mixed-case variables are safer.
I also recommend against using \t and \n in strings, and counting on echo to translate them into tabs and newlines, respectively. Some versions of echo do this automatically, some require the -e option to tell them to do it, some will print "-e" as part of their output... it's a mess. In bash, you can use $'...' to translate those escape sequences directly, e.g:
tusage="$tusage"$' \t--help (shows this help output)\n' # Note mixed quoting modes
echo "$tusage" # Note that double-quoting is *required* for this to work right
You should also fix the file listing so it doesn't depend on being unquoted (see chepner's comment). If you don't need to scan subdirectories of $DIR/scripts, you can do this with a simple wildcard (note lowercase vars and that the var is double-quoted, but the wildcard isn't):
arr=( "$dir/scripts"/*.sh )
If you need to look in subdirectories, it's more complicated. If you have bash v4 you can use a globstar wildcard, like this:
shopt -s globstar
arr=( "$dir/scripts"/**/*.sh )
If your script might have to run under bash v3, see BashFAQ #20: "How can I find and safely handle file names containing newlines, spaces or both?", or just use this:
while IFS= read -r -d '' f <&3; do
if [ -f $f ]
# ... etc
done 3< <(find "$dir/scripts" -type f -name '*.sh' -print0)
(That's my favorite it-just-works idiom for iterating over find's matches. Although it does require bash, not some generic POSIX shell.)

Why doesn't counting files with "for file in $0/*; let i=$i+1; done" work?

I'm new in ShellScripting and have the following script that i created based on a simpler one, i want to pass it an argument with the path to count files. Cannot find my logical mistake to make it work right, the output is always "1"
#!/bin/bash
i=0
for file in $0/*
do
let i=$i+1
done
echo $i
To execute the code i use
sh scriptname.sh /path/to/folder/to/count/files
$0 is the name with which your script was invoked (roughly, subject to several exceptions that aren't pertinent here). The first argument is $1, and so it's $1 that you want to use in your glob expression.
#!/bin/bash
i=0
for file in "$1"/*; do
i=$(( i + 1 )) ## $(( )) is POSIX-compliant arithmetic syntax; let is deprecated.
done
echo "$i"
That said, you can get this number more directly:
#!/bin/bash
shopt -s nullglob # allow globs to expand to an empty list
files=( "$1"/* ) # put list of files into an array
echo "${#files[#]}" # count the number of items in the array
...or even:
#!/bin/sh
set -- "$1"/* # override $# with the list of files matching the glob
if [ -e "$1" ] || [ -L "$1" ]; then # if $1 exists, then it had matches
echo "$#" # ...so emit their number.
else
echo 0 # otherwise, our result is 0.
fi
If you want to count the number of files in a directory, you can run something like this:
ls /path/to/folder/to/count/files | wc -l

Recursively list hidden files without ls, find or extendedglob

As an exercise I have set myself the task of recursively listing files using bash builtins. I particularly don't want to use ls or find and I would prefer not to use setopt extendedglob. The following appears to work but I cannot see how to extend it with /.* to list hidden files. Is there a simple workaround?
g() { for k in "$1"/*; do # loop through directory
[[ -f "$k" ]] && { echo "$k"; continue; }; # echo file path
[[ -d "$k" ]] && { [[ -L "$k" ]] && { echo "$k"; continue; }; # echo symlinks but don't follow
g "$k"; }; # start over with new directory
done; }; g "/Users/neville/Desktop" # original directory
Added later: sorry - I should have said: 'bash-3.2 on OS X'
Change
for k in "$1"/*; do
to
for k in "$1"/* "$1"/.[^.]* "$1"/..?*; do
The second glob matches all files whose names start with a dot followed by anything other than a dot, while the third matches all files whose names start with two dots followed by something. Between the two of them, they will match all hidden files other than the entries . and ...
Unfortunately, unless the shell option nullglob is set, those (like the first glob) could remain as-is if there are no files whose names match (extremely likely in the case of the third one) so it is necessary to verify that the name is actually a file.
An alternative would be to use the much simpler glob "$1"/.*, which will always match the . and .. directory entries, and will consequently always be substituted. In that case, it's necessary to remove the two entries from the list:
for k in "$1"/* "$1"/.*; do
if ! [[ $k =~ /\.\.?$ ]]; then
# ...
fi
done
(It is still possible for "$1"/* to remain in the list, though. So that doesn't help as much as it might.)
Set the GLOBIGNORE file to exclude . and .., which implicitly turns on "shopt -u dotglob". Then your original code works with no other changes.
user#host [/home/user/dir]
$ touch file
user#host [/home/user/dir]
$ touch .dotfile
user#host [/home/user/dir]
$ echo *
file
user#host [/home/user/dir]
$ GLOBIGNORE=".:.."
user#host [/home/user/dir]
$ echo *
.dotfile file
Note that this is bash-specific. In particular, it does not work in ksh.
You can specify multiple arguments to for:
for k in "$1"/* "$1"/.*; do
But if you do search for .* in directories , you should be aware that it also gives you the . and .. files. You may also be given a nonexistent file if the "$1"/* glob matches, so I would check that too.
With that in mind, this is how I would correct the loop:
g() {
local k subdir
for k in "$1"/* "$1"/.*; do # loop through directory
[[ -e "$k" ]] || continue # Skip missing files (unmatched globs)
subdir=${k##*/}
[[ "$subdir" = . ]] || [[ "$subdir" = .. ]] && continue # Skip the pseudo-directories "." and ".."
if [[ -f "$k" ]] || [[ -L "$k" ]]; then
printf %s\\n "$k" # Echo the paths of files and symlinks
elif [[ -d "$k" ]]; then
g "$k" # start over with new directory
fi
done
}
g ~neville/Desktop
Here the funky-looking ${k##*/} is just a fast way to take the basename of the file, while local was put in so that the variables don't modify any existing variables in the shell.
One more thing I've changed is echo "$k" to printf %s\\n "$k", because echo is irredeemably flawed in its argument handling and should be avoided for the purpose of echoing an unknown variable. (See Rich's sh tricks for an explanation of how; it boils down to -n and -e throwing a spanner in the works.)
By the way, this will NOT print sockets or fifos - is that intentional?

How to test filename expansion result in bash?

I want to check whether a directory has files or not in bash.
My code is here.
for d in {,/usr/local}/etc/bash_completion.d ~/.bash/completion.d
do
[ -d "$d" ] && [ -n "${d}/*" ] &&
for f in $d/*; do
[ -f "$f" ] && echo "$f" && . "$f"
done
done
The problem is that "~/.bash/completion.d" has no file.
So, $d/* is regarded as simple string "~/.bash/completion.d/*", not empty string which is result of filename expansion.
As a result of that code, bash tries to run
. "~/.bash/completion.d/*"
and of course, it generates error message.
Can anybody help me?
If you set the nullglob bash option, through
shopt -s nullglob
then globbing will drop patterns that don't match any file.
# NOTE: using only bash builtins
# Assuming $d contains directory path
shopt -s nullglob
# Assign matching files to array
files=( "$d"/* )
if [ ${#files[#]} -eq 0 ]; then
echo 'No files found.'
else
# Whatever
fi
Assignment to an array has other benefits, including desirable (correct!) handling of filenames/paths containing white-space, and simple iteration without using a sub-shell, as the following code does:
find "$d" -type f |
while read; do
# Process $REPLY
done
Instead, you can use:
for file in "${files[#]}"; do
# Process $file
done
with the benefit that the loop is run by the main shell, meaning that side-effects (such as variable assignment, say) made within the loop are visible for the remainder of script. Of course, it's also way faster, if performance is an issue.
Finally, an array can also be inserted in command line arguments (without splitting arguments containing white-space):
$ md5sum fileA "${files[#]}" fileZ
You should always attempt to correctly handle files/paths containing white-space, because one day, they will happen!
You could use find directly in the following way:
for f in $(find {,/usr/local}/etc/bash_completion.d ~/.bash/completion.d -maxdepth 1 -type f);
do echo $f; . $f;
done
But find will print a warning if some of the directory isn't found, you can either put a 2> /dev/null or put the find call after testing if the directories exist (like in your code).
find() {
for files in "$1"/*;do
if [ -d "$files" ];then
numfile=$(ls $files|wc -l)
if [ "$numfile" -eq 0 ];then
echo "dir: $files has no files"
continue
fi
recurse "$files"
elif [ -f "$files" ];then
echo "file: $files";
:
fi
done
}
find /path
Another approach
# prelim stuff to set up d
files=`/bin/ls $d`
if [ ${#files} -eq 0 ]
then
echo "No files were found"
else
# do processing
fi

Resources