How to avoid an error when listing nonexistent files? - bash

Question

I'm grouping files by dates in the filenames and processing them by groups.
for m in {01..12}; do
for d in {01..31}; do
f=`ls ./mydir/2018.${m}.${d}T*.jpg`
# process files
done
done
However, the code raises error if no files exist for some dates, e.g.,
ls: cannot access '2018.01.20T*.jpg': No such file or directory
How can I skip missing dates?

Answer 1

Enable nullglob so non-matching wildcards expand to nothing. Then you can skip parsing ls altogether and simply iterate over the matching files.
shopt -s nullglob
for m in {01..12}; do
for d in {01..31}; do
for f in ./mydir/2018.${m}.${d}T*.jpg; do
# process file
done
done
done
If you want all the file names at once, store them in an array. Arrays are better than a plain strings because they can handle file names with spaces and other special characters.
shopt -s nullglob
for m in {01..12}; do
for d in {01..31}; do
files=(./mydir/2018.${m}.${d}T*.jpg)
# process files
echo "processing ${files[#]}..."
done
done
What's the cleanest way to localise the shopt so as to restore nullglob to its original (unknown) value after this block?
Use a subshell: surround the block with parentheses. A subshell creates a child process which ensures changes don't leak into the parent.
(
shopt -s nullglob
...
)
It's polite to do this whenever you're changing shell options, and it's an elegant alternative to pushd+popd. Note that any variable assignments will be local to the subshell, so be careful there.

Answer 2

Here is another way, using find:
Assume the following dir:
$ ls -l mydir/
total 0
-rw-r--r-- 1 0 Jan 23 16:46 2018.01.20Thellowet.jpg
-rw-r--r-- 1 0 Jan 23 16:47 2018.04.24Thellowet.jpg
-rw-r--r-- 1 0 Jan 23 16:46 some_random_crap
-rw-r--r-- 1 0 Jan 23 16:46 wet
-rw-r--r-- 1 0 Jan 23 16:46 when
-rw-r--r-- 1 0 Jan 23 16:46 who
-rw-r--r-- 1 0 Jan 23 16:46 wtf
Using find:
find ./mydir/ -type f -regextype sed -regex ".*2018\.[0-9]\{,2\}\.[0-9]\{,2\}T.*\.jpg.*" -exec echo "---{}" \;
Gives (minor processing of data by appending --- to the file name):
---./mydir/2018.04.24Thellowet.jpg
---./mydir/2018.01.20Thellowet.jpg
NOTE: This will also return files that have 2018.00.xy or 2018.xy.00 where x and y can be any number between 0 and 9
Regex explained:
.* : any pattern
[0-9]{,2}: a 2 digit number
The \ are used to escape special characters.