Bash rsync script to recursively copy video & dir structure from FUSE? - bash

Question

I am trying to copy all video files recursively from a locally synced FUSE file system (GDrive, Insync, Cryptomator, Ubuntu 20.04), recreating the source directory structure in the destination for any video that is moved - only. Although it is working for some video others are left behind with the same file extension as some that were successfully moved.
Can I adjust my script somehow to make this work properly please? I was also using --ignore-existing but I removed it and have been testing with an empty source directory just in case that helped but it did not.
#!/bin/bash
# Define source and destination directories
src_dir="/home/user/cloud"
dest_dir="/home/user/testmove"
# Use rsync to move files and create directories recursively
rsync --progress --ignore-existing -avm -f'+ *[mM][pP]3$' -f'+ *[fF][lL][aA][cC]$' -f'+ *[mM][4][aA]$' -f'+ *[wW][aA][vV]$' -f'+ *[aA][iI][fF][fF]$' -f'+ *[pP][cC][mM]$' -f'+ *[aA][aA][cC]$' -f'+ *[oO][gG][gG]$' -f'+ *[aA][lL][aA][cC]$' -f'+ */' -f'- *' "$src_dir"/ "$dest_dir"/ --prune-empty-dirs
I found using the $ stopped any files being copied but I am testing following which seems to work, and Eric's much more comprehensive and flexible version from his answer below:
#!/bin/bash
# Define source and destination directories
src_dir="/home/user/cloud"
dest_dir="/home/user/testmove"
# Use rsync to copy files and create directories recursively
rsync -ahPv \
-f'+ *.[mM][pP]3' \
-f'+ *.[mM][4][aA]' \
-f'+ *.[wW][aA][vV]' \
-f'+ *.[oO][gG][gG]' \
-f'+ *.[wW][mM][aA]' \
-f'+ *.[fF][lL][aA][cC]' \
-f'+ *.[aA][iI][fF][fF]' \
-f'+ *.[pP][cC][mM]' \
-f'+ *.[aA][aA][cC]' \
-f'+ *.[aA][lL][aA][cC]' \
-f'+ */' \
-f'- *' \
"$src_dir"/ "$dest_dir" --prune-empty-dirs --stats -n
# To delete empty directories left in source when transferring:
# find "$src_dir" -depth -type d -empty -delete

Answer 1

There were some elements of the command structure which were conflicting, preventing the desired restrictions, and overlooking the inclusions. The following form provides the desired output.
Note that I have some hardcoded values that you need to remove.
#!/bin/bash
audio=1
video=0
DryRun=""
while [ $# -gt 0 ]
do
case $1 in
--source ) src_dir="$2" ; shift ; shift ;;
--target ) dest_dir="$2" ; shift ; shift ;;
--audio ) audio=1 ; video=0 ; shift ;;
--video ) audio=0 ; video=1 ; shift ;;
--dry ) DryRun="--dry-run --stats" ; shift ;;
* ) echo -e "\n Invalid option used on command line. Only valid options: [ --source {src_dir} | --target {dest_dir} | --audio | --video | --dry ]\n Bye!\n" ; exit 1 ;;
esac
done
# Define source and destination directories
#src_dir="/home/user/cloud"
src_dir="`pwd`/Output"
#dest_dir="/home/user/testmove"
dest_dir="`pwd`/Dupl"
# Define rules for include or exclude
cat >"EXCLUDES.filters" <<"EnDoFiNpUt"
- *
EnDoFiNpUt
#filters="--filter \"merge AUDIO_TYPES.filters\" " ### THIS FORM DOES NOT WORK !!!
if [ ${audio} -eq 1 ]
then
### NOTE: Trailing "$" to match end of string is implicit, unless there is a trailing "*"
cat >"AUDIO_TYPES.filters" <<"EnDoFiNpUt"
+ */
+ *\.[mM][pP][3]
+ *\.[fF][lL][aA][cC]
+ *\.[mM][4][aA]
+ *\.[wW][aA][vV]
+ *\.[aA][iI][fF][fF]
+ *\.[pP][cC][mM]
+ *\.[aA][aA][cC]
+ *\.[oO][gG][gG]
+ *\.[aA][lL][aA][cC]
EnDoFiNpUt
filters="--include-from=AUDIO_TYPES.filters --exclude-from=EXCLUDES.filters"
fi
if [ ${video} -eq 1 ]
then
### NOTE: Trailing "$" to match end of string is implicit, unless there is a trailing "*"
cat >"VIDEO_TYPES.filters" <<"EnDoFiNpUt"
+ */
+ *\.[mM][pP][4]
+ *\.[aA][vV][iI]
+ *\.[mM][kK][vV]
+ *\.[fF][lL][vV]
+ *\.[wW][mM][vV]
+ *\.[mM][oO][vV]
+ *\.[Aa][Vv][Cc][Hh][Dd]
+ *\.[wW][eE][bB][mM]
+ *\.[hH][2][6][4]
+ *\.[mM][pP][eE][gG][4]
+ *\.[Aa][Vv][Cc][Hh][Dd]
EnDoFiNpUt
filters="--include-from=VIDEO_TYPES.filters --exclude-from=EXCLUDES.filters"
fi
# Use rsync to move files and create directories recursively
### NOTE: I prefer long-form options for understanding code at first glance; shortform doesn't do that for me.
#rsync -rlptgoDvP ### shortform for universal options
rsync \
--verbose \
--recursive \
--links \
--owner \
--group \
--perms \
--times \
--devices \
--specials \
--prune-empty-dirs \
--partial \
--progress \
${DryRun}\
${filters} \
"${src_dir}/" "${dest_dir}/"
Log of session output:
building file list ...
3 files to consider
created directory /0__WORK/Dupl
./
DEF.mP4
0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/3)
abc.mp4
0 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=0/3)
sent 196 bytes received 126 bytes 644.00 bytes/sec
total size is 0 speedup is 0.00
Session log for version of script modified per requestor's (saltyeggs) reformulated command (also works, differently, but as expected):
building file list ...
3 files to consider
created directory /0__WORK/Dupl
./
DEF.mP4
abc.mp4
Number of files: 3 (reg: 2, dir: 1)
Number of created files: 3 (reg: 2, dir: 1)
Number of deleted files: 0
Number of regular files transferred: 2
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 128
Total bytes received: 94
sent 128 bytes received 94 bytes 444.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)