BASH multiple nested IFs inside DO - bash

Question

The script I am trying to write needs to do the following
Go to specific directory and find all files with the *.mp4 extension that is newer than 30 minutes and move them into a variable.
All files within the variable needs to be uploaded to an FTP and an email notification goes out. The next if will need to look through the files and find all files with the extension of *1280x720_3500_h32.mp4 these files will then be uploaded to a different FTP server.
Lastly the script needs to move all files to a separate location on the network for archiving.
The process then needs to run every 30 minutes. Which is why I have added a separate script that runs the 30 minute sleep process and calls the initial script. So in effect I have a Upload script and a sleep script. So two questions springs to mind:
I am not sure about the 2nd IF where the script is looking for all the *1280x720_3500_h32.mp4 files - will it upload all files with this file name that is newer than 30 minutes
Race conditions where the script will try to upload files that have already been processed or will skip files?
Sleep script:
#!/bin/bash
ScripLocal='/Users/jonathan/scripts/Event_Scripts'
while true
do
echo "running Upload script: find_newerthan_mv_toFtp_v2.sh"
cd $ScripLocal
. find_newerthan_mv_toFtp_v2.sh
echo "finished running script: ffind_newerthan_mv_toFtp_v2.sh - next run is in 30 minutes"
sleep 30m
done
Upload Script:
#!/bin/bash
#File Origin
GSPORIGIN='/Volumes/HCHUB/Episode/Watchfolders/Output/Profile_01/'
#To location on the SAN where the files are being stored
DESTDIRSAN='/Volumes/HCHUB/Digitalmedia/online_encodes'
# 1st FTP Details
HOST='ftp.upload01.com'
USER='xxxx'
PASSWD='xxxx'
DESTDIR="/8619/_!/intlod/upload/"
#2nd FTP details
HOST01='ftp.upload02.com'
USER01='xxxx'
PASSWd01='xxxx'
cd $GSPORIGIN
for file in `find . -type f -name "*.mp4" -mmin -30`
do
echo $file
if [ -f $file ] ; then
echo "Uploading to FTP 01"
ftp -n -v $HOST << EOT
ascii
user $USER $PASSWD
prompt
cd $DESTDIR
mput $file
EOT
echo "$file has been copied to FTP 01" | mail -s "$file has been copied to FTP 01 in Directory $DESTDIR" xxxx#xxxx.com xxxx#xxxx.com xxxx#xxxx.com;
if [[ $file == *1280x720_3500_h32.mp4 ]] ; then
echo "Uploading FTP 02 version"
ftp -n -v $HOST01 << EOT
ascii
user $USER01 $PASSWD01
prompt
mput $file
EOT
echo "$file has been copied to FTP 02" | mail -s "$file has been copied to FTP 02" xxxx#xxxx.com;
fi
echo "moving files to SAN $DESTDIRSAN"
mv -v $file $DESTDIRSAN
else exit 1
fi
done

Answer 1

If you're worried about race conditions, don't use FTP. FTP is a very, very bad protocol for data exchange - it doesn't give you proper error messages when something goes wrong, the exit code is always 0 (even when something didn't work), there is no protection against duplicate or missed uploads, etc.
Use rsync instead. rsync will start writing the target file under a temporary name (so the receiver won't see it until the upload is complete), it can resume an upload (important for big files), it uses checksums to make sure the upload is correct and it fails with a non-null error code if anything is wrong.
On top of that, it can use ssh to secure the connection so you don't have to send plain text passwords over the network (which is a very bad idea today even in an Intranet).
rsync has powerful filtering options (what to upload) and can it can archive any processed file so you won't upload something twice. It can even detect when a user uploads a file a second time and fail (or silently skip if the file is 100% identical).
To make sure that you don't run two uploads at the same time, use a lock:
mkdir .lock || { echo "Can't lock - process is already running" ; exit 1 ; }
trap "rmdir .lock" EXIT
You might also want to have a look at FTAPI which does all this and more.

Answer 2

Use crontab, not sleeping threads. sleep is prone to failure, and nohupping is a hassle anyway.
Use rsync instead of looking for changes yourself, if you at all can. Also, you don't need to script rsync like you do with ftp.

Answer 3

Got the python variation of the script working (thank you Daniel!!).
#!/usr/bin/python
# Created in a rush, on the 31st October
# HALLOWEEN SPECIAL - QUICK AND DIRTY
import os
import shutil
from ftplib import FTP
# File Origin
ENCODE_OUTPUT='xxxxxxx'
# To location on the SAN where the files are being cached on the network
CACHE_DIR='xxxxxxxx'
#ENCODE_OUTPUT='encode/'
#CACHE_DIR='cache/'
#FTP01 FTP Details
FTP01_HOST='upload.FTP01.com'
FTP01_USER='xxxxx'
FTP01_PASSWD='xxxxxxx'
FTP01_DESTDIR="dir/dir/"
#Fingerprinting FTP details
FINGERPRINT_HOST='ftp.ftp02.com'
FINGERPRINT_USER='xxxxx'
FINGERPRINT_PASSWD='xxxxxx'
FINGERPRINT_DEST_DIR='/'
dirList = os.listdir(ENCODE_OUTPUT)
dirList.sort()
# Move the files out of the encoded located and into the cache dir
for video_file in dirList:
print "Moving file %s to %s" % (video_file, CACHE_DIR)
shutil.move(ENCODE_OUTPUT + video_file, CACHE_DIR)
# Upload the file to FTP01
print "Logging into %s FTP" % FTP01_HOST
ftp = FTP(FTP01_HOST)
ftp.login(FTP01_USER, FTP01_PASSWD)
ftp.cwd(FTP01_DESTDIR)
print "Uploading file %s to %s" % (video_file, FTP01_HOST+FTP01_DESTDIR)
ftp.storbinary('STOR ' + FTP01_DESTDIR+video_file, open(CACHE_DIR+video_file, "rb"), 1024)
# Close the connection
ftp.close
# If the file ands in "1280x720_3500_h32.mp4" copy it to the finger printing service
if video_file[-21:]=="1280x720_3500_h32.mp4":
# Upload the file to the fingerprinting service
print "File is HD. Logging into %s FTP" % FINGERPRINT_HOST
ftp = FTP(FINGERPRINT_HOST)
ftp.login(FINGERPRINT_USER, FINGERPRINT_PASSWD)
ftp.cwd(FINGERPRINT_DEST_DIR)
print "Uploading file %s to %s" % (video_file, FINGERPRINT_HOST)
ftp.storbinary('STOR ' + FINGERPRINT_DEST_DIR+video_file, open(CACHE_DIR+video_file, "rb"), 1024)
ftp.close