Sort files into sub folders by date - bash - bash

Question

Basically my HDD crashed, I was able to recover all the files, but, all the files have retained their meta & some have retained their names, I have 274000 images, which I need to more or less, sort into folders by date.
So let's say it starts with the first files, it would get the date from the file, create a sub folder, and until the date changes, keep moving that file into the created folder, once the date changes, it would create a new folder and keep doing the same thing.
I'm sure this is possible, I really didn't want to have to do this manually as it would take weeks...
Lets say I have a target folder /target/
Target contains, 274000 files, in no sub folders at all.
The folders structure should be /target/YY/DD_MM/filenames
I would like to create a bash script for this, but I'm not really sure where to proceed from here.
I've found this:
#!/bin/bash
DIR=/home/data
target=$DIR
cd "$DIR"
for file in *; do
dname="$( date -d "${file%-*}" "+$target/%Y/%b_%m" )"
mkdir -vp "${dname%/*}"
mv -vt "$dname" "$file"
done
Would creating a folder without checking if it exists delete files inside that folder?
I'm also not quite sure what adding an asterix to the dir pathname would do?
I'm not quite familiar with bash, but I'd love to get this working if someone could please explain to me a little more what's going on?
Thankyou!

Answer 1

I seemed to have found an answer that suited me, this worked on OSX just fine on three files, before I run it on the massive folder, can you guys just check that this isn't going to fail somewhere?
#!/bin/bash
DIR=/Users/limeworks/Downloads/target
target=$DIR
cd "$DIR"
for file in *; do
# Top tear folder name
year=$(stat -f "%Sm" -t "%Y" $file)
# Secondary folder name
subfolderName=$(stat -f "%Sm" -t "%d-%m-%Y" $file)
if [ ! -d "$target/$year" ]; then
mkdir "$target/$year"
echo "starting new year: $year"
fi
if [ ! -d "$target/$year/$subfolderName" ]; then
mkdir "$target/$year/$subfolderName"
echo "starting new day & month folder: $subfolderName"
fi
echo "moving file $file"
mv "$file" "$target/$year/$subfolderName"
done

Answer 2

I've had issues with the performance of the other solutions since my filesystem is remotely mounted and access times are big.
I've worked on some improved solutions in bash and python:
Bash version:
record # cat test.sh
for each in *.mkv
do
date=$(date +%Y-%d-%m -r "$each");
_DATES+=($date);
FILES+=($each);
done
DATES=$(printf "%s\n" "${_DATES[#]}" | sort -u);
for date in ${DATES[#]}; do
if [ ! -d "$date" ]; then
mkdir "$date"
fi
done
for i in ${FILES[#]}; do
dest=$(date +%Y-%d-%m -r "$i")
mv $i $dest/$i
done
record # time bash test.sh
real 0m3.785s
record #
Python version:
import os, datetime, errno, argparse, sys
def create_file_list(CWD):
""" takes string as path, returns tuple(files,date) """
files_with_mtime = []
for filename in [f for f in os.listdir(CWD) if os.path.splitext(f)[1] in ext]:
files_with_mtime.append((filename,datetime.datetime.fromtimestamp(os.stat(filename).st_mtime).strftime('%Y-%m-%d')))
return files_with_mtime
def create_directories(files):
""" takes tuple(file,date) from create_file_list() """
m = []
for i in files:
m.append(i[1])
for i in set(m):
try:
os.makedirs(os.path.join(CWD,i))
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
def move_files_to_folders(files):
""" gets tuple(file,date) from create_file_list() """
for i in files:
try:
os.rename(os.path.join(CWD,i[0]), os.path.join(CWD,(i[1] + '/' + i[0])))
except Exception as e:
raise
return len(files)
if __name__ == '__main__':
parser = argparse.ArgumentParser(prog=sys.argv[0], usage='%(prog)s [options]')
parser.add_argument("-e","--extension",action='append',help="File extensions to match",required=True)
args = parser.parse_args()
ext = ['.' + e for e in args.extension]
print "Moving files with extensions:", ext
CWD = os.getcwd()
files = create_file_list(CWD)
create_directories(files)
print "Moved %i files" % move_files_to_folders(files)
record # time python sort.py -e mkv
Moving files with extensions: ['.mkv']
Moved 319 files
real 0m1.543s
record #
Both scripts are tested upon 319 mkv files modified in the last 3 days.

Answer 3

I worked on a little script and tested it.Hope this helps.
#!/bin/bash
pwd=`pwd`
#list all files,cut date, remove duplicate, already sorted by ls.
dates=`ls -l --time-style=long-iso|grep -e '^-.*'|awk '{print $6}'|uniq`
#for loop to find all files modified on each unique date and copy them to your pwd
for date in $dates; do
if [ ! -d "$date" ]; then
mkdir "$date"
fi
#find command will find all files modified at particular dates and ignore hidden files.
forward_date=`date -d "$date + 1 day" +%F`
find "$pwd" -maxdepth 1 -not -path '*/\.*' -type f -newermt "$date" ! -newermt "$forward_date" -exec cp -f {} "$pwd/$date" \;
done
You must be in your working directory where your files to be copied according to date are present.