Fill backup drive with newest files - windows

Question

I have about 12TB (and growing) library, distributed over 3 HDDs, of video files and I would like to back them up to an external harddrive. I am using Windows 10 Pro.
My backup drive has only 8TB and I would like to always backup the newest 8TB of video files. So far I have tried about 10 sync tools but none of them allowed me to copy files according to creation date.
Also with robocopy I haven't found a way to copy only the latest 8TB of files.
Any suggestions?

Answer 1

I have 2 batch scripts for you that do what you have asked for. The main difference between them is, that the 1st script is using where to locate all the files which will use the timestamp of the last change. To use another timestamp e.g. last access or time of creation you have to use the 2nd script I attached which uses dir.
1.
Batch script using where to locate files:
It takes minimum 2 arguments:
Usage: batch.bat <dst> <src_1> [<src_...> <src_n>]
It uses the where /r <src_n> * /t command which will build a list of all files in all subdirectories with a timestamp of the last change in the following format:
<size> <date> <time> <file name>
5397 11.07.2017 14:32:09 C:\Users\...\foo.txt
10860 12.07.2017 11:25:15 C:\Users\...\bar.log
The timestamp is of last change if you need the time of the creation or last access take the 2nd batch script below which is using dir as there is the possibility to choose between different timestamps.
This output will be written (without the column size) into a temporary file under the temp dir %TEMP% (will be deleted automatically after script) for every source that is passed via the argument list. The complete temporary file is sorted by the date and time, newest first.
This sorted output will be used to copy it into the destination folder.
Example:
Current directory in cmd is C:\...\Desktop):
batch.bat "backup_folder" "F:\folder" "F:\folder with space"
Current directory somewhere:
batch.bat "C:\...\Desktop\backup_folder" "F:\folder" "F:\folder with space"
The C:\...\Desktop\backup_folder will contain 2 folders folder and folder with space which contain all files of these source folders after the scripts operation.
In your case the batch script would copy only 8 TB of the newest files because then the drive will be full and the batch script will exit because it recognizes copy errors.
The batch script:
#echo off
setlocal enabledelayedexpansion
rem Check if at least 2 arguments are passed
set INVARGS=0
if [%1] == [] set INVARGS=1
if [%2] == [] set INVARGS=1
if %INVARGS% == 1 (
echo Usage: %0 ^<dst^> ^<src_1^> ^<src_...^> ^<src_n^>
goto :EOF
)
rem Store Carriage return character in CR (used for spinner function, see below)
for /f %%a in ('copy /Z "%~dpf0" nul') do set "CR=%%a"
rem Set temp file name
set "uniqueFile=%TEMP%\%0_%RANDOM%.tmp"
rem Set destination path and shift to next argument
set "dstpath=%~1"
shift
rem Store src_1 to src_n arguments into src array
set idx=0
:again
rem If %1 is blank, we are finished
if not [%1] == [] (
if not exist "%~1" (
echo The following source folder doesn't exist:
echo "%~1"!
echo The program will terminate!
goto :end
)
pushd "%~1"
set "src[%idx%]=!cd!"
if [!src[%idx%]:~-1!] == [\] (
set src[%idx%]=!src[%idx%]:~0,-1!
)
set /a idx=%idx%+1
popd
rem Shift the arguments and examine %1 again
shift
goto :again
)
rem Build command string:
rem where /r "src_1" * /t ^& where /r "src_..." * /t ^& where /r "src_n" * /t ^& break"
set "command="
for /F "tokens=2 delims==" %%s in ('set src[') do (
set "command=!command!where /r "%%s" * /t ^& "
)
set "command=!command!break"
echo "Command: ^<!command!^>"
rem Clear temp file and write files to copy in it
break>"%uniqueFile%"
for /f "tokens=2,3,4,* delims= " %%F in ('!command!') do (
call :spinner
echo %%F %%G %%H %%I>>"%uniqueFile%"
)
rem Open the built file list
echo File list will be opened...
%uniqueFile%
rem Ask user if copying should start
echo Should this files copied into %dstpath%? (Y/N)
set INPUT=
set /P INPUT=Answer?:%=%
if not [%INPUT%] == [y] (
if not [%INPUT%] == [Y] (
goto :end
)
)
set "dstpathPart="
rem Sort files newest first (last changed timestamp)
for /f "tokens=3,* delims= " %%F in ('type "%uniqueFile%" ^| sort /r') do (
rem Build destination part from source folder names
call :buildDestinationPath dstpathPart "%%F %%G"
rem Prepend the destination folder
set "dstFile=!dstpath!\!dstpathPart!"
rem Make directories that doesn't exists
for %%F in ("!dstFile!") do set "directory=%%~dpF"
if not exist "!directory!" ( mkdir "!directory!" )
rem Copy files and echo it
echo copy /y "!file!" "!dstFile!"
copy /y "!file!" "!dstFile!"
echo.
rem If copying failed exit program
if errorlevel 1 (
echo Copying failed... Maybe there is no more space on the disk!
echo The program will terminate!
goto :end
)
)
goto :end
rem Function definitions
:buildDestinationPath
for /F "tokens=2 delims==" %%s in ('set src[') do (
rem Go into folder e.g.: C:\src_1\ --> C:\
pushd "%%s"
cd ..
rem file contains full path of the where command
set "file=%~2"
rem Remove trailing space
if "!file:~-1!" == " " (
call set "file=%%file:~0,-1%%"
)
rem remove src folder from file to make it relative
rem e.g. C:\src_1\foo\bar\file1.txt --> src_1\foo\bar\file1.txt
call set "dstpathPart_temp=%%file:!cd!=%%"
rem Switch back to origin folder
popd
rem If the folder name changed the substring taken was right so take next
if not [!dstpathPart_temp!] == [!file!] (
set "%~1=!dstpathPart_temp!"
goto :next
)
)
:next
goto :EOF
:spinner
set /a "spinner=(spinner + 1) %% 4"
set "spinChars=\|/-"
<nul set /p ".=Building file list... !spinChars:~%spinner%,1!!CR!"
goto :EOF
:end
del "%uniqueFile%"
goto :EOF
2.
Batch script using dir to locate files:
It takes one more argument so that you need minimum 3 arguments:
Usage: batch.bat <timeordering> <dst> <src_1> [<src_...> <src_n>]
It will generate the same list as explained above but with the help of the dir command. The dir command takes the following parameter which is the <timeordering> parameter of the script:
/tc Creation
/ta Last access
/tw Last written
The batch script:
#echo off
setlocal enabledelayedexpansion
rem Check if at least 2 arguments are passed
set INVARGS=0
if [%1] == [] set INVARGS=1
if [%2] == [] set INVARGS=1
if [%3] == [] set INVARGS=1
if %INVARGS% == 1 (
echo Usage: %0 ^<timeordering^> ^<dst^> ^<src_1^> ^<src_...^> ^<src_n^>
goto :EOF
)
rem Store Carriage return character in CR (used for spinner function, see below)
for /f %%a in ('copy /Z "%~dpf0" nul') do set "CR=%%a"
rem Set temp file name
set "uniqueFile=%TEMP%\%0_%RANDOM%.tmp"
rem Set timeordering and destination path and shift to next argument
set "timeordering=%~1"
shift
set "dstpath=%~1"
shift
rem Store src_1 to src_n arguments into src array
set idx=0
:again
rem If %1 is blank, we are finished
if not [%1] == [] (
if not exist "%~1" (
echo The following source folder doesn't exist:
echo "%~1"!
echo The program will terminate!
goto :end
)
pushd "%~1"
set "src[%idx%]=!cd!"
if [!src[%idx%]:~-1!] == [\] (
set src[%idx%]=!src[%idx%]:~0,-1!
)
set /a idx=%idx%+1
popd
rem Shift the arguments and examine %1 again
shift
goto :again
)
rem Clear temp file and write files to copy in it
break>"%uniqueFile%"
rem call commands with all sources:
rem call :getFileInformation /TC "src_1\*"
rem /TW
rem /TA
for /F "tokens=2 delims==" %%s in ('set src[') do (
call :getFileInformation %timeordering% "%%s\*" "%uniqueFile%""
)
rem Open the built file list
echo Unsorted file list will be opened...
%uniqueFile%
rem Ask user if copying should start
echo Should this files copied into %dstpath%? (Y/N)
set INPUT=
set /P INPUT=Answer?:%=%
if not [%INPUT%] == [y] (
if not [%INPUT%] == [Y] (
goto :end
)
)
set "dstpathPart="
rem Sort files newest first (last changed timestamp)
for /f "tokens=3,* delims= " %%F in ('type "%uniqueFile%" ^| sort /r') do (
rem Build destination part from source folder names
call :buildDestinationPath dstpathPart "%%F %%G"
rem Prepend the destination folder
set "dstFile=!dstpath!\!dstpathPart!"
rem Make directories that doesn't exists
for %%F in ("!dstFile!") do set "directory=%%~dpF"
if not exist "!directory!" ( mkdir "!directory!" )
rem Copy files and echo it
echo copy /y "!file!" "!dstFile!"
copy /y "!file!" "!dstFile!"
echo.
rem If copying failed exit program
if errorlevel 1 (
echo Copying failed... Maybe there is no more space on the disk!
echo The program will terminate!
goto :end
)
)
goto :end
rem Function definitions
:buildDestinationPath
for /F "tokens=2 delims==" %%s in ('set src[') do (
rem Go into folder e.g.: C:\src_1\ --> C:\
pushd "%%s"
cd ..
rem file contains full path of the where command
set "file=%~2"
rem Remove trailing space
if "!file:~-1!" == " " (
call set "file=%%file:~0,-1%%"
)
rem remove src folder from file to make it relative
rem e.g. C:\src_1\foo\bar\file1.txt --> src_1\foo\bar\file1.txt
call set "dstpathPart_temp=%%file:!cd!=%%"
rem Switch back to origin folder
popd
rem If the folder name changed the substring taken was right so take next
if not [!dstpathPart_temp!] == [!file!] (
set "%~1=!dstpathPart_temp!"
goto :next
)
)
:next
goto :EOF
:getFileInformation
for /f "delims=" %%a in ('dir /s /b /a-d %~1 "%~2"') do (
for /f "tokens=1,2 delims= " %%b in ('dir %~1 "%%a" ^| findstr /l /c:"%%~nxa"') do (
echo %%b %%c %%a>>"%~3"
call :spinner
)
)
goto :EOF
:spinner
set /a "spinner=(spinner + 1) %% 4"
set "spinChars=\|/-"
<nul set /p ".=Building file list... !spinChars:~%spinner%,1!!CR!"
goto :EOF
:end
del "%uniqueFile%"
goto :EOF