Windows 7/8/10 (x86/x64)
Window Batch
Hi all,
Firstly, the task I have been given:
A music library drive is backed up to a secondary drive every day using Robocopy.
Due to the possibility of deleting a file on the music library, the two cannot be mirror clones. Only different/new files are copied over to the backup drive.
Sometimes, it is discovered that a file(s) are in an incorrect folder on the music library and are moved to their correct folder. When the backup script is engaged, it then copies these 'new' files and their 'new' folder to the backup drive. The backup drive then has two copies of the file(s), one in it's correct 'new' position in the tree, and the other is the old erroneous one. The old is an unwanted duplicate.
I have so far managed to script the following:
Run through the latest Robocopy Log file (RClog), converting source file path to correct path on the backup drive. So it can be compared.
DIR the backup drive and dump to text file.
Put the converted RClog into an 'array' of variables.
Run through each line in the DIR list and compare the filename against the array. If it's a match, see if the full path matches. If not a fullpath match, then it's a possible* duplicate. Output full path to txt file.
So, I have my current code below.
It is painfully slow however. I think this is because I am performing close to 1 million actions! There are around 10k mp3 files, and about 100 newly copied files in the RClog.
I am thinking that I need to start, instead, with the smaller list of file, the RClog, and find those files in the DIR_List of the backup drive. However, I notice that FINDSTR doesn't like backlashes and needs them escaping. This would require a FOR loop to do variable substring substitution, which could slow things down some for both lists.
There is a final task I must perform, and that will be to MD5 hash the proposed unwanted duplicate file, and see if it's the same as the newly copied. *It could be that it happens to share the same name, but be from a different artist, so that needs to be checked. Some human intervention will likely be required at some point here, but for the most part automation is key, and quicker automation.
Whilst I really would prefer to keep this pure batch without 3rd party tools, I would consider using PowerShell as long as it's up to v3.0. I don't want to have to update things on the systems, and force changes. I am trying to work with what is there, and pure batch is available.
I apologise in advance for the rather casual commenting style.
Thanks in advance for any contributions.
::#ECHO OFF
#ECHO:
#ECHO: ===========================================================
#ECHO:
#ECHO: Music_library backup duplicate files finder
#ECHO:
#ECHO: Finds duplicate files to those included in the specified
#ECHO: robocopy log and lists them in a text file on the desktop
#ECHO:
#ECHO: ===========================================================
#ECHO:
:: Define Duplicates log
SET "DUPE_CHECK_LOG=%userprofile%\desktop\duplicates_log.txt"
:: Define Directory to scan
SET "IN_DIR=U:\Backups\Music_Library"
:: Define temporary DIR txt file
SET "TMP_DIR_TXT=%TMP%\Temp_Dir_list.txt"
:: FOR THE ROBOCCOPY LOG COPIED FILE PATH CONVERTER
:: Define Temporary robocopy log converted path file
SET "TMP_RC=%TMP%\Temp_RC_Conv.txt"
:: Define common root folder between both locations
:: copied from (in the Rc log) and backup location.
SET "CommonSplit=\Digital_Music_Library\"
:: END SECTION
:: Put rc log file path into a variable for easier reading
SET /P "RC_LOG=Drag and drop desired Robocopy Log here > "
:: strip rc log down to name.ext only
FOR /F "tokens=*" %%a IN ("%RC_LOG%") DO SET "RC_LOG_name=%%~nxa"
:: Print rc log name to new entry in dupes log
(
#ECHO:
#ECHO: %RC_LOG_name%
)>>"%DUPE_CHECK_LOG%"
:: Debug
::PAUSE
:: Delete old converted robocopy log if present
IF EXIST "%TMP_RC%" DEL /Q "%TMP_RC%"
:: Delete old dir txtif present
IF EXIST "%TMP_DIR_TXT%" DEL /Q "%TMP_DIR_TXT%"
:: Convert Robocopy Log to a list of those files in their new location
:: on the target drive. RC log lists 'from' location not 'to', so all
:: files wouldn't share the same path - different drive altogether.
:: Need to string path of copied files down to a common root folder
:: between the two locations. eg. Digital_Music_Library
FOR /F "tokens=3 delims= " %%a IN ('FINDSTR /C:"New File" "%RC_LOG%"') DO (
CALL :RC_CONVERTER "%%~fa"
)
:: Debug
::PAUSE
:: Take all of the data from text file and jam it into an array of variables
:: Each number of the var corresponds to that line in the text file...sheesh!
:: One var name for full path, the other for filenameext.
:: I just don't want yet another FOR loop to play with paramters/arguments - eg. %%~dpnxa
:: Shove this bit after the RC log is created, but really before anything else.
:: EnableDelayedExpansion must persist, can't ENDLOCAL on it yet, as it'll clear the var array!
SETLOCAL EnableDelayedExpansion
SET /A i=0
FOR /F "usebackq tokens=*" %%a IN ("%TMP_RC%") DO (
SET /A i+=1
SET "ArrayFullPath[!i!]=%%~fa"
SET "ArrayFilenameExt[!i!]=%%~nxa"
)
:: debug
::PAUSE
:: Dumps contents of music library into a txt file, all mp3s.
DIR /b/s "%IN_DIR%\*.mp3">>"%TMP_DIR_TXT%"
:: Loop through each line in Dir.txt checking if file is dupe
FOR /F "usebackq tokens=*" %%a IN ("%TMP_DIR_TXT%") DO (
CALL :CHECKER "%%~fa"
REM PAUSE
)
:: End out commands
:: Delete old converted robocopy log if present
IF EXIST "%TMP_RC%" DEL /Q "%TMP_RC%"
:: Delete old dir txtif present
IF EXIST "%TMP_DIR_TXT%" DEL /Q "%TMP_DIR_TXT%"
#ECHO:
#ECHO: Finished finding duplicates.
ENDLOCAL
PAUSE
GOTO :EOF
:CHECKER
:: Start at 1 and go to the maximum number of array vars created earlier.
:: For each one compare the found filename.ext from above against the filename.ext in the array
:: If it matches, make sure the path doesn't, and then print full path to duplicate file to a text file.
:: End subroutine and go back for the next found file.
FOR /L %%a IN (1,1,%i%) DO (
IF /I "!ArrayFilenameExt[%%a]!" == "%~nx1" (
IF /I NOT "!ArrayFullPath[%%a]!" == "%~f1" (
FOR /F "tokens=*" %%z IN ("%~f1") DO (
ECHO %%~fz>>"%DUPE_CHECK_LOG%"
)
)
)
)
EXIT /B
:RC_CONVERTER
:: Take the full path from the RC log sent as paramter/argument
:: Use variable substring substitution to remove everything before and
:: including the common folder.
:: Print out to text file target drive, path, common folder and remaining path of
:: copied files.
SET "TMP_STR=%~f1"
CALL SET "TempStr=%%TMP_STR:*%CommonSplit%=%%"
FOR /F "tokens=*" %%g in ("%TempStr%") DO (
#ECHO %IN_DIR%%CommonSplit%%%~g>>"%TMP_RC%"
)
EXIT /B
Related
I am just beginning to write my own batch scripts and currently I am trying to develop a small batch script to read a text file containing a list of specific file names with extensions one at a time. After reading the file name, search the hard drive for that file and if found, move it to a specific directory.
This is the code I have developed thus far. It works as long as the files you are looking for are in the same directory. The OS I am using is Windows 8.1.
#echo off
cls
echo.
echo.
echo.
for /F "delims=" %%a in (VIRUS_LIST.txt) do (
echo searching for %%a
echo.
For /R %%G in (%%a) do (
if exist %%a (
copy "%%G" "c:\vault"
echo moving file at %%G to c:\vault
echo.
)
)
)
pause
Here is how I would like it to work; 1. read the file name from the VIRUS_LIST.TXT file, 2. scan drive C for that file, 3. If found copy or move it to a directory on C drive called C:\vault.
The list file I am using is called VIRUS_LIST.TXT that holds the file names to search for, and if found they would be moved to C:\VAULT to be processed later. If anyone can offer any help it would be greatly welcomed.
I am new to windows batch scripting .. please help on this scenario
I have file structures as below ::
dir1:
c:\workspace\changeset\com\folder
subfolder1
one.java
subfolder-2
two.java
dir2:
c:\workspace\target\class\com\folder
subfolder1
one.class
subfolder2
two.class
Subfolder3
three.class
Need to find and replace dir1 files in respective subfolders i.e one.java and two.java from dir2 files i.e one.class and two.class ( need to replace certain .java files with .class files from specific folder )
much appreciated for your help.
Thanks
Arjun
for /f "delims=" %%a in ('dir /b /a-d "c:\workspace\changeset\com\folder\*.java"') do (
if exist "c:\workspace\target\class\com\folder\%%~na.class" (
echo copy "c:\workspace\target\class\com\folder\%%~na.class" "c:\workspace\changeset\com\folder\%%a")
)
The required COPY commands are merely ECHOed for testing purposes. After you've verified that the commands are correct, change ECHO COPY to COPY to actually copy the files. Append >nul to suppress report messages (eg. 1 file copied)
Note that execution directly from the prompt and as lines with a batch file are different. The metavariable (loop-control variable) %%x must be referenced as %%x for a batch line and %x if executed from the command prompt. Since it makes little sense to repeatedly execute a line containing a for from the prompt (it's easier to create a batch file), I post the batch-file version. User's responsibility to adjust for direct-from-prompt if desired.
Read each filename in /b basic form /a-d without directories and assign the filename+extension to %%a.
If a file in the other directory called thenamepartofthefile.class exists, then copy that file to the first directory.
Please post the entire problem to be solved. The approach can change radically, as in this case.
#ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
SET "sourcedir1=U:\sourcedir\changeset"
SET "sourcedir2=U:\sourcedir\target"
FOR /f "delims=" %%a IN (
'dir /b /s /a-d "%sourcedir1%\*.java" '
) DO (
SET "javadir=%%~dpa"
SET "classfile=!javadir:%sourcedir1%=%sourcedir2%!%%~na.class"
IF EXIST !classfile! ECHO COPY "!classfile!" "%%a"
)
GOTO :EOF
You would need to change the settings of sourcedir1 and sourcedir2 to suit your circumstances.
Essentially, the same approach and the same comments re messages. The difference is that this procedure uses files and subdirectories in the dir list and substitutes the first part of the directoryname in deriving the expected name of the .class file.
I know there is a way in linux to filter all file generated after a specific time.
But how could we do that in windows command line ? Or in bash .
For example, I have three files in a folder. After 10/10/2016, 12:12:54, a new files was generated into this folder and I need to know the new file's name,size and path.
Or,
I don't know when the new files will be generated. I want to check each 10 mins. If there are some new files generated after a specific , I can get the file's name, path and size.
I search something about that , I know I can use forfiles /P directory /S /D +08/01/2013 to do that. But it will displays all the files which are modified after 08/01/2013 under directory. But I want it displays the folders in directory and all files in directory folder(not in its sub directory).
Although you did not show any own efforts to solve your task, I decided to provide a script that returns a list of files created since the previous execution. It does not check the file creation time stamp, because date/time maths is not natively supported in pure batch-file solutions. Instead it generates a list of files, stores it in a temporary file and compares it with a previously saved list.
Opposed to relying on file time stamps, this will for sure recognise every new files. When checking the time stamps, files may be considered as new since the last run erroneously or new files may not be recognised erroneously, particularly such files that are created during execution of the script.
So here is the code:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem // Define constants here:
set "TARGET=D:\Data" & rem // (path to the directory to observe)
set "PATTERN=*.*" & rem // (search pattern for matching files)
set "LIST=%TEMP%\%~n0" & rem // (file base name of the list files)
set "FIRSTALL=#" & rem /* (defines behaviour upon first run:
rem set to anything to return all files;
rem set to empty to return no files) */
rem /* Determine which list file exists, ensure there is only one,
rem then toggle between file name extensions `.one`/`.two`: */
set "LISTOLD=%LIST%.two"
set "LISTNEW=%LIST%.one"
if exist "%LIST%.one" (
if not exist "%LIST%.two" (
set "LISTOLD=%LIST%.one"
set "LISTNEW=%LIST%.two"
) else (
erase "%LIST%.one"
if defined FIRSTALL (
> "%LIST%.two" rem/
) else (
erase "%LIST%.two"
)
)
) else (
if not exist "%LIST%.two" (
if defined FIRSTALL (
> "%LIST%.two" rem/
)
)
)
rem /* Create new list file, containing list of matching files
rem sorted by creation date in ascending order: */
> "%LISTNEW%" dir /B /A:-D /O:D /T:C "%TARGET%\%PATTERN%"
if not exist "%LISTOLD%" (
> nul 2>&1 copy /Y "%LISTNEW%" "%LISTOLD%"
)
rem // Search new list file for items not present in old one:
2> nul findstr /V /I /X /L /G:"%LISTOLD%" "%LISTNEW%"
if ErrorLevel 2 type "%LISTNEW%"
rem // Delete old list file:
erase "%LISTOLD%"
endlocal
exit /B
The first time the script runs, all files in the monitored directory are returned, unless you change set "FIRSTALL=#" to set "FIRSTALL=", in which case no files are returned the first time.
The core command is findstr which is configured so that the old list file provides literal search strings for being used to search the new list file and to return not matching lines, so the output are those lines of the new list file which do not occur in the old one.
Supposing the script is saved as new-files-since-last-run.bat, you could wrap around another tiny script that constitutes an endless loop with a polling rate of 10 minutes, like this:
#echo off
:LOOP
> nul timeout /T 600 /NOBREAK
call "%~dp0new-files-since-last-run.bat"
goto :LOOP
Setting up Windows Task Scheduler might be a better option though.
I have some folders with different names. Each folder has a specific structure as listed below:
Folder1
Contents
x64
Folder1.aaxplugin
TransVST_Fixer.exe
Folder 2
Contents
x64
Folder 2.aaxplugin
TransVST_Fixer.exe
There are two files within each subfolder x64. One file has the same name as the folder two folder levels above. The other file is an .exe file whose name is the same in all folders.
Now I need to run file with file extension aaxplugin on each specific .exe file. It would be obviously very time consuming opening each and every single folder and drag & drop each file on .exe to run it on this file.
That's why I am trying to create a batch script to save some time.
I looked for solutions here on Stack Overflow. The only thing I have found so far was a user saying this: When I perform a drag & drop, the process 'fileprocessor.exe' is executed. When I try to launch this exe, though, CMD returns error ('not recognized or not batch file' stuff).
How can I do this?
UPDATE 12/22/2015
I used first a batch file with following line to copy the executable into x64 subfolder of Folder1.
for /d %%a in ("C:\Users\Davide\Desktop\test\Folder1\*") do ( copy "C:\Program Files\Sugar Bytes\TransVST\TransVST_Fixer.exe" "%%a\x64\" 2> nul )
After asking here, I tried the following script:
for /f "delims=" %%F in ('dir /b /s x64\*.aaxplugin') do "%%~dpFTransVST_Fixer.exe" "%%F"
Unfortunately, the output is as following
C:\Users\Davide\Desktop>for /F "delims=" %F in ('dir /b /s x64\*.aaxplugin') do "%~dpFTransVST_Fixer.exe" "%F"
The system cannot find the file specified.
Try the following batch code:
#echo off
setlocal EnableDelayedExpansion
for /R "%USERPROFILE%\Desktop\test" %%F in (*.aaxplugin) do (
set "FilePath=%%~dpF"
if not "!FilePath:\x64\=!" == "!FilePath!" "%ProgramFiles%\Sugar Bytes\TransVST\TransVST_Fixer.exe" "%%F"
)
endlocal
The command FOR with option/R searches recursive in all directories of directory %USERPROFILE%\Desktop\test being expanded on your machine to C:\Users\Davide\Desktop for files with file extension aaxplugin. The loop variable F contains on each loop run the name of the found file with full path without surrounding double quotes.
The drive and path of each found file is assigned to environment variable FilePath.
Next a case-sensitive string comparison is done between file path with all occurrences of string \x64\ case-insensitive removed with unmodified file path.
Referencing value of environment variable FilePath must be done here using delayed expansion because being defined and evaluated within a block defined with ( ... ). Otherwise command processor would expand %FilePath% already on parsing the entire block resulting in a syntax error on execution because string substitution is not possible as no environment variable FilePath defined above body block of FOR loop.
The strings are not equal if path of file contains a folder with name x64. This means on provided folder structure that the file is in folder x64 and not somewhere else and therefore the application is executed next from its original location to fix the found *.aaxplugin file.
The line with IF is for the folder structure example:
if not "C:\Users\Davide\Desktop\test\Folder1\Contents" == "C:\Users\Davide\Desktop\test\Folder1\Contents\x64\"
if not "C:\Users\Davide\Desktop\test\Folder 2\Contents" == "C:\Users\Davide\Desktop\test\Folder 2\Contents\x64\"
So for both *.aaxplugin files the condition is true because the compared strings are not identical
Also possible would be:
#echo off
setlocal EnableDelayedExpansion
for /F "delims=" %%F in ('dir /A-D /B /S "%USERPROFILE%\test\*.aaxplugin" 2^>nul') do (
set "FilePath=%%~dpF"
if not "!FilePath:\x64\=!" == "!FilePath!" "%ProgramFiles%\Sugar Bytes\TransVST\TransVST_Fixer.exe" "%%F"
)
endlocal
But command DIR is not really necessary as it can be seen on first provided code.
But if the application TransVST_Fixer.exe for some unknown reason does its job right only with directory of file being also the current directory, the following batch code could be used instead of first code using the commands pushd and popd:
#echo off
setlocal EnableDelayedExpansion
for /R "%USERPROFILE%\test" %%F in (*.aaxplugin) do (
set "FilePath=%%~dpF"
echo !FilePath!
if /I "!FilePath:~-5!" == "\x64\" (
pushd "%%~dpF"
"%ProgramFiles%\Sugar Bytes\TransVST\TransVST_Fixer.exe" "%%~nxF"
popd
)
)
endlocal
There is one more difference in comparison to first code. Now the last 5 characters of path of file are compared case-insensitive with the string \x64\. Therefore the file must be really inside a folder with name x64 or X64. A folder with name x64 or X64 anywhere else in path of file does not result anymore in a true state for the condition as in first two batch codes.
But if for some unknown reason it is really necessary to run the application in same folder as the found *.aaxplugin and the directory of the file must be the current directory, the following batch code could be used:
#echo off
setlocal EnableDelayedExpansion
for /R "%USERPROFILE%\test" %%# in (*.aaxplugin) do (
set "FilePath=%%~dp#"
if /I "!FilePath:~-5!" == "\x64\" (
pushd "%%~dp#"
"%%~dp#TransVST_Fixer.exe" "%%~nx#"
popd
)
)
endlocal
The path of the file referenced with %%~dpF always ends with a backslash which is the reason why there is no backslash left of TransVST_Fixer.exe (although command processor could handle also file with with two backslashes in path).
In batch code above character # is used as loop variable because %%~dp#TransVST_Fixer.exe is easier to read in comparison to %%~dpFTransVST_Fixer.exe. It is more clear for a human with using # as loop variable where the reference to loop variable ends and where name of application begins. For the command processor it would not make a difference if loop variable is # or upper case F.
A lower case f would work here also as loop variable, but is in general problematic as explained on Modify variable within loop of batch script.
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
dir /?
echo /?
endlocal /?
for /?
if /?
popd /?
pushd /?
set /?
setlocal /?
Your question isn't quite clear, but it seems, something like this should work:
for /f "delims=" %%f in ('dir /b /s X64\*.ext') do "%%~dpfMyExe.exe" "%%f"
Maybe you have to change directory to each folder (depends on your .exe):
for /f "delims=" %%d in ('dir /B /ad') do (
pushd "%%d"
for /f "delims=" %%f in ('dir /b "contents\x64\*.ext"') do (
cd Contents\x64
MyExe.exe "%%f"
)
popd
)
Assuming:
The Directory structure is fixed and the files are indeed in a subfolder contents\X64\.
MyExe.exe is the same (name) in every folder.
There is only one file *.ext in every folder.
I'll give you the script I created for doing so, hope it works for you
for /d %%d IN (./*) do (cd "%%d/Contents/x64" & "../../../TransVST_Fixer.exe" "%%d" & cd "/Program Files (x86)\Common Files\Avid\Audio\Plug-Ins")
Please note that I placed the fixer inside the root folder so I just have to copy it once. You have to place it inside your root folder and execute it. What it does:
iterate over each folder
for each one it enters /Contents/x64, executes the fixer (wich is 3 levels above) and after that returns to the original folder.
If you have your plugins in a different folder, you just have to change this part replacing the path for the one you have your plugins in.
cd "/Program Files (x86)\Common Files\Avid\Audio\Plug-Ins"
REMEMBER to place the script on that folder. For this example I place my script on the folder "/Program Files (x86)\Common Files\Avid\Audio\Plug-Ins" and run it (as a .bat).
PS: the fixer will place the fixed plugins in "C:\Users\Public\modified" (just read the screen while executing, it gives you the new files path. If you want to move them to the right path, you can execute this from the new files path ("C:\Users\Public\modified")
for %%d IN (*.aaxplugin) do (mkdir "%%d_temp/Contents\x64" & move "%%d" "%%d_temp/Contents\x64/%%d" & rename "%%d_temp" "%%d")
with that, I iterate over every plugin and create a folder with the same name (I create _temp because of name colision, after moving the file I rename it to the correct one), also with the subfolder "/Contents/x64", and move the plugin inside. Once donde, you can just take the resulting folders and place them in their correct path.
Hope it works, for me it works like a charm.
I want to parse a directory and its subdirectories and copy all files in a target directory ignoring the original folder structure. All files shall be copied directly in the target directory.
I found a nice solution here for the Windows command line:
for /r FolderA %f in (*) do #copy "%f" target
https://stackoverflow.com/a/1502183/772434
That in general works fine, but I have duplicate files in the source directory which have the same name. There are too many files, so I can not handle those exceptions by hand.
How can I handle those files in the script automatically?
Options:
overwrite files during copying
keep copies and rename file e. g. by adding "__123" a number at the end of the file name
compare files (MDS5 sum or similar) and create copy only if files are different.
I would prefer option 3 (compare file content and keep numbered copies only if files are really different), but I'm also interested in solutions for options 1 and 2 (more pragmatic).
Here's an approach. i haven't tested it, so i hope it works as expected.
There is a global variable called CURRENT_DUPLICATE_SUFFIX. every time a file with the same name is found, but with different content, this value gets incremented and appended to the resulting file name
#echo off
SET SOURCE_DIR=C:\temp\LogAnalyzer
SET TARGET_DIR=C:\temp\test\output
SET CURRENT_DUPLICATE_SUFFIX=1
for /r "%SOURCE_DIR%" %%f in (*) do CALL :SUB1 "%%f"
GOTO END
:SUB1
SET PURE_FILE_NAME=%~nx1
SET TARGET_FILE_NAME=%TARGET_DIR%\%PURE_FILE_NAME%
IF NOT EXIST "%TARGET_FILE_NAME%" (
COPY %1 "%TARGET_DIR%"
GOTO :EOF
)
REM if we are here, the target file exists.
echo n |comp %1 "%TARGET_FILE_NAME%" >NUL 2>NUL
REM in case the files have the same content
IF %ERRORLEVEL%==0 GOTO :EOF
SET TARGET_FILE_NAME=%TARGET_FILE_NAME%_%CURRENT_DUPLICATE_SUFFIX%
SET /A CURRENT_DUPLICATE_SUFFIX=%CURRENT_DUPLICATE_SUFFIX%+1
COPY %1 "%TARGET_FILE_NAME%
GOTO :EOF
:END