Batch: For directories not working with delayed expansion - windows

I have a CI system that regularly publishes new files.
In a script, I want to take the latest version of several (separately published) files. The general file structure is the following: C:\Path\to\publish\<token>\<flavor>\file. There are 3 variable parts in this, per published file. (And all 3 can in some cases have wildcards in it)
The setup I currently have, makes use of delayed expansion to make this configurable:
set A_PATH=C:\Path\to\publish\*
set A_SUBDIR=<flavor>
set A_NAME=file_*.txt
call :searchFile A
goto :logicUsingA
:searchFile
SETLOCAL ENABLEDELAYEDEXPANSION
set SEARCH_PATH=%~1_PATH
set SEARCH_NAME=%~1_NAME
set SEARCH_SUBDIR=%~1_SUBDIR
FOR /D %%I IN ('DIR !%SEARCH_PATH%! /O:-D /B') do (
echo I:%%I
FOR %%J IN (!%SEARCH_SUBDIR%!) do (
echo J:%%J
FOR /F "delims=" %%K IN ('DIR %%I\%%J\!%SEARCH_NAME%! /A:-D /O:-D /B') do (
echo K:%%K
ENDLOCAL
set "%~1=%%I\%%J\%%K"
GOTO :EOF
)
)
) 2> nul
This system almost did what it needed to do, namely, based on a configuration pick some files and fill a variable with them.
Unfortunately, we discovered that something goes wrong with executing this script.
The FOR /D %%I IN ('DIR !%SEARCH_PATH%! /O:-D /B') should sort the directories with the newest first. However, the print below it, prints: (assume 001 was created before 002 ...)
I:'DIR
I:C:\Path\to\publish\001
I:C:\Path\to\publish\002
I:C:\Path\to\publish\003
This makes me suspect that the delayed expansion doesn't play nice with the for-loop, as this is the only noticeable difference with other solutions I find at stackoverflow, like https://stackoverflow.com/a/45104510/2466431
Would it be possible to instruct the for-loop to execute this dir-command iso treating it as a list of items?
EDIT: What the code is trying to achieve is a windows batch variant of the bash one-liner: ls -t C/Path/to/publish/*/<flavor>/file_*.txt

The loop for /D%%I in ('dir !%SEARCH_PATH%! /O:-D /B') do ( ... ) is wrong as you must use for /F to capture and parse the output of a command like dir in our situation. So you may either use for /D %%I in ("!%SEARCH_PATH%!") do ( ... ) or for /F "delims=" %%I in ('dir "!%SEARCH_PATH%!" /O:-D /B') do ( ... ) to be syntactically correct.
But neither will probably be useful for you, because:
for /D does not access the file system unless wildcards are present;
for /D does not provide sort options (it returns directories as it gets them from the file system);
dir, when a directory name is given, lists its contents rather than just returning its name;
Nevertheless, there is a way to work around the said problem with dir, namely appending \ and trying to list the contents of the given directory; if wildcards are given, dir returns an error; if a dedicated directory name is given but it does not exist, dir returns an error too; conditional execution can be applied to make use of that behaviour:
dir /B /A:D "%SomeDir%\" && (echo "%SomeDir%") || dir /B /A:D /O:-D /T:C "SomeDir%"
This provides the following results:
if %SomeDir% points to a dedicated existing directory, the first dir command lists its contents, so it succeeds, hence the echo command is executed, returning the (quoted) variable contents, but the second dir command is skipped; (quotation of the echo string is done here in order to protect whitespaces or other special characters; the surrounding quotation marks "" are no problem though as they can easily be removed later anyway;)
if %SomeDir% points to a dedicated directory that cannot be found, the first dir command fails (error: The system cannot find the file specified.), hence the echo command is skipped, but the second dir command is executed, which also fails (error: File Not Found);
if %SomeDir% contains a wildcard in the last path element, the first dir command fails (error: The filename, directory name, or volume label syntax is incorrect.), hence the echo command is skipped, but the second dir command is executed, which lists all matching directory names, sorted by creation date/time (newest first);
The listing returned by the first dir command as well as any error messages can be suppressed using redirection, so the remaining output is the one by echo or by the second dir:
> nul 2>&1 dir /B /A:D "%SomeDir%\" && (echo "%SomeDir%") || 2> nul dir /B /A:D /O:-D /T:C "SomeDir%"
This can now be applied to your script:
#echo off
set "A_PATH=C:\Path\to\publish\*"
set "A_SUBDIR=<flavor>"
set "A_NAME=file_*.txt"
call :searchFile A
echo A:%A%
rem goto :logicUsingA
goto :EOF
:searchFile
set "%~1="
setlocal EnableDelayedExpansion
set "SEARCH_PATH=%~1_PATH"
set "SEARCH_NAME=%~1_NAME"
set "SEARCH_SUBDIR=%~1_SUBDIR"
cd /D "!%SEARCH_PATH%!\.." || exit /B 1
for /F "delims=" %%I in ('
^> nul 2^>^&1 dir /B /A:D "!%SEARCH_PATH%!\" ^
^&^& ^(echo "!%SEARCH_PATH%!"^) ^
^|^| 2^> nul dir /B /A:D /O:-D /T:C "!%SEARCH_PATH%!"
') do (
echo I:%%~nxI
for /F "delims=" %%J in ('
^> nul 2^>^&1 dir /B /A:D "%%~nxI\!%SEARCH_SUBDIR%!\" ^
^&^& ^(echo "!%SEARCH_SUBDIR%!"^) ^
^|^| 2^> nul dir /B /A:D /O:-D /T:C "%%~nxI\!%SEARCH_SUBDIR%!"
') do (
echo J:%%~J
for /F "delims=" %%K in ('
^> nul 2^>^&1 dir /B /A:-D "%%~nxI\%%~J\!%SEARCH_NAME%!\" ^
^|^| 2^> nul dir /B /A:-D /O:-D /T:C "%%~nxI\%%~J\!%SEARCH_NAME%!"
') do (
echo K:%%K
endlocal
set "%~1=%CD%\%%~nxI\%%~J\%%K"
goto :EOF
)
)
)
goto :EOF
In the inner-most loop that enumerates files rather than directories, I applied the same technique, but I skipped the echo command as it should never be executed anyway; the only reason why I kept the two dir commands here is to handle the situation when you specify a certain file name (no wildcards) but there is actually a directory with that name (a single dir would then unintentionally return its contents).
There have also a few other things changed:
the variable %~1 is initially cleared (in the sub-routine :searchFile, just in case no file could be found at all);
the quoted set syntax is consistently used (set "VAR=Value");
all paths are now quoted to avoit trouble with whitespaces or other special characters;
the dir option /T:C is added to regard the creation date/time rather than the date/time of the last modification; just remove it if you want to regard the latter;
cd /D"!%SEARCH_PATH%!\.." ||exit/B 1 has been added to change to the parent directory, because the later used dir /B commands just return directory or file names but not full paths; that is also the reason why the ~nx-modifier is used in %%~nxI, so there is no more difference whether the value comes from dir /B or echo; the exit /B 1 part makes sure to skip the remaining code in case the parent directory could not be found/accessed;
~-modifiers are used in %%~J to ensure unquoted directory names (remember that I put quotes in the echo command lines);
2> nul has been removed (from the end of the body of the outer-most loop) to not suppress error messages in general;

Related

(Batch) How to recursively delete all files/folders in a directory except those with a leading .?

I have a directory, src. I want to recursively delete all of its contents except for files (.gitignore, ...) and folders (.git, .vscode, ...) whose names begin with .. Matching that pattern in subdirectories is neither necessary nor harmful.
What is the cleanest way to do this in a batch file?
#ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir"
:: step 1 : delete all files NOT starting "."
FOR /f "tokens=1*delims=" %%a IN (
'dir /s /b /a-d "%sourcedir%\*" '
) DO (
ECHO %%~nxa|FINDSTR /b /L "." >nul
IF ERRORLEVEL 1 ECHO(DEL "%%a"
)
:: step 2 : delete all directories NOT starting "."
FOR /f "tokens=1*delims=" %%a IN (
'dir /s /b /ad "%sourcedir%\*" ^|sort /r'
) DO (
ECHO %%~nxa|FINDSTR /b /L "." >nul
IF ERRORLEVEL 1 ECHO(RD "%%a"
)
GOTO :EOF
You would need to change the setting of sourcedir to suit your circumstances.
The required DEL commands are merely ECHOed for testing purposes. After you've verified that the commands are correct, change ECHO(DEL to DEL to actually delete the files.
The required RD commands are merely ECHOed for testing purposes. After you've verified that the commands are correct, change ECHO(RD to RD to actually delete the directories.
for each filename in the entire subtree, see whether it starts with ., setting errorlevel to non-0 if not and hence delete the file.
Once this has been done, repeat the operation with directorynames, but sort the names found in reverse so that a subdirectoryname of any directory will appear before the directoryname. Attempt to remove the directory with an rd - it will remain if it contains files or subdirectories (which implicitly will start .). Append 2>nul to the rd line to suppress error messages (where the directory cannot be removed as it still contains files/subdirectories)
Try this to exclude folders with a leading dot (on a per folder base):
for /f "tokens=*eol=." %%A in ('dir /B /AD') do Echo=%%A
This doesn't affect folder names containing dots at other positions.
A relatively slow variant recursing all folders and using findstr to filter all folders containing \. both chars need escaping with a backslash
for /r /D %%A in (*) do #echo %%A|findstr /V "\\\."

How to rename files in subfolder into a specific format

I have files named as RabcdYYMMKKACCOUNT.TXT in the Subfolders of a folder where YYMM is year, month this will change. KK is another identifier, I want all the files to be renamed to MSFKKDNB.ABC, the KK is the identifier in the input file.
Below is the one i tried and the result of it:
FOR /R %%f IN (*account.txt) DO REN "%%f" *dnb.abc
R00531706AUAccount.txt is renamed to R00531706AUAccount.txtdnb.abc
but the output should be MSFAUDNB.abc
This could be done for example with:
#echo off
setlocal EnableExtensions EnableDelayedExpansion
for /R %%I in (???????????account.txt) do (
set "FileName=%%~nI"
set "NewFileName=MSF!FileName:~9,2!DNB.abc"
if not exist "%%~dpI!NewFileName!" (
ren "%%~fI" "!NewFileName!" 2>nul
if not exist "%%~dpI!NewFileName!" echo Failed to rename file: "%%~fI"
) else (
echo Cannot rename file: "%%~fI"
)
)
endlocal
The file name of found account text file is assigned to environment variable FileName.
The new name for the file is created by concatenating the fixed parts MSF and DNB.abc with the 2 characters to keep from file name using string substitution and delayed expansion.
Next it is checked if a file with new name does not already exist. Is this the case the file renaming is done otherwise an error message is output.
After renaming the file it is checked if that was successful. A slightly different error is output if renaming failed for example because of a sharing violation.
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
echo /?
endlocal /?
for /?
if /?
ren /?
set /?
setlocal /?
Read also the Microsoft article about Using Command Redirection Operators.
Try this:
#Echo Off
For %%A In ("*account.txt") Do (Set "_=%%~nA"
SetLocal EnableDelayedExpansion
Ren "%%A" "MSF!_:~-9,2!DNB.abc"
EndLocal)
I would probably do it the following way, provided that the files to rename are located in immediate sub-directories (YYMM) of the given root directory and nowhere else:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "_ROOT=." & rem // (specify path to the root directory)
for /D %%D in ("%_ROOT%\????") do (
for %%F in ("%_ROOT%\%%~nxD\R??????????Account.txt") do (
set "FDIR=%%~nxD" & set "FILE=%%~nxF"
setlocal EnableDelayedExpansion
ECHO ren "!_ROOT!\!FDIR!\!FILE!" "MSF!FILE:~9,2!DNB.abc"
endlocal
)
)
endlocal
exit /B
If you want to check whether both the sub-directory name and the year/month portion of the file names are purely numeric, you could use the following script:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "_ROOT=." & rem // (specify path to the root directory)
for /F "delims= eol=|" %%D in ('
dir /B /A:D "%_ROOT%\????" ^| ^
findstr "^[0123456789][0123456789][0123456789][0123456789]$"
') do (
for /F "delims= eol=|" %%F in ('
dir /B /A:-D "%_ROOT%\%%~nxD\R??????????Account.txt" ^| ^
findstr "^R....[0123456789][0123456789][0123456789][0123456789].."
') do (
set "FDIR=%%~nxD" & set "FILE=%%~nxF"
setlocal EnableDelayedExpansion
ECHO ren "!_ROOT!\!FDIR!\!FILE!" "MSF!FILE:~9,2!DNB.abc"
endlocal
)
)
endlocal
exit /B
If you want to check whether the sub-directory name matches the year/month (YYMM) portion of the file names, replace the pattern R??????????Account.txt by R????%%~nxD??Account.txt (for both scripts).
After having verified the correct output of either script, remove the upper-case ECHO commands to actually rename any files!
Basically, both scripts use sub-string expansion to extract the identifier part (KK) from the file names. Since there are variables set and read in the same block of code, delayed expansion is required for that. The second approach does not list the sub-directories and files by standard for loops, it uses the dir command, findstr to filter their names and a for /F loop to capture the resulting output for both sub-directories and files.

How to return the newest file from each sub-directory in Windows command line using batch

I am using batch script for windows command line on windows 7. I am currently trying to get the newest file from each sub directory and print them to the screen. For example, if I have:
C:\Home\Hi\Folder1\a 01/05/2016
C:\Home\Hi\Folder1\b 01/10/2016
C:\Home\Hi\Folder2\x 03/05/2016
C:\Homeh\Hi\Folder2\y 03/1/2016
It would return: folders b and x.
I have written some script to attempt this, but it only returns the newest file in the last directory which in my example would be x. The script is:
FOR /R "C:\Users\Maxx\Desktop\tools" %%G in (.) DO (
Pushd %%G
FOR /F %%I IN ('DIR %%G /B /O:D') DO SET NEWEST=%%I
ECHO %NEWEST%
Popd )
Echo "back home"
If anyone knows how to get the newest file from each subdirectory that would be great.
Note: I have looked at various other examples such as: Generate a list of the newest file in each subdirectory in windows batch, which has been very helpful in building what I have now, but ultimately it did not work.
You need to apply delayed expansion as you are writing and reading variable NEWEST within the same block of code:
setlocal EnableDelayedExpansion
for /R "C:\Users\Maxx\Desktop\tools" %%G in (.) do (
pushd "%%~G"
for /F %%I in ('dir "%%~G" /B /O:D') do set "NEWEST=%%I"
echo !NEWEST!
)
popd
echo back home
endlocal
Alternatively, replace echo %NEWEST% by call echo %%NEWEST%%:
for /R "C:\Users\Maxx\Desktop\tools" %%G in (.) do (
pushd "%%~G"
for /F %%I in ('dir "%%~G" /B /O:D') do set "NEWEST=%%I"
call echo %%NEWEST%%
)
popd
echo back home
In addition, I improved quoting.
#ECHO OFF
SETLOCAL
FOR /R "C:\106x" %%G in (.) DO (
Pushd "%%G"
SET "first="
FOR /F "delims=" %%I IN ('DIR . /a-d /B /O:-D 2^>nul') DO IF NOT DEFINED first SET first=Y&ECHO %%~dpI
Popd
)
Echo "back home"
GOTO :EOF
For each directory, go to the directory, clear a flag first, scan the directory for files (not directories), basic format, reverse-date order, suppressing any "file not found" reports from empty directories.
On finding the first (ie youngest) file set the flag first and report the filename - or parts of the filename you require.
Once first has been set, the report mechanism will not be invoked as if defined works on the current value of the variable.
Please note that when specifying dates, it's advisable to state the format that you are using - or at least use a date where the day number is >12, which should be adequate. For instance, your date "01/10/2016" may be interpreted as Jan 10th or Oct 1st, depending on the respondent's locale. "13/10/2016" or "10/13/2016" would make the issue obvious.

Keep X amount of files in folder, forfiles

I would like to keep the X latest files from a folder and delete the rest. Is this possible with FORFILES? If it's not I can fallback to another solution I seen here. Thanks for help.
I did this but it takes by dates: EDIT
forfiles /p [path] /s /d -5 /c "cmd /c echo #file"
(echo file for testing purpose)
#ECHO OFF
SETLOCAL
SET "targetdir=U:\destdir"
SET /a retain=10
FOR /f "skip=%retain%delims=" %%a IN (
'dir /b /a-d /o-d "%targetdir%\*" '
) DO ECHO (DEL "%targetdir%\%%a"
GOTO :EOF
You would need to change the setting of targetdir to suit your circumstances. Equally, this procedure targets all files - change the filemask to suit.
The required DEL commands are merely ECHOed for testing purposes. After you've verified that the commands are correct, change ECHO(DEL to DEL to actually delete the files.
method is to simply execute a dir in basic form without directories, sorted in reverse-date order.
Skip the first 10 entries, and delete the rest.
With forfiles I see no chance to accomplish your task of returning the newest (most recent) number of files.
So my idea for this approach is this:
to use dir /B /A:-D /T:C /O:-D to retrieve a bare list (/B) of files (no directories, /A:-D), sorted by creation date (/T:C; if you want to use the last modification date, simply remove the /T:C portion) in decending order (/O:-D), meaning newest items first;
to put over a for /F "eol=| delims=" loop to gather and parse the dir output line by line, meaning file by file, not excluding file names beginning with ; (eol=|, | is illegal for file names) and not splitting file names containing white-spaces like SPACE or TAB (delims=);
to establish a variable that constitutes a counter, incremented per each loop iteration;
to place an if condition inside of the loop to check if the counter reached the desired limit number and in case it is fulfilled, to break the for /F loop by goto;
Here is the related code:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem Define global constants here:
set "TARGETPATH=\path\to\files\*.*"
set /A "TOTAL=10"
set /A "COUNT=0"
for /F "eol=| delims=" %%F in ('
dir /B /A:-D /T:C /O:-D "%TARGETPATH%"
') do (
echo(%%F
set /A COUNT+=1
setlocal EnableDelayedExpansion
if !COUNT! GEQ %TOTAL% (
endlocal
goto :NEXT
) else (
endlocal
)
)
:NEXT
endlocal
exit /B
I toggled the delayed variable expansion within the for /F loop to avoid trouble in case file names contain exclamation marks !, which would get lost in the line echo(%%F in case it is on.
Update
The following code accomplishes the original task of your question, namely to delete files in a given directory but to keep the most recent number of files:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem Define global constants here:
set "TARGETPATH=\path\to\files\*.*"
set /A "TOTAL=10"
set "SKIPOPT=" & if %TOTAL% GTR 0 set "SKIPOPT=skip=%TOTAL% "
for /F "%SKIPOPT%eol=| delims=" %%F in ('
dir /B /A:-D /T:C /O:-D "%TARGETPATH%"
') do (
del /P "%%F"
)
endlocal
exit /B
Since for /F supports a skip= to skip the given number of lines, and so files in our situation, let us make use of it. It is given indirectly here via variable SKIPOPT, which holds the entire option string like skip=10 (given that TOTAL is set to 10). The if %TOTAL% GTR 0 query is implemented for the script not to fail in case TOTAL is 0, because for /F does not accept the option skip=0.
The /P switch at the del command lets appear a prompt Delete (Y/N)? for testing purposes. If you do not want any prompts, simply remove it.

Copying files with name containing just numbers using a batchfile

I have a Directory with a deep Directory->Sub-directory tree structure. I need to write a batch file to copy all the numbered files (files with names as digits and no alphabetic characters) from all the sub-directories.
For example, a sub-directory might contain the following files:
WR10091.txt
AX10091.htm
10091.txt
AX10091.xml
10091.xml
I need to copy 10091.txt and 10091.xml to another location. I can copy files like AX10091.xml and AX10091.htm by specifying AX*.*. But I cannot figure out how to copy just numbered files with no alphabetic characters. There are thousands of directories and the directory structure does not have any pattern (the depth of a tree branch can vary considerably).
Any help will be appreciated.
#echo off
setlocal enableextensions disabledelayedexpansion
set "source=%cd%"
set "target=x:\target\folder"
for /r "%source%" %%a in (*) do (
(for /f "delims=0123456789" %%b in ("%%~na") do (
break
)) || echo copy "%%~fa" "%target%"
)
In this code the for %%a will iterate over all the files under the indicated folder. For each of them, the for /f %%b will try to tokenize the file name (%%~na) using numbers as delimiters. If the file name only contains numbers, there will be nothing to process (only delimiters) and the inner for raises errorlevel. This is checked with conditional execution (the code after the || is executed if the previous command fails) and if errorlevel was raised the copy operation is echoed to console.
If the output is correct, remove the echo to perform the copy.
note: the break in the inner for loop is included just to have a command that does nothing when files with non numeric names are found.
#echo off
for /f "tokens=* delims=" %%a in ('dir /b /s /a:-d "*"') do (
echo %%~na|findstr /e /b /r "[1-9]*" >nul 2>nul && (
copy %%~fa c:\somewhere\
)
)
should be executed in the same directory as the files.
for /f "delims=" %%a in ('dir /b/s/a-d ^| findstr /reic:"\\[0-9][0-9]*\..*" /c:"\\[0-9][0-9]*"') do copy "%%~a" "targetDir"
This might not work with XP and/or Vista, but this can be fixed if needed (see What are the undocumented features and limitations of the Windows FINDSTR command).

Resources