Findstr too slow on UNC batch script - for-loop

I am running a script that goes through network folders and saves the found files, however it is taking too long to run. I have tried isolating the findstr to a single folder and it runs at an okay time, so I assume it has something to do with the FOR loop.
#echo off
setlocal
set SERVERS=server1 server2 server3 server4
cls
echo Type below the query parameters:
set /p year=Year (4 digits):
set /p month=Month (2 digits):
set /p day=Day (2 digits):
set /p query=Query string:
cls
echo Results:
del /F /Q "C:\Users\%USERNAME%\Desktop\found_files\*" 2>nul
if not exist "C:\Users\%USERNAME%\Desktop\found_files" mkdir "C:\Users\%USERNAME%\Desktop\found_files"
for /f "tokens=*" %%a in ('for %%i in ^(%SERVERS%^) do #findstr /S /I /M /C:"%query%" "\\%%i\folder_structure\*%year%-%month%-%day%*.xml"') do copy /Y "%%a" "C:\Users\%USERNAME%\Desktop\found_files" >nul & echo %%a & set found=1
echo.
if "%found%"=="1" (
echo File^(s^) saved successfully!
) else (
echo No files found!
)
echo.
pause
if "%found%"=="1" explorer C:\Users\%USERNAME%\Desktop\found_files

Your script is already optimized pretty well. I don't think there is much you can do to speed things up.
I suspect your problem is that FINDSTR is running on your local machine, and it must scan the files on all UNC paths (almost surely not local). This means the entire content of every file must be transmitted across your network. If your system is anything like where I work, that could be a nightmare. Our network drive performance is pathetic (more than a factor of 100 slower than local drive)!
Squashman (and SomethingDark) were somewhat concerned about your outer FOR /F executing a nested FOR statement. But I believe that is the most efficient way. When FOR /F iterates command output, it must launch a new process to execute the command. Your current script only needs one sub-process.
The more "traditional" approach would be to move the %SERVERS% iteration outside the inner loop as follows:
for %%i in (%SERVERS%) do for /f "tokens=*" %%a in (
'findstr /S /I /M /C:"%query%" "\\%%i\folder_structure\*%year%-%month%-%day%*.xml"'
) do copy /Y "%%a" "C:\Users\%USERNAME%\Desktop\found_files" >nul & echo %%a & set found=1
But this is actually less efficient because it must launch a new sub-process for each UNC path within %SERVERS%. That being said, I don't think the difference is significant compared to the actual bottle neck of transmitting the file content across the network.
To show the impact of one vs. 100 sub-processes, I ran a quick comparison of the following logically equivalent (but meaningless) commands:
for /f "delims=" %%F in (
'for /l %%N in ^(1 1 100^) do #findstr /m "^" *'
) do echo %%F>nul
:: This took 39 seconds on my machine
for /l %%N in (1 1 100) do for /f %%F in (
'findstr /m "^" *'
) do echo %%F>nul
:: This took 60.9 seconds on my machine

#echo off
set SERVERS=server1,server2,server3,server4
cls
echo Type below the query parameters:
:: Type echo %date% on command prompt if its returns the current date dd/mm/yyyy format, you can load the variables using a substring:
set year=%date:~6,4%
set month=%date:~3,2%
set day=%date:~0,2%
set /p query=Query string:
:: set counter for files founded
set found=0
cls
echo Results:
if not exist "C:\Users\%USERNAME%\Desktop\found_files" (mkdir "C:\Users\%USERNAME%\Desktop\found_files") else (del /F /Q "C:\Users\%USERNAME%\Desktop\found_files\*" 2>nul)
for /f %%i in (%SERVERS%) do ('#find /i /c "%query%" "\\%%i\folder_structure\*%year%-%month%-%day%*.xml"') do (
if "%%i"=="1" (set /a found=%found%+1 && copy /Y "\\%%i\folder_structure\*%year%-%month%-%day%*.xml" "C:\Users\%USERNAME%\Desktop\found_files" >nul && echo File^(s^) saved successfully! & echo.) else (echo No files found!)
)
echo.
pause
if %found% gtr 0 (explorer C:\Users\%USERNAME%\Desktop\found_files)

Related

Batch File to execute all .exe in a folder with count

I have quite a few exe files, I want to run them with a single batch file. As far as I understand, these two codes work for me;
for %%a in ("\*.exe") do start "" "%%\~fa"
for %%i in (\*.exe) do start "" /b "%%i"
But that cmd screen closes when all files are run. What I want is this: That cmd screen will not close when the process is finished and will show me the result (counting if possible), a code that can count how many of these .exe files work and how many fail.
So for example;
87 files blocked
13 files could not be blocked
Something like this? Is this possible?
Maybe you can get an inspiration from this batch below. It works through program exit code. It spawns all executable, wait for their completion, then count how much failed / succeeded.
It should also work with well-designed GUI program, not only command-line based ones.
It's a rough/basic answer, you may need to refine it according to your exact needs.
#echo off
setlocal enableextensions enabledelayedexpansion
REM Use marker files for getting results.
set OK_EXT=.SUCCESS
set FAIL_EXT=.FAILURE
REM Purge all possible marker files.
del /q *!OK_EXT! *!FAIL_EXT! > NUL 2>&1
set /a count=0
REM Parse all executables
for %%E in (*.exe) do (
echo Launching: %%~nxE
REM Create two marker files for each executable.
echo.>%%~nE!OK_EXT!
echo.>%%~nE!FAIL_EXT!
REM Start the executable, delete the WRONG marker.
REM I would have prefered to use "touch" to create the good one instead, but not standard on Windows.
start %comspec% /C "%%~nxE && ( del /q %%~nE!FAIL_EXT! ) || ( del /q %%~nE!OK_EXT! )"
set /a count +=1
)
REM Now, "count" contains the number of executables launched.
echo All processes launched.
echo.
:loop
echo Waiting for results...
set /a curr=0
REM Simply count the number of marker files. Must be equal to "count" when everything is finished.
for /F "usebackq tokens=*" %%C in (`dir /b *!OK_EXT! *!FAIL_EXT!`) do (
set /A curr+=1
)
if !curr! GTR !count! (
set /a curr-=!count!
echo Still !curr! processes running...
timeout /t 2
goto :loop
)
echo All results found.
echo.
echo Parsing results...
set /a ok_exe=0
set ok_exe_list=
set /a fail_exe=0
set fail_exe_list=
REM Parse all marker files.
for /F "usebackq tokens=*" %%C in (`dir /b *!OK_EXT! *!FAIL_EXT!`) do (
REM And set counters + list according to the marker file type (OK or FAILED).
if /I "%%~xC"=="!OK_EXT!" (
set /A ok_exe+=1
set ok_exe_list=!ok_exe_list! %%~nC
) else (
set /A fail_exe+=1
set fail_exe_list=!fail_exe_list! %%~nC
)
)
REM Simple display.
echo Programs without error: !ok_exe!/!count!
echo !ok_exe_list!
echo.
echo Programs with error: !fail_exe!/!count!
echo !fail_exe_list!
echo.
goto :eof

Extract a variable from another variable in a FOR loop

:yes
SetLocal EnableDelayedExpansion
SET /P saizo="Introduzca el valor (en KB) minimo para realizar la busqueda: "
SET /A sss=%saizo%+0
IF %sss% EQU 0 (
ECHO Valor invalido. Introduzca puros numeros.
PAUSE
CLS
GOTO yes
)
forfiles /s /c "cmd /c echo #fsize #file" >"tumtum.txt"
FOR /F "tokens=*" %%I IN (tumtum.txt) DO (
SET ae=%%I
FOR /F "tokens=1 delims=" %%J IN ("!ae!") DO (
SET fsf=%%J
SET fnf=!ae:%fsf%=!
ECHO !fnf!
)
PAUSE
)
Description of the code: it asks you for a number (SET /P), then checks if it's a number and starts looking for the files on the path where it's being run. The machine needs to look for files that has a greater value (in KBs) and ECHO them. My only issue comes in the part of extracting both the size and full file name. Tried doing a FOR /F "tokens=1,2" but for files names with a space the values breaks up. What could be done to fix this?
for /f "tokens=1,*delims= " %%I in ('forfiles /s /p u:\ /c "cmd /c echo #fsize #file"') do echo size: %%I name:%%J
See the documentation about for (for /? from the prompt).
Placing the forfiles command within single-quotes makes for run the command as a source of input data.
Tokens=1* means "choose the first token, delimited by any of the delims characters, and assign it to the metavariable" (%I in this case); Choose the remaining characters on the line and assign them to the next metavariable.
forfiles is very slow. You would probably be better off using a dir/s/b command to select the filenames and %~zI to show the size of the file whose name is "%I"

Batch How to avoid a same 2nd FOR loop?

I have created a batch script to backup some subfolders from a path A to a path B (Z:\Folder1\… > U:\Backup\…).
The script lists the subfolders inside path A and increments a number for each of them.
Then, I just have to enter the number(s) of the subfolder(s) I want to backup and xcopy does the rest.
The problem is that sometimes I have thousands of subfolders in path A and only a few to backup (10 or 15).
What I would like is that once I enter the number of these folders, it will go straight to the backup without having to loop all the subfolders inside path A AGAIN (which take time).
Here is my batch :
#echo off
setlocal EnableDelayedExpansion
rem Script for backuping some subfolders from path A to path B
set BackupLocation=U:\Backup
set subfolder_no=1
FOR /F "delims=" %%a IN ('dir /b "Z:\Folder1\*"') DO (
set subfolder=%%a
if defined subfolder (
echo !subfolder_no! !subfolder!
set /a subfolder_no+=1
)
)
set /a subfolder_no=%subfolder_no%-1
set /a index=0
set /a choice=-1
echo.
set /p choice=Enter the number(s) of the subfolder(s) you want to backup:
FOR /F "delims=" %%a IN ('dir /b "Z:\Folder1\*"') DO (
set subfolder=%%a
if defined subfolder (
set /a index+=1
)
FOR %%f IN (%choice%) DO if %%f==!index! (
echo.
echo Backuping subfolder !subfolder!
xcopy "Z:\Folder1\!subfolder!" "%BackupLocation%\!subfolder!\" /e /i /y
)
)
echo.
pause
exit
How can I do this ? Is it possible to get the subfolders' name from their matching number and store them in variables or something ?
Thanks a lot for your help !
Here's an example which only enumerates the directories once.
Please note, whilst it technically answers your question, and performs the task you require of it, this version is designed to work as intended on Windows 10. The specific part you asked about works in other versions too, but the :BackUp labelled section uses a new and undocumented Windows 10 feature of the sort command, to return only unique items from the selection list. Without that, your end user could tehnically provide the same number multiple times, and thus trigger multiple backups of the same directory. As this part of the code is technically outside of the scope of your question, I will leave it up to you to modify the code section yourself, should you be deploying this on older Operating Systems.
#Echo Off
SetLocal EnableExtensions DisableDelayedExpansion
Rem Script for backing up some subdirectories of a Source path to a Backup path.
Set "SourceLocation=Z:\Folder1"
Set "BackupLocation=U:\Backup"
If Not Exist "%Sourcelocation%\." Exit /B
If Not Exist "%Backuplocation%\." Exit /B
:ShowSet
For /F "Delims==" %%G In ('"(Set subdirectory[) 2>NUL"') Do Set "%%G="
Set "index=0"
For /F "EOL=? Delims=" %%G In ('Dir /B /A:D /O:N "%SourceLocation%" 2^>NUL'
) Do (Set /A index += 1
Set "subdirectory=%%G"
SetLocal EnableDelayedExpansion
Echo !index! !subdirectory!
For /F "Tokens=1,*" %%H In ("!index! !subdirectory!") Do (EndLocal
Set "subdirectory[%%H]=%%I"))
If Not Defined subdirectory[1] Exit /B
:Select
Echo(
Echo Please type the number(s) for the subdirectories you want to backup,
Echo(
Echo For multiple selections please separate each with spaces e.g. 1 3 6
Echo For none please type 0 or press [ENTER].
Set "selection=0"
Set /P "selection=>"
Set "selection=%selection:"=%"
Set "selection=%selection:)=%"
If Not Defined selection GoTo Select
If "%selection%" == "0" GoTo :EOF
(Set selection) | "%SystemRoot%\System32\findstr.exe"^
/X /R /C:"selection=[0123456789 ][0123456789 ]*" 1>NUL || GoTo Select
Set "selection=%selection% "
:BackUp
For /F %%G In (
'"(Echo(%selection: =^&Echo(%) | "%SystemRoot%\System32\sort.exe" /Unique"'
) Do If Not Defined subdirectory[%%G] (Echo Selection %%G was not valid) Else (
SetLocal EnableDelayedExpansion
Echo(&Echo Backing up subdirectory %%G !subdirectory[%%G]!
"%SystemRoot%\System32\Robocopy.exe" ^
"%SourceLocation%\!subdirectory[%%G]!" ^
"%BackupLocation%\!subdirectory[%%G]!" /E /NC /NDL /NJH /NJS /NS ^
| %SystemRoot%\System32\find.exe /V " (0x00000005) "
EndLocal)
Pause
In the :BackUp section, I have used Robocopy.exe instead of the deprecated xcopy.exe utility you had used.
If you wish to still use xcopy.exe, replace:
"%SystemRoot%\System32\Robocopy.exe" ^
"%SourceLocation%\!subdirectory[%%G]!" ^
"%BackupLocation%\!subdirectory[%%G]!" /E /NC /NDL /NJH /NJS /NS ^
| %SystemRoot%\System32\find.exe /V " (0x00000005) "
with:
"%SystemRoot%\System32\xcopy.exe" ^
"%SourceLocation%\!subdirectory[%%G]!" ^
"%BackupLocation%\!subdirectory[%%G]!\" /E /I /Y

How to write multiple path and file of a folder in once command line argument dos batch

On windows 7 x64 with a batch files (*.bat):
I need to check in a folder the last file (less than 5 days) and path to add in Mycommand as argument.
I need i think :
"FORFILES -5"
perhaps "Call"
i begin with :
#echo off
REM
set folder="D:\temp"
set vararg="-a"
cd /d %folder%
for /F "delims=" %%i in ('dir /b') do (D:\stockage\Mycommand %vararg% %folder%\\"%%i" -v > c:\temp\test_log.txt)
How to output like that :
D:\stockage\Mycommand -a D:\temp\file.jpg -a D:\temp\file2.jpg -a D:\temp\file3.jpg etc...
don't put non existing file or blank argument, i think i must Loop the argument "-a D:\temp\filexxx.jpg" but i don't know how to do it.
And if possible escape file name probleme more than x chars or space inside.
Can you give me an example ?
Thank.
echo produces a linefeed and you can not stop it from doing so. As an alternative, first assemble the complete line:
setlocal enabledelayedexpansion
set commandline="D:\stockage\Mycommand"
set maxfiles=5
for /f "delims=" %%i in ('dir /b /o-d') do (
set /a maxfiles-=1
set commandline=!commandline! -a "%%i"
if !maxfiles! leq 0 goto :enough
)
:enough
echo %commandline%
This gives you the latest 5 files (or fewer, if the folder has not enough files)
Edited to reverse the order of the Parameters:
setlocal enabledelayedexpansion
set "commandline="
set maxfiles=5
for /f "delims=" %%i in ('dir /b /o-d') do (
set /a maxfiles-=1
set commandline=-a "%%i" !commandline!
if !maxfiles! leq 0 goto :enough
)
:enough
set commandline="D:\stockage\Mycommand" %commandline%
echo %commandline%

How to count no of lines in text file and store the value into a variable using batch script?

I want to count the no of lines in a text file and then the value has to be stored into a environment variable. The command to count the no of lines is
findstr /R /N "^" file.txt | find /C ":"
I refered the question How to store the result of a command expression in a variable using bat scripts?
Then I tried,
set cmd="findstr /R /N "^" file.txt | find /C ":" "
I am getting the error message,
FIND: Parameter format not correct
How could i get rid of this error.
There is a much simpler way than all of these other methods.
find /v /c "" filename.ext
Holdover from the legacy MS-DOS days, apparently. More info here: https://devblogs.microsoft.com/oldnewthing/20110825-00/?p=9803
Example use:
adb shell pm list packages | find /v /c ""
If your android device is connected to your PC and you have the android SDK on your path, this prints out the number of apps installed on your device.
You could use the FOR /F loop, to assign the output to a variable.
I use the cmd-variable, so it's not neccessary to escape the pipe or other characters in the cmd-string, as the delayed expansion passes the string "unchanged" to the FOR-Loop.
#echo off
cls
setlocal EnableDelayedExpansion
set "cmd=findstr /R /N "^^" file.txt | find /C ":""
for /f %%a in ('!cmd!') do set number=%%a
echo %number%
Inspired by the previous posts,
a shorter way of doing so:
CMD.exe
C:\>FINDSTR /R /N "^.*$" file.txt | FIND /C ":"
The number of lines
Try it. It works in my console.
EDITED:
(the "$" sign removed)
FINDSTR /R /N "^.*" file.txt | FIND /C ":"
$ reduces the number by 1 because it is accepting the first row as Field name and then counting the number of rows.
Try this:
#Echo off
Set _File=file.txt
Set /a _Lines=0
For /f %%j in ('Find "" /v /c ^< %_File%') Do Set /a _Lines=%%j
Echo %_File% has %_Lines% lines.
It eliminates the extra FindStr and doesn't need expansion.
- edited to use ChrisJJ's redirect suggestion. Removal of the TYPE command makes it three times faster.
#Tony: You can even get rid of the type %file% command.
for /f "tokens=2 delims=:" %%a in ('find /c /v "" %_file%') do set /a _Lines=%%a
For long files this should be even quicker.
I usually use something more like this
for /f %%a in (%_file%) do (set /a Lines+=1)
for /f "usebackq" %A in (`TYPE c:\temp\file.txt ^| find /v /c "" `) do set numlines=%A
in a batch file, use %%A instead of %A
The perfect solution is:
FOR /F %%i IN ('TYPE "Text file.txt" ^| FIND /C /V ""') DO SET Lines=%%i
I found this solution to work best for creating a log file that maintains itself:
setlocal enabledelayedexpansion
SET /A maxlines= 10
set "cmd=findstr /R /N "^^" "filename.txt" | find /C ":""
for /f %%a in ('!cmd!') do set linecount=%%a
GOTO NEXT
:NEXT
FOR /F %%A IN ("filename.txt") DO (
IF %linecount% GEQ %maxlines% GOTO ExitLoop
echo %clientname% %Date% %Time% >> "filename.txt")
EXIT
:ExitLoop
echo %clientname% %Date% %Time% > "filename.txt"
EXIT
Environmental variables included are %clientname% the computername of the remote client %Date% is the current date and %Time% the current time. :NEXT is called after getting the number of lines in the file. If the file line count is greater than the %maxlines% variable it goes to the :EXITLOOP where it overwrites the file, creating a new one with the first line of information. if it is less than the %maxlines% variable it simply adds the line to the current file.
You don't need to use find.
#echo off
set /a counter=0
for /f %%a in (filename) do set /a counter+=1
echo Number of lines: %counter%
This iterates all lines in the file and increases the counter variable by 1 for each line.
The :countLines subroutine below accepts two parameters: a variable name; and a filename. The number of lines in the file are counted, the result is stored in the variable, and the result is passed back to the main program.
The code has the following features:
Reads files with Windows or Unix line endings.
Handles Unicode as well as ANSI/ASCII text files.
Copes with extremely long lines.
Isn’t fazed by the null character.
Raises an error on reading an empty file.
Counts beyond the Batch max int limit of (31^2)-1.
#echo off & setLocal enableExtensions disableDelayedExpansion
call :countLines noOfLines "%~1" || (
>&2 echo(file "%~nx1" is empty & goto end
) %= cond exec =%
echo(file "%~nx1" has %noOfLines% line(s)
:end - exit program with appropriate errorLevel
endLocal & goto :EOF
:countLines result= "%file%"
:: counts the number of lines in a file
setLocal disableDelayedExpansion
(set "lc=0" & call)
for /f "delims=:" %%N in ('
cmd /d /a /c type "%~2" ^^^& ^<nul set /p "=#" ^| (^
2^>nul findStr /n "^" ^&^& echo(^) ^| ^
findStr /blv 1: ^| 2^>nul findStr /lnxc:" "
') do (set "lc=%%N" & call;) %= for /f =%
endlocal & set "%1=%lc%"
exit /b %errorLevel% %= countLines =%
I know it looks hideous, but it covers most edge-cases and is surprisingly fast.
Just:
c:\>(for /r %f in (*.java) do #type %f ) | find /c /v ""
Font: https://superuser.com/questions/959036/what-is-the-windows-equivalent-of-wc-l
One nice surprise is for one who has git bash on his windows: just plain old linux wc -l <filename> will works for you there
In the below code, the variable name are SalaryCount and TaxCount
#ECHO OFF
echo Process started, please wait...
for /f %%C in ('Find /V /C "" ^< "D:\Trial\Salary.txt"') do set SalaryCount=%%C
echo Salary,%SalaryCount%
for /f %%C in ('Find /V /C "" ^< "D:\Trial\Tax.txt"') do set TaxCount=%%C
echo Tax,%TaxCount%
Now if you need to output these values to a csv file, you could use the below code.
#ECHO OFF
cd "D:\CSVOutputPath\"
echo Process started, please wait...
echo FILENAME,FILECOUNT> SUMMARY.csv
for /f %%C in ('Find /V /C "" ^< "D:\Trial\Salary.txt"') do set Count=%%C
echo Salary,%Count%>> SUMMARY.csv
for /f %%C in ('Find /V /C "" ^< "D:\Trial\Tax.txt"') do set Count=%%C
echo Tax,%Count%>> SUMMARY.csv
The > will overwrite the existing content of the file and the >> will append the new data to existing data. The CSV will be generated in D:\CSVOutputPath
You can pipe the output of type into find inside the in(…) clause of a for /f loop:
for /f %%A in ('
type "%~dpf1" ^| find /c /v ""
') do set "lineCount=%%A"
But the pipe starts a subshell, which slows things down.
Or, you could redirect input from the file into find like so:
for /f %%A in ('
find /c /v "" ^< "%~dpf1"
') do set "lineCount=%%A"
But this approach will give you an answer 1 less than the actual number of lines if the file ends with one or more blank lines, as teased out by the late foxidrive in counting lines in a file.
And then again, you could always try:
find /c /v "" example.txt
The trouble is, the output from the above command looks like this:
---------- EXAMPLE.TXT: 511
You could split the string on the colon to get the count, but there might be more than one colon if the filename had a full path.
Here’s my take on that problem:
for /f "delims=" %%A in ('
find /c /v "" "%~1"
') do for %%B in (%%A) do set "lineCount=%%B"
This will always store the count in the variable.
Just one last little problem… find treats null characters as newlines. So if sneaky nulls crept into your text file, or if you want to count the lines in a Unicode file, this answer isn’t for you.
You can also try
set n=0 & for /f "tokens=*" %a in (text.txt) do set/a n=!n!+1
echo !n!
You can also mark with a wildcard symbol * to facilitate group files to count.
Z:\SQLData>find /c /v "" FR_OP133_OCCURENCES_COUNT_PER_DOCUMENTS_*.txt
Result
---------- FR_OP133_OCCURENCES_COUNT_PER_DOCUMENTS_AVIFRS01_V1.TXT: 2041
---------- FR_OP133_OCCURENCES_COUNT_PER_DOCUMENTS_AVIOST00_V1.TXT: 315938
---------- FR_OP133_OCCURENCES_COUNT_PER_DOCUMENTS_AVIFRS00_V1.TXT: 0
---------- FR_OP133_OCCURENCES_COUNT_PER_DOCUMENTS_CNTPTF00_V1.TXT: 277

Resources