This question already has answers here:
Parallel execution of shell processes
(6 answers)
Closed 4 years ago.
I have this batch command
#echo off
FOR %%i in (C:\input\*.*) DO (
echo processing %%i
if not exist "C:\output\%%i" process.exe "%%i" -out "C:\output\%%i"
)
echo ---- finished ----
pause
Here my tool process.exe processes in a loop all files within a directory - if a result doesn't already exist.
Now my CPU is fast enough to run this process.exe on 2 or 3 files at the same time which would make the processing of the files much faster.
Question: How do I have to change the command to make my batch file processing 2-3 files at the same time?
The following starts processes up to a max count of %bunch%. Whenever one of them finishes, another one will be started.
#ECHO off
setlocal enabledelayedexpansion
set bunch=3
for %%a in (C:\input\*) do (
call :loop
echo processing: %%a
start "MyCommand" cmd /c timeout 60
REM if not exist "C:\output\%%i" start "MyCommand" cmd /c process.exe "%%i" -out "C:\output\%%i"
)
call :loop
goto :eof
:loop REM waits for available slot
echo on
for /f %%x in ('tasklist /fi "windowtitle eq MyCommand" ^| find /c "cmd.exe"') do set x=%%x
if %x% geq %bunch% goto :loop
echo off
goto :eof
I don't have your process.exe, so I have to guess. but the REMed line should work for you. (the timeout command is just to show the princip)
Related
I currently have to compress several thousand files (~40-80MB each) with brotli and get them ready for an s3 bucket.
From what i've researched so far, brotli can't multithread the compression so, brotli.exe uses ~10% of the cpu. How can I iterate through the files in a folder and spawn multiple (brotli).exe files to work at the same time (8-10 processes should fill the cpu)?
windows/powershell/vbs, I can try any suggestions
At the moment, I'm running this batch
for /R %%f in (*.) do (
"brotli" -Z "--output=E:\output\brotli\%%~nf" "%%f"
)
#ECHO OFF
SETLOCAL
:: set limit to #jobs
SET /a limit=8
:: establish a subdirectory in %temp%
SET "control=%temp%\brotlicontrol"
MD "%control%" 2>NUL
:: Dummy for testing
for %%f IN (fred anna george bill betty carl celia daphne john kelly ian zoe brian
tracey susan colin jane selina valerie david stephen) DO (
rem for /R %%f in (*.) do (
CALL :wait
START /min "brotli %%~nf" q75403766_2 "%%f"
)
GOTO :EOF
:wait
SET /a running=0
FOR /f %%y IN ('DIR /a-d /b "%control%\*.flg" 2^>nul ^|FIND /c ".flg" ') DO SET /a running=%%y
IF %running% geq %limit% timeout /t 1 >nul&GOTO wait
GOTO :eof
Here's a main batch which starts a subsidiary batch
#echo off
setlocal
ECHO.>"%control%\%~n1.flg"
REM "brotli" -Z "--output=E:\output\brotli\%~n1" %1
:: Dummy - variable timeout 5-20 seconds
SET /a exectime=(%RANDOM% %% 16) + 5
timeout /t %exectime% >nul
del "%control%\%~n1.flg"
EXIT
I had %%f iterate through a list of names for testing. All you need to do is to remove that test code and use your original code which I remmed out to process your list of files.
The process calls the :wait routine, which counts the .flg files in the temporary directory, and sets running to that value.
If the number running is greater than or equal to (geq) the limit established in the initialisation, wait 1 second and try again, otherwise the :wait routine terminates and the subsidiary batch q75403766_2 is started /min minimised and with the name brotli nameoffile. It's important that the first quoted parameter to start exists as it's used as the title of the started process. You could use "" if you want (for no title) but you should not omit this title string.
The sub-process started (q75403766_2) first creates a .flg file with the name of the file being processed in the control directory, then runs the brotli job (remmed out again) - I added a few lines to create a variable timeout to simulate the brotli process-time - and deletes the control file and exits.
The carets before the redirectors in the for loops tell cmd that the redirection is to be applied to the command being executed, not the for. 2>nul (+caret) says "redirect error messages (file not found) to nowhere (ie. discard them)".
I have quite a few exe files, I want to run them with a single batch file. As far as I understand, these two codes work for me;
for %%a in ("\*.exe") do start "" "%%\~fa"
for %%i in (\*.exe) do start "" /b "%%i"
But that cmd screen closes when all files are run. What I want is this: That cmd screen will not close when the process is finished and will show me the result (counting if possible), a code that can count how many of these .exe files work and how many fail.
So for example;
87 files blocked
13 files could not be blocked
Something like this? Is this possible?
Maybe you can get an inspiration from this batch below. It works through program exit code. It spawns all executable, wait for their completion, then count how much failed / succeeded.
It should also work with well-designed GUI program, not only command-line based ones.
It's a rough/basic answer, you may need to refine it according to your exact needs.
#echo off
setlocal enableextensions enabledelayedexpansion
REM Use marker files for getting results.
set OK_EXT=.SUCCESS
set FAIL_EXT=.FAILURE
REM Purge all possible marker files.
del /q *!OK_EXT! *!FAIL_EXT! > NUL 2>&1
set /a count=0
REM Parse all executables
for %%E in (*.exe) do (
echo Launching: %%~nxE
REM Create two marker files for each executable.
echo.>%%~nE!OK_EXT!
echo.>%%~nE!FAIL_EXT!
REM Start the executable, delete the WRONG marker.
REM I would have prefered to use "touch" to create the good one instead, but not standard on Windows.
start %comspec% /C "%%~nxE && ( del /q %%~nE!FAIL_EXT! ) || ( del /q %%~nE!OK_EXT! )"
set /a count +=1
)
REM Now, "count" contains the number of executables launched.
echo All processes launched.
echo.
:loop
echo Waiting for results...
set /a curr=0
REM Simply count the number of marker files. Must be equal to "count" when everything is finished.
for /F "usebackq tokens=*" %%C in (`dir /b *!OK_EXT! *!FAIL_EXT!`) do (
set /A curr+=1
)
if !curr! GTR !count! (
set /a curr-=!count!
echo Still !curr! processes running...
timeout /t 2
goto :loop
)
echo All results found.
echo.
echo Parsing results...
set /a ok_exe=0
set ok_exe_list=
set /a fail_exe=0
set fail_exe_list=
REM Parse all marker files.
for /F "usebackq tokens=*" %%C in (`dir /b *!OK_EXT! *!FAIL_EXT!`) do (
REM And set counters + list according to the marker file type (OK or FAILED).
if /I "%%~xC"=="!OK_EXT!" (
set /A ok_exe+=1
set ok_exe_list=!ok_exe_list! %%~nC
) else (
set /A fail_exe+=1
set fail_exe_list=!fail_exe_list! %%~nC
)
)
REM Simple display.
echo Programs without error: !ok_exe!/!count!
echo !ok_exe_list!
echo.
echo Programs with error: !fail_exe!/!count!
echo !fail_exe_list!
echo.
goto :eof
This question already has answers here:
Windows Batch Variables Won't Set
(2 answers)
Variables are not behaving as expected
(1 answer)
Example of delayed expansion in batch file
(5 answers)
ERRORLEVEL vs %ERRORLEVEL% vs exclamation mark ERRORLEVEL exclamation mark
(1 answer)
Closed 2 years ago.
I'm trying to make a batch script to kill LGHub after it launches on startup (LGHub needs to load in order to start services that run my mouse's macros/binds, but I don't need the full program/window sitting open on my desktop after those services launch). According to directions I found to launch the batch file at startup, I made a shortcut to the file and threw the shortcut in the Windows Startup folder.
Now, I've never written a batch script before, but from my research and the code taken from other threads, this should do the job. If not, well, A) there's that, but B) the file seems to launch on startup, but then close immediately, instead of waiting like it should.
My current script is
#ECHO off
set /A counter=0
:while
IF %counter% le 30 (
tasklist /FI "IMAGENAME eq lghub.exe" 2>NUL | find /I /N "lghub.exe">NUL
IF "%ERRORLEVEL%"=="0" (
taskkill /im "lghub.exe"&exit
) ELSE (
set /A counter+=1
SLEEP 1
GOTO :while
)
)
echo "Could not kill LG Hub after 30 seconds."&exit
#echo off
cd /d "%~dp0" & set "_cnt=0"
title <nul && title ..\%~nx0
Setlocal EnableDelayEdexpansion
:while
call set /a "_cnt+=1" & cls & echo\ & if !_cnt! leq 29 (
2>nul tasklist /nh | 2>nul find /i "lghub.exe" >nul || (
timeout 1 /nobreak | echo\ Trying to kill LG Hub: !_cnt! & goto:while )
)
echo "Could not kill LG Hub after !_cnt! seconds." & endlocal
Why not try to replace le to leq, and use Setlocal EnableDelayEdexpansion
IF %counter% le 30 (...
Here is what I am trying to do. Suppose I have a program called myprogram.exe, which I have to execute 1000 times.
Under Windows, I could usually do something as simple as:
for /L %n in (1,1,1000) do start /myfolder/myprogram.exe
However, suppose I only have 5 CPU threads I can devote to running the 1000 instances of myprogram.exe, such that I launch only 5, then when one of these finishes another one is launched, etc until the whole 1000 end.
Under Linux and using GNU Parallel, I could simply do:
seq 1000 | parallel -N0 -j5 "nohup myprogram.exe"
How could I achieve something like that in Windows command line? Notice that in my case using Cygwin is not an option, so resorting to xargs and GNU Parallel under Windows are not options either.
Here is a way, by using powershell to do the process count. and using a simply set /a as counter.
#echo off
setlocal enabledelayedexpansion
set /a cnt=0
:counter
if !cnt! lss 1000 (
for /F "tokens=*" %%i in ('powershell ^(Get-Process -Name 'myprogram'^).count') do set proc=%%i
if !proc! lss 5 (
start "C:\myfolder\myprogram.exe"
set /a cnt+=1
)
goto :counter
)
You could add echo !cnt! in the line before goto :counter if you want to see it count.
It can be done without using delayedexpansion but I prefer to use it here.
This will run five processes in parallel. Each time one of them finishes, the next process will be started (so there are always 5 of them until they all are done)
#ECHO off
setlocal enabledelayedexpansion
set bunch=5
for /l %%a in (1,1,1000) do (
call :loop
echo processing: %%a
start "MyCommand" cmd /c timeout !random:~-1!
)
call :loop
goto :eof
:loop REM waits for available slot
for /f %%x in ('tasklist /fi "windowtitle eq MyCommand" ^| find /c "cmd.exe"') do set x=%%x
if %x% geq %bunch% goto :loop
goto :eof
Add the /min switch to start to minimize the started processes.
Give them a unique windowtitle (MyCommand here) to be able to count them.
Replace cmd /c timeout !random:~-1! with your actual command.
EDIT a slightly modified script, which may work better, if myprogram is a GUI (script above will work better with CLI applications):
#ECHO off
setlocal enabledelayedexpansion
set bunch=5
for /l %%a in (1,1,100) do (
call :loop
echo processing: %%a
start notepad.exe
)
call :loop
goto :eof
:loop REM waits for available slot
for /f %%x in ('tasklist /fi "imagename eq Notepad.exe"^|find /c "."') do set x=%%x
if %x% geq %bunch% goto :loop
goto :eof
I have a situation very similar to the one described in this question (but in batch, not shell). I made a simple batch script to iterate through the lines of a tile and download data from a server using a python script (the process itself is more complicated than just a simple download, it has to authenticate with an API and fetch several URLs).
The first version was as follows:
for /F "tokens=*" %%A in (client_name_list.txt) do python download_metadata.py "%%A"
The way it is it waits until each iteration is done to move on, so I updated it to the following:
for /F "tokens=*" %%A in (client_name_list.txt) do start cmd /C python download_metadata.py "%%A"
The second versions does what I want to but, as the file client_name_list.txt is about 30,000 lines long, a lot of command prompts start spawning and the computers freezes within seconds.
How do I limit the number of running instances of CMD (to, for example 10) and make the script wait until there is a "free CMD slot" to go the next line?
Adapted from my answer to "Parallel execution of shell processes". Follow the link to get an explanation.
#echo off
setlocal enableDelayedExpansion
:: Display the output of each process if the /O option is used
:: else ignore the output of each process
if /i "%~1" equ "/O" (
set "lockHandle=1"
set "showOutput=1"
) else (
set "lockHandle=1^>nul 9"
set "showOutput="
)
:: Define the maximum number of parallel processes to run.
set "maxProc=10"
:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
set "lock="
for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
set "lock=%%T"
goto :break
)
:break
set "lock=%temp%\lock%lock%_%random%_"
:: Initialize the counters
set /a "startCount=0, endCount=0"
:: Clear any existing end flags
for /l %%N in (1 1 %maxProc%) do set "endProc%%N="
:: Launch the commands in a loop
set launch=1
for /f "delims=" %%A in (client_name_list.txt) do (
if !startCount! lss %maxProc% (
set /a "startCount+=1, nextProc=startCount"
) else (
call :wait
)
set cmd!nextProc!=%%A
if defined showOutput echo -------------------------------------------------------------------------------
echo !time! - proc!nextProc!: starting %%A
2>nul del %lock%!nextProc!
%= Redirect the lock handle to the lock file. The CMD process will =%
%= maintain an exclusive lock on the lock file until the process ends. =%
start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 python download_metadata.py "%%A"
)
set "launch="
:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
:: redirect stderr to null to suppress any error message if redirection
:: within the loop fails.
for /l %%N in (1 1 %startCount%) do 2>nul (
%= Redirect an unused file handle to the lock file. If the process is =%
%= still running then redirection will fail and the IF body will not run =%
if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
%= Made it inside the IF body so the process must have finished =%
if defined showOutput echo ===============================================================================
echo !time! - proc%%N: finished !cmd%%N!
if defined showOutput type "%lock%%%N"
if defined launch (
set nextProc=%%N
exit /b
)
set /a "endCount+=1, endProc%%N=1"
)
)
if %endCount% lss %startCount% (
1>nul 2>nul ping /n 2 ::1
goto :wait
)
2>nul del %lock%*
if defined showOutput echo ===============================================================================
echo Done
In each iteration of your for loop you can count the number of CMD task open. If the value is lesser than the limit you start a new task else you wait until one slot is free.
#echo off
set $Limit=11
setlocal enabledelayedexpansion
for /F "tokens=*" %%A in (client_name_list.txt) do (call:wait %%A)
exit/b
:wait
set "$cmd="
for /f %%a in ('tasklist ^| findstr /i "cmd"') do set /a $cmd+=1
if !$cmd! lss %$Limit% (
start cmd /C python download_metadata.py "%1"
goto:eof)
ping localhost -n 2 >nul
goto:wait