Batch file multitasking issues sharing variables with parent - windows

Question

I'm trying to mass encode video files in x265 and wanted to come up with a batch file that could automate the process. In order to expedite things, I found that having 3 instances of ffmpeg called with 2 threads resulted in ideal encoding times, however I've tried all day yesterday to come up with a way to get a batch file that will call 3 instances and then call new ones when they complete. So far, this is where I am:
PARENT BATCH
#echo off
Setlocal EnableDelayedExpansion
SET /A COUNT=0
for %%a in (*.mkv) do (
CALL :CHECK
SET /A COUNT+=1
START CALL "child.bat" "%%a"
)
EXIT /B 0
:CHECK
IF !COUNT! EQU 3 (
TIMEOUT /T 5
GOTO :CHECK
)
EXIT /B 0
CHILD BATCH
ffmpeg <COMMAND LINE ARGS>
SET /A COUNT-=1
EXIT /B 0
I have two problems. 1) The COUNT variable isn't being updated in the parent process and it never spawns new instances when the child processes finish. 2) The child process doesn't cleanly exit. It leaves a separate cmd.exe window open with a DOS prompt.
Any ideas?
Edit: Replaced nested GOTO to prevent FOR loop breakage
Workaround below
Setlocal EnableDelayedExpansion
SET /A COUNT=0
for %%a in (*.mkv) do (
IF !COUNT! EQU 3 CALL :CHECK
SET /A COUNT+=1
START CALL "child.bat" "%%a"
)
EXIT /B 0
:CHECK
IF EXIST DONE.TXT (
SET /A COUNT-=1
DEL DONE.TXT
EXIT /B 0
) ELSE (
TIMEOUT /T 5
GOTO :CHECK
)
EXIT /B 0
CHILD BATCH
ffmpeg <COMMAND LINE ARGS>
:DONE
IF EXIST DONE.TXT (
TIMEOUT /T 1
GOTO :DONE
)
echo 1 >> DONE.TXT
EXIT 0

Answer 1

Regarding your stated problems:
1) A child process cannot modify environment variables in the parent process. You will need a different mechanism to detect when the child has terminated. Also, as Squashman states in his comment, a GOTO within a loop will break (terminate) the loop, which is why no new child processes are launched after the first 3.
2) Your child window does not terminate because you use EXIT /B. Use EXIT instead and the window will close.
You have a long way to go before you have a working solution ;-)
Perhaps the biggest hurdle is detecting when a child process terminates.
I know of 3 strategies:
1) Use TASKLIST coupled with FIND /C to count the number of ffmpeg processes that are currently running. This is perhaps the simplest solution, but it cannot differentiate between processes that your script launches vs processes that may have been launched by some other mechanism.
2) Use a file as a signal. Create an empty file for each process, and then when the process finishes, have it delete the file. Your main script can monitor which processes are active by looking for the files. This is also simple, but it does not behave well if one of your processes crashes before it can delete the file. That leaves your system in an unhealthy state.
3) My favorite is to use lock files. Each child process locks a file via redirection, and when the process terminates (crash, normal exit, it doesn't matter how), then the lock is released. The main process can attempt to lock the same files. It knows the child has terminated if the lock is successful, else the child is still running. This strategy is the most complicated, and it uses arcane syntax, but I find it highly effective.
I have already implemented a solution at Parallel execution of shell processes that uses option 3). Below is an adaptation/simplification of that code for your situation.
I launch each child process in the parent window using START /B, and I redirect all output to the lock file. When finished, I type the output file so you can see what happened. I also list the start and stop times for each child process.
You just need to adjust the 3 top environment variables to suit your needs. The remainder should be good to go. However, the code as written will fail if any file names contain the ! character. This limitation can be removed with a bit more work.
There is extensive documentation within the script. The %= COMMENT =%syntax is one way of safely embedding comments within a loop without using REM.
#echo off
setlocal enableDelayedExpansion
:: Define the command that will be run to obtain the list of files to process
set listCmd=dir /b /a-d *.mkv
:: Define the command to run for each file, where "%%F" is an iterated file name from the list
:: something like YOUR_COMMAND -i "%%F"
set runCmd=ffmpeg [** command arguments go here **]
:: Define the maximum number of parallel processes to run.
set "maxProc=3"
::---------------------------------------------------------------------------------
:: The remainder of the code should remain constant
::
:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
set "lock="
for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
set "lock=%%T"
goto :break
)
:break
set "lock=%temp%\lock%lock%_%random%_"
:: Initialize the counters
set /a "startCount=0, endCount=0"
:: Clear any existing end flags
for /l %%N in (1 1 %maxProc%) do set "endProc%%N="
:: Launch the commands in a loop
set launch=1
for /f "tokens=* delims=:" %%F in ('%listCmd%') do (
if !startCount! lss %maxProc% (
set /a "startCount+=1, nextProc=startCount"
) else (
call :wait
)
set cmd!nextProc!=%runCmd%
echo -------------------------------------------------------------------------------
echo !time! - proc!nextProc!: starting %runCmd%
2>nul del %lock%!nextProc!
%= Redirect the lock handle to the lock file. The CMD process will =%
%= maintain an exclusive lock on the lock file until the process ends. =%
start /b "" cmd /c >"%lock%!nextProc!" 2^>^&1 %runCmd%
)
set "launch="
:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
:: redirect stderr to null to suppress any error message if redirection
:: within the loop fails.
for /l %%N in (1 1 %startCount%) do 2>nul (
%= Redirect an unused file handle to the lock file. If the process is =%
%= still running then redirection will fail and the IF body will not run =%
if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
%= Made it inside the IF body so the process must have finished =%
echo ===============================================================================
echo !time! - proc%%N: finished !cmd%%N!
type "%lock%%%N"
if defined launch (
set nextProc=%%N
exit /b
)
set /a "endCount+=1, endProc%%N=1"
)
)
if %endCount% lss %startCount% (
timeout /t 1 /nobreak >nul
goto :wait
)
2>nul del %lock%*
echo ===============================================================================
echo Thats all folks

Answer 2

Easy solution with a random window title (based on user2956477 idea):
set windowtitle=Worker_%random%%random%%random%
start /min cmd /c "title %windowtitle% & dosomething.bat file1"
start /min cmd /c "title %windowtitle% & dosomething.bat file2"
start /min cmd /c "title %windowtitle% & dosomethingelse.exe"
timeout 3 >nul
:LOOP
tasklist /V /FI "imagename eq cmd.exe" /FI "windowtitle eq %windowtitle%*" | findstr %windowtitle% >nul
if "%errorlevel%"=="0" timeout 1 >nul && goto LOOP