I currently have to compress several thousand files (~40-80MB each) with brotli and get them ready for an s3 bucket.
From what i've researched so far, brotli can't multithread the compression so, brotli.exe uses ~10% of the cpu. How can I iterate through the files in a folder and spawn multiple (brotli).exe files to work at the same time (8-10 processes should fill the cpu)?
windows/powershell/vbs, I can try any suggestions
At the moment, I'm running this batch
for /R %%f in (*.) do (
"brotli" -Z "--output=E:\output\brotli\%%~nf" "%%f"
)
#ECHO OFF
SETLOCAL
:: set limit to #jobs
SET /a limit=8
:: establish a subdirectory in %temp%
SET "control=%temp%\brotlicontrol"
MD "%control%" 2>NUL
:: Dummy for testing
for %%f IN (fred anna george bill betty carl celia daphne john kelly ian zoe brian
tracey susan colin jane selina valerie david stephen) DO (
rem for /R %%f in (*.) do (
CALL :wait
START /min "brotli %%~nf" q75403766_2 "%%f"
)
GOTO :EOF
:wait
SET /a running=0
FOR /f %%y IN ('DIR /a-d /b "%control%\*.flg" 2^>nul ^|FIND /c ".flg" ') DO SET /a running=%%y
IF %running% geq %limit% timeout /t 1 >nul&GOTO wait
GOTO :eof
Here's a main batch which starts a subsidiary batch
#echo off
setlocal
ECHO.>"%control%\%~n1.flg"
REM "brotli" -Z "--output=E:\output\brotli\%~n1" %1
:: Dummy - variable timeout 5-20 seconds
SET /a exectime=(%RANDOM% %% 16) + 5
timeout /t %exectime% >nul
del "%control%\%~n1.flg"
EXIT
I had %%f iterate through a list of names for testing. All you need to do is to remove that test code and use your original code which I remmed out to process your list of files.
The process calls the :wait routine, which counts the .flg files in the temporary directory, and sets running to that value.
If the number running is greater than or equal to (geq) the limit established in the initialisation, wait 1 second and try again, otherwise the :wait routine terminates and the subsidiary batch q75403766_2 is started /min minimised and with the name brotli nameoffile. It's important that the first quoted parameter to start exists as it's used as the title of the started process. You could use "" if you want (for no title) but you should not omit this title string.
The sub-process started (q75403766_2) first creates a .flg file with the name of the file being processed in the control directory, then runs the brotli job (remmed out again) - I added a few lines to create a variable timeout to simulate the brotli process-time - and deletes the control file and exits.
The carets before the redirectors in the for loops tell cmd that the redirection is to be applied to the command being executed, not the for. 2>nul (+caret) says "redirect error messages (file not found) to nowhere (ie. discard them)".
Related
I have quite a few exe files, I want to run them with a single batch file. As far as I understand, these two codes work for me;
for %%a in ("\*.exe") do start "" "%%\~fa"
for %%i in (\*.exe) do start "" /b "%%i"
But that cmd screen closes when all files are run. What I want is this: That cmd screen will not close when the process is finished and will show me the result (counting if possible), a code that can count how many of these .exe files work and how many fail.
So for example;
87 files blocked
13 files could not be blocked
Something like this? Is this possible?
Maybe you can get an inspiration from this batch below. It works through program exit code. It spawns all executable, wait for their completion, then count how much failed / succeeded.
It should also work with well-designed GUI program, not only command-line based ones.
It's a rough/basic answer, you may need to refine it according to your exact needs.
#echo off
setlocal enableextensions enabledelayedexpansion
REM Use marker files for getting results.
set OK_EXT=.SUCCESS
set FAIL_EXT=.FAILURE
REM Purge all possible marker files.
del /q *!OK_EXT! *!FAIL_EXT! > NUL 2>&1
set /a count=0
REM Parse all executables
for %%E in (*.exe) do (
echo Launching: %%~nxE
REM Create two marker files for each executable.
echo.>%%~nE!OK_EXT!
echo.>%%~nE!FAIL_EXT!
REM Start the executable, delete the WRONG marker.
REM I would have prefered to use "touch" to create the good one instead, but not standard on Windows.
start %comspec% /C "%%~nxE && ( del /q %%~nE!FAIL_EXT! ) || ( del /q %%~nE!OK_EXT! )"
set /a count +=1
)
REM Now, "count" contains the number of executables launched.
echo All processes launched.
echo.
:loop
echo Waiting for results...
set /a curr=0
REM Simply count the number of marker files. Must be equal to "count" when everything is finished.
for /F "usebackq tokens=*" %%C in (`dir /b *!OK_EXT! *!FAIL_EXT!`) do (
set /A curr+=1
)
if !curr! GTR !count! (
set /a curr-=!count!
echo Still !curr! processes running...
timeout /t 2
goto :loop
)
echo All results found.
echo.
echo Parsing results...
set /a ok_exe=0
set ok_exe_list=
set /a fail_exe=0
set fail_exe_list=
REM Parse all marker files.
for /F "usebackq tokens=*" %%C in (`dir /b *!OK_EXT! *!FAIL_EXT!`) do (
REM And set counters + list according to the marker file type (OK or FAILED).
if /I "%%~xC"=="!OK_EXT!" (
set /A ok_exe+=1
set ok_exe_list=!ok_exe_list! %%~nC
) else (
set /A fail_exe+=1
set fail_exe_list=!fail_exe_list! %%~nC
)
)
REM Simple display.
echo Programs without error: !ok_exe!/!count!
echo !ok_exe_list!
echo.
echo Programs with error: !fail_exe!/!count!
echo !fail_exe_list!
echo.
goto :eof
The Issue: Group policies don;t allow me to change the lock screen to slideshow or spotlight but i am local admin on the pc and by replacing C:\Windows\Web\Screen\Screen.jpg i can change the lock screen picture.
Solution: create a batch/CMD/PS script that runs every xx minutes and copies a random picture from a source folder to replace C:\Windows\Web\Screen\Screen.jpg
i found a possible script in this article that could work but how do i modify it for my purpose and if i schedule it in task scheduler to run every 30 minutes would a Batch file run in the background without interference or would a CMD script or Powershell script be a better solutions?
see code below:
#echo off & setlocal
set "workDir=C:\source\folder"
::Read the %random%, two times is'nt a mistake! Why? Ask Bill.
::In fact at the first time %random% is nearly the same.
#set /a "rdm=%random%"
set /a "rdm=%random%"
::Push to your path.
pushd "%workDir%"
::Count all files in your path. (dir with /b shows only the filenames)
set /a "counter=0"
for /f "delims=" %%i in ('dir /b ^|find "."') do call :sub1
::This function gives a value from 1 to upper bound of files
set /a "rdNum=(%rdm%*%counter%/32767)+1"
::Start a random file
set /a "counter=0"
for /f "delims=" %%i in ('dir /b ^|find "."') do set "fileName=%%i" &call :sub2
::Pop back from your path.
popd "%workDir%"
goto :eof
:: end of main
:: start of sub1
:sub1
::For each found file set counter + 1.
set /a "counter+=1"
goto :eof
:: end of sub1
:: start of sub2
:sub2
::1st: count again,
::2nd: if counted number equals random number then start the file.
set /a "counter+=1"
if %counter%==%rdNum% (
:: OUTPUT ALERT BOX with FILENAME
MSG * "%fileName%"
)
goto :eof
:: end of sub2
IMO PowerShell is by far superior to batch in this case
dir C:\source\folder\*.jpg |Get-Random|Copy -Dest C:\Windows\Web\Screen\Screen.jpg -Force
You might wrap it in a batch/cmd line
powershell -Nop -C "dir C:\source\folder\*.jpg |Get-Random|Copy -Dest C:\Windows\Web\Screen\Screen.jpg -Force"
I'm trying to mass encode video files in x265 and wanted to come up with a batch file that could automate the process. In order to expedite things, I found that having 3 instances of ffmpeg called with 2 threads resulted in ideal encoding times, however I've tried all day yesterday to come up with a way to get a batch file that will call 3 instances and then call new ones when they complete. So far, this is where I am:
PARENT BATCH
#echo off
Setlocal EnableDelayedExpansion
SET /A COUNT=0
for %%a in (*.mkv) do (
CALL :CHECK
SET /A COUNT+=1
START CALL "child.bat" "%%a"
)
EXIT /B 0
:CHECK
IF !COUNT! EQU 3 (
TIMEOUT /T 5
GOTO :CHECK
)
EXIT /B 0
CHILD BATCH
ffmpeg <COMMAND LINE ARGS>
SET /A COUNT-=1
EXIT /B 0
I have two problems. 1) The COUNT variable isn't being updated in the parent process and it never spawns new instances when the child processes finish. 2) The child process doesn't cleanly exit. It leaves a separate cmd.exe window open with a DOS prompt.
Any ideas?
Edit: Replaced nested GOTO to prevent FOR loop breakage
Workaround below
Setlocal EnableDelayedExpansion
SET /A COUNT=0
for %%a in (*.mkv) do (
IF !COUNT! EQU 3 CALL :CHECK
SET /A COUNT+=1
START CALL "child.bat" "%%a"
)
EXIT /B 0
:CHECK
IF EXIST DONE.TXT (
SET /A COUNT-=1
DEL DONE.TXT
EXIT /B 0
) ELSE (
TIMEOUT /T 5
GOTO :CHECK
)
EXIT /B 0
CHILD BATCH
ffmpeg <COMMAND LINE ARGS>
:DONE
IF EXIST DONE.TXT (
TIMEOUT /T 1
GOTO :DONE
)
echo 1 >> DONE.TXT
EXIT 0
Regarding your stated problems:
1) A child process cannot modify environment variables in the parent process. You will need a different mechanism to detect when the child has terminated. Also, as Squashman states in his comment, a GOTO within a loop will break (terminate) the loop, which is why no new child processes are launched after the first 3.
2) Your child window does not terminate because you use EXIT /B. Use EXIT instead and the window will close.
You have a long way to go before you have a working solution ;-)
Perhaps the biggest hurdle is detecting when a child process terminates.
I know of 3 strategies:
1) Use TASKLIST coupled with FIND /C to count the number of ffmpeg processes that are currently running. This is perhaps the simplest solution, but it cannot differentiate between processes that your script launches vs processes that may have been launched by some other mechanism.
2) Use a file as a signal. Create an empty file for each process, and then when the process finishes, have it delete the file. Your main script can monitor which processes are active by looking for the files. This is also simple, but it does not behave well if one of your processes crashes before it can delete the file. That leaves your system in an unhealthy state.
3) My favorite is to use lock files. Each child process locks a file via redirection, and when the process terminates (crash, normal exit, it doesn't matter how), then the lock is released. The main process can attempt to lock the same files. It knows the child has terminated if the lock is successful, else the child is still running. This strategy is the most complicated, and it uses arcane syntax, but I find it highly effective.
I have already implemented a solution at Parallel execution of shell processes that uses option 3). Below is an adaptation/simplification of that code for your situation.
I launch each child process in the parent window using START /B, and I redirect all output to the lock file. When finished, I type the output file so you can see what happened. I also list the start and stop times for each child process.
You just need to adjust the 3 top environment variables to suit your needs. The remainder should be good to go. However, the code as written will fail if any file names contain the ! character. This limitation can be removed with a bit more work.
There is extensive documentation within the script. The %= COMMENT =%syntax is one way of safely embedding comments within a loop without using REM.
#echo off
setlocal enableDelayedExpansion
:: Define the command that will be run to obtain the list of files to process
set listCmd=dir /b /a-d *.mkv
:: Define the command to run for each file, where "%%F" is an iterated file name from the list
:: something like YOUR_COMMAND -i "%%F"
set runCmd=ffmpeg [** command arguments go here **]
:: Define the maximum number of parallel processes to run.
set "maxProc=3"
::---------------------------------------------------------------------------------
:: The remainder of the code should remain constant
::
:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
set "lock="
for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
set "lock=%%T"
goto :break
)
:break
set "lock=%temp%\lock%lock%_%random%_"
:: Initialize the counters
set /a "startCount=0, endCount=0"
:: Clear any existing end flags
for /l %%N in (1 1 %maxProc%) do set "endProc%%N="
:: Launch the commands in a loop
set launch=1
for /f "tokens=* delims=:" %%F in ('%listCmd%') do (
if !startCount! lss %maxProc% (
set /a "startCount+=1, nextProc=startCount"
) else (
call :wait
)
set cmd!nextProc!=%runCmd%
echo -------------------------------------------------------------------------------
echo !time! - proc!nextProc!: starting %runCmd%
2>nul del %lock%!nextProc!
%= Redirect the lock handle to the lock file. The CMD process will =%
%= maintain an exclusive lock on the lock file until the process ends. =%
start /b "" cmd /c >"%lock%!nextProc!" 2^>^&1 %runCmd%
)
set "launch="
:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
:: redirect stderr to null to suppress any error message if redirection
:: within the loop fails.
for /l %%N in (1 1 %startCount%) do 2>nul (
%= Redirect an unused file handle to the lock file. If the process is =%
%= still running then redirection will fail and the IF body will not run =%
if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
%= Made it inside the IF body so the process must have finished =%
echo ===============================================================================
echo !time! - proc%%N: finished !cmd%%N!
type "%lock%%%N"
if defined launch (
set nextProc=%%N
exit /b
)
set /a "endCount+=1, endProc%%N=1"
)
)
if %endCount% lss %startCount% (
timeout /t 1 /nobreak >nul
goto :wait
)
2>nul del %lock%*
echo ===============================================================================
echo Thats all folks
Easy solution with a random window title (based on user2956477 idea):
set windowtitle=Worker_%random%%random%%random%
start /min cmd /c "title %windowtitle% & dosomething.bat file1"
start /min cmd /c "title %windowtitle% & dosomething.bat file2"
start /min cmd /c "title %windowtitle% & dosomethingelse.exe"
timeout 3 >nul
:LOOP
tasklist /V /FI "imagename eq cmd.exe" /FI "windowtitle eq %windowtitle%*" | findstr %windowtitle% >nul
if "%errorlevel%"=="0" timeout 1 >nul && goto LOOP
Here is part of a Windows batch file that I already have:
#echo off
setlocal ENABLEDELAYEDEXPANSION
set files=0
for /F "tokens=*" %%A in (filelist.csv) do (
SET /A files=!files! + 1
set var!files!=%%A
)
set var
findstr /i /c:"%var5%" trans.log
if %errorlevel% EQU 0 del %var5%
In this batch file, "filelist.csv" is just a list of files in a particular folder that is automatically generated. The quantity and names of the files in this folder change on a daily basis and when they change the list is updated. There is one file per line in "filelist.csv". The contents of each line in the file are assigned a variable named var1 through varN, and values of these variables correspond to actual file names.
The batch file then searches through another file called "trans.log". In the specific case above it searches for the file that corresponds to "var5". If it finds that file mentioned in "trans.log" then it gets deleted.
What I need to figure out is a way for this last bit of the batch:
findstr /i /c:"%var5%" trans.log
if %errorlevel% EQU 0 del %var5%
to loop for as many times as there are variables "var1" through "varN" and each time it loops, to automatically replace the variable with the next one in the series. So in the case above, if there were 10 files in the list then there would be variables "var1" through "var10". The next time the loop runs from where it is above, it would replace "var5" with "var6" and then "var7", all the way to "var10" and then it would exit the loop and move on to any other commands that come after it in the batch.
Can anyone help me with this, please? TIA
#echo off
setlocal
for /F "skip=4tokens=*" %%A in (filelist.csv) do (
findstr /i /c:"%%A" trans.log > nul
if not errorlevel 1 ECHO del %%A
)
This should do what you appear to want to do. Without sample data, testing is impossible.
The skip=4 skips the first 4 lines of filelist.csv and feeds each succeeding line to FINDSTR which sets errorlevel to 0 if the filename is found in trans.log. the if not errorlevel 1 tests for errorlevel being 0 - an the set/test cycle is executed for each line after the first 4 (skipped) lines.
Note that your original scheme would fail (if I understand your purpose correctly) if file5 was not in trans.log. but file6 (or any later) was in trans.log - file5 would be erroneously deleted and the target retained. You'd also get a series of errors for hits after file5 because, having deleted file5, re-executing the delete on file5 would find it was absent...
Naturally, the del command is simply echoed - change echo del to del after verification to actually delete the targets.
I think this does what you're asking.
#ECHO OFF
#echo off
setlocal ENABLEDELAYEDEXPANSION
set files=0
for /F "tokens=*" %%A in (text.txt) do (
SET /A files=!files! + 1
set var!files!=%%A
for %%i in (!files!) do (CALL :MyFunction !var%%i!)
)
REM
REM Other code can go here...
REM
GOTO END
:MyFunction
ECHO findstr /i /c:"%1" trans.log
ECHO if %errorlevel% EQU 0 del %1
:END
Remove the ECHO statements in MyFunction to actually run findstr and delete the file.
I am trying to create a batch file to take photos off a camera or sd drive (which are specific and don't change.
I then want to move all the *.jpg to a different drive on the company's internal network.
The way it's set up it will look like n:\jobnumberfolder\pictures.jpg
I need the files to be able to be renamed like: "vpi(1).jpg", "vpi(2).jpg" and so on (the same thing you accomplish in Windows by highlighting multiple files and clicking rename.
The batch file will prompt the user for a job number (which will be the folder it will be moved to), and a description (what to name the file).
My programming experience is limited and in php and python, but I do understand the very basics: loops, if-else, arrays,... things like that.
My problem is that with the batch file, I cannot seem to find a good way to create a for loop to get what I want. I think I could do this in python using the os module, but I feel like I'm missing something simple (like a simple command or something I could be using for Windows).
I cannot even get a variable to increment, even with delayed expansion command. Even if I could, would it be possible to add it to the file name? I tried to find an answer for this, but have not been able to.
So my question are:
Can i actually do this in a batch file?
Would it be easier to just write a python script to do it, which will cause me to have to install python on the company's computer? I just want to be able to rename a file with a incrementing number at the end.
Something like this is what I'm looking for
i = 0
name = "whatever"
for each jpg in camera/images
rename each to whatever + i
i+=1
Any help would be appreciated.
Change n:\ in two places and f:\cam-folder\ where needed. It's untested:
#echo off
:loop
cls
echo Enter "jobnumberfolder,description" with comma and no quotes:
echo similar to 986,vpi
set "var="
set /p "var="
for /f "tokens=1,* delims=," %%a in ("%var%") do set "job=%%a"&set "desc=%%b"
echo "%job%" "%desc%" - correct? press enter to continue or N to start again:
set "var="
set /p "var="
if defined var goto :loop
md "n:\%job%\" 2>nul
setlocal enabledelayedexapansion
echo.
set c=0
for %%a in ("f:\cam-folder\*.jpg") do (
set /a c+=1
echo copying "n:\%job%\%desc%(!c!)%%~xa"
copy "%%a" "n:\%job%\%desc%(!c!)%%~xa" >nul
)
echo done
pause
#echo off
setlocal enableextensions enabledelayedexpansion
rem configuration
set "sourceFolder=w:\images"
set "targetFolder=n:\jobs"
rem test if there is something to copy
if not exist "%sourceFolder%\*.jpg" (
echo NO FILES FOUND
goto endProcess
)
:askJob
rem ask job number
set /p "jobNumber=Job Number : "
rem test if no job number
if "%jobNumber%"=="" (
echo NO JOB NUMBER - End of process
goto endProcess
)
rem test for existing job number
if exist "%targetFolder%\%jobNumber%" (
echo DUPLICATE JOB
goto askJob
)
rem ask for description
set /p "description=Description : "
rem test for no description
if "%description%"=="" (
echo NO DESCRIPTION - End of process
goto endProcess
)
rem WARNING - No test for valid job number nor valid description
rem WARNING - Don't shoot your own foot, or add code to validate
rem WARNING - according to your needs
rem create target job folder
set "target=%targetFolder%\%jobnumber%"
mkdir "%target%" > nul 2>nul
rem DO THE COPY
rem Generate a file list (dir), numerate it (findstr /n) and for each line
rem in generated list, split the number from the filename (for with delims)
rem and copy source file to numbered target file
for /f "tokens=1,* delims=:" %%f in ('dir /tc /od /b "%sourceFolder%\*.jpg" ^| findstr /n "^"') do (
echo transfer: "%sourceFolder%\%%g" == "%target%\%description%(%%f).*"
copy "%sourceFolder%\%%g" "%target%\%description%(%%f).*" >nul 2>nul
)
echo FILES COPIED
:endProcess
rem Clean
endlocal
rem and exit
exit /b