I recently started converting all my video files to MP4 for streaming purposes, and was using batch so far without issues.
Now I have 6000+ videos to convert and needed some sort of queue system I could manipulate.
This is my current code to convert everything from directories into commands in a separate folder.
#echo off
setlocal disabledelayedexpansion
set dirs=D:\ANIME E:\ANIME F:\ANIME
for %%a in (%dirs%) do (
echo %%a
pushd %%a
for /f "tokens=* delims= " %%f IN ('dir/b/s *.avi *.mkv *.m4v *.mpeg *.mpg') do (
echo D:\handbrake\HandBrakeCLI.exe -i "%%f" -e qsv_h264 -q 20 -E fdk_haac -B 196 -R Auto -D 0,0,0,0 --audio-copy-mask aac,ac3,dtshd,dts,mp3 --audio-fallback copy:* --subtitle=1 --subtitle-default --subtitle-forced --subtitle-burn -o "%%~dpf%%~nf.mp4" > "D:\batch\%%~nf.bat"
echo del /F "%%f" >> "D:\batch\%%~nf.bat"
)
)
This works great and makes me 1000's of bat files to call and that is where the issue arises.
A lot of files have special characters like ! in their names which messes up the for loop I use to loop through these. I set the file to read-only before processing, so my other batch won't try to convert the same file (I run 3 at once).
I found a workaround using ENABLEDELAYEXPANSION, but after a few loops I will get the error saying I exceeded the maximum calls for that.
#echo off
setlocal enableextensions
echo Starting loop
goto loop
:loop
for %%f in (D:\batch\*.bat) do (
set ATTRIBS=%%~af
setlocal enabledelayedexpansion
set READ_ATTRIB=!ATTRIBS:~1,1!
if !READ_ATTRIB!==- (
setlocal disabledelayedexpansion
attrib +R "%%f"
call "%%f"
)
timeout /t 2 /nobreak
goto loop
)
endlocal
Normally you'd say just change the ! variables to % and remove delayedexpansion, but that gives me errors in getting the file attributes.
set READ_ATTRIB=%ATTRIBS:~1,1%
if %READ_ATTRIB%==- (
Will give me no result and the error ( is unexpected
If I echo READ_ATTRIB the result will be ~1,1 always and when echoing others I will get ECHO IS OFF.
Is there any way to not use delayedexpansion or in a way that I won't get errors after a few loops?
32 is the deepest level of setlocal when used in batch file. You need to use setlocal - endlocal pair analogously to a pair of ( left and ) right parentheses. See capitalized ENDLOCAL below.
#echo off
setlocal enableextensions
echo Starting loop
goto loop
:loop
for %%f in (D:\batch\*.bat) do (
set ATTRIBS=%%~af
setlocal enabledelayedexpansion
set READ_ATTRIB=!ATTRIBS:~1,1!
if !READ_ATTRIB!==- (
setlocal disabledelayedexpansion
attrib +R "%%f"
call "%%f"
ENDLOCAL
)
ENDLOCAL
timeout /t 2 /nobreak
REM this was totally mazy: goto loop
)
endlocal
Moreover, latter goto loop in your original code causes that for %%f loop is initialised over and over indefinitely against the same data set, i.e. %%f in the loop body refers to the only file all the time. See capitalized REM.
Related
I want to start down mixing hundreds of surround media files in a folder to stereo, but this is a process that will take a lot of time. I'd like to create a batch file that executes my ffmpeg command to this set of files (probably listed in a .txt with dir /s /b) that I can run whenever my PC is on, but also keeps a record of already processed files to be excluded on the next run.
I know I can easily keep track of already processed files by simply adding something like if errorlevel 0 echo "%%~fg">>processed.txt to my loop, but I'm finding it challenging to come up with a way to ignore these files when running the script the next time.
Of course I could always manually edit the file list to be looped and remove the ones already processed, but I wonder if there is a clever way to do it programatically
An example of using a log with findstr. replace the definition of Set "SourceList=%~dp0list.txt" with the filepath of the file used to store the list of files for processing, or modify the for /f loop options to iterate over the output of your Dir command.
#Echo off
If not exist "%TEMP%\%~n0.log" break >"%TEMP%\%~n0.log"
Set "SourceList=%~dp0list.txt"
For /f "usebackq delims=" %%G in ("%sourceList%")Do (
%SystemRoot%\System32\Findstr.exe /xlc:"%%G" "%TEMP%\%~n0.log" >nul && (
Rem item already processed and appended to log. do nothing.
) || (
Rem item does not exist in log. Proccess and append to log.
Echo(Proccess %%G
>>"%TEMP%\%~n0.log" Echo(%%G
)
)
I managed to produce a way that doesn't rely on external tools and executes the job only with for loops. As it was needed to use UTF-8, extra steps were taken to properly revert the codepage after the batch file ended:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
for /F "tokens=*" %%G in ('%SystemRoot%\System32\chcp.com') do for %%H in (%%G) do set /A "CodePage=%%H" 2>nul
%SystemRoot%\System32\chcp.com 65001 >nul 2>&1
if not exist "%~dpn0.log" break >"%~dpn0.log"
set "FileList=%~dp0FileList.txt"
set "LogFile=%~dpn0.log"
for /f "usebackq delims=" %%G in ("%FileList%") do (
set "SkipFile="
for /f "usebackq delims=" %%H in ("%LogFile%") do (
if "%%G" == "%%H" (
set "SkipFile=1"
)
)
if not defined SkipFile (
echo --^> Processing file "%%~nG"
rem file is processed
) else (
echo "%%~nG" has already been processed
rem file is not processed
)
)
%SystemRoot%\System32\chcp.com %CodePage% >nul
endlocal
As can be seen, part of this answer is inspired by #T3RROR's!
I made this script that finds all directories and echoes the directorie's name to a .txt file. The script is working but it ends up echoing only %A without any value. My script is below!
set /a count=0
setlocal EnableDelayedExtensions
FOR /D %%A in ("*") DO (call :sub)
endlocal
pause
exit
:sub
(echo [DIR] %%A)>>%count%.txt
set /a count+=1
The output in the .txt files is [DIR] %A.
Any idea how to fixe this? Thanks -David
First remark, you are using an invalid option for setlocal but that is probably just a typo.
The problem is that you are try to use a for-parameter where it cannot be used.
The rule is "A for-parameter can only be used within the command or command block () of a for loop"
Your subroutine is not within the command block of a for loop, but you can start a dummy for loop in the subroutine which will give you access to all available for-parameters.
set /a count=0
setlocal EnableDelayedExpansion
FOR /D %%A in ("*") DO (call :sub)
endlocal
pause
exit/b
:sub
For %%. in (.) do (echo [DIR] %%A)>>%count%.txt
set /a count+=1
You need to pass the parameter to the subroutine.
From https://www.informit.com/articles/article.aspx?p=1154761&seqNum=11 :
for %%f in (*.dat) do call :onefile %%f
exit /b
:onefile
echo Processing file %1...
echo ... commands go here ...
exit /b
As you've already enabled delayed expansion, there's no need to use a Call to a label, just do it within the loop.
#Echo Off
SetLocal EnableExtensions EnableDelayedExpansion
Set "count=0"
For /D %%G In (*) Do (
Set /A count += 1
(Echo [DIR] %%G) 1>"!count!.txt"
)
On this version, I've started at 1 instead of 0 for the first text file name, if you really want to start at 0, change line 3 to Set "count=-1"
I need a big help from the community, please if somebody can give me some hints. I have the following windows batch script which is supposed to read more than 10 million records as different CSV files and merge them all together. I am running the script on the server. So it's not very slow. But the problem is that the code doesn't handle duplicated records. I am not sure how to change the script in order to handle the duplication records and only passed unique records. I would be very very appreciated for your help.
rem Set current working directory to Task folder
set FilePath=%~dp0
set FolderPath=%FilePath:~0,-1%
rem Set Space environment variables
call "%FolderPath%"\..\SpaceEnv.bat
rem Set Task specific environment variables
set TaskName=MergeCSVfiles
set fileName=result.csv
set LogFile=%TaskName%_%LogDateTime%.log
:begin
cd ..
cd "Source Files\DCM_Source\Inbox"
echo Staring merge %fileName% at: %time%
setlocal enabledelayedexpansion
set "first=1"
>%fileName% (
for %%F in (msource*.csv) do (
if not "%%F"=="%fileName%" (
set /p "header="<"%%F"
if defined first (
type "%%F"
set "first="
) else (
type "%%F" |find /V "!header!"
)
)
)
)
endlocal
echo Finish merging %fileName% at: %time%
******UPDATED******
Example of CSV file
Sites|Level 2 sites|Date-time (visit start)|Visit ID|Unique visitor ID|Date-time (event)|Sources|Visitor categories|Visitor ID|Visits
SE Romania|PRM|2018-01-01T00:30:04|1|-6427177464|2018-01-01T00:30:04|Portal sites|-|0|2
SE Romania|PRM|2018-01-01T00:30:04|1|-6427177464|2018-01-01T00:30:04|Portal sites|-|0|2
This code will dedupe a file. In order to do that it must be sorted. This means any header record at the top of the file will be sorted into the file. This is code I received from dbenham. I can't remember if he originally posted it on StackOverflow or DosTips.com. If the file is very large it will more than likely crash with an out of memory error.
#echo off
:: Call function to dedupe file
CALL :DEDUPE "filename.txt"
goto :eof
:DEDUPE
:: DEDUPE file
setlocal disableDelayedExpansion
set "file=%~1"
set "sorted=%file%.sorted"
set "deduped=%file%.deduped"
::Define a variable containing a linefeed character
set LF=^
::The 2 blank lines above are critical, do not remove
sort "%file%" >"%sorted%"
>"%deduped%" (
set "prev="
for /f usebackq^ eol^=^%LF%%LF%^ delims^= %%A in ("%sorted%") do (
set "ln=%%A"
setlocal enableDelayedExpansion
if /i "!ln!" neq "!prev!" (
endlocal
(echo %%A)
set "prev=%%A"
) else endlocal
)
)
>nul move /y "%deduped%" "%file%"
del "%sorted%"
GOTO :EOF
#ECHO OFF
SETLOCAL
SETLOCAL ENABLEDELAYEDEXPANSION
SET "sourcedir=U:\sourcedir"
SET "filenamecommon=q49264647*.csv"
:: switch to required source directory
PUSHD "%sourcedir%"
:: get header line
FOR %%f IN (%filenamecommon%) DO FOR /f "delims=" %%h IN (%%f) DO SET "header=%%h"&goto gotheader
:gotheader
COPY %filenamecommon% atempfilename
SET "lastline="
>resultfilename (
ECHO %header%
SETLOCAL enabledelayedexpansion
FOR /f "delims=" %%d IN ('sort atempfilename' ) DO (
IF "%%d" neq "!lastline!" IF "%%d" neq "%header%" ECHO %%d
SET "lastline=%%d"
)
endlocal
)
DEL atempfilename
popd
GOTO :EOF
You would need to change the setting of sourcedir to suit your circumstances.
I used file/dirctorynames that suit my system for testing.
Note : datafiles containing the characters ! or ^ or unbalanced " will not be processed correctly.
First, find the header line by setting header from any matching filename. Once header is set, forcibly abort the for loops.
copy and concatenate all of the required files to a tempfile.
output the header line, then sort the tempfile to group identical lines. Read the result and output only those lines that differed from the previous and were not header lines.
Applying /i to the if statements will make the entire routine disregard character-case.
Sort the tempfile
Ok. Give this code a try. I think this code would generate the result file with not duplicated records not matters its size. However, the time the program will take depends on several factors, although IMHO it should not be excessive because the core part of the process is based on findstr.exe command.
#echo off
setlocal
del result.csv 2>NUL
rem Process all input files
for /F "delims=" %%f in ('dir /B /O:-S msource*.csv') do (
echo Merging file: %%f
if not exist result.csv (
rem Initialize output file with first input file
copy "%%f" result.csv > NUL
) else (
rem Get records in this file that are not in result file
findstr /V /G:result.csv "%%f" > newRecords.csv
rem and add they to the result file
type newRecords.csv >> result.csv
)
)
del newRecords.csv
You may also try to eliminate the dash in /O:-S switch of dir command; perhaps this change will speed up the process a little...
In below code i am tring to fetch the line no of string "AXX0000XXXA" from file data.txt,then fetching line by line and printing target.txt file,in between if the line reach the find line no i am adding one more line from file temp.txt.The code is working fine with the less nos of records(tested with 150 lines-File Size 100 kb),but when i am processing with 50K records(File Size 25MB) it is taking more then 25 minutes to process.could you please help me how i will process same in less time.
#echo off
setlocal enabledelayedexpansion
for /f "delims=:" %%a in ('findstr /n "AXX0000XXXA" "C:\Users\23456\Desktop\data.txt"') do (set find_line=%%a)
set /a counter=0
for /f "usebackq delims=" %%b in (`"findstr /n ^^ C:\Users\23456\Desktop\data.txt"`) do (
set curr_line=%%b
set /a counter=!counter!+1
if !counter! equ !find_line! (
type temp.txt >> target.txt
)
call :print_line curr_line
)
endlocal
:print_line
setlocal enabledelayedexpansion
set line=!%1!
set line=!line:*:=!
echo !line!>>target.txt
endlocal
Your code uses three Batch file constructs that are inherently slow: call command, >> append redirection and setlocal/endlocal, and these constructs are executed once per each file line! It would be faster to include the subroutine into the original code to avoid the call and setlocal commands, and an echo !line!>>target.txt command imply open the file, search for the end, append the data and close the file, so it is faster to use this construct: (for ...) > target.txt that just open the file once. An example of a code with such changes is in Compo's answer.
This is another method to solve this problem that may run faster when the search line is placed towards the beginning of the file:
#echo off
setlocal enabledelayedexpansion
for /f "delims=:" %%a in ('findstr /n "AXX0000XXXA" "C:\Users\23456\Desktop\data.txt"') do (set /A find_line=%%a-1)
call :processFile < "C:\Users\23456\Desktop\data.txt" > target.txt
goto :EOF
:processFile
rem Duplicate the first %find_line%-1 lines
for /L %%i in (1,1,%find_line%) do (
set /P "line="
echo !line!
)
rem Insert the additional line
type temp.txt
rem Copy the rest of lines
findstr ^^
exit /B
This should create target.txt with content matching data.txt except for an inserted line taken from tmp.txt immediately above the line matching the search string, AXX0000XXXA.
#Echo Off
Set "fSrc=C:\Users\23456\Desktop\data.txt"
Set "iSrc=temp.txt"
Set "sStr=AXX0000XXXA"
Set "fDst=target.txt"
Set "iStr="
Set/P "iStr="<"%iSrc%" 2>Nul
If Not Defined iStr Exit/B
Set "nStr="
For /F "Delims=:" %%A In ('FindStr/N "%sStr%" "%fSrc%" 2^>Nul') Do Set "nStr=%%A"
If Not Defined nStr Exit/B
( For /F "Tokens=1*Delims=:" %%A In ('FindStr/N "^" "%fSrc%"') Do (
If "%%A"=="%nStr%" Echo %iStr%
Echo %%B))>"%fDst%"
I have made it easy for you to change your variable data, you only need to alter lines 3-6.
I have assumed that this was your intention, your question was not clear, please accept my apologies if I have assumed incorrectly.
i am writing a batch script monotonic file renamer. basically, it makes the titles of all the files 1 2 3 4 .... and so on. i have since expanded it to be able to handle files of different types (txt, doc, flv, etc) but not everything is working out.
my main concern is i have broken the delayed expansion calls i was making before. now using !var1! is never expanded, or never recognized as a variable.
here is a verbosely commented version of my script
::a monotonic file renamer
#echo off
SETLOCAL ENABLEDELAYEDEXPANSION
SET tempfile=temp.txt
SET exttemp=exttemp.txt
if [%1] == [] goto usage
::make sure your dont overwrite something useful
if EXIST %tempfile% (
ECHO Temp file already exists, are you sure you want to delete?
del /P %tempfile%
)
if EXIST %exttemp% (
ECHO EXT Temp file already exists, are you sure you want to delete?
del /P %exttemp%
)
::initialize
SET /a counter=0
SET type=
SET /a ender=%1
::write filenames to tempfile
DIR /B /ON > %tempfile%
::read lines one by one
for /f "usebackq delims=" %%a in (%tempfile%) do (
REM make sure we do not rename any of the working files
if NOT "%%a"=="renamer.bat" (
if NOT "%%a"=="temp.txt" (
if NOT "%%a"=="exttostr.bat" (
SET /a counter+=1
REM get file extension
exttostr %%a > %exttemp%
SET /P type= < %exttemp%
REM housekeeping
del /F %exttemp%
REM rename
ren %%a !counter!.!type!
ECHO Renamed "%%a" to "!counter!.!type!"
)))
REM exit when we have run enough
if "!counter!"=="!ender!" goto exit
)
goto exit
:usage
echo Usage: renamer NUMFILES
:exit
::final housekeeping
DEL temp.txt
the idea is i drop my two files, renamer.bat(this file) and exttostr.bat(helper to get the file extension) into the folder and run it, it will rename files sorted alphabetically from 1 to how ever many files i specify.
when i run the code, it never uses the variables marked for delayed expansion appropriately, always leaving them as "!varname!", so it renames the first file "!counter!.!type!" and throws errors for the rest because there is already a file in the directory with that name.
this brings me to a secondary issue. sorting the dir list alphabetically results in a poor handling of numbered files. for example the list:
"1 7 15 75 120"
is sorted:
"1 120 15 7 75"
i have not been able to find a way around this yet, only that it is indeed the intended result of the dir sort. the only workaround i have is padding numbers with enough zeroes in the front.
thanks in advance for any insight!
everything is sorted but the second problem. i think i have not spoken well. i have this issue when i take IN the directory file names, not when writing out. so they already need to be padded. i has hoping there was some other way to read the directory and have it be sorted appropriately.
the most promising thing i have found is here: http://www.dostips.com/DtCodeBatchFiles.php#Batch.SortTextWithNumbers
#ECHO OFF
if "%~1"=="/?" (
echo.Sorts text by handling first number in line as number not text
echo.
echo.%~n0 [n]
echo.
echo. n Specifies the character number, n, to
echo. begin each comparison. 3 indicates that
echo. each comparison should begin at the 3rd
echo. character in each line. Lines with fewer
echo. than n characters collate before other lines.
echo. By default comparisons start at the first
echo. character in each line.
echo.
echo.Description:
echo. 'abc10def3' is bigger than 'abc9def4' because
echo. first number in first string is 10
echo. first number in second string is 9
echo. whereas normal text compare returns
echo. 'abc10def3' smaller than 'abc9def4'
echo.
echo.Example:
echo. To sort a directory pipe the output of the dir
echo. command into %~n0 like this:
echo. dir /b^|%~n0
echo.
echo.Source: http://www.dostips.com
goto:EOF
)
if "%~1" NEQ "~" (
for /f "tokens=1,* delims=," %%a in ('"%~f0 ~ %*|sort"') do echo.%%b
goto:EOF
)
SETLOCAL ENABLEDELAYEDEXPANSION
set /a n=%~2+0
for /f "tokens=1,* delims=]" %%A in ('"find /n /v """') do (
set f=,%%B
(
set f0=!f:~0,%n%!
set f0=!f0:~1!
rem call call set f=,%%%%f:*%%f0%%=%%%%
set f=,!f:~%n%!
)
for /f "delims=1234567890" %%b in ("!f!") do (
set f1=%%b
set f1=!f1:~1!
call set f=0%%f:*%%b=%%
)
for /f "delims=abcdefghijklmnopqrstuwwxyzABCDEFGHIJKLMNOPQRSTUWWXYZ~`##$*_-+=:;',.?/\ " %%b in ("!f!") do (
set f2=00000000000000000000%%b
set f2=!f2:~-20!
call set f=%%f:*%%b=%%
)
echo.!f1!!f2!!f!,%%B
rem echo.-!f0!*!f1!*!f2!*!f!*%%a>&2
)
this code can sort the filenames with one number in them (i.e. video100.mov is fine, video100video10.mov would break it)
the issue i have is i think adding a call to this helper fn will break it again, so i will be trying to include this in my modified renamer.bat now. any help is appreciated.
Probably the batch for extracting the extension reset the local environment.
But, you don't need it. You may extract the extension with the ~x option. Something similar to this ....
:monotonicrename
set /a counter = 0
for %%a in (%1\*.*) do (
if exist %%~fa (
set /a counter += 1
echo ren %%~fa !counter!%%~xa
)
)
goto :eof
to include leading zeroes in the counter, so that the directory sorts correctly, replace the previous rename command with three lines
set zcounter=0000!counter!
set zcounter=!zcounter:~-4!
echo ren %%~fa !counter!%%~xa
So putting all pieces together, add the monotonicrename function you just created in the batch file that can be as simpler as...
#echo off
setlocal enabledelayedexpansion
call :monotonicrename %1
goto :eof
:monotonicrename
set /a counter = 0
for %%a in (%1\*.*) do (
if exist %%~fa (
set /a counter += 1
set zcounter=0000!counter!
set zcounter=!zcounter:~-4!
echo ren %%~fa !zcounter!%%~xa
)
)
goto :eof
I didn't experience any issues with delayed expansion, everything worked fine for me (except, of course, for the fact that I didn't have the exttostr.bat helper script.)
Anyway, there are several things that could be improved about your script:
You don't need to store the result of DIR into a file to read it afterwards. You can read the output directly in the FOR loop.
You don't need the helper batch script. The extension can be extracted from %%a by using the ~x modifier with the loop variable: %%~xa. You can read more about modifiers by issuing HELP FOR from the command prompt.
The renamer batch file's own name can be referenced in the script as %0. You can apply the ~n modifier where you only need to use the name without the extension. The combined modifier of ~nx will give you the name with the extension.
So, here's how your script might look like with the above issues addressed:
::a monotonic file renamer
#echo off
SETLOCAL ENABLEDELAYEDEXPANSION
IF [%1] == [] GOTO usage
::initialize
SET /A counter=0
SET type=
SET /A ender=%1
::read lines one by one
FOR /F "usebackq delims=" %%a IN (`DIR /B /ON`) DO (
REM make sure we do not rename any of the working files
IF NOT "%%~a"=="%~nx0" (
SET /A counter+=1
RENAME "%%~a" "!counter!%%~xa"
ECHO Renamed "%%~a" to "!counter!%%~xa"
)
REM exit when we have run enough
IF "!counter!"=="!ender!" GOTO :EOF
)
GOTO :EOF
:usage
ECHO Usage: %~n0 NUMFILES
As for your secondary issue, it can be easily resolved like this:
Use something like 100000 as counter's initial value. (Use however many 0s you like, but possibly no more than nine.) Add the same value to ender as well.
When renaming files, instead of !counter! use the expression that removes the first character (the 1): !counter:~1! (in fact, this is not about removal, but about extracting a substring starting from the offset of 1, learn more about it with the HELP SET command).
Here's the modified version of the above script:
::a monotonic file renamer
#echo off
SETLOCAL ENABLEDELAYEDEXPANSION
IF [%1] == [] GOTO usage
::initialize
SET /A counter=1000
SET type=
SET /A ender=%1
SET /A ender+=counter
::read lines one by one
FOR /F "usebackq delims=" %%a IN (`DIR /B /ON`) DO (
REM make sure we do not rename any of the working files
IF NOT "%%~a"=="%~nx0" (
SET /A counter+=1
RENAME "%%~a" "!counter:~1!%%~xa"
ECHO Renamed "%%~a" to "!counter:~1!%%~xa"
)
REM exit when we have run enough
IF "!counter!"=="!ender!" GOTO :EOF
)
GOTO :EOF
:usage
ECHO Usage: renamer NUMFILES
You can also see that I made some other enhancements, like making sure the file name is enclosed in double quotes, and using GOTO :EOF instead of GOTO exit (:EOF is a special pre-defined label that points at the end of the batch script so you don't need to define your own).