Batch file for /f isnt reading files - windows

Lots of issues With batch coding... Things just don't seem to work the way i expect them to.
So I am Using a batch file to Extract Text data from .as Files. I managed to get that working However It creates a bunch of Junk/empty txt files that don't have any useful content extracted from them. So I made another batch file that gets called from main.bat and is supposed to Clean the empty files, however the variables are incorrect even though the for loop in main.bat is almost identical.
Full Copies of the batch files main.bat and Clean.bat
The Issue Is the Bit of Code below (From Clean.bat line 33) #note the following code had most of its counters and echos removed
#echo off
cls
setlocal enabledelayedexpansion
for /r %%Z in (*.txt) do (
SET /a count=0
for /F "eol= tokens=1 delims=," %%A in ("%%Z") do (
if /i %%A==LN set /a count+=1
)
if !count! EQU 0 (
rem del "%%Z")
)
pause
In this Code %%A and %%Z Are both Equal to The Full File Path of the file that should be getting read from. When %%A should be token 1 of The txt file in question. Because of this count always = 0 so it always deletes the file (thats why del is commented out).
Here's an Example of the file its supposed to read from
LN,296,textE("海沿いに立つ高級フィットネスリゾート施設。");
LN,299,textE("夏休み、俺たち兄妹は、陸上の強化合宿という「名目」のもと、\nこの施設を訪れていた。");
LN,302,textE("莉 央\nすごい、すごい!!おしゃれなところだねー!");
Basically Its supposed to check each line and if LN doesnt Exist as the first token to any line it deletes the file. (Because that means the file is empty except for a Line count of the Original .as file)

This may work for you given the things you have said:
I interpreted your point to mean that if the file doesn't contain LN as the first token on any line, then it is to be deleted.
#echo off
for /r %%Z in (*.txt) do (
SET "count="
for /F "usebackq delims=," %%A in ("%%Z") do (
if /i "%%A"=="LN" set count=1
)
if not defined count echo del "%%Z"
)
pause
This is another way to do the same thing:
#echo off
for /r %%Z in (*.txt) do (
findstr "^LN," "%%Z" >nul || echo del "%%Z"
)
pause

Related

How to make a batch file loop resume from where it was by looping through the files listed on a .txt?

I want to start down mixing hundreds of surround media files in a folder to stereo, but this is a process that will take a lot of time. I'd like to create a batch file that executes my ffmpeg command to this set of files (probably listed in a .txt with dir /s /b) that I can run whenever my PC is on, but also keeps a record of already processed files to be excluded on the next run.
I know I can easily keep track of already processed files by simply adding something like if errorlevel 0 echo "%%~fg">>processed.txt to my loop, but I'm finding it challenging to come up with a way to ignore these files when running the script the next time.
Of course I could always manually edit the file list to be looped and remove the ones already processed, but I wonder if there is a clever way to do it programatically
An example of using a log with findstr. replace the definition of Set "SourceList=%~dp0list.txt" with the filepath of the file used to store the list of files for processing, or modify the for /f loop options to iterate over the output of your Dir command.
#Echo off
If not exist "%TEMP%\%~n0.log" break >"%TEMP%\%~n0.log"
Set "SourceList=%~dp0list.txt"
For /f "usebackq delims=" %%G in ("%sourceList%")Do (
%SystemRoot%\System32\Findstr.exe /xlc:"%%G" "%TEMP%\%~n0.log" >nul && (
Rem item already processed and appended to log. do nothing.
) || (
Rem item does not exist in log. Proccess and append to log.
Echo(Proccess %%G
>>"%TEMP%\%~n0.log" Echo(%%G
)
)
I managed to produce a way that doesn't rely on external tools and executes the job only with for loops. As it was needed to use UTF-8, extra steps were taken to properly revert the codepage after the batch file ended:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
for /F "tokens=*" %%G in ('%SystemRoot%\System32\chcp.com') do for %%H in (%%G) do set /A "CodePage=%%H" 2>nul
%SystemRoot%\System32\chcp.com 65001 >nul 2>&1
if not exist "%~dpn0.log" break >"%~dpn0.log"
set "FileList=%~dp0FileList.txt"
set "LogFile=%~dpn0.log"
for /f "usebackq delims=" %%G in ("%FileList%") do (
set "SkipFile="
for /f "usebackq delims=" %%H in ("%LogFile%") do (
if "%%G" == "%%H" (
set "SkipFile=1"
)
)
if not defined SkipFile (
echo --^> Processing file "%%~nG"
rem file is processed
) else (
echo "%%~nG" has already been processed
rem file is not processed
)
)
%SystemRoot%\System32\chcp.com %CodePage% >nul
endlocal
As can be seen, part of this answer is inspired by #T3RROR's!

How to merge ONLY unique records from several CSVs with windows batch scripting?

I need a big help from the community, please if somebody can give me some hints. I have the following windows batch script which is supposed to read more than 10 million records as different CSV files and merge them all together. I am running the script on the server. So it's not very slow. But the problem is that the code doesn't handle duplicated records. I am not sure how to change the script in order to handle the duplication records and only passed unique records. I would be very very appreciated for your help.
rem Set current working directory to Task folder
set FilePath=%~dp0
set FolderPath=%FilePath:~0,-1%
rem Set Space environment variables
call "%FolderPath%"\..\SpaceEnv.bat
rem Set Task specific environment variables
set TaskName=MergeCSVfiles
set fileName=result.csv
set LogFile=%TaskName%_%LogDateTime%.log
:begin
cd ..
cd "Source Files\DCM_Source\Inbox"
echo Staring merge %fileName% at: %time%
setlocal enabledelayedexpansion
set "first=1"
>%fileName% (
for %%F in (msource*.csv) do (
if not "%%F"=="%fileName%" (
set /p "header="<"%%F"
if defined first (
type "%%F"
set "first="
) else (
type "%%F" |find /V "!header!"
)
)
)
)
endlocal
echo Finish merging %fileName% at: %time%
******UPDATED******
Example of CSV file
Sites|Level 2 sites|Date-time (visit start)|Visit ID|Unique visitor ID|Date-time (event)|Sources|Visitor categories|Visitor ID|Visits
SE Romania|PRM|2018-01-01T00:30:04|1|-6427177464|2018-01-01T00:30:04|Portal sites|-|0|2
SE Romania|PRM|2018-01-01T00:30:04|1|-6427177464|2018-01-01T00:30:04|Portal sites|-|0|2
This code will dedupe a file. In order to do that it must be sorted. This means any header record at the top of the file will be sorted into the file. This is code I received from dbenham. I can't remember if he originally posted it on StackOverflow or DosTips.com. If the file is very large it will more than likely crash with an out of memory error.
#echo off
:: Call function to dedupe file
CALL :DEDUPE "filename.txt"
goto :eof
:DEDUPE
:: DEDUPE file
setlocal disableDelayedExpansion
set "file=%~1"
set "sorted=%file%.sorted"
set "deduped=%file%.deduped"
::Define a variable containing a linefeed character
set LF=^
::The 2 blank lines above are critical, do not remove
sort "%file%" >"%sorted%"
>"%deduped%" (
set "prev="
for /f usebackq^ eol^=^%LF%%LF%^ delims^= %%A in ("%sorted%") do (
set "ln=%%A"
setlocal enableDelayedExpansion
if /i "!ln!" neq "!prev!" (
endlocal
(echo %%A)
set "prev=%%A"
) else endlocal
)
)
>nul move /y "%deduped%" "%file%"
del "%sorted%"
GOTO :EOF
#ECHO OFF
SETLOCAL
SETLOCAL ENABLEDELAYEDEXPANSION
SET "sourcedir=U:\sourcedir"
SET "filenamecommon=q49264647*.csv"
:: switch to required source directory
PUSHD "%sourcedir%"
:: get header line
FOR %%f IN (%filenamecommon%) DO FOR /f "delims=" %%h IN (%%f) DO SET "header=%%h"&goto gotheader
:gotheader
COPY %filenamecommon% atempfilename
SET "lastline="
>resultfilename (
ECHO %header%
SETLOCAL enabledelayedexpansion
FOR /f "delims=" %%d IN ('sort atempfilename' ) DO (
IF "%%d" neq "!lastline!" IF "%%d" neq "%header%" ECHO %%d
SET "lastline=%%d"
)
endlocal
)
DEL atempfilename
popd
GOTO :EOF
You would need to change the setting of sourcedir to suit your circumstances.
I used file/dirctorynames that suit my system for testing.
Note : datafiles containing the characters ! or ^ or unbalanced " will not be processed correctly.
First, find the header line by setting header from any matching filename. Once header is set, forcibly abort the for loops.
copy and concatenate all of the required files to a tempfile.
output the header line, then sort the tempfile to group identical lines. Read the result and output only those lines that differed from the previous and were not header lines.
Applying /i to the if statements will make the entire routine disregard character-case.
Sort the tempfile
Ok. Give this code a try. I think this code would generate the result file with not duplicated records not matters its size. However, the time the program will take depends on several factors, although IMHO it should not be excessive because the core part of the process is based on findstr.exe command.
#echo off
setlocal
del result.csv 2>NUL
rem Process all input files
for /F "delims=" %%f in ('dir /B /O:-S msource*.csv') do (
echo Merging file: %%f
if not exist result.csv (
rem Initialize output file with first input file
copy "%%f" result.csv > NUL
) else (
rem Get records in this file that are not in result file
findstr /V /G:result.csv "%%f" > newRecords.csv
rem and add they to the result file
type newRecords.csv >> result.csv
)
)
del newRecords.csv
You may also try to eliminate the dash in /O:-S switch of dir command; perhaps this change will speed up the process a little...

Reading line by line from one file and write to another file using batch script

In below code i am tring to fetch the line no of string "AXX0000XXXA" from file data.txt,then fetching line by line and printing target.txt file,in between if the line reach the find line no i am adding one more line from file temp.txt.The code is working fine with the less nos of records(tested with 150 lines-File Size 100 kb),but when i am processing with 50K records(File Size 25MB) it is taking more then 25 minutes to process.could you please help me how i will process same in less time.
#echo off
setlocal enabledelayedexpansion
for /f "delims=:" %%a in ('findstr /n "AXX0000XXXA" "C:\Users\23456\Desktop\data.txt"') do (set find_line=%%a)
set /a counter=0
for /f "usebackq delims=" %%b in (`"findstr /n ^^ C:\Users\23456\Desktop\data.txt"`) do (
set curr_line=%%b
set /a counter=!counter!+1
if !counter! equ !find_line! (
type temp.txt >> target.txt
)
call :print_line curr_line
)
endlocal
:print_line
setlocal enabledelayedexpansion
set line=!%1!
set line=!line:*:=!
echo !line!>>target.txt
endlocal
Your code uses three Batch file constructs that are inherently slow: call command, >> append redirection and setlocal/endlocal, and these constructs are executed once per each file line! It would be faster to include the subroutine into the original code to avoid the call and setlocal commands, and an echo !line!>>target.txt command imply open the file, search for the end, append the data and close the file, so it is faster to use this construct: (for ...) > target.txt that just open the file once. An example of a code with such changes is in Compo's answer.
This is another method to solve this problem that may run faster when the search line is placed towards the beginning of the file:
#echo off
setlocal enabledelayedexpansion
for /f "delims=:" %%a in ('findstr /n "AXX0000XXXA" "C:\Users\23456\Desktop\data.txt"') do (set /A find_line=%%a-1)
call :processFile < "C:\Users\23456\Desktop\data.txt" > target.txt
goto :EOF
:processFile
rem Duplicate the first %find_line%-1 lines
for /L %%i in (1,1,%find_line%) do (
set /P "line="
echo !line!
)
rem Insert the additional line
type temp.txt
rem Copy the rest of lines
findstr ^^
exit /B
This should create target.txt with content matching data.txt except for an inserted line taken from tmp.txt immediately above the line matching the search string, AXX0000XXXA.
#Echo Off
Set "fSrc=C:\Users\23456\Desktop\data.txt"
Set "iSrc=temp.txt"
Set "sStr=AXX0000XXXA"
Set "fDst=target.txt"
Set "iStr="
Set/P "iStr="<"%iSrc%" 2>Nul
If Not Defined iStr Exit/B
Set "nStr="
For /F "Delims=:" %%A In ('FindStr/N "%sStr%" "%fSrc%" 2^>Nul') Do Set "nStr=%%A"
If Not Defined nStr Exit/B
( For /F "Tokens=1*Delims=:" %%A In ('FindStr/N "^" "%fSrc%"') Do (
If "%%A"=="%nStr%" Echo %iStr%
Echo %%B))>"%fDst%"
I have made it easy for you to change your variable data, you only need to alter lines 3-6.
I have assumed that this was your intention, your question was not clear, please accept my apologies if I have assumed incorrectly.

Windows Batch renaming from text document

I made a Batch script to rename a large amount of files. It takes their name and searches for it in a text document, copies the line and takes the data I need from it and then renames the file.
It seems to work fine for the most part, but I can't check to see how it's doing because it is constantly producing errors/warnings in the console.
#echo off
set ogg=.ogg
Setlocal EnableDelayedExpansion
for %%a in (*.ogg) do (
set fileNameFull=%%a
set fileName=!fileNameFull:~0,-4!
for /F "delims=" %%a in ('findstr /I !fileName! strings.txt') do (
endlocal
set "stringLine=%%a%ogg%"
)
Setlocal EnableDelayedExpansion
set fullString=!stringLine:~26!
ren %%a "!fullString!"
)
pause
The code works, I'd just like to be able to track progress, as 10,000s of files are being renamed at a time and I've no indication of how far along the process is.
The errors are:
"FINDSTR: Cannot open [...]"
"The syntax of the command is incorrect."
#echo off
Setlocal EnableDelayedExpansion
for %%a in (*.ogg) do (
for /F "delims=" %%q in ('findstr /I /L /c:"%%~na" strings.txt') do (
set "stringLine=%%q"
)
ECHO ren "%%a" "!stringLine:~26!.ogg"
)
pause
This code should be equivalent, but fixed, to the code you've posted.
Fixes:
Removed the endlocal/setlocal complication - not required
changed the inner `for` metavariable - must not be duplicate `%%a`
Changed the `findstr` switches - add `/L` for literal and `/c:` to force single token in case of a separator-in-name; use `%%~na` to specify "the name part of `%%a`" to avoid the substringing gymnastics.
removed said gymnastics
Removed 2-stage string manipulation of destination filename
Removed superfluous setting of `ogg`
The resultant code should duplicate what you have originally, except that it will simply report the rename instruction. You should test this against a small representative sample to verify.
for counting/progress:
set /a count=0
for %%a in (*.ogg) do (
for /F "delims=" %%q in ('findstr /I /L /c:"%%~na" strings.txt') do (
set "stringLine=%%q"
)
ECHO ren "%%a" "!stringLine:~26!.ogg"
set /a count +=1
set /a stringline= count %% 1000
if %stringline% equ 0 echo !count! Processed
)
pause
which should show you progress each 1000.
You could use
if %stringline% equ 0 echo !count! Processed&pause
to wait for user-action before progressing...
BTW -I'm making the assumption that the newname is from column 27+ in your file, since you've not shown us a sample.Also, you should be aware that a simple findstr would locate the target string as a substring anywhere within the file - either as the newname or the oldname. If you invoke the /B switch on the findstr, then the string will match at the very beginning of the line only.

Insert Into Multiple Files in Windows shell script

Basically I'm trying to write a batch file to insert some code into multiple files. Here are the details of what I'm tring to accomplish:
1. The input string comes from a file test.txt.
2. The string needs to be inserted as the second line of destination files.
3. Destination files are all the .xml files under the same direction as the batch file.
I suppose I should use a FOR loop to go through all .xml files. Something like
for /f %%i in ('dir /b *.xml') do ()
I've read though some tutorials and posts but can't find a way to add anything to files in a loop. Using Echo or TYPE doesn't seems to work for each file in a loop. How do I modify files in a loop?
Also to insert to a certain number of line some post say the file needs to be put into a variable. But my files are pretty large, which I don't want to put into variables. Is there another way to insert into a certain line in a file?
#ECHO OFF
SETLOCAL
FOR /f "delims=" %%i IN ('dir /b *.xml') DO (
SET line2=Y
(
FOR /f "usebackqdelims=" %%x IN ("%%i") DO (
ECHO(%%x
IF DEFINED line2 TYPE Line2.txt
SET "line2="
)
)>"%%~ni.lmx"
)
GOTO :EOF
This should work for you - but it will delete empty lines.
#echo off
set /P string=< test.txt
for %%a in (*.xml) do (
(for /F "usebackq tokens=1* delims=:" %%b in ('findstr /N "^" "%%a"') do (
if %%b equ 2 echo %string%
set "line=%%c"
setlocal EnableDelayedExpansion
echo(!line!
endlocal
)) > "%%a.new"
)
New files have .xml.new extension; you may add a couple lines to delete original .xml files and rename .xml.new ones to .xml.

Resources