loop through file saving to variable - windows

I now have the following bat file working (which allows one to add text to the end of each line of a file) -- please see also:
bat file: Use of if, for and a variable all together
#echo off
setLocal EnableDelayedExpansion
IF EXIST "%FileToModify1%" (
for /f "tokens=* delims= " %%a in (%FileToModify1%) do (
echo %%a Note: certain conditions apply >> "%SaveFile1%"
)
)
However, I would like to save each line to a variable (including the new line symbol(s)) and then echo the variable to a file at the end. Since there are several lines in the file it is really inefficient to save to a file with each line.
I tried googling this, but the answers do not fit my situation...
essentially I need the syntax for concatenating and saving to a variable (cumulatively like "+=" in C#), and also using the new lines...

Actually you do not need to put everything into a variable, you just need to place the redirection at another position.
Try this:
#echo off
setlocal EnableDelayedExpansion
if exist "%FileToModify1%" (
for /F "usebackq delims=" %%a in ("%FileToModify1%") do (
echo %%a Note: certain conditions apply
)
) > "%SaveFile1%"
endlocal
Note that empty lines in the original file are ignored by for /F, so they are not transferred to the new file. Also lines starting with ; are ignored by for /F (unless you change the eol option -- see for /?).
I modified the for /F options:
no delims are allowed, so the each line is output as is (with "tokens=* delims= ", leading spaces are removed from each line if present);
usebackq allows to surround the file specification in "" which is helpful if it contains spaces;
Appendix A
If you still want to store the file content into a variable, you can do this:
#echo off
setlocal EnableDelayedExpansion
rem the two empty lines after the following command are mandatory:
set LF=^
if exist "%FileToModify1%" (
set "FileContent="
for /F "usebackq delims=" %%a in ("%FileToModify1%") do (
set "FileContent=!FileContent!%%a Note: certain conditions apply!LF!"
)
(echo !FileContent!) > "%SaveFile1%"
)
endlocal
The file content is stored in variable FileContent, including the appendix Note: certain conditions apply. LF holds the new-line symbol.
Note:
The length of a variable is very limited (as far as I know, 8191 bytes since Windows XP and 2047 bytes earlier)!
[References:
Store file output into variable (last code fragment);
Explain how dos-batch newline variable hack works]
Appendix B
Alternatively, you could store the file content in a array, like this:
#echo off
setlocal EnableDelayedExpansion
if exist "%FileToModify1%" (
set /A cnt=0
for /F "usebackq delims=" %%a in ("%FileToModify1%") do (
set /A cnt+=1
set "Line[!cnt!]=%%a Note: certain conditions apply"
)
(for /L %%i in (1,1,!cnt!) do (
echo !Line[%%i]!
)) > "%SaveFile1%"
)
endlocal
Each line of the file is stored in an array Line[1], Line[2], Line[3], etc., including the appendix Note: certain conditions apply. cnt contains the total number of lines, which is the array size.
Note:
Actually this is not a true array data type as such does not exist in batch, it is a collection of scalar variables with an array-style naming (Line[1], Line[2],...); therefore one might call it pseudo-array.
[References:
Store file output into variable (first code fragment);
How to create an array from txt file within a batch file?]

you can write the output file in one shot:
(
for /l %%i in (0,1,10) do (
echo line %%i
)
)>outfile.txt
(much quicker than appending each line separately)

Related

For command token doesn't work when there are empty fields in the record

I am trying to extract the values from the third field of a file which has data records.
The fields are separated by vertical bar characters:
9001||10454145||60|60
9001|234467|10454145||60|60
9001|234457|10454145||60|60
Command is -
for /f "tokens=3 delims=|" %%A IN ('Findstr /i "9001" .\itemloc\%%~nf.dat') do (
echo %%A >> log.txt
)
But the output I am getting is
60
10454145
10454145
The empty fields are messing up my output. Any suggestions how to make the for token work with empty fields in the record?
#ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
rem The following settings for the directories and filenames are names
rem that I use for testing and deliberately includes spaces to make sure
rem that the process works using such names. These will need to be changed to suit your situation.
SET "sourcedir=u:\your files"
SET "destdir=u:\your results"
SET "filename1=%sourcedir%\q75199035.txt"
SET "outfile=%destdir%\outfile.txt"
(
FOR /f "usebackqtokens=1*delims=" %%e IN ("%filename1%") DO (
SET "line=%%e"
FOR /f "tokens=3 delims=|" %%y IN ("!line:||=|(missing)|!") DO ECHO %%y
)
)>"%outfile%"
TYPE "%outfile%"
GOTO :EOF
Always verify against a test directory before applying to real data.
Note that if the filename does not contain separators like spaces, then both usebackq and the quotes around %filename1% can be omitted.
The magic is that for each line, || is replaced by |(missing)|.
This simple solution has its faults - for instance if there is ||| in the source data, or the usual suspects (some punctuation symbols like !) but should be quite happy with alphameric source text.
Another way would be to use a third-party utility like sed to pre-process the source data.
The fundamental reason for this phenomenon is that for/f parses the line as [delimiters]token1[delimiters]token2..., where [delimiters] is any sequence of any of the delimiter characters.

How to extract content of a CSV file into individual files with folder and file name based on the first two data columns?

I have the following request for help with CSV file extracts:
C:\TestDirectory\Parse.csv contains:
User, FileName, HTMLContent
Tom, Doc1, <HTML etc etc etc
Tom, Doc2, <HTML etc etc>
Sue, Other, <HTML etc etc>
I would like a script to create for the example above:
D:\ResultsDirectory\Tom\Doc1.html
D:\ResultsDirectory\Tom\Doc2.html
D:\ResultsDirectory\Sue\Other.html
So far I have:
#echo off
Set file= c:\TestDirectory\Parse.csv
REM get header:
set /p header=<%file%
REM ignore header Skip 1:
REM Process Filename Tokens 2:
for /f "skip=1 delims=" %%A in (%file%) do (
for /f "tokens=2 delims=," %%B in ("%%A") do (
if not exist "%%B.html" echo %header%>"%%B.html"
>>"%%B.html" echo %%A
)
)
pause
Problems: These are my searching limits.
I get the filenames without the correct path.
The file is populated by the whole line from the CSV, and I don't know how to get just what I require.
#ECHO Off
SETLOCAL ENABLEDELAYEDEXPANSION
rem The following settings for the directories and filename are names
rem that I use for testing and deliberately include names which include spaces to make sure
rem that the process works using such names. These will need to be changed to suit your situation.
SET "sourcedir=u:\your files"
SET "destdir=u:\your results"
SET "filename1=%sourcedir%\q72682391.txt"
SET "header="
FOR /f "usebackqdelims=" %%b IN ("%filename1%") DO (
IF DEFINED header (
FOR /f "tokens=1,2,*delims=," %%u IN ("%%b") DO (
FOR /f "tokens=*" %%e IN ("%%v") DO (
MD "%destdir%\%%u" 2>nul
(
ECHO !header!
FOR /f "tokens=*" %%y IN ("%%w") DO ECHO %%y
)>"%destdir%\%%u\%%e.html
)
)
) ELSE SET "header=%%b"
)
GOTO :EOF
Always verify against a test directory before applying to real data.
Outer loop: reads each line of file into %%b. As header is initialised to nothing, on reading the first line, header is not defined, so it is set by the else clause of the if statement to the header line. For every other line, header will be defined, so processing continues:
Next inner loop: %%u is set to the first token (name), %%v to the second (filename) and %%w to the remainder of the line (html).
Next loop: "tokens=*" will strip leading spaces from the string %%v, which includes the space after the comma-delimiter from the previous forso%%e` will contain the filename without the leading space.
Create the destination (name) directory from %%u. The 2>nul suppresses the error reports (like directory already exists)
Enclosing a sequence of lines in parentheses enables us to redirect all of the echoed output to a new file using > rather than having to create using a > and then append using >>.
So, output the contents of header using delayedexpansion syntax Stephan's DELAYEDEXPANSION link
then use the tokens=* mechanism again to output the contents of %%w with the leading spaces suppressed.
Personally, I can't see the point of including the header in the file generated, but that's what you appear to have expected to do with your original code. If you don't want it, simply remove that line.

copying most recent file set from txt in unknown directory

i have a notepad.txt document that lists the files that need to be copied to the folder that holds the batch file. the files are located in several sub directories and with my code, it copies all of the files with specified name.
for /f "delims=" %%i in (testlist.txt) do echo |robocopy "%dir1%." "C:\temporary" "%%i.*" /s /ndl /njs /njh /nc /ts /ns
how do i set this up properly so it will search the most recent file, and copy only the file not the folder and subfolder?
How to get file's last modified date on Windows command line?
for %a in (MyFile.txt) do set FileDate=%~ta
Compare 2 dates in a Windows batch file
set "sdate1=%olddate:~-4%%olddate:~3,2%%olddate:~0,2%"
set "sdate2=%newdate:~-4%%newdate:~3,2%%newdate:~0,2%"
if %sdate1% GTR %sdate2% (goto there) else echo here
So given that you can already read the file and do the copy, here is pseudo code of the logic I would write for putting it all together:
set oldTimeStamp = "1901-01-01" //so first comparison wins and doesn't throw a null error
for each filename in list.txt
set newTimestamp = getTimeStamp(filename)
if newTimeStamp > oldTimeStamp then set fileToCopy = filename
set oldTimeStamp = newTimeStamp
next
doCopy(fileToCopy)
Basically loop through each filename and get the timestamp. Store the timestamp of the previous file and compare the new and old timestamps. If the current one is newer, save the filename to a variable that you will use to copy. At the end of the loop, fileToCopy should contain the name of the file with the most recent modified time.
The following code snippet retrieves the most recent file and echos its path. Here the wmic command is used to get standardised locale-independent timestamps, which can immediately be compared as strings, so it is not necessary to convert them to numbers. So here it is:
#echo off
setlocal EnableExtensions EnableDelayedExpansion
set "RECENT=00000000000000.000000+000"
set "RECENTFILE="
for /F "usebackq eol=| delims=" %%L in ("testlist.txt") do (
setlocal DisableDelayedExpansion
set "CURRFILE=%%~fL"
if exist "%%~fL" (
setlocal EnableDelayedExpansion
for /F "skip=1 tokens=1 delims= " %%T in ('
wmic DATAFILE ^
WHERE Name^="!CURRFILE:\=\\!" ^
GET LastModified ^
/FORMAT:TABLE
') do (
for /F "delims=" %%S in ("%%T") do (
if %%S GTR !RECENT! (
endlocal
endlocal
set "RECENT=%%S"
set "RECENTFILE=%%~fL"
) else (
endlocal
endlocal
)
)
)
) else (
endlocal
)
)
if defined RECENTFILE (
rem Perform your action here:
echo(!RECENTFILE!
)
endlocal
exit /B
What happens:
there are two variables RECENT and RECENTFILE which hold the timestamp of and the path to most recent file, respectively;
the outer for /F loop walks through the items in the list file testlist.txt;
for each existing item, a wmic query is executed to get the last modify date, and its output is parsed by two nested for /F loops, each iterating once only; since wmic returns Unicode strings, a single for /F loop is not enough because it leaves some orphaned carriage-return characters, which may impact the remaining code, but a second loop removes them;
the retrieved file date is compared to the buffered one in RECENT, and if it is greater, meaning that the file is newer, it is stored in RECENT and the respective file path is stored in RECENTFILE;
if variable RECENTFILE is finally not defined, the list testlist.txt does not point to existing files, or it is empty;
the toggling of delayed expansion is necessary to avoid trouble with any special characters;
Besides the fact, that the wmic queries are worse in terms of performance compared to getting the timestamps using for (for instance for %F in ("*.*") do echo %~tF), the following restriction applies:
The , character must not occur in any of the listed file paths!
According to this answer, there is a way to overcome this, but then the ) character is disallowed: to replace the clause WHERE Name^="!CURRFILE:\=\\!" by WHERE ^(Name^="!CURRFILE:\=\\!"^) (the escaping ^ of the parenthesis is only required as the wmic command line is placed within a for /F set). So you can either have , or ) within a wmic command line, but not both of these characters.

Batch File - Insert Line into file

I'm trying to insert a line into a file using the following code (from Write batch variable into specific line in a text file)
#echo off
setlocal enableextensions enabledelayedexpansion
set inputfile=variables.txt
set tempfile=%random%-%random%.tmp
copy /y nul %tempfile%
set line=0
for /f "delims=" %%l in (%inputfile%) do (
set /a line+=1
if !line!==4 (
echo WORDS YOU REPLACE IT WITH>>%tempfile%
) else (
echo %%l>>%tempfile%
)
)
del %inputfile%
ren %tempfile% %inputfile%
endlocal
My problem is the file has comment lines (which start with semicolons) which need to be kept
; directory during network startup. This statement must indicate a local disc
; drive on your PC and not a network disc drive.
LOCALDRIVE=C:\TEMP;
; PANELISATION PART/NET NAMING CONVENTION
; When jobs are panelised, parts/nets are renamed for each panel step by
When I run the batch file, it ignores the semicolon lines, So I only get:
LOCALDRIVE=C:\TEMP;
What do I need to do to keep the semicolon lines?
The EOL option determines what lines are to be ignored. The default value is a semicolon. If you know a character that can never appear in the first position of a line, then you can simply set EOL to that character. For example, if you know a line can't start with |, then you could use
for /f "eol=| delims=" %%l in (%inputfile%) do ...
There is an awkward syntax that disables EOL completely, and also disables DELIMS:
for /f delims^=^ eol^= %%l in (%inputfil%) do ...
Note that FOR /F always discards empty lines, so either of the above would result in:
; directory during network startup. This statement must indicate a local disc
; drive on your PC and not a network disc drive.
LOCALDRIVE=C:\TEMP;
; PANELISATION PART/NET NAMING CONVENTION
; When jobs are panelised, parts/nets are renamed for each panel step by
A trick is used if you want to preserve empty lines. Use FIND or FINDSTR to insert the line number before each line, and then use expansion find/replace to remove the line number. Now you know the line never begins with ;, so you can ignore the EOL option.
for /f "delims=" %%L in ('findstr /n "^" "%inputfile%"') do (
set "ln=%%L"
set "ln=!ln:*:=!"
REM You now have the original line, do whatever needs to be done here
)
But all of the above have a potential problem in that you have delayed expansion enabled when you expand the FOR variable, which means that any content containing ! will be corrupted. To solve this you must toggle delayed expansion on and off within the loop:
setlocal disableDelayedExpansion
...
for /f "delims=" %%L in (findstr /n "^" "%inputfile%") do (
set "ln=%%L"
setlocal enableDelayedExpansion
set "ln=!ln:*:=!"
REM You now have the original line with ! preserved, do whatever needs done here
endlocal
)
Also, when ECHOing an empty line, it will print out ECHO is off unless you do something like
echo(!ln!
It takes time to open and position the write cursor to the end every time you use >> within the loop. It is faster to enclose the entire operation in one set of parentheses and redirect once. Also, you can replace the DEL and REN with a single MOVE command.
Here is a final robust script:
#echo off
setlocal disableDelayedExpansion
set "inputfile=variables.txt"
set line=0
>"%inputfile%.new" (
for /f "delims=" %%L in (findstr /n "^" "%inputfile%") do (
set "txt=%%L"
set /a line+=1
setlocal enableDelayedExpansion
set "txt=!txt:*:=!"
if !line! equ 4 (
echo New line content here
) else (
echo(!txt!
)
endlocal
)
)
move /y "%inputfile%.new" "%inputfile%" >nul
endlocal
That is an awful lot of work for such a simple task, and it requires a lot of arcane knowledge.
There is a much quicker hack that works as long as
your first 4 lines do not exceed 1021 bytes
none of your first 3 lines have trailing control characters that need to be preserved
the remaining lines do not have <tab> characters that must be preserved (MORE converts <tab> into a string of spaces.
#echo off
setlocal enableDelayedExpansion
set "inputfile=variables.txt"
>"%inputfile%.new" (
<"%inputfile%" (
for /l %%N in (1 1 3) do (
set "ln="
set /p "ln="
echo(!ln!
)
)
echo New line content here
more +4 "%inputfile%"
)
move /y "%inputfile%.new" "%inputfile%"
That is still a lot of work and arcane knowledge.
I would use my JREPL.BAT utility
Batch is really a terrible tool for text processing. That is why I developed JREPL.BAT to manipulate text using regular expressions. It is a hybrid JScript/batch script that runs natively on any Windows machine from XP onward. It is extremely versatile, robust, and fast.
A minimal amount of code is required to solve your problem with JREPL. Your problem doesn't really require the regular expression capabilities.
jrepl "^" "" /jendln "if (ln==4) $txt='New content here'" /f "variables.txt" /o -
If used within a batch script, then you must use call jrepl ... because JREPL.BAT is also a batch script.
By default, the FOR command treats ; as the end-of-line character, so all those lines that start with ; are being ignored.
Add eol= to your FOR command, like this:
for /f "eol= delims=" %%l in (%inputfile%) do (
It looks like you're echoing just the line delimiter, not the whole line:
echo %%l>>%tempfile%
I'm rusty on ms-dos scripts, so I can't give you more than that.

Nesting for loop in batch file

I want to nest a for loop inside a batch file to delete carriage return.
I tried it like you can see below but it does not work.
#echo off
setLocal EnableDelayedExpansion
for /f "tokens=* delims= " %%a in (Listfile.txt) do (
set /a N+=1
set v!N!=%%a
)
for /l %%i in (1, 1, %N%) do (
echo !v%%i!
for /r "tokens=* delims=" %%i in (windows.cpp) do (
echo %%i >> Linux11.cpp
)
)
pause
Here I want to check with windows.cpp. If its working I like to change windows .cpp with !v%%i!
You cannot do this in a batch file. You have no way of addressing or writing arbitrary characters. Every tool on Windows normally makes sure to output Windows line breaks (i.e. CR+LF). Some can read Unix-style line breaks just fine, which is why you can easily convert from them. But to them isn't possible.
Also as a word of caution: Source code files often contain blank lines (at least mine do) that are for readability. for /f skips empty lines which is why you're mangling the files for your human readers there. Please don't do that.
As for your question: When nesting two loops you have to make sure that they don't use the same loop variable. Show me a language where code like you wrote actually works.
Something like
for /l %%i in (1, 1, %N%) do (
echo !v%%i!
for /f "tokens=* delims=" %%l in ("!v%%i!") do (
rem do whatever you want to do with the lines
)
)
should probably work better (you missed the final closing parenthesis as well). Thing to remember: If you want to use a certain variable instead of a fixed file name it surely helps replacing that fixed file name by that variable.
It would be probably easiest to use some unix2dos/dos2unix converter to do that or some win32 flavor of sed.
The intrinsic issue of your code is already addressed by another answer, hence I am going to focus on the main task you are trying to accomplish, namely converting DOS/Windows-style end-of-line markers (or line-breaks) to Unix-style ones.
Doing this is very tricky in a batch file, but give the following script a try. Supposing it is called convert.bat, and the original text file is named convert.txt, run the script using the following command line:
convert.bat "convert.txt" LF
The name of the returned file will get the original file name with _converted_EOL appended. The second argument LF specifies Unix-style line-breaks; omitting it will return DOS/Windows-style ones.
So here is the code:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
rem check whether or not an existing file is given as the first argument
>&2 (
if "%~1"=="" (
echo No file specified.
exit /B 2
) else if not exist "%~1" (
echo File "%~1" not found.
exit /B 1
)
)
rem get carriage-return character
for /F %%A in ('copy /Z "%~0" nul') do set "CR=%%A"
rem get line-feed character (the two empty lines afterwards are mandatory!)
(set ^"LF=^
%= blank line =%
^")
rem check which line-break is given by the second argument
rem (`CR` - carriage return (Mac); `LF` - line feed (Unix);
rem anything else or nothing - CR+LF (Windows, default))
setlocal EnableDelayedexpansion
set "BR=!CR!!LF!"
if /I "%~2"=="CR" set "BR=!CR!" & (>&2 echo CR not supported.) & exit /B 3
if /I "%~2"=="LF" set "BR=!LF!"
rem convert line-breaks; append `_converted_EOL` to file name
setlocal DisableDelayedExpansion
> "%~n1_converted_EOL%~x1" (
for /F delims^=^ eol^= %%L in ('
findstr /N /R "^" "%~1"
') do (
set "LINE=%%L"
rem firstly, precede every line with a dummy character (`:`) and
rem append the specified line-break in order to avoid the loss of
rem leading white-spaces or trouble with leading equal-to signs,
rem all caused by `set /P`, which is needed here to return the
rem line without a trailing DOS/Windows-style line-break (opposed
rem to `echo`); then, let `pause` strip off that character;
rem lastly, let `findstr` return the remainder;
rem (the `rem` suffix is just there to fix syntax highlighting)
cmd /V /C ^< nul set /P #="!LINE:*:=:!!BR!" | (> nul pause & findstr "^") & rem/ "^"
)
)
endlocal
endlocal
endlocal
exit /B
The following restrictions apply:
no line must be longer than about 8190 characters (this is a general limitation of batch files);
the file must not contain any null-bytes (well, a normal text file should not hold such, but Unicode-encoded do);
the last line of the returned file will always be terminated by a line-break, even if the respective original line is not;
And here is another solution for line-break conversions: Convert all CR to CRLF in text file using CMD

Resources