Batch command to find/replace text inside file - windows

I have a template file (say myTemplate.txt) and I need to make some edits to create my own file (say myFile.txt) from this template.
So the template contains lines like
env.name=
env.prop=
product.images.dir=/opt/web-content/product-images
Now I want this to be replaced as follows;
env.name=abc
env.prop=xyz
product.images.dir=D:/opt/web-content/product-images
So I am looking for batch commands to do the following;
1. Open the template file.
2. Do a kind of find/replace for the string/text
3. Save the updates as a new file
How do I achieve this ?

The easiest route is to modify your template to look something like this:
env.name=!env.name!
env.prop=!env.prop!
product.images.dir=/opt/web-content/product-images
And then use a FOR loop to read and write the file while delayed expansion is enabled:
#echo off
setlocal enableDelayedExpansion
set "env.name=abc"
set "env.prop=xyz"
(
for /f "usebackq delims=" %%A in ("template.txt") do echo %%A
) >"myFile.txt"
Note it is much faster to use one over-write redirection > for the entire loop then it is to use append redirection >> within the loop.
The above assumes that no lines in template begin with ;. If they do, then you need to change the FOR EOL option to a character that will never start a line. Perhaps equal - for /f "usebackq eol== delims="
Also the above assumes the template doesn't contain any blank lines that you need preserved. If there are, then you can modify the above as follows (this also eliminates any potential EOL issue)
#echo off
setlocal enableDelayedExpansion
set "env.name=abc"
set "env.prop=xyz"
(
for /f "delims=" %%A in ('findstr /n "^" "template.txt"') do (
set "ln=%%A"
echo(!ln:*:=!
)
) >"myFile.txt"
There is one last potential complicating isse - you could have problems if the template contains ! and ^ literals. You could either escape the chars in the template, or you could use some additional substitution.
template.txt
Exclamation must be escaped^!
Caret ^^ must be escaped if line also contains exclamation^^^!
Caret ^ should not be escaped if line does not contain exclamation point.
Caret !C! and exclamation !X! could also be preserved using additional substitution.
extract from templateProcessor.bat
setlocal enableDelayedExpansion
...
set "X=^!"
set "C=^"
...

Related

Extracting URL from text file in Batch

I have a script that needs to extract a YouTube URL from a text file.
Here's what I have in the text file (output.txt):
---------- NUMBER11.TXT
<link itemprop="url" href="http://www.youtube.com/channel/UCnxGkOGNMqQEUMvroOWps6Q">
Note the text file has a line of empty space to start, which is annoying, and the URL is on line 3. Something that doesn't show up in the formatting for this site is the 11 spaces before the actual href starting as well. I'd like to separate it from the mass of other junk.
I've tried something like this:
set /p long= < output.txt
echo %long%
set short1=%long:^<link itemprop^="url" href^="=%
echo %short1% > o1.txt
I thought this would remove the selected text from the file, but I think this is a little over my head.
I'm getting the output.txt from firstly a curl of a youtube video page, and secondly from a find command here:
find "href=""http://www.youtube.com/channel/" %vd% > output.txt
Maybe I'm making this more complicated than it is?
Using batch-files to access files with special characters, like redirect, it can cause some problems, so it is not recommended, but I felt like posting an answer anyway, so given you exact example, here is one way. If your example is not as per your post, which I highly expect it to be, then this probably would not work.
#echo off
setlocal enabledelayedexpansion
for /f "usebackq delims=" %%i in ("output.txt") do for %%a in (%%i) do (
set "var=%%~a"
set "var=!var:>=!"
set "var=!var:"=!"
if "!var:~0,4!" == "http" echo !var!
)
#ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir"
SET "filename1=%sourcedir%\q64572433.txt"
set "url="
FOR /f "tokens=4,5delims=>= " %%a IN (%filename1%) DO if "%%~a"=="href" set "url=%%~b"
echo URL=%url%
GOTO :EOF
You would need to change the setting of sourcedir to suit your circumstances. The listing uses a setting that suits my system.
I used a file named q64572433.txt containing your data for my testing.
The for command tokenises each line of the file, using =, > and space as delimiters (the 3 characters between delims= and ")
On the line of interest, token 4 would be href and token 5 the url - and this is the only line where href is the fourth token. When that is detected, assign the 5th token (in %%b) to the variable, removing the quotes with ~ for good measure.
I would suggest you parse the results directly from your curl command instead of outputting them to a text file, and then using find against that output.
However, instead of using find.exe, I would suggest you use the following method using findstr.exe instead, to get the URL assigned to any line containing href= followed by "http: or "https and subsequently followed by youtube.com.
#Echo Off
SetLocal EnableExtensions DisableDelayedExpansion
For /F Tokens^=*EOL^= %%G In (
'%__APPDIR__%findstr.exe /IR "href=\"http[s:].*youtube\.com" "output.txt"'
) Do (Set "Line=%%G" & SetLocal EnableDelayedExpansion
For /F Tokens^=2Delims^=^" %%H In ("!Line:*href=!") Do EndLocal & Echo %%H)
Pause
If you want the output stored as a variable, instead of Echoing it, change Echo %%H to Set "URL=%%H". You could then use %URL%, (or "%URL%" if you need it doublequoted), elsewhere in your script.

Windows Batch file - strip leading characters

I have a batch file which copies some local files up to a google storage area using the gsutil tool. The gsutil tool produces a nice log file showing the details of the files that were uploaded and if it was OK or not.
Source,Destination,Start,End,Md5,UploadId,Source Size,Bytes Transferred,Result,Description
file://C:\TEMP\file_1.xlsx,gs://app1/backups/file_1.xlsx,2018-12-04T15:25:48.428000Z,2018-12-04T15:25:48.804000Z,CPHHZfdlt6AePAPz6JO2KQ==,,18753,18753,OK,
file://C:\TEMP\file_2.xlsx,gs://app1/backups/file_2.xlsx,2018-12-04T15:25:48.428000Z,2018-12-04T15:25:48.813000Z,aTKCOQSPVwDycM9+NGO28Q==,,18753,18753,OK,
What I would like to do is to
check the status result in column 8 (OK or FAIL)
If the status is OK then move the source file to another folder (so that it is not uploaded again).
The problem is that the source filename is appended with "file://" which I can't seem to remove, example
file://C:\TEMP\file_1.xlsx
needs to be changed into this
C:\TEMP\file_1.xlsx
I am using a for /f loop and I am not sure if the manipulation of the variables %%A is different within a for /f loop.
#echo off
rem copy the gsutil log file into a temp file and remove the header row using the 'more' command.
more +1 raw_results.log > .\upload_results.log
rem get the source file name (column 1) and the upload result (OK) from column 8
for /f "tokens=1,8 delims=," %%A in (.\upload_results.log) do (
echo The source file is %%A , the upload status was %%B
set line=%%A
set line=!line:file://:=! >> output2.txt echo !line!
echo !line!
)
The output is like this.
The source file is file://C:\TEMP\file_1.xlsx , the upload status was OK
The source file is file://C:\TEMP\file_2.xlsx , the upload status was OK
I'm expecting it to dump the altered values out into a new file but it is not producing anything at the moment.
Normally I would extract from a specific character to the end of the string with something like this but it doesn't work with my For/f loop.
%var:~7%
Any pointers or a different way of doing it greatly appreciated.
Since the part to remove seems fixed it is easier to use substrings.
Also using for /f "skip=1" evades he neccessity of the external command more +1 and another intermediate file.
#echo off & setlocal EnableDelayedExpansion
type NUL>output2.txt
for /f "skip=1 eol=| tokens=1,8 delims=," %%A in (.\upload_results.log) do (
echo The source file is %%A , the upload status was %%B
set "line=%%A"
set "line=!line:~7!"
echo(!line!>>output2.txt
echo(!line!
)
File names and paths can contain also one or more exclamation marks. The line set line=%%A is parsed by Windows command processor a second time before execution with enabled delayed expansion. See How does the Windows Command Interpreter (CMD.EXE) parse scripts? Every ! inside the string assigned to loop variable A is on this line interpreted as begin or end of a delayed expanded environment variable reference. So the string of loop variable A is assigned to environment variable line with an unwanted modification if file path/name contains one or more exclamation marks.
For that reason it is best to avoid usage of delayed expansion. The fastest solution is for this task using a second FOR to get file:// removed from string assigned to loop variable A.
#echo off
del output2.txt 2>nul
for /F "skip=1 tokens=1,8 delims=," %%A in (upload_results.log) do (
echo The source file is %%A , the upload status was %%B.
for /F "tokens=1* delims=/" %%C in ("%%~A") do echo %%D>>output2.txt
)
Even faster would be without the first echo command line inside the loop:
#echo off
(for /F "skip=1 delims=," %%A in (upload_results.log) do (
for /F "tokens=1* delims=/" %%B in ("%%~A") do echo %%C
))>output2.txt
The second solution can be written also as single command line:
#(for /F "skip=1 delims=," %%A in (upload_results.log) do #for /F "tokens=1* delims=/" %%B in ("%%~A") do #echo %%C)>output2.txt
All solutions do following:
The outer FOR processes ANSI (fixed one byte per character) or UTF-8 (one to four bytes per character) encoded text file upload_results.log line by line with skipping the first line and ignoring always empty lines and lines starting with a semicolon which do not occur here.
The line is split up on every occurrence of one or more commas into substrings (tokens) with assigning first comma delimited string to specified loop variable A. The first solution additionally assigns eighth comma delimited string to next loop variable B according to ASCII table.
The inner FOR processes the string assigned to loop variable A with using / as string delimiter to get assigned to specified loop variable file: and to next loop variable according to ASCII table the rest of the string after first sequence of forward slashes which is the full qualified file name.
The full qualified file name is output with command echo and appended either directly to file output2.txt (first solution) or first to a memory buffer which is finally at once written into file output2.txt overwriting a perhaps already existing file with that file name in current directory.
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
del /?
echo /?
for /?
See also the Microsoft article about Using command redirection operators for an explanation of the redirections >, >> and 2>nul

How to ignore the ampersand within an url for replacing a string in a batch script?

In need to write a batch script, that replaces a variable with a URL in an html document. The URL comes from the users input.
#echo off
setlocal DisableDelayedExpansion
set /p vargs="Google Search: "
set "search=xxgs"
set "replace=%vargs%"
del output.html
for /F "delims=" %%a in (test.html) DO (
set line=%%a
setlocal EnableDelayedExpansion
>> output.txt echo(!line:%search%=%replace%!
endlocal
)
If I paste a Google URL (e.g. https://www.google.de/search?q=test&oq=test&aqs=chrome..69i57j69i60j69i65l3j69i60.704j0j9&sourceid=chrome&ie=UTF-8), it'll returned an error, because of the ampersand.
How can i stop the interpretation of the ampersand and just replace my variable xxgs with the url?
The problem is that the ampersand (and also some other characters) have special meaning to cmd, together with the fact that you are expanding the variable replace using normal (immediate) percent expansion (%replace%).
Since you are using this variable nested within a delayed expansion expression (namely the sub-string replacement !line:%search%=%replace%!), you cannot use delayed expansion for replace. However, you can delay the value expansion by using another for /F loop, as for variables are expanded also after special characters (like ^, &, (, ), ", <, >, |) are handled:
#echo off
setlocal DisableDelayedExpansion
set "vargs="
set /p vargs="Google Search: "
set "search=xxgs"
set "replace=%vargs%"
> "output.txt" (
for /F "usebackq delims=" %%a in ("test.html") do (
set "line=%%a"
setlocal EnableDelayedExpansion
for /F "delims=" %%b in (""!replace!"") do (
echo(!line:%search%=%%~b!
)
endlocal
)
)
The inner for /F loop receives the value of !replace!, which is available in %%~b later; since %%~b is still expanded before the actual sub-string replacement of line is done, it provides the expected output. The outer pair of quotation marks in the expression ""!replace!"" is removed by for /F and tells it to consider the remainder as a literal string (as there is no usebackq option); the inner pair of "" is processed by for /F together with the value of replace and is later removed by the ~ modifier in %%~b; with this trick, the loop even executes in case replace is empty, as the string appears as "" to for /F (note that the loop ignores empty strings/lines), so the script works even if the user enters nothing in the prompt. If you do not want this behaviour and skip the sub-string replacement upon empty user input, simply remove the outer pair of quotation marks "" and replace %%~b with %%b.
Note that exclamation marks in the value of replace still lead to unexpected results, because for variables are expanded before delayed expansion is processed, which consumes ! characters.
In addition to that, I changed the following:
the variable vargs is cleared before the prompt set /P vargs=, so when the user enters nothing, vargs is truly empty, as set /P keeps the previous value in case of no input;
instead of writing every single line to the output file, I redirected the entire for /F loop structure to the file output.txt once only; since this allows to use the > operator rather than >>, the output file does not have to be deleted in advance any more (although you deleted output.html rather than output.txt, but I assume this is just a typo);
I added the usebackq option to for /F and quoted the file path/name, so it may even contain white-spaces or special characters;
the syntax of set within the loop structure is changed to the quoted one set "line=%%a";

for loop doesn't iterate through the lines of a text file

I have a for loop that is supposed to print each line of a text file. Instead it's printing the logPath.
This is the code:
set enabledelayedexpansion
for %%G in (C:\ExecutionSDKTest_10.2.2\*.properties) DO (
Set fileName=%%~nxG
...
set logPath="C:/ExecutionSDKTest_10.2.2/Logs/!fileName!.log"
...
For /f "tokens=*" %%B in (!logPath!) Do (
echo Inside the for loop for printing each line!!
set logLine=%%B
print !logLine! REM this prints the logPath instead of each logLine and jumps out of this for loop after the 1st iteration!
)
)
Any help?
echo off
For %%G in (C:\ExecutionSDKTest_10.2.2\*.properties) DO (
FOR /F "tokens=*" %%i in (%%G) do #echo %%i
)
Use backslashes instead of forward slashes.
set "logPath=C:\ExecutionSDKTest_10.2.2\Logs\!fileName!.log"
While usually you can use them interchangeably in Windows, cmd is a special case as the forward slash is used for switches and options to built-in commands. And its parser often stumbles over forward slashes. You usually can safely pass such paths to external commands, though.
you don't tell us which line is issuing the "invalid switch" error message, but I see several potential problems:
to use !variables! you need to enable delayed expansion
SetLocal EnableDelayedExpansion
don't use '/' in filenames, change to '\'
set logPath="C:\ExecutionSDKTest_10.2.2\Logs\!fileName!.log"
print command sends a text file to the printer. Change it to echo
echo !logLine!

Remove trailing spaces from a file using Windows batch?

How could I trim all trailing spaces from a text file using the Windows command prompt?
The DosTips RTRIM function that Ben Hocking cites can be used to create a script that can right trim each line in a text file. However, the function is relatively slow.
DosTips user (and moderator) aGerman developed a very efficient right trim algorithm. He implemented the algorithm as a batch "macro" - an interesting concept of storing complex mini scripts in environment variables that can be executed from memory. The macros with arguments are a major discussion topic in and of themselves that is not relevent to this question.
I have extracted aGerman's algorithm and put it in the following batch script. The script expects the name of a text file as the only parameter and proceeds to right trim the spaces off each line in the file.
#echo off
setlocal enableDelayedExpansion
set "spcs= "
for /l %%n in (1 1 12) do set "spcs=!spcs!!spcs!"
findstr /n "^" "%~1" >"%~1.tmp"
setlocal disableDelayedExpansion
(
for /f "usebackq delims=" %%L in ("%~1.tmp") do (
set "ln=%%L"
setlocal enableDelayedExpansion
set "ln=!ln:*:=!"
set /a "n=4096"
for /l %%i in (1 1 13) do (
if defined ln for %%n in (!n!) do (
if "!ln:~-%%n!"=="!spcs:~-%%n!" set "ln=!ln:~0,-%%n!"
set /a "n/=2"
)
)
echo(!ln!
endlocal
)
) >"%~1"
del "%~1.tmp" 2>nul
Assuming the script is called rtrimFile.bat, then it can be called from the command line as follows:
rtrimFile "fileName.txt"
A note about performance
The original DosTips rtrim function performs a linear search and defaults to trimming a maximum of 32 spaces. It has to iterate once per space.
aGerman's algorithm uses a binary search and it is able to trim the maximum string size allowed by batch (up to ~8k spaces) in 13 iterations.
Unfotunately, batch is very SLOW when it comes to processing text. Even with the efficient rtrim function, it takes ~70 seconds to trim a 1MB file on my machine. The problem is, just reading and writing the file without any modification takes significant time. This answer uses a FOR loop to read the file, coupled with FINDSTR to prefix each line with the line number so that blank lines are preserved. It toggles delayed expansion to prevent ! from being corrupted, and uses a search and replace operation to remove the line number prefix from each line. All that before it even begins to do the rtrim.
Performance could be nearly doubled by using an alternate file read mechanism that uses set /p. However, the set /p method is limited to ~1k bytes per line, and it strips trailing control characters from each line.
If you need to regularly trim large files, then even a doubling of performance is probably not adequate. Time to download (if possible) any one of many utilities that could process the file in the blink of an eye.
If you can't use non-native software, then you can try VBScript or JScript excecuted via the CSCRIPT batch command. Either one would be MUCH faster.
UPDATE - Fast solution with JREPL.BAT
JREPL.BAT is a regular expression find/replace utility that can very efficiently solve the problem. It is pure script (hybrid batch/JScript) that runs natively on any Windows machine from XP onward. No 3rd party exe files are needed.
With JREPL.BAT somewhere within your PATH, you can strip trailing spaces from file "test.txt" with this simple command:
jrepl " +$" "" /f test.txt /o -
If you put the command within a batch script, then you must precede the command with CALL:
call jrepl " +$" "" /f test.txt /o -
Go get yourself a copy of CygWin or the sed package from GnuWin32.
Then use that with the command:
sed "s/ *$//" inputFile >outputFile
Dos Tips has an implementation of RTrim that works for batch files:
:rTrim string char max -- strips white spaces (or other characters) from the end of a string
:: -- string [in,out] - string variable to be trimmed
:: -- char [in,opt] - character to be trimmed, default is space
:: -- max [in,opt] - maximum number of characters to be trimmed from the end, default is 32
:$created 20060101 :$changed 20080219 :$categories StringManipulation
:$source http://www.dostips.com
SETLOCAL ENABLEDELAYEDEXPANSION
call set string=%%%~1%%
set char=%~2
set max=%~3
if "%char%"=="" set char= &rem one space
if "%max%"=="" set max=32
for /l %%a in (1,1,%max%) do if "!string:~-1!"=="%char%" set string=!string:~0,-1!
( ENDLOCAL & REM RETURN VALUES
IF "%~1" NEQ "" SET %~1=%string%
)
EXIT /b
If you're not used to using functions in batch files, read this.
There is a nice trick to remove trailing spaces based on this answer of user Aacini; I modified it so that all other spaces occurring in the string are preserved. So here is the code:
#echo off
setlocal EnableDelayedExpansion
rem // This is the input string:
set "x= This is a text string containing many spaces. "
rem // Ensure there is at least one trailing space; then initialise auxiliary variables:
set "y=%x% " & set "wd=" & set "sp="
rem // Now here is the algorithm:
set "y=%y: =" & (if defined wd (set "y=!y!!sp!!wd!" & set "sp= ") else (set "sp=!sp! ")) & set "wd=%"
rem // Return messages:
echo input: "%x%"
echo output: "%y%"
endlocal
However, this approach fails when a character of the set ^, !, " occurs in the string.
Good tool for removing trailing spaces in files in windows:
http://mountwhite.net/en/spaces.html
I just found a very nice solution for trimming off white-spaces of a string:
Have you ever called a sub-routine using call and expanded all arguments using %*? You will notice that any leading and/or trailing white-spaces are removed. Any white-spaces occurring in between other characters are preserved; so are all the other command token separators ,, ;, = and also the non-break space (character code 0xFF). This effect I am going to utilise for my script:
#echo off
set "STR="
set /P STR="Enter string: "
rem /* Enable Delayed Expansion to avoid trouble with
rem special characters: `&`, `<`, `>`, `|`, `^` */
setlocal EnableDelayedExpansion
echo You entered: `!STR!`
call :TRIM !STR!
echo And trimmed: `!RES!`
endlocal
exit /B
:TRIM
set "RES=%*"
exit /B
This script expects a string entered by the user which is then trimmed. This can of course also be applied on lines of a file (which the original question is about, but reading such line by line using for /F is shown in other answers anyway, so I skip this herein). To trim the string on one side only, add a single character to the opposite side prior to trimming and remove it afterwards.
This approach has got some limitations though: it does not handle characters %, !, ^ and " properly. To overcome this, several intermediate string manipulation operations become required:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "STR="
set /P STR="Enter string: "
setlocal EnableDelayedExpansion
echo You entered: `!STR!`
set "STR=!STR:%%=%%%%!"
set "STR=!STR:"=""!^"
if not "%STR%"=="%STR:!=%" set "STR=!STR:^=^^^^!"
set "STR=%STR:!=^^^!%"
call :TRIM !STR!
set "RES=!RES:""="!^"
echo And trimmed: `!RES!`
endlocal
endlocal
exit /B
:TRIM
set "RES=%*"
exit /B
Update
Both of the above scripts cannot handle the characters &, <, > and |, because call seems to become aborted as soon as such a character appears in an unquoted and unescaped manner.
However, I finally found a way to fix that and come up with an approach that can successfully deal with all characters (except perhaps some control characters, which I did not test):
#echo off
setlocal EnableExtensions EnableDelayedExpansion
rem // The last white-space in `STRING` is a tabulator:
set "RESULT=" & set "STRING= (<&>"^|)^^!^^^^;,= ^"
echo Input string: `!STRING!`
rem // Double quotes to avoid troubles with unbalanced ones:
if defined STRING set "STRING=!STRING:"=""!^"
rem // Particularly handle carets and exclamation marks as delayed expansion is enabled:
if defined STRING set "STRING=!STRING:^=^^^^!"
if defined STRING set "STRING=%STRING:!=^^^!%" !
if defined STRING (
rem // Escape all characters that `call` has got troubles with:
set "STRING=!STRING:^=^^!"
set "STRING=!STRING:&=^&!"
set "STRING=!STRING:<=^<!"
set "STRING=!STRING:>=^>!"
set "STRING=!STRING:|=^|!"
)
rem /* Call the sub-routine here; the strigs `!=!` constitute undefined dummy variables
rem with an illegal name, which eventually become removed; the purpose of them us to
rem enable usage of that `call` inside of a `for` loop with the meta-variable `%%S`,
rem which would otherwise become unintentionally expanded rather than `%%STRING%%`,
rem which literally contained `%%S`; the `!=!` at the end is just there in case you
rem want to append another string that could also match another `for` meta-variable;
rem note that `!!` is not possible as this would be collapsed to a single `!`, so
rem a (most probably undefined) variable `!STRING%!` would then become expanded: */
call :TRIM %%!=!STRING%%!=!
rem /* The caret doubling done by `call` does not need to be reverted, because due to
rem doubling of the quotes carets appear unquoted, so implicit reversion occurs here;
rem of course the doubling of the quotes must eventually be undone: */
if defined RESULT set "RESULT=!RESULT:""="!^"
echo Now trimmed: `!RESULT!`
endlocal
exit /B
:TRIM
rem // This is the effective line that does the left- and right-trimming:
set "RESULT=%*" !
exit /B
I use this Python 2 script to print lines with trailing whitespace and remove them manually:
#!/usr/bin/env python2
import sys
if not sys.argv[1:]:
sys.exit('usage: whitespace.py <filename>')
for no, line in enumerate(open(sys.argv[1], 'rb').read().splitlines()):
if line.endswith(' '):
print no+1, line
I know that Python is not preinstalled for Windows, but at least it works cross-platform.

Resources