Batch file: Escape questionmark in for loop - windows

This for loop (reduced minimal example);
#echo off
for %%a in (help -help --help /help ? /?) do (
echo %%a
)
chokes on the 2 elements with a '?' character. It outputs
C:\Temp>test.bat
help
-help
--help
/help
C:\Temp>
So it just quits the loop when it hits the first '?'.
What is the proper escape sequence for this set? Tried a bunch of stuff, double quotes, carets, backslash, etc. but nothing seems to work.

Another option is to use linefeeds within a FOR /F string. FOR /F will treat each line as an independent string. Below I show four ways to do the same thing.
#echo off
setlocal enableDelayedExpansion
:: Define LF to contain a linefeed character
set ^"LF=^
^" The above empty line is critical. DO NOT REMOVE
:: Option 1
:: Embed linefeeds directly in the string literal
for /f %%A in ("help!LF!-help!LF!--help!LF!/help!LF!?!LF!/?") do (
echo(%%A
)
echo(
:: Option 2
:: Define a variable with spaces and use search and replace
:: to substitue linefeeds
set "help=help -help --help /help ? /?"
for %%L in ("!LF!") do for /f %%A in ("!help: =%%~L!") do (
echo(%%A
)
echo(
:: Option 3
:: Embed linefeed directly in string without LF variable
for /f %%A in (^"help^
-help^
--help^
/help^
?^
/?^") do (
echo(%%A
)
echo(
:: Option 4
:: Embed linefeed directly in search and replace without LF variable
for /f %%A in (^"!help:^ ^=^
!^") do (
echo(%%A
)
I prefer option 2. I find it to be the easiest to read, yet still be compact.
Note that MC ND and I both use echo(%%A. This is necessary to prevent echo /? from displaying the help for the ECHO command.

Nothing seems to work because nothing will work.
In its simplest form (for %%x in (set) do ...), when the for command iterates over the elements in the set, it will test if the element contains a * or a ?. In this case, it is considered a file wildcard and will search for files matching the indicated expression. You can test it in your case placing a file with a single character name and no extension in the folder containing the batch file, and the ? will match it and you will see it in the output.
And as far as i know there is no way to avoid it. This is the way for command is intended to work.
You can construct something similar to what you are trying but will need another approach
#echo off
setlocal
set "opt.a=help"
set "opt.b=-help"
set "opt.c=--help"
set "opt.d=/help"
set "opt.e=?"
set "opt.f=/?"
for /f "tokens=2 delims==" %%a in ('set opt.') do (
echo(%%a
)
endlocal
In this sample each value is in a environment variable and the set is retrieved via a set command and output processed with the options of for command. It is just one option. You need the values as line separated strings to be processed with a for /f.

While I personally think dbenham has provided the best answer, I want to add that in some cases FOR /F loop should be provided with options:
tokens=* - this would disable tokenization of lines (or, in case you need it, you could set delims to whatever you want), allowing you to have spaces in strings;
usebackq - this would allow you to have double quotes in strings. However, in this case, single quotes need to be escaped with caret - ^', and surrounding double quotes should be replaced with single quotes:
(set RN=^
%= \r\n =%
)
for /f "usebackq tokens=*" %%A in ('^'quotes^'!RN!"double quotes"!RN!`backquotes`') do (
echo:%%A
)
As for the original question, there are few other options available. Alex was onto something with his answer, but it didn't cover how to actually iterate through the list. To do it you need to chain echo commands. You start with echo: and use ^&echo: to echo all subsequent strings/variables:
set "var=line1"
for /F "usebackq delims=" %%G in (`
echo:%var%^&
echo: line2 * ? ^^^^ ^^^& ^^^< ^^^> ^^^| ^^^" ' ^` ^) ^&
echo: ^ ^ line3 ^ ^ ^&echo:li
ne4^&echo ^&echo: ^&echo:/?
`) do (
echo "%%G"
)
This will give you:
"line1"
" line2 * ? ^ & < > | " ' ` ) "
" line3 "
"li ne4"
"ECHO is on." # System language specific
" "
"/? "
As shown, you can break lines in any way, as long as you keep ^& and echo: intact (be wary of spaces).
Using colon instead of space after echo will sanitize values like ON, OFF or /?. You can also use some other characters, like (, but i find colon more readable.
Line-breaks are allowed, but all successive whitespaces (spaces and line-breaks) are replaced with a single space (additional spaces can be enforced by escaping them with single caret ^). echo ^& will count as echo with no parameters and will display current echo setting, echo:^& will give an empty line (FOR /F ignores empty lines), and echo: ^& will give you .
^ & < > | " need to be escaped with three carets. ` ) need to be escaped with a single caret.
Without usebackq option you don't have to escape `, but you'll need to escape ' with three carets. You'll also need to use single quotes instead of surrounding backquotes.
You can avoid excessive escaping if you put the expression inside double quotes like this:
for /f "usebackq delims=" %%G in (`"echo:%var%&echo: line2 * ? ^^ ^& ^< ^> ^| ' ` "abc" ^) &echo &echo: &echo:/?"`) do (
echo "%%G"
)
This will give you:
"line1"
" line2 * ? ^ & < > | ' ` "abc" ) "
"ECHO is on." # System language specific
" "
"/?"
To chain echo commands you use &echo: (note, no caret).
Line-breaks are not allowed. Spaces are preserved.
" cannot be escaped, but could be used if there is an even number of them. You don't need to escape ' and `.
Again, without usebackq option you'll need to use single quotes instead of surrounding backquotes.

If you use a for /f loop it works without escaping the ? characters, although the loop doesn't work the same way:
C:\Users\Alex\Downloads\test>for /f "tokens=* delims=" %a in ('echo help -help --help /help ? /?') do echo %a
C:\Users\Alex\Downloads\test>echo help -help --help /help ? /?
help -help --help /help ? /?

Related

Find and replace algorithm for string in text file using batch script, works, but stopping when `<`, `>`, or `|` characters appear

I've been trying to figure out how to replace an entire line in a text file that contains a certain string using a Batch Script. I've found this solution provided by another user on Stack Overflow, which does the job, however, it just stops iterating through the text file at some random point and in turn, the output file is left with a bunch of lines untransferred from the original file. I've looked character by character, and line by line of the script to figure out what each part exactly does, and can not seem to spot what is causing this bug.
The code provided, thanks to Ryan Bemrose on this question
copy nul output.txt
for /f "tokens=1* delims=:" %%a in ('findstr /n "^" file.txt') do call :do_line "%%b"
goto :eof
:do_line
set line=%1
if {%line:String =%}=={%line%} (
echo.%~1 >> output.txt
goto :eof
)
echo string >> output.txt
The lines it is stopping at always either contain < or > or both and lines with | will either cause it to stop, or sometimes it will delete the line and continue.
To do this robustly, Delayed expansion is necessary to prevent "poison" characters such as < > & | etc being interpreted as command tokens.
Note however that delayed expansion should not be enabled until after the variable containing the line value is defined so as to preserve any ! characters that may be present.
The following will robustly handle all Ascii printable characters *1, and preserve empty lines present in the source file:
#Echo off
Set "InFile=%~dp0input.txt"
Set "OutFile=%~dp0output.txt"
Set "Search=String "
Set "Replace="
>"%OutFile%" (
for /F "delims=" %%G in ('%SystemRoot%\System32\findstr.exe /N "^" "%InFile%"') do (
Set "line=%%G"
call :SearchReplace
)
)
Type "%OutFile%" | More
goto :eof
:SearchReplace
Setlocal EnableDelayedExpansion
Set "Line=!Line:*:=!"
If not defined Line (
(Echo()
Endlocal & goto :eof
)
(Echo(!Line:%Search%=%Replace%!)
Endlocal & goto :eof
*1 Note - Due to how substring modification operates, You cannot replace Search strings that:
contain the = Operator
Begin with ~

escape double quotes in a variable when using echo|set /p=

echo|set /p= can output a variable without a newline.
I am trying to loop a text file like this
for /f %%a in (zen.txt) do (
set var=%%a
echo !var!
echo|set /p=!var!
)
There are some lines with only one ", for example:
"he said...
echo outputs the line like above correctly while echo|set /p= output nothing.
Is it possible to escape double quotes in a variable when using echo|set /p=.
We will need to provide set/p with additional quotes to consume. You can try with something like (without the test file creation, of course)
#echo off
setlocal enableextensions disabledelayedexpansion
rem TESTING - Create a input file
>zen.txt (
echo "He said...
echo It was not me!
echo It was "the thing"!
echo ...
echo ^& it was all" ( The end )
)
for /f "delims=" %%a in (zen.txt) do (
<nul set /p="%%a "
)
echo|set /p simulates a return key press to terminate the set/p prompt, but it creates separate cmd instances for each side of the pipe making it slow. A <nul input redirection is enough to get the same result but faster and with less resources usage.
set /p is a bit nasty with handling whitespaces, quotes and equal signs.
A quote at the beginning or the end has to be doubled, BUT when you want to display quotes, the expression should be enclosed in quotes, too.
To display a single quote use
<nul set /p ="""
set /p seems to strip one time the outer quotes.
Your code can be changed to <NUL set /p="!var!" that should work with normal text and also with quotes.
If the text starts with whitespaces, they will be dropped. (But not up to XP, there set /p "= Hello" shows the spaces).
set /p seems to use two times a quote remover.
First for the extended set syntax
<nul set /p "=hello" Text after the last quote will be dropped
But also for the content, the outer quotes will be dropped
<nul set /p ="hello" Text after the last quote will be dropped
And even combining both works
<nul set /p "="hello" Text after the last inner quote will be dropped "
Note: I use <nul set /p, it's much faster, because the echo | set /p version uses a pipe and spawns two new cmd.exe instances. (Already mentioned by MC ND)

whitespace to newline in batchfile (windows)

i want to convert every whitespace in input to a new line character and save to a file (say tmp.txt) with batch file windows .
input:
http://www.example1.com wwwexample2com wwwexample3com wwwexample4com
to output in tmp.txt :
http://www.example1.com
wwwexample2com
wwwexample3com
wwwexample4com
how do I do that?
If the input is a file named "input file.txt" and the urls contain "=":
#echo off
setlocal enableDelayedExpansion
>"output file.txt" (for /f "usebackq delims=" %%a in ("input file.txt") do (
set line=%%a
call :listtokens "!line: =" "!"
))
pause & exit
:listtokens
if .%1==. (exit /b) else (echo %~1 & shift & goto listtokens)
If the input is a variable with alphanumeric words:
>"output file.txt" (for %%a in (%VARIABLE%) do echo %%a)
If the input is a file named "input file.txt" with alphanumeric words:
>"output file.txt" (for /f "usebackq delims=" %%a in ("input file.txt") do (
for %%b in (%%a) do echo %%b
))
If it's from a console utility that outputs alphanumeric words:
>"output file.txt" (for /f "delims=" %%a in ('run some command') do (
for %%b in (%%a) do echo %%b
))
The following pure batch script will put each space delimited "word" onto a separate line in the output. Empty lines will be stripped. The script is very robust. It should handle any ASCII text as long as every source line is <8kb. Note this will not convert other whitespace characters like tab into newlines.
#echo off
setlocal disableDelayedExpansion
>"input.txt.new" (
for /f "usebackq eol= tokens=* delims= " %%A in ("input.txt") do (
set "ln=%%A"
setlocal enableDelayedExpansion
echo(!ln: =^
!%= This line and the empty line above must remain exactly as written=%
endlocal
)
)
findstr . "input.txt.new" >"output.txt"
del "input.txt.new"
type output.txt
However, I would not use the above because I think pure batch makes text manipulation overly complex.
I would use a regular expression text processing utility called JREPL.BAT. It is pure script (hybrid JScript/batch) that runs natively on any Windows machine from XP onward. It is much simpler, faster, and more robust than any pure batch solution. The utility is extensively documented, and there are many regular expression tutorials available on the web.
The following one liner will efficiently put each whitespace delimtted "word" onto a separate line in the output.
jrepl "\S+" "$0" /jmatch /f "input.txt" /o "output.txt"
If you really want to convert every whitespace character into a newline, as you say in your comment, then you could use the following. But I don't think it is really what you want.
jrepl "\s" "\n" /x /f "input.txt" /o "output.txt"

I can process a file-and-command,file-and-string or few files with FOR /F but no other combinations.Why?

Here's my code:
#echo off
echo ->minus
echo _>underscore
rem if this file is used it tries to execute a command
echo not-at-all>'notacomand'
echo processing two files at once
for /f "delims=" %%# in (minus underscore ) do echo %%#
echo processing file and command
for /f "delims=" %%# in (minus '"echo ++"') do echo %%#
echo processing files and string
for /f "delims=" %%# in (minus underscore "zzz") do echo %%#
rem searches for file "zzz" minus
rem for /f "delims=" %%# in ("zzz" minus) do echo %%#
rem searches for file zzz" minus
rem for /f "delims=" %%# in ("zzz" minus) do echo %%#
echo the command at the end is not processed
for /f "delims=" %%# in (minus "zzz" 'echo ---') do echo %%#
rem searches for file 'echo
rem for /f "delims=" %%# in (minus 'echo ---' "zzz") do echo %%#
Tested on Windows 8.1 and Windows XP. ]:-).
I've tested very few combinations but my conclusions so far are:
You can process as many files as you want at once.Wild cards are not
accepted.For files with spaces useback should be used.
You can process as many files as you want at once puls ONE command (but with additional double quotes)
or string.Everything after the string or command will be ignored
except syntax errors.
Have no idea why (most probably without the cmd code answer is impossible).Also have no idea if this can be useful or is known behavior , but definitely is curious.
I'm pretty sure that you've run into a case of ambiguous documentation and a bug or two.
I believe what the documentation is trying to convey is that /f has three mutually exclusive choices: fileset, string, command output. One paragraph in the help begins with "You can also use the FOR /F parsing logic on an immediate string". I think "an" really means "one". Similarly the next paragraph says, "...output of a command...". Again I think "a" means one. My reasoning for mutually exclusive is that is how it is implemented more or less.
One of the bugs I'm claiming is that a single string works as a string if it's at the end of a fileset. The other might be the odd parsing of the quote characters that's been noted.
I think the high level parsing algorithm is something like: 1) look for open parenthesis, 2) if first non blank is quote character use appropriate handler for either string or command cases. 3) if first character is not quote or usebackq is specified and first character is double quote, then do fileset case 4) if previous token was a file specification then keep parsing, but if previous token was not a file specification then stop parsing
4) would explain the behavior of a single string at the end of a file set
String parsing algorithm seems to be: 1) look for beginning quote as first quote after open paren, 2) look for end quote as last quote before close paren, 3) everything in between is the string (or command) 4) the quotes don't even have to be balanced. The following all behave consistently with this explanation:
rem space is delimiter
for /f "tokens=1,2*" %i in ( "zzz" "yyy" ) do echo i %i j %j k %k
rem unbalanced quotes
for /f "tokens=1,2* delims=" %i in ( 'echo zzz' yyy' ) do echo i %i j %j k %k
rem apostrophe is delimiter
for /f "tokens=1,2* usebackq delims='" %i in ( 'zzz' 'yyy' ) do echo i %i j %j k %k

Spurious spaces in output of for loop

I'm trying to use a batch file to convert a file containing sql code into a single environment variable for use with the MSSQL utility bcp.
For example, if InFile.sql contains
-- This is a simple statement
SELECT *
FROM table
The output of ECHO %query% should be
SELECT * FROM people
The code below works for me most of the time
SETLOCAL=ENABLEDELAYEDEXPANSION
:: Replace VarOld with VarNew
FOR /f "delims=" %%a IN ('TYPE InFile.sql') DO ( SET line=%%a & ECHO !line:table=people! >> TmpFile1 )
:: Remove comment lines starting with '-' and remove newline characters
(FOR /f "eol=- delims=" %%a in (TmpFile1) DO SET/p=%%a ) <nul >TmpFile2
:: Create variable 'Query'
FOR /f "delims=" %%a IN ('TYPE TmpFile2') DO SET query=%%a
however, the first FOR loop adds 3 space characters at the end of each line and the second FOR loop adds another space character so the result is
SELECT * FROM people
I could cope with the additional spaces (although the purist in me wasn't happy!) until I had to use it with a long SQL query and multiple replacement steps - every line in the file was having 12 space characters added. The additional spaces are enough to make the resulting query around 8300 characters long - too much for Windows' 8196 character limit for a batch file line.
Can anybody see how I can remove these spurious spaces?
Using tokens=* in a for loop should trim whitespace as you're capturing a line of infile.sql. Here's a proof of concept, echoing %query% contained within quotation marks to illustrate the trimming:
#echo off
setlocal enabledelayedexpansion
set query=
if "%~1"=="" goto usage
if not exist "%~1" goto usage
for /f "usebackq eol=- tokens=*" %%I in ("%~f1") do (
set "sub=%%I"
set query=!query! !sub:table=people!
)
:: strip the leading space from %query%
echo "%query:~1%"
goto :EOF
:usage
echo Usage: %~nx0 sqlfile
Example output:
C:\Users\me\Desktop>type infile.sql
-- This is a simple statement
SELECT *
FROM table
C:\Users\me\Desktop>test.bat infile.sql
"SELECT * FROM people"
The fundamental issue is that trailing spaces ARE significant in SET statements and ECHO statements before the redirectors.
In your code, you need to remove the spaces after %%a and people! in the first FOR Thus:
FOR /f "delims=" %%a IN ('TYPE InFile.sql') DO (SET line=%%a&ECHO !line:table=people!>> TmpFile1)
The next problem is a little more subtle. In
(FOR /f "eol=- delims=" %%a in (TmpFile1) DO SET/p=%%a ) <nul >TmpFile2
the space following /p=%%a is REQUIRED because it provides the separator between the text taken from the lines when building TmpFile2 - and that leads to a superfluous trailing space. Try replacing that space with a Q for instance - just for testing.
Hence, you need to delete the final space from QUERY after it's been constructed in your final FOR
SET query=%query:~0,-1%

Resources