For example, this:
ffmpeg -i %1 -c:v libx264 -preset medium -crf 30 -c:a aac "%~n1_OUT.mp4"
Does not need %%1 and %%~n1 respectively.
But this:
FOR /f %%f in ('dir /b .') DO somecommand %%f
Needs %%, otherwise it's not working.
Can somebody explain any logical reason for this chaotic design?
The percent-sign % is used by cmd.exe and also its predecessor command.com to mark expansion1 of environment variables (like %VAR%), for-loop meta-variables (like %I/%%I), and argument references in batch files (like %1).
There are two distinct parsing modes, which behave differently when it comes to %-expansion:
Command line mode:
There are no command line arguments, hence no argument references are supported.
Empty (undefined) variables are not expanded, meaning that %VAR% is kept as is when an environment variable VAR is not set.
There is no escaping of %-signs supported, so %%VAR%% results in % + value of VAR + %.
Batch file mode:
Batch files may have arguments, which are expanded by argument references like %1.
Empty (undefined) variables are expanded to a blank string.
Escaping of the %-sign is supported in that %% represents a literal %.
Of course I cannot tell why the developers decided the parser to behave that way, but I believe there are several factors contributing:
The command line mode existed before batch file mode, the latter of which required introduction of support for arguments, which in turn led to the demand to let the parser distinguish between variable or argument references.
When the for command was introduced2, the developers decided to use the %-sign too to mark loop meta-variables, leading to conflicts in command line mode, particularly because empty variables remain: For instance, an expression like %foo%bar results in the value of foo + bar when variable foo is set, but otherwise to value of %f + oo + value of %b + ar when loop variables %f and %b exist, and so on. This is particularly because for parses the command line a second time after the initial potential expansion of environment variables.
Introduction of batch file mode and the conflicts coming from the for parser probably led to the decision of improved handling of %-expansion in that undefined variables are treated differently by expanding such to empty strings. In order to still be able to yield literal %-symbols, escaping like %% was introduced: For example, a text like 10% plus 20% could not be returned without escaping since a variable named SPACE + plus 20 is most likely undefined; however, specifying 10%% plus 20%% results in the literal string 10% plus 20%.
Anyway, the %-escaping in batch file mode is the intrinsic reason for why for meta-variables need to be specified like %%I (in contrast to %I in command line mode) in order for them to survive the escaping, resulting in %I, which is then recognised by the for command parser that comes into play after %-expansion.
For detailed information about what exactly happens, refer to: How does the Windows Command Interpreter (CMD.EXE) parse scripts?
1) The term expansion means to replace an expression (like %VAR%) by the string value it refers to (like the one stored in the respective environment variable named VAR) while parsing.
2) Although I do not know whether the for command was introduced before or after implementation of the batch file mode.
Command started from console window use single percent sign in for loop varible, but script files uses doubled percent sign. Its by design, well documented. Nothing chaotic..
https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/for
Related
When I run the following batch script:
#echo off
REM %~ will strip surrounding quotes if any
echo HERE
I get the following error:
C:\>test.cmd
The following usage of the path operator in batch-parameter
substitution is invalid: %~ will strip surrounding quotes if any
For valid formats type CALL /? or FOR /?
Same effect if REM is changed to ::.
Seems like the parser is ignoring the comment indicator and parsing the %~. If I put a space between the % and ~ then it works fine.
Windows 7 Enterprise (have not checked any other versions).
Seems like a bug to me, but am I missing something?
The %-expansion, hence expanding normal environment variables (like %VAR%) as well as command line arguments (like %0), is the very first step after having read a line, therefore it happens even before the rem command is recognised. Thus you need to avoid the %~ (by writing rem % + ~ ..., for instance).
Given that the command extensions are enabled, which is the default anyway, %~ is recognised as invalid argument syntax (the ~ is expected to be followed by a decimal digit denoting the argument position or by a valid modifier like f, d, p, n, x, etc.; see Command Line arguments (Parameters)) and results in a fatal error, meaning that an error message is thrown and batch file processing is aborted (the %ErrorLevel% is not set though).
The same effect comes up when you try to do sub-string substitution but specifying an empty search string (like %VAR:=replace% or %VAR:*=replace%, given that VAR is defined), also with command extensions enabled.
See also this thread: How does the Windows Command Interpreter (CMD.EXE) parse scripts?
I think it is clearly covered in quite a few docs that cmd will interpret the arguments before comments, see the example in #LotPings comment as well as #aschiphl's post. That being said, you can momentarily disableextensions and then turn it back on when needed. The below example shows how disabling it will allow you to use it in the REM comment and then enabled again after to show allow extensions:
#echo off
setlocal disableextensions
REM %~ will strip surrounding quotes if any"
endlocal
echo my batch file is %~0
I need to output the first X characters of the content of OLD_ENTRY, say 33 chars. I grab the number of chars with another script. What is, in the following command in a Windows cmd script, the correct syntax to use a variable, say POS, instead of the hardcoded value 33?
echo %OLD_ENTRY:~0,33%
Thanks for any help,
Rip
Alright, I tend to call something like this as "nested variables". Anyway, to expand such nested variables, you need to establish a second parsing or expansion phase, and you need to ensure that the inner variable (POS) becomes expanded first, and the outer one (OLD_ENTRY) becomes expanded during the second phase. There are some options:
Using call:
This option avoids delayed variable expansion, which could be problematic with literal ! symbols, but it is quite slow, and it doubles quoted ^ characters:
In command prompt window:
call echo %^OLD_ENTRY:~0,%POS%%
This looks like "escaping" (^) the outer variable, but actually, this has got nothing to do with true escaping. In command prompt, an undefined variable does not become replaced by an empty string, it is just kept literally. So in the first pass, the undefined variable ^OLD_ENTRY is simply kept (you can verify that by defining such a variable by set "^OLD_ENTRY=something"), scanning for the closing % is skipped after the : for undefined variables strangely, the variable %POS% becomes expanded, and the last (orphaned) % is kept too; immediately after this phase, the escape sequence ^O is recognised, which results in a literal O; so we get echo %OLD_ENTRY:~0,33%, which becomes expanded in the second pass.
In a batch file:
call echo %%OLD_ENTRY:~0,%POS%%%
In the first pass, the first two consecutive percent symbols become replaced by one literal % sign, the variable %POS% becomes expanded, and the remaining two consecutive percent symbols become replaced by one literal % sign too, so we have echo %OLD_ENTRY:~0,33%, which becomes expanded in the second pass.
Using delayed variable expansion:
This is the better option, I think, because it is faster and does not mess around with ^:
echo !OLD_ENTRY:~0,%POS%!
This option works in both command prompt window and batch files. Here the first pass is the normal/immediate expanssion (%) which handles the %POS% variable, so the second pass the delayed expansion (!) receives echo !OLD_ENTRY:~0,33! to expand.
Refer also to this post: How does the Windows Command Interpreter (CMD.EXE) parse scripts?
Special characters in batch files are a pain, but I haven't found the right workaround for properly escaping the first two characters of this particular string I'm trying to pass the application.
SET pass=^&AntiBatchfileString
A_Program.exe /pass=%pass%
Things I have tried:
:: Escaping the escape twice, first for ^, second for &.
SET pass=^^^^&AntiBatchfileString
echo %pass%
:: Combining escapes.
SET first=^^
SET second=^^&AntiBatchfileString
SET pass=%first%%second%
echo %pass%
:: Preventing expansion
SET first=^^
SET second=^^&AntiBatchfileString
SET pass=!first!%second%
echo %pass%
:: I got this to print correctly
SET "pass=^&AntiBatchfileString"
echo ^^%pass%
Still when passing the last one it doesn't accept the login, I don't know what the final output is. That got me thinking maybe it was trying to do another expansion when passing the parameter to the application, so I quoted that as well.
SET "pass=^&AntiBatchfileString"
A_Program.exe "/pass=^^%pass%"
It's still not working, I'm not sure what I'm missing at this point.
Supposing you want the string ^&AntiBatchfileString literally, this is the best set syntax, as most special characters (^ & ( ) < > | and also the standard delimiters , ; = SPACE TAB) lose their particular meaning as soon as ther are placed in between "", and the "" themselves do not become part of the variable value:
set "pass=^&AntiBatchfileString"
This works only as long as the command extensions are on, which is the Windows default anyway (type cmd /? and see the /E option).
When expanding (reading) a variable like "%pass%" (with enclosing ""), special characters are still treated literally.
However, as soon as you expand it like %pass% (no ""), they get back their special meaning. So you have the following options:
Use set "pass=^^^&AntiBatchfileString", where ^^ escapes the literal ^ and ^& the literal & when reading like %pass%.
Enable delayed expansion (see set /? about how it works and setlocal /? or cmd /? about how to enable it), where the variable value is expanded (read) at a point of time where parsing of special characters has already been completed.
I prefer the latter approach, because no special escaping is necessary, and it can also deal with " appearing in the string value (even if unsymmetrically present).
By the way, " can also be escaped by ^", as long as this does not appear within unescaped "".
Nevertheless, % signs cannot be escaped like ^% in a batch file, because percent expansion happens before escaping, but you need to double them like %% to get one literal one each, independent whether or not the string is in between "".
Note that on the console, %% does not work.
Finally, literal ! are consumed by the delayed expansion feature when enabled, therefore you need to pay particular attention to those in case, by escaping them like ^!, or also by intelligently toggling delayed expansion (hence to enable it only when it is actually needed and to disable it otherwise, when a literal string is provided, like in a set command line, for instance, when expanding a standard variable like %pass% and when reading a for variable like %%I (batch file) or %I (console), for example). Of course this is also not the ultimate solution, because you need setlocal and endlocal to enable/disable delayed expansion, which are intended to localise environment changes, so any variable changes since the most recent setlocal command are lost as soon as endlocal is executed (there are some tricks for passing a variable value over the endlocal barrier though).
If you want to use % as a string without escaping in a batch file:
Like %20, you can use %%%20.
git clone "https:// abc.com /D%%%220an"
I am trying to use the rem command to place a remark in a command line that contains several commands. Here are some examples to illustrate what I mean:
echo Hello & rem.Comment & echo world!
(echo Hello & rem.Comment) & echo world!
This works perfectly fine, both echo commands in each line are executed as I expect. The . seems to modify the behaviour of the rem command so that it does not treat the remaining line as comment:
Hello
world!
If I placed a SPACE (or any other delimiter TAB, ,, ;, =) instead of the ., the remaining line and therefore the second echo would be ignored (for the second example a More? prompt appears, because the ) is part of the remark and cmd expects a closing ) because of the ():
Hello
I found out that beside ., the following characters work as well: :, /, \, [, ] and +.
What else works is escaped delimiters: ^SPACE, ^TAB, ^,, ^; and ^=.
Nevertheless, is there a secure and reliable way to do that?
I would be very glad about a solution that works for both command prompt and batch-files.
According to this external reference, the familiar syntax echo. for returning a blank line fails under certain circumstances, hence using echo( is recommended as this is the only reliable method.
However, for rem, the ( does not work, everything after rem( is not recognised as a command.
Since I am aware of a weird bug of the rem command in Windows XP (reference this external link: rem %~), I am interested in a solution that applies to Windows Vista, Windows 7 or higher.
The "weird" REM %~ "bug" is not limited to XP. It is present in all modern versions of Windows that use CMD.EXE. After reading your question, I wrote Simon of SS64 a note to give clarification on the issue. REM can also fail if variable var exists, and you have rem %var:=.
So technically, there is no guaranteed safe way to blindly use REM.
But, if you are willing to accept the fatal % expansion risk, most of your listed hacks are safe to use, but only if the line includes at least one additional command via & or &&.
REM. is never safe to use in any situation if there exists a file named REM (without extension).
The folder dividers \ and / always fail if the current folder contains a file named test.bat and you use REM\..\test.bat.
In a similar fashion, REM:\..\test.bat always fails.
Every one of the other hacks can fail stand-alone in a similar situation. For example, REM^[tab]\..\test.bat fails stand-alone, but works if concatenated with another command. This is the only type of situation I've found where +, [, ], or ^[tab] can fail.
There are additional cases where some of the other hacks can fail.
Any character in the set C (^[space], ^,, ^;, ^=) that are valid in file names can fail stand-alone if remC.bat exists. For example, the following fails stand-alone:
rem^ Fails if "rem .bat" exists
Yet they are all safe when concatenated with another command:
echo OK&rem^ This is safe
rem^ This is safe &echo OK
Temporary Update
Some of the above is wrong. Investigations are ongoing at http://www.dostips.com/forum/viewtopic.php?f=3&t=6895&p=44813#p44813.
I believe the following are the simplest forms that are guaranteed to work in all cases (disregarding invalid % expansion)
REM: At least one space (or other token delimiter) must be after :
REM\ At least one space (or other token delimiter) must be after \
REM/ At least one space (or other token delimiter) must be after /
REM^[tab] At lease one space (or other token delimiter) must be after [tab]
But I won't correct the earlier info until the dust has settled
End Temporary Update
My favorite way to use inline comments is to use impossible variables. Only dynamic pseudo variables can contain = in a name, and no variable name can ever contain two =. So I like to use %= Remark goes here =%. The beauty of this form is it can be used pretty much anywhere with impunity, as long as the comment does not contain % or :. It can even be used safely within parenthesized blocks of code.
for %%F in (*) do (
%= Comment within code block =%
%= 2nd comment within code block =%
FINDSTR /B %=Must match beginning of line=% "string" %= Search string =% "%%F" %= File to search =%
)
This variants of REM seems to be a safe way to enable the & sign in the comment part.
REM/
REM\
REM:
Despite of #dbenham's comment, I can't create any file which would iterfere with these REM variants (I tried REM.bat, REM;.bat and so on).
It's always a good idea to add a space after the REM^<char>.
The problem with %~ can't be solved, as the cmd.exe uses multiple parser phases for each line.
And the %~ error is detected in an early phase (percent expansion phase), just before the phase where a REM would be detected.
But at all, I prefere percent comments for inline comments, described by dbenham
EDIT:
I removed the carets from REM^<char> as it's doesn't matter.
Normally a REM remarks the rest of the line, as the batch parser detects the REM keyword in phase2 of the parser and switches to a specialized parser only for REM.
But when a character is appended to REM the keyword will nt be detected in phase2.
If the character is one of \/;,=+( the parser will remove it later and executes a normal REM command.
That's the cause why the command operators &, &&, |, || can be recognized in this case.
Why rem/ | break fails, but (REM/) | break works?
It's because the pipe starts two seperate cmd child processes.
With surrounding parenthesis the command will be parsed the first time in the child process.
But without parenthesis, the parent process has already parsed the REM/ and checks if the file exists (but doesn't execute it).
But when such a file exists then the parser is smart enough to remove the seperator character and detects that REM is an internal command.
This behaviour looks a bit strange.
I have the following command in a windows batch script
echo =%%k-16,INDIRECT.EXT^("'C:\Users\...\Analysis\[ObsStreamflow.xlsx]Sheet1'^!A%%k"^),INDIRECT.EXT^("'C:\Users\...\Analysis\[sim%%j.xlsx]Sheet1'^!B!val!"^),^=C%%k/1000,^=D%%k-B%%k,^=ABS^(E%%k^),^=(E%%k^)^^2,=^(B%%k-B10^),=Sqrt^(B%%k^),=SQRT^(D%%k^),=^(J%%k - B13^)^^2 >>t%%j.csv
where the omitted file path is 38 characters long (I don't think I'm hitting the line limits, but just in case this is the problem). This is a single line in my .bat file, shown here as multiple lines just to make things more readable.
The output is mostly correct, except that where I have ^^2, it just becomes 2 (so I have =(E1)2 and =(J1-B13)2. If I omit the Indirect.Ext text, and just have
echo =%%k-16,a1,b1,^=C%%k/1000,^=D%%k-B%%k,^=ABS^(E%%k^),^=(E%%k^)^^2,=^(B%%k-B10^),=Sqrt^(B%%k^),=SQRT^(D%%k^),=^(J%%k - B13^)^^2 >>t%%j.csv
it prints correctly, so the relevant comments show as =(E1)^2 and =(J1-B13)^2, which is what I am after.
I've not had any luck finding an answer, everything I have found just points to using ^^ to get echo to return ^. I cannot break this command into multiple lines, I need it to be a single row in csv format.
Any suggestions for a fix much appreciated, I only really need to use this for a week or so, don't need an elegant solution, just one that works. - I'm very new to bat scripts (and indeed programming in general), will keep trying different ideas in the mean time.
It's only the exclamation mark that creates the problems for you.
If at least one ! is in your line (and delayed expansion is enabled), then a second caret escape phase will be started.
In this phase quotes aren't regarded, only carets.
A small test
setlocal EnableDelayedExpansion
echo one^1
echo two^^2
echo two^^2 With exclam!
echo five^^^^^& With exclam!
Output
one1
two^2
two2 With exclam
four^& With exclam
So in your sample, you need five carets.
Four to create one caret and the last one to escape the ), as the escape of the special character is only once required.
Not sure what your specific problem is but you can use a trick in Windows to emulate echo -n (echo without a newline).
The commands:
<nul: >file.csv set /p junk=first field
<nul: >>file.csv set /p junk=,second field
>>file.csv echo ,third field
will result in a single line:
first field,second field,third field
That may make it easier for you to avoid the specific problem and, as a bonus, clean up your script so it's a little more readable (such as one field per script line).
It works because set /p var=prompt is the input command. It first outputs prompt without a newline then waits for the user to enter something, assigning it to the var environment variable.
By getting input from nul:, you basically give it an empty string so it doesn't wait. The prompt is output to file.csv without the newline.
In any case, for something this complex, I'd be bypassing cmd.exe for something a little more powerful such as the UNIX text processing tools under CygWin or MinGW (which require installation but are well worth it), or even VBScript scripts (which should be on Windows by default), where you can more easily control the output.