I've got a batch script (app1.bat) calling another batch script (app2.bat) which itself calls a program in windows (program.exe).
app2.bat calls program.exe with a parameter after a flag in this way:
program.exe -f Parameter with whitespaces coming into the program
What I want to do is to pass the phrase that comes to program.exe from app1.bat into app2.bat but i don't know how to properly handle the doublequotes. Currently I am passing the phrase from app1.bat to app2.bat in double quotes and inside an app2.bat (prior to executing program.exe) I get rid of the quotes like that:
inside app1.bat
call app2.bat "Parameter with whitespaces coming into the program"
inside app2.bat
set old_phrase=%1%
set new_phrase=%old_phrase:"=%
program.exe -f %new_phrase%
old_phrase is
"Parameter with whitespaces coming into the program"
and new_phrase I end up with is
Parameter with whitespaces coming into the program
Is there any standard way to handle such a situation (being passing a string to an external program which expects a tring without quotes and being ok with whitespaces, whereas batch does not allow for no-quotes-and-whitespaces strings)
When you execute call /? from cmd to launch the help you will see quite a bit around expansion of %n
The first one states:
%~1 - expands %1 removing any surrounding quotes (")
You can therefore dump all the other set commands and simply run this in your batch file:
program.exe -f %~1
Related
I need to pass a password with special characters from powershell script automation.ps1 to batch script batch_script.bat which pipes it to main.py. Piping from batch_script.bat to main.py works fine, that is authentication succeeds. However, when I run the entire procedure described above, authentication fails, but echoing the password shows the correct password string.
My guess is that there are issues with special characters. What is a safe way to pass these strings?
Background
I want to automate the daily download from some external source via a Python script main.py. This process requires a password. So I wrote a batch_script.bat which pipes the password to the Python script when prompted for it. However, I don't want to store the password as plain text in the batch script, so I encrypted the password and wrote another layer automation.ps1 which decrypts the password and passes it as plain text to batch_script.bat.
automation.ps1
# get password
$securePassword = Get-Content "<file_path>" | ConvertTo-SecureString
$credentials = New-Object System.Management.Automation.PsCredential("<user_name>",$securePassword)
$unsecurePassword = ($credentials).GetNetworkCredential().Password
# call script
$script = "<path>\batch_script.bat"
$args = #("`"<user name>`"","`"$unsecurePassword`"")
start-process $script $args
batch_script.bat
(I am aware that in this example I discard the passed username, just wanted to preserve the fact that I pass multiple arguments in case there is any relevance to it)
#echo off
SET username=%~1
SET password=%~2
echo(%password%|python main.py
With following, all special characters should be handled very well. If any character required to be escaped, check this
$pass could be any string but check for special characters of powershell
$pass="%^&<>|'\``,;=(\)![]\/";
# Wait till it ends with -Wait when using -NoNewWindow.
# It may be comprehensible to use `" instead of "" to denote we are enclosing string in quotes.
(thanks #mklement0 for elaboration).
start-process -Wait -NoNewWindow .\script.cmd "`"$pass`""
script.cmd
setlocal
rem Remove Double quotes
set "arg=%~1"
rem Test result with base64 encoding
echo|set/p="%arg%"|openssl base64
rem echo is used with set/p to prevent trailing new line.
echo|set/p="%arg%"|python main.py
rem Test with following, argument is in double quotes
rem script "%^&<>|'\`,;=(\)![]\/"
rem Expected result
rem %^&<>|'\`,;=(\)![]\/
tl;dr:
Unless you specifically need the batch file to run in a new window, avoid Start-Process (whose built-in aliases are start and saps), and invoke the batch file directly.
To avoid problems with special characters in $unsecurePassword, do not pass it as an argument, pass it via stdin (the standard input stream), which your batch file will pass through to your python script:
automation.ps1:
# ...
$script = "<path>\batch_script.bat"
# Pass the password via *stdin*
$unsecurePassword | & $script 'userName'
Note: It is the $OutputEncoding preference variable that controls what character encoding PowerShell uses for sending text to an external program's stdin. In Windows PowerShell, that variable defaults to ASCII(!) encoding, meaning that any characters outside the 7-bit ASCII-range of Unicode characters, such as accented characters, are unsupported (they turn to literal ?); fortunately, PowerShell [Core] v6+ now defaults to UTF-8. Assign the required encoding to $OutputEncoding as needed.
batch_script.bat:
#echo off
SET username=%~1
REM The batch file will pass its own stdin input through to Python.
python main.py
Read on for background information.
Invoking a batch file from PowerShell:
Unless you truly need to launch a batch file in a new window, the best approach is to invoke it directly from PowerShell; that way, it runs:
in the same console window, synchronously.
with its output streams connected to PowerShell's (which allows you to capture or redirect the output).
Because your batch-file path is stored in a variable, direct invocation requires use of &, the call operator:
# Note: The " chars. around $unsecurePassword are only needed if the variable
# value contains cmd.exe metacharacters - see next section.
& $script 'userA' `"$unsecurePassword`"
Start-Process is usually the wrong tool for invoking console applications, batch files, and other console-based scripts; see this answer for more information.
If you do need the batch file to run in a new window (which is only an option on Windows), use Start-Process as follows (the command will execute asynchronously, unless you also pass -Wait):
# The string with the arguments to pass is implicitly bound
# to the -ArgumentList parameter. Use only " for embedded quoting.
Start-Process $script "userA `"$unsecurePassword`""
Note: While the (implied) -ArgumentList (-Args) parameter is array-valued ([string[]]) and passing the arguments individually is arguably the cleaner approach, this generally does not work properly, due to a longstanding bug that probably won't get fixed; for instance,
Start-Process foo.exe -Args 'one', 'two (2)' passes 3 arguments rather than 2; that is, it passes single string 'two (2)' as two arguments - see this GitHub issue.
Therefore, it is ultimately simpler and more predictable to pass a single argument with embedded quoting to -ArgumentList, but be sure to use only " (not ') for embedded quoting:
Start-Process foo.exe -Args "one `"two (2)`""
Passing arguments robustly to cmd.exe / batch files:
Note:
The limitations of cmd.exe (the legacy command processor that interprets batch files) prevent fully robust solutions; notably, you cannot prevent the interpretation of tokens such as %foo% as environment-variable references when you call a batch file from PowerShell (at least not without altering the argument to %foo^%, which will retain the ^).
In your specific case, since you're trying to echo an argument unquoted, embedded double quotes (") in such an argument - which need to be escaped as "" - aren't properly supported: they are passed through as "".
Passing an unquoted argument to cmd.exe / a batch file breaks, if that argument contains one of cmd.exe's metacharacters, i.e., characters with special syntactic meaning; in this context, they are: & | < > ^ "
The solution is to enclose the argument in double quotes ("..."), with the added need to double " chars. that are embedded (a part of the value).
PowerShell, after performing its own parsing of the command line (notably evaluating variable references and expressions), constructs the command line that is ultimately used to invoke the external target program, behind the scenes.
However, it only automatically double-quotes an argument if it contains spaces, not if it only contains cmd.exe metacharacters; e.g., a variable with verbatim string content two (2) is passed double-quoted - $val = 'two 2'; .\foo.bat $val results in command line .\foo.bat "two 2" - whereas string content a&b is not - $val = 'a&b'.\foo.bat $val results in .\foo.bat a&b - which breaks.
The solution - as shown in your question - is to enclose the variable reference in literal, embedded " characters, because such a "pre-quoted" value instructs PowerShell to pass the value as-is:
$val = 'a&b'; .\foo.bat `"$val`" results in .\foo.bat "a&b"
Note: .\foo.bat "`"$val`"" has the same effect; I'm taking advantage of the fact that PowerShell in argument (parsing) mode (generally) implicitly treats arguments as if they were double-quoted; in expression (parsing) mode, such as in the array-construction statement in the question (#(..., ...)), you do need the "`"$val`"" form.
The problem with your specific batch file:
A properly "..."-enclosed argument (with any embedded " chars. escaped as "") is properly seen as a parameter (e.g., %1) inside a batch file.
However, it is seen with the enclosing double quotes and with any doubled embedded " chars.
If you were to pass this parameter to the target program (python in this case) as an argument, everything would work as expected.
However, since you're passing the value via stdin using echo, you need to strip the enclosing double quotes so that they're not passed as part of the value, which is what your batch file attempts (e.g., %~2)
However, passing the stripped value causes the echo command to break.
There is no good solution to this problem with echo, short of performing cumbersome explicit ^-escaping (^ being cmd.exe's escape character):
$escapedUnsecurePassword = $unsecurePassword -replace '[&|<>^]' -replace '"', '""'
& $script 'userA' `"$escapedUnsecurePassword`"
That alone still isn't enough, however - your batch_script.bat file needs a modification too:
Because the assignment itself in your SET password=%~2 command isn't protected with double quotes, it breaks with values that contain metacharacters; somewhat paradoxically, you must use the form SET "password=%~2" in order to safely strip the embedded enclosing " chars.:
#echo off
REM Strip the enclosing "..." from the arguments (%~<n>)
REM !! The *assignment itself* must be in "..." so that
REM !! it does not break if the value has cmd.exe metacharacters.
set "username=%~1"
set "password=%~2"
echo(%password%|python main.py
Note that that will work as intended for all metacharacters except the - of necessity doubled - embedded ", which are passed through as "".
However, there is a workaround for echoing a string with metacharacters unquoted, as also demonstrated in subcoder's helpful answer:
If you define batch_script.bat as follows:
#echo off
set "username=%~1"
REM Do NOT strip quotes from the password argument
set password=%2
REM Using a trick with set /p, echo the password unquoted.
REM Note: Put the "|" directly after the ":" to avoid trailing spaces.
<NUL set /p=%password% & echo:|python main.py
The workaround repurposes set's /p option, which accepts a prompt message to print when interactively prompting the user for a value and prints the message without quotes; the actual interactive prompt is suppressed via <NUL, so that only the message is printed.
echo: prints a newline (line break), which is necessary, because the set /p command prints its message without a trailing newline (if you don't want to send a newline, simply omit & echo:).
Caveat: In addition to the problem with embedded "" applying here too, this workaround has a side effect: it trims leading whitespace; e.g., " foo " results in output foo (only trailing whitespace is preserved); however, given that arguments with leading whitespace are rare, this may not matter in practice.
Given how cumbersome / obscure the above approaches are, the stdin-based approach shown at the top is preferable.
You pass arguments to batch files in powershell using the -argumentlist switch of start/saps. For you you could use:
saps "$script" -argumentlist $args
But I would suggest first breaking $args up as it may not work since to pass arguments, you usually want to pass the arguments one at a time like:
saps "$script" -argumentlist "1","2","3"
Passing $args will work most of the time, but there are some cases where where it won't work. Most of the time you are fine though
I need to call a batch file from inside CYGWIN however one of it's parameters is a path-like string containing semicolons. Normally in windows command line one could enclose that parameter in quotes (which would need to be trimmed later on). However this approach doesn't wok in cygwin
Example batch (echoes first 3 parameters)
echo %1
echo %2
echo %3
Windows cmd call
file.bat "a;b" c
Ouput
"a;b"
c
empty
Cygwin call
./file.bat "a;b" c
Output
a
b
c
Including space anywhere inside quotes will ensure that parameter with semicolon or comma is passed correctly. Although I have to admit that I do not understand this behavior whatsoever, it seems to be working flawlessly.
./file.bat "a;b " c
Output
"a;b"
c
As #jeb mentioned in his comment, enclosing quotes can be trimmed by accessing parameter variable like this
%~1
Recent battles with quoting led me to another technique.
Create a temporary batch file and pass it to cmd. (I used "filex.bat" in this example).
echo 'call file.bat "a;b" c' > filex.bat ; cmd /c filex.bat ; rm filex.bat
The Windows command prompt (cmd.exe) has an optional /s parameter, which modifies the behavior of /c (run a particular command and then exit) or /k (run a particular command and then show a shell prompt). This /s parameter evidently has something to do with some arcane quote handling.
The docs are confusing, but as far as I can tell, when you do cmd /csomething, and the something contains quotation marks, then by default cmd will sometimes strip off those quotes, and /s tells it to leave them alone.
What I don't understand is when the quote removal would break anything, because that's the only time /s ("suppress the default quote-removal behavior") would be necessary. It only removes quotes under a certain arcane set of conditions, and one of those conditions is that the first character after the /c must be a quotation mark. So it's not removing quotes around arguments; it's either removing quotes around the path to the EXE you're running, or around the entire command line (or possibly around the first half of the command line, which would be bizarre).
If the path to the EXE is quoted, e.g. cmd /c "c:\tools\foo.exe" arg1 arg2, then quotes are unnecessary, and if cmd wants to remove them, fine. (It won't remove them if the path has a space in the name -- that's another of the arcane rules.) I can't imagine any reason to suppress the quote removal, so /s seems unnecessary.
If the entire command line is quoted, e.g. cmd /c "foo.exe arg1 arg2", then it seems like quote removal would be a necessity, since there's no EXE named foo.exe arg1 arg2 on the system; so it seems like opting out of quote removal using /s would actually break things. (In actual fact, however, it does not break things: cmd /s /c "foo.exe arg1 arg2" works just fine.)
Is there some subtlety to /s that's eluding me? When would it ever be necessary? When would it even make any difference?
Cmd /S is very useful as it saves you having to worry about "quoting quotes". Recall that the /C argument means "execute this command as if I had typed it at the prompt, then quit".
So if you have a complicated command which you want to pass to CMD.exe you either have to remember CMD's argument quoting rules, and properly escape all of the quotes, or use /S, which triggers a special non-parsing rule of "Strip first and last " and treat all other characters as the command to execute unchanged".
You would use it where you want to take advantage of the capabilities of the CMD shell, rather than directly calling another program. For example environment variable expansion, output or input redirection, or using CMD.exe built-ins.
Example:
Use a shell built-in: This executes as-if you had typed DEL /Q/S "%TMP%\TestFile" at the prompt:
CMD.exe /S /C " DEL /Q/S "%TMP%\TestFile" "
This executes SomeCommand.exe redirecting standard output to a temp file and standard error to the same place:
CMD.exe /S /C " "%UserProfile%\SomeCommand.exe" > "%TMP%\TestOutput.txt" 2>&1 "
So what does /S give you extra? Mainly it saves you from having to worry about quoting the quotes. It also helps where you are unsure whether for example an environtment variable contains quote characters. Just say /S and put an extra quote at the beginning and end.
Vaguely Related: $* in Bourne Shell.
Some background
Recall that the list of arguments to main() is a C-ism and Unix-ism. The Unix/Linux shell (e.g. Bourne Shell etc) interprets the command line, un-quotes the arguments, expands wildcards like * to lists of files, and passes a list of arguments to the called program.
So if you say:
$ vi *.txt
The vi command sees for example these arguments:
vi
a.txt
b.txt
c.txt
d.txt
This is because unix/linux operates internally on the basis of "list of arguments".
Windows, which derives ultimately from CP/M and VAX, does not use this system internally. To the operating system, the command line is just a single string of characters. It is the responsibility of the called program to interpret the command line, expand file globs (* etc) and deal with unquoting quoted arguments.
So the arguments expected by C, have to be hacked up by the C runtime library. The operating system only supplies a single string with the arguments in, and if your language is not C (or even if it is) it may not be interpreted as space-separated arguments quoted according to shell rules, but as something completely different.
Here's an example of how it can make a difference.
Suppose you have two executables: c:\Program.exe and c:\Program Files\foo.exe.
If you say
cmd /c "c:\Program Files\foo"
you'll run foo.exe (with no arguments) whereas if you say
cmd /s /c "c:\Program Files\foo"
you'll run Program.exe with Files\foo as the argument.
(Oddly enough, in the first example, if foo.exe didn't exist, Program.exe would run instead.)
Addendum: if you were to type
c:\Program Files\foo
at the command prompt, you would run Program.exe (as happens with cmd /s /c) rather than foo.exe (as happens with just cmd /c). So one reason for using /s would be if you want to make sure a command is parsed in exactly the same way as if it were being typed at the command prompt. This is probably more likely to be desirable in the scenario in the question Michael Burr linked to, where cmd.exe is being launched by CreateProcess rather than from a batch file or the command line itself..
That is, if you say
CreateProcess("cmd.exe", "cmd /s /c \"" MY_COMMAND "\"", ...)
then the string MY_COMMAND will be parsed exactly as if it were typed at the command prompt. If you're taking command-line input from the user, or if you're a library processing a command line provided by an application, that's probably a good idea. For example, the C runtime library system() function might be implemented in this way.
In all but one specific case, the /S won't actually make any difference.
The help for cmd.exe is accurate, if a bit complicated:
If /C or /K is specified, then the remainder of the command line after
the switch is processed as a command line, where the following logic is
used to process quote (") characters:
If all of the following conditions are met, then quote characters
on the command line are preserved:
no /S switch
exactly two quote characters
no special characters between the two quote characters,
where special is one of: &<>()#^|
there are one or more whitespace characters between the
two quote characters
the string between the two quote characters is the name
of an executable file.
Otherwise, old behavior is to see if the first character is
a quote character and if so, strip the leading character and
remove the last quote character on the command line, preserving
any text after the last quote character.
I'd summarize as follows:
Normal behavior:
If the rest of the command line after /K or /C starts with a quote, both that quote and the final quote are removed. (See exception below.) Other than that, no quotes are removed.
Exception:
If the rest of the command line after /K or /C starts with a quote, followed by the name of an executable file, followed by another quote, AND if those are the only two quotes, AND if the file name contains spaces but contains no special characters, then the quotes are not removed (even though they normally would have been removed according to the rule above).
The only effect of /S is to override this one exception, so that the two quote characters are still removed in that case.
If you always use /S, you can forget about the exception and just remember the "normal" case. The downside is that cmd.exe /S /C "file name with spaces.exe" argument1 won't work without adding an extra set of quotes, whereas without /S it would have worked... until you decide to replace argument1 with "argument1".
I have a question about the DOS start command.
I have already read this topic:
Using the DOS “start” command with parameters passed to the started program
Using the "start" command with parameters passed to the started program
but my question is a little different.
I have this problem: I need to pass paths that need to be quoted.
For example, if path have no quotes this works fine:
start "" app.exe -option c:\myapp\myfile.txt
but if path have double quotes it doesn't works.
I have this line in my BATCH file:
start "" myapp.exe -option %mypath%
and when %mypath% contains double quotes (paths that have spaces or other characters in names) the start command returns very strange results.
Thanks
Sandro
Normally it's not a problem to use parameters there with quotes, but you get problems if your app-path has also quotes.
Then you need to add an extra CALL statement.
start "" app.exe -option c:\myapp\myfile.txt - Works
start "" app.exe -option "c:\myapp\myfile.txt" - Works
start "" "app.exe" -option c:\myapp\myfile.txt - Works
start "" "app.exe" -option "c:\myapp\myfile.txt" - Don't works
start "" CALL "app.exe" -option "c:\myapp\myfile.txt" - Works
This might help, but it is a bit way round about method and slight modification may required to suit your need.
The idea is to:
Dump the environment variable which has quotes to a text file with a predefined name. Like:"set mypath2 > withQt.bat"
Use windows power shell or some third party tool to find and replace quotes in that file.
Create another text file (one time step only) containing string "Set "
Use copy command to append the file mentioned in step2 with the file created in step3 and create a batch file with a predefined name. Like: copy base.bat + withQt.bat withtqt.bat
Run the batch file, which creates another/replaces the environment variable with value without quotes.
Sorry, I couldn't get something more elegant at this time.
There have been a lot of questions asked and answered about batch file parameters with regards to the %*, but I haven't found an answer for this.
Is there an equivalent syntax in batch files that can perform the same behavior as "$#" in Unix?
Some context:
#echo off
set MYPATH=%~dp0
set PYTHON=%MYPATH%..\python\python
set BASENAME=%~n0
set XTPY=%MYPATH%..\SGTools\bin\%BASENAME%.py
"%PYTHON%" "%XTPY%" %*
This is the .bat file that is being used a proxy to call a Python script. So I am passing all the parameters (except the script name) to the Python script. This works fine until there is a parameter in quotes and/or contains spaces.
In shell scripts you can use "$#" to take each parameter and enclose it in quotes. Is there something I can do to replicate this process?
Example calls:
xt -T sg -t "path with possible spaces" -sum "name with spaces" -p <tool_name> -o lin32 lin64 win32 <lots of other options with possibilities of spaces>
The command/file xt simply contains the code listed above, because the actual executable is Python code in a different folder. So the point is to create a self-contained package where you only add one directory (xbin directory) to your path.
I'm not sure what the cleanest solution is, but here is how I worked around the problem:
setlocal EnableDelayedExpansion
for %%i in (%*) do set _args= !_args! "%%~i"
echo %_args%
%_args% will now contain a quoted list of each individual parameter. For example, if you called the batch file as follows:
MYBATFILE "test'1'file" "test'2'file" "test 3 file"`
echo %_args%
will produce the original quoted input.
I needed this for CMD files that take unfriendly file or directory names and pass them to Cygwin Bash shell scripts which do the heavy lifting, but I couldn't afford to have the embedded single quotes or spaces lost in the transition.
Note the ~i in %%~i% which is necessary to remove quotes before we apply quotes. When a parameter containing spaces is passed (e.g., "test 3 file" above), the CMD shell will already have applied quotes to it. The tilde here makes sure that we don't double-quote parameters containing spaces.