My GitHub Actions workflow "runs-on" windows-latest. I want a custom Action which executes a PowerShell (core or legacy is fine) script. I have a parallel action that runs on Linux and MacOS. So, my .github/actions/install-tf-win directory contains this action.yml
name: install_tf_win
description: installs Terraform/TerraGrunt for Windows
runs:
using: "composite"
steps:
- run: install-tf-win\install-tf-win.ps1
shell: pwsh
The directory also contains install-tf-win.ps1. I have tried all sorts of variations on that run statement. Starting with "&" and without, variations in paths used with forwards and backwards slashes. I originally started with $GITHUB_ACTION_PATH/install-tf-win.ps1 (works for Linux/MacOS), however it seemed that GITHUB_ACTION_PATH was getting evaluated to be an empty string and then there were complaints about /install-tf-win.ps1 not being found. I tried both pwsh and powershell for the shell key value.
The form shown above results in this error:
Run ./.github/actions/install-tf-win
install-tf-win\install-tf-win.ps1: D:\a\_temp\c1d3d7fa-074b-4f90-ade0-799dcebd84ec.ps1:2
Line |
2 | install-tf-win\install-tf-win.ps1
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| The module 'install-tf-win' could not be loaded. For more information, run 'Import-Module
| install-tf-win'.
I can obviously code my way around this by just putting the PowerShell statements in a step. But, the documentation suggests this should work. https://docs.github.com/en/free-pro-team#latest/actions/creating-actions/metadata-syntax-for-github-actions#runsstepsshell .
So, how do I get this to work, please?
The error message implies that you cannot refer to your script with a relative path:
The command is passed to pwsh, PowerShell [Core]'s CLI, via its -Command (-c) parameter, which interprets the install-tf-win part of
install-tf-win\install-tf-win.ps1 as a module name rather than as a subdirectory in the absence of an actual subdirectory by that name on Windows[1], so the implication is that such a path doesn't exist.
The linked documentation, suggests that you need an absolute path, based on the GITHUB_ACTION_PATH environment variable, which in PowerShell must be referenced as $env:GITHUB_ACTION_PATH (untested):
# ...
- run: '& $env:GITHUB_ACTION_PATH/install-tf-win/install-tf-win.ps1'
shell: pwsh
Note:
The need to use &, the call operator, which is a syntactic necessity due to the script path containing a(n environment-)variable reference; the same would apply if the path were quoted - see this answer for background information.
Since & is a metacharacter in YAML when used unquoted at the start of a value, the entire string is quoted. Single-quoting ('...') is employed in this case, so that YAML doesn't interpret the contents of the string up front.
As an aside: The implication of '...'-quoting working (as confirmed by Kevin, the OP) is that when pwsh -c is ultimately called on Windows, the string's content is (properly) double-quoted ("..."), because PowerShell's CLI only recognizes " as having syntactic function for command-line argument parsing. By contrast, a '...'-quoted -c argument would be interpreted as a verbatim string rather than as a command, causing its content to be simply echoed.
[1] How PowerShell interprets a path such as foo\bar.ps1 when executed as a command, as of PowerShell 7.1
Interpreted as a command - both inside a PowerShell session and via the -Command / -c parameter of the PowerShell CLI, as used by GitHub Actions - the form foo\bar.ps1 is ambiguous:
It could refer to a module named foo, and its bar.ps1 command (even though module commands don't have .ps1 name extensions).
It could refer to a subdirectory foo and file bar.ps1 in it.
This applies to Windows only:
PowerShell first interprets the path as a file-system path.
If the path doesn't exist, it falls back to interpreting it as a module-command reference.
On Unix-like platforms - even though PowerShell generally allows you to use \ and / interchangeably as the path separator on all supported platforms - this particular path form is always interpreted as a module reference; in other words: you cannot use this syntax form to invoke a script this way (see below).
This surprising inconsistency is discussed in GitHub issue #14307.
There are two ways to avoid this ambiguity, which work on all supported platforms; the following cause PowerShell to unequivocally treat the path as file-system script path:
Use / instead of \: foo/bar.ps1
Prefix the path with .\ (or ./): .\foo\bar.ps1
(Of course, you could also use a full path.)
Note that when the -File parameter of PowerShell's is used (rather than -Command / -c), this ambiguity doesn't arise, as the argument is then always considered a (potentially relative) file-system path, even on Unix-like platforms, irrespective of whether you use \ or /.
Related
I have created a simple .bat file. In this batch file i have a variable named urlExample which is equal to "example.com".
In the same batch file i want to use this variable urlExample in a powershell command.
Specifically, consider the following code:
#echo off
set urlExample = "example.com"
powershell -ExecutionPolicy Bypass -Command "& {$WebClient = New-Object System.Net.WebClient;$WebClient.DownloadFile($urlExample,"C:\.....somePath")
How could i achieve using the urlExample inside the WebClient command?
P.s. I don't want to simply put the url in the DownloadFile's first argument. I want to pass it with a batch variable.
Thanks in advance
In cmd.exe, all variables are also environment variables, such as %urlExample% in your case, and child processes - such as a call to powershell.exe, the Windows PowerShell CLI, inherit environment variables.
By contrast, PowerShell also has shell(-only) variables (e.g., $urlExample, limited to that session only), whereas environment variables must be accessed via the env namespace (e.g. $env:urlExample - see the conceptual about_Environment_Variables help topic).
While you can use string interpolation on the cmd.exe side to "bake in" the values of cmd.exe-defined environment variables, by embedding %urlExample% in the -Command argument, it is more robust to let PowerShell access the environment variable, by referencing $env:urlExample.
Therefore:
#echo off
:: Note: No spaces around "=", double-quote the name *and* the value.
set "urlExample=example.com"
:: Note the reference to $env:urlExample
:: Embedded " chars. are escaped as \"
powershell -ExecutionPolicy Bypass -Command "$WebClient = New-Object System.Net.WebClient; $WebClient.DownloadFile($env:urlExample, \"C:\.....somePath\")"
Note:
-ExecutionPolicy Bypass isn't strictly needed in this case, given that no execution of a script file is (.ps1) is involved here (whether directly with -File or indirectly, as part of a -Command argument).
However, given that the effective execution policy also applies in less obvious scenarios when you use -Command (the default parameter of powershell.exe)[1] - such as (possibly implicitly) loading a module that is either a script module (*.psm1) and / or contains formatting / type-definition data (*.ps1xml) - using -ExecutionPolicy Bypass is a good habit to form to ensure predictable execution, assuming that you trust the code you're invoking.
As Compo points out, another good habit to form to ensure a predictable execution environment is to use -NoProfile, which bypasses loading of PowerShell's profile files. In addition to preventing potentially unnecessary / unwanted modifications of the execution environment by the profiles, bypassing profile loading also speeds up the command.
[1] Note that pwsh, the PowerShell (Core) CLI, now defaults to -File.
What specific syntax needs to be changed in the windows command line commands below in order for git remote add origin %repoWithToken% to resolve to the intended valid URL?
COMMANDS THAT ARE FAILING:
The following commands are run in a pwsh shell on a windows-latest GitHub runner.
set repoWithToken="https://"$GIT_PAT"#github.com/accountName/repoName.git"
git init
git remote add origin %repoWithToken%
ERROR MESSAGE:
The following error message is given by the GitHub windows-latest runner when the above code is run:
fatal: '%repoWithToken%' does not appear to be a git repository
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Error: Process completed with exit code 1.
The answer can just be simple windows command line command. No need for any fancy PowerShell.
Also, $GIT_PAT is correct GitHub syntax, though it is possible that the concatenation in set repoWithToken="https://"$GIT_PAT"#github.com/accountName/repoName.git" might need to be done differently in the code above. This is in the process of being translated from Bash.
Building on Olaf's helpful comments:
set <varName>=<value is used in cmd.exe for assigning variables; while in PowerShell set is a built-in alias for Set-Variable, the latter has very different syntax.
More importantly, its use is rarely needed, because variables are assigned as $<varName> = <value>
PowerShell uses the same syntax on setting and getting variable values, $<varName>; thus, after having assigned to a variable named repoWithToken with $repoWithToken = <value>, you also refer to its value later with $repoWithToken (by contrast, %repoWithToken% is again cmd.exe syntax).
Assuming that GIT_PAT is the name of an environment variable, you must refer to it as $env:GIT_PAT in PowerShell variable (with explicit name delineation: ${env:GIT_PAT})
Unlike Bash (and POSIX-compatible shells in general), PowerShell only allows you to compose a single string from a mix of quoted and unquoted tokens if the first token is unquoted.
Something like "https://"$GIT_PAT"#github.com/accountName/repoName.git" therefore doesn't work as a single string argument, because its first token is quoted, causing PowerShell to break it into two arguments in this case.
Since string interpolation is needed here in order to replace ${env:GIT_PAT} with its value, simply enclose the entire value in "...", i.e. make it an expandable string
Therefore:
$repoWithToken = "https://${env:GIT_PAT}#github.com/accountName/repoName.git"
git init
git remote add origin $repoWithToken
I would like to know if there is any way to achieve this behavior on windows, for example:
/b?n/ca? /etc/pa??wd -> executes 'cat /etc/passwd'
With limited exceptions in PowerShell, on Windows there is no support for shell-level globbing - target commands themselves must perform resolution of wildcard patterns to matching filenames; if they don't, globbing must be performed manually, up front, and the results passed as literal paths; see the bottom section for background information.
PowerShell:
Perhaps surprisingly, you can invoke an executable by wildcard pattern, as zett42 points out, though that behavior is problematic (see bottom section):
# Surprisingly DOES find C:\Windows\System32\attrib.exe
# and invokes it.
C:\Windows\System3?\attr*.exe /?
Generally, you can discover commands, including external programs, via the Get-Command cmdlet.
Many file-processing cmdlets in PowerShell do perform their own globbing (e.g., Get-ChildItem, Remove-Item); if you're calling commands that do not, notably external programs that don't, you must perform globbing manually, up front, except on Unix-like platforms when calling _external programs, where PowerShell does perform automatic globbbing (see bottom section):
Use Convert-Path to get the full, file-system-native paths of matching files or directories.
While Resolve-Path may work too, it returns objects whose .ProviderPath property you need to access to get the same information (stringifying these objects, as happens implicitly when you pass them to external programs, yields their .Path property, which may be based on PowerShell-only drives that external programs and .NET APIs know nothing about.)
For more control over what is matched, use Get-ChildItem and access the result objects' .Name or .FullName property, as needed; for instance, Get-ChildItem allows you to limit matching to files (-File) or directories (-Directory) only.
PowerShell makes it easy to use the results of manually performed globbing programmatically; the following example passes the full paths of all *.txt files in the current directory to the cmd.exe's echo command as individual arguments; PowerShell automatically encloses paths with spaces in "...", if needed:
cmd /c echo (Get-ChildItem -Filter *.txt).FullName
Generally, note that PowerShell's wildcard patterns are more powerful than those of the host platform's file-system APIs, and notably include support for character sets (e.g. [ab]) and ranges (e.g. [0-9]); another important difference is that ? matches exactly one character, whereas the native file-system APIs on Windows match none or one.
However, when using the -Filter parameter of file-processing cmdlets such as Get-ChildItem, the host platform's patterns are used, which - while limiting features - improves performance; a caveat is that on Unix-like platforms ? then seemingly acts like on Windows, i.e causing it to match none or one character.
cmd.exe (Command Prompt, the legacy shell):
cmd.exe does not support calling executables by wildcard pattern; some of cmd.exe's internal commands (e.g., dir and del) and some standard external programs (e.g., attrib.exe) do perform their own globbing; otherwise you must perform globbing manually, up front:
where.exe, the external program for discovering external programs fundamentally only supports wildcard patterns in executable names (e.g. where find*.exe), not in paths, which limits wildcard-based lookups to executables located in directories listed in the PATH environment variable.
:: OK - "*" is part of a *name* only
where.exe find*.exe
:: !! FAILS: "*" or "?" must not be part of a *path*
:: !! -> "ERROR: Invalid pattern is specified in "path:pattern"."
where.exe C:\Windows\System32\find*.exe
Globbing via dir appears to be limited to wildcard characters in the last path component:
:: OK - "*" is only in the *last* path component.
dir C:\Windows\System32\attri*
:: !! FAILS: "*" or "?" must not occur in *non-terminal* components.
:: !! -> "The filename, directory name, or volume label syntax is incorrect."
dir C:\Windows\System3?\attri*
Using manual globbing results programmatically is quite cumbersome in cmd.exe and requires use of for statements (whose wildcard matching has the same limitations as the dir command); for example, using the syntax for batch files (.cmd or .bat files):
To use the resolved executable file path for invocation (assuming only one file matches):
#echo off
setlocal
:: Use a `for` loop over a wildcard pattern to enumerate
:: the matching filenames - assumed to be just *one* in this case,
:: namely attrib.exe, and save it in a variable.
for %%f in (C:\Windows\System32\attr*.exe) do set "Exe=%%f"
:: Execute the resolved filename to show its command-line help.
"%Exe%" /?
To pass matching filenames as multiple arguments to a single command:
#echo off
setlocal enableDelayedExpansion
:: Use a `for` loop over a wildcard pattern to enumerate
:: matching filenames and collect them in a single variable.
set files=
for %%f in (*.txt) do set files=!files! "%%f"
:: Pass all matching filenames to `echo` in this example.
echo %files%
Background information:
On Unix-like platforms, POSIX-compatible shells such as Bash themselves perform globbing (resolving filename wildcard patterns to matching filenames), before the target command sees the resulting filenames, as part of a feature set called shell expansions (link is to the Bash manual).
On Windows, cmd.exe (the legacy shell also known as Command Prompt) does NOT perform such expansions and PowerShell mostly does NOT.
That is, it is generally up to each target command to interpret wildcard patterns as such and resolve them to matching filenames.
That said, in PowerShell, many built-in commands, known as cmdlets, do support PowerShell's wildcard patterns, notably via the -Path parameter of provider cmdlets, such as Get-ChildItem.
Additionally and more generally, cmdlet parameters that represent names often support wildcards too; e.g., Get-Process exp* lists all processes whose image name start with exp, such as explorer.
Note that the absence of Unix-style shell expansions on Windows also implies that no semantic distinction is made between unquoted and quoted arguments (e.g., *.txt vs. "*.txt"): a target command generally sees both as verbatim *.txt.
In PowerShell, automatic globbing DOES occur in these limited cases:
Perhaps surprisingly, an executable file path can be invoked via a wildcard pattern:
as-is, if the pattern isn't enclosed in '...' or "..." and/or contains no variable references or expressions; e.g.:
C:\Windows\System3?\attri?.exe
via &, the call operator, otherwise; e.g.:
& $env:SystemRoot\System32\attri?.exe
However, this feature is of questionable utility - When would you not want to know up front what specific executable you're invoking? - and it is unclear whether it was implemented by design, given that inappropriate wildcard processing surfaces in other contexts too - see GitHub issue #4726.
Additionally, up to at least PowerShell 7.2.4, if two or more executables match the wildcard pattern, a misleading error occurs, suggesting that no matching executable was found - see GitHub issue #17468; a variation of the problem also affects passing a wildcard-based path (as opposed to a mere name) that matches multiple executables to Get-Command.
In POSIX-compatible shells, the multi-match scenario is handled differently, but is equally useless: the first matching executable is invoked, and all others are passed as its arguments.
On Unix-like platforms only, PowerShell emulates the globbing feature of POSIX-compatible shells when calling external programs, in an effort to behave more like the platform-native shells; if PowerShell didn't do that, something as simple as ls *.txt would fail, given that the external /bin/ls utility would then receive verbatim *.txt as its argument.
However, this emulation has limitations, as of PowerShell 7.2.4:
The inability to use wildcard patterns that contain spaces - see GitHub issue #10683.
The inability to include hidden files - see GitHub issue #4683.
A still experimental feature, available in preview versions of 7.3, PSNativePSPathResolution, automatically translates wildcard patterns based on PowerShell-only drives to their underlying native file-system paths; however, this feature is currently overzealous - see GitHub issue #13640 - and inherently bears the risk of false positives - see GitHub issue #13644
In PowerShell you can use Resolve-Path which Resolves the wildcard characters in a path, and displays the path contents.
Example: I want to locate signtool.exe from the Windows SDK which typically resides in "c:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64\signtool.exe" where there could be any other version(s) installed.
So I could use: Resolve-Path 'c:\program*\Windows Kits\10\bin\*\x64\signtool.exe'
EDIT:
If you want to execute it directly you can use the & invocation operator e.g.
&(Resolve-Path 'c:\wind?ws\n?tepad.exe')
Out of curiosity I called which myscript.sh for a script I have under ~/bin/myscript.sh (in macOS's bash). I can execute the script with no problems from any directory without giving its full path. ~/bin is on my PATH and the script has executable flags set, ls -l outputs -rwxr-xr-x.
I would have expected which to show me the script's full path, but it didn't output anything.
Is this intended behaviour or is something odd happening here?
This happens when you add a literal tilde ~ to your PATH, instead of the actual path to your home directory. Rewriting ~ into /home/youruser is the job of the shell, not of the tool or filesystem. If you e.g. quote the "~", this rewrite doesn't happen and other tools like which get confused.
Here's information on this issue from the shellcheck wiki:
Literal tilde in PATH works poorly across programs.
Problematic code:
PATH="$PATH:~/bin"
Correct code:
PATH="$PATH:$HOME/bin"
Rationale:
Having literal ~ in PATH is a bad idea. Bash handles it, but nothing else does.
This means that even if you're always using Bash, you should avoid it because any invoked program that relies on PATH will effectively ignore those entries.
For example, make may say foo: Command not found even though foo works fine from the shell and Make and Bash both use the same PATH. You'll get similar messages from any non-bash scripts invoked, and whereis will come up empty.
Use $HOME or full path instead.
It is fine to run
evince ./result/demo_1000000_10000*.ps
on a shell window. But when I put it into a scripts file, then run that file, it can not find all those files ./result/demo_1000000_10000*.ps... here * is meant to be a wildcard and following is the scripts.
evince ./result/demo_1000000_10000"*.ps"
So are there any changes that should be made when putting commands into scripts?
It should work the same way in a script or on the command line. The quotation marks prevent the wildcard from being expanded. Just remove them from the script. (Why did you add them in the first place?)
If the command runs from the prompt as shown, then it should also run from a shell script if the current directory of the invoking process is the same - with exactly the same notation. There is no reason to include quotes in the scripted version if you want it to do the same as the unscripted version. And if you ran the quoted version at the command line, it would fail the same as the quoted version in the scripted version does.
However, in a script, you do have to worry about whether the Postscript files you plan to work on are in the correct location. Sometimes, the script uses an absolute pathname, sometimes the script uses cd to change directory to the correct place, sometimes there's an argument or environment variable that locates the files.
So, if used carefully, you don't have to change anything for the script to work - but there are many ways you can prevent the script from working. One of those is by adding quotes around wildcard characters.