EC2 Windows User Data: Powershell does not run as expected - windows

Question

I am trying to bootstrap EC2 Windows instances via Powershell commands inside user data. The steps I am trying to execute via user data are:
Install chocolatey
Use chocolatey to install Python
Use chocolatey to install AWS CLI
Use AWS CLI to download a Powershell script from S3
Run that Powershell script
The user data is pretty straightforward:
<powershell>
Set-ExecutionPolicy Bypass -Force;
iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'));
choco install python3 -y;
choco install awscli -y
refreshenv
$BootstrapScript = (Join-Path $env:TEMP "NewBootstrap.ps1")
& aws s3api get-object --bucket my-bucket-name --key bootstrap/WindowsBootstrap.ps1 "$BootstrapScript"
iex "$BootstrapScript"
</powershell>
The bootstrap script is never downloaded or executed. If I logon to the instance and view the logs, the output suggests something strange is happening:
At the top of the logs there are errors complaining that 'aws' is not recognized as the name of a cmdlet. Why should this error before it's even tried to install the aws cli?
After that the term
'C:\Users\Administrator\AppData\Local\Temp\NewBootstrap.ps1' is not recognized as the name of a cmdlet, function,
script file. Again - why is this erroring before we've tried to get that file?
Then the logs show that choco, python and awscli actually were installed correctly.
I don't understand what is happening with the order of execution. If I login to the box and execute the exact same user data script that is contained in C:\Windows\Temp it runs completely as expected.
Any help on understanding or debugging this would be most appreciated.
Output from C:\ProgramData\Amazon\EC2-Windows\Launch\Log\UserdataExecution.log with some choco output elided for brevity
2017/11/06 12:11:49Z: Userdata execution begins
2017/11/06 12:11:49Z: Zero or more than one <persist> tag was not provided
2017/11/06 12:11:49Z: Unregistering the persist scheduled task
2017/11/06 12:11:54Z: Zero or more than one <runAsLocalSystem> tag was not provided
2017/11/06 12:11:54Z: Zero or more than one <script> tag was not provided
2017/11/06 12:11:54Z: Zero or more than one <powershellArguments> tag was not provided
2017/11/06 12:11:54Z: <powershell> tag was provided.. running powershell content
2017/11/06 15:13:42Z: Userdata execution begins
2017/11/06 15:13:42Z: Zero or more than one <persist> tag was not provided
2017/11/06 15:13:42Z: Unregistering the persist scheduled task
2017/11/06 15:13:54Z: Zero or more than one <runAsLocalSystem> tag was not provided
2017/11/06 15:13:54Z: Zero or more than one <script> tag was not provided
2017/11/06 15:13:54Z: Zero or more than one <powershellArguments> tag was not provided
2017/11/06 15:13:55Z: <powershell> tag was provided.. running powershell content
2017/11/06 15:16:11Z: Userdata: is currently executing. To end it kill the process with id: 2828
2017/11/06 15:17:40Z: Message: The errors from user scripts: & : The term 'aws' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the
spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\Windows\TEMP\UserScript.ps1:15 char:3
+ & aws s3api get-object --bucket my-bucket-name --key bootstra ...
+ ~~~
+ CategoryInfo : ObjectNotFound: (aws:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
C:\Users\Administrator\AppData\Local\Temp\NewBootstrap.ps1 : The term
'C:\Users\Administrator\AppData\Local\Temp\NewBootstrap.ps1' is not recognized as the name of a cmdlet, function,
script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is
correct and try again.
At line:1 char:1
+ C:\Users\Administrator\AppData\Local\Temp\NewBootstrap.ps1
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (C:\Users\Admini...ewBootstrap.ps1:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
2017/11/06 15:17:40Z: Message: The output from user scripts: This is the new bootstrap
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 11/6/2017 3:14 PM chocInstall
Getting latest version of the Chocolatey package for download.
Getting Chocolatey from https://chocolatey.org/api/v2/package/chocolatey/0.10.8.
... Chocolatey Install output ...
Chocolatey (choco.exe) is now ready.
You can call choco from anywhere, command line or powershell by typing choco.
Run choco /? for a list of functions.
You may need to shut down and restart powershell and/or consoles
first prior to using choco.
Ensuring chocolatey commands are on the path
Ensuring chocolatey.nupkg is in the lib folder
Installing awscli
Chocolatey v0.10.8
Installing the following packages:
python3
.. Python Download / Install output ...
Download of python-3.6.3-amd64.exe (30.16 MB) completed.
Hashes match.
Installing python3...
python3 has been installed.
Installed to 'C:\Python36'
python3 can be automatically uninstalled.
Environment Vars (like PATH) have changed. Close/reopen your shell to
see the changes (or in powershell/cmd.exe just type `refreshenv`).
The install of python3 was successful.
Software installed as 'EXE', install location is likely default.
Chocolatey installed 1/1 packages.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Chocolatey v0.10.8
Installing the following packages:
awscli
.. AWS CLI Download / Install output ...
The install of awscli was successful.
Software installed as 'msi', install location is likely default.
Chocolatey installed 1/1 packages.
See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Refreshing environment variables from registry for cmd.exe. Please wait...Finished..
2017/11/06 15:17:40Z: Userdata execution done

Answer 1

The issue was that when installed as part of the cloud init process, the Powershell profile fails to import Chocolatey. This means that packages will install fine via choco install, but are not made available in the environment, even if you call refreshenv (hence my call to aws failed even though it installed successfully.)
To fix this problem you can manually force Powershell to import the Chocolatey module by editing your Powershell profile.
# Updated profile content to explicitly import Choco
$ChocoProfileValue = #'
$ChocolateyProfile = "$env:ChocolateyInstall\helpers\chocolateyProfile.psm1"
if (Test-Path($ChocolateyProfile)) {
Import-Module "$ChocolateyProfile"
}
'#
# Write it to the $profile location
Set-Content -Path "$profile" -Value $ChocoProfileValue -Force
# Source it
. $profile
You can read more on this in the chocolatey troubleshooting guide