How do I gracefully take a web app offline during Octopus deployment? - octopus-deploy

I was a bit remiss to find that Octopus, as amazing as it is, doesn't do anything cute or clever about shutting down your web app before it is upgraded.
In our solution we have two web apps (a website and a separate API web app) that rely on the same database, so while one is being upgraded the other is still live and there is potential that web or API requests are still being serviced while the database is being upgraded.
Not clean!
Clean would be for Octopus to shut down the web apps, wait until they are shut-down and then go ahead with the upgrade, bring the app pools back online once complete.
How can that be achieved?

Selfie-answer!
It is easy to make Octopus-deploy take a little extra care with your deployments, all you need is a couple of extra Execute-Powershell steps in your deployment routine.
Add a new first step to stop the app pool:
# Settings
#---------------
$appPoolName = "PushpayApi" # Or we could set this from an Octopus environment setting.
# Installation
#---------------
Import-Module WebAdministration
# see http://technet.microsoft.com/en-us/library/ee790588.aspx
cd IIS:\
if ( (Get-WebAppPoolState -Name $appPoolName).Value -eq "Stopped" )
{
Write-Host "AppPool already stopped: " + $appPoolName
}
Write-Host "Shutting down the AppPool: " + $appPoolName
Write-Host (Get-WebAppPoolState $appPoolName).Value
# Signal to stop.
Stop-WebAppPool -Name $appPoolName
do
{
Write-Host (Get-WebAppPoolState $appPoolName).Value
Start-Sleep -Seconds 1
}
until ( (Get-WebAppPoolState -Name $appPoolName).Value -eq "Stopped" )
# Wait for the apppool to shut down.
And then add another step at the end to restart the app pool:
# Settings
#---------------
$appPoolName = "PushpayApi"
# Installation
#---------------
Import-Module WebAdministration
# see http://technet.microsoft.com/en-us/library/ee790588.aspx
cd IIS:\
if ( (Get-WebAppPoolState -Name $appPoolName).Value -eq "Started" )
{
Write-Host "AppPool already started: " + $appPoolName
}
Write-Host "Starting the AppPool: " + $appPoolName
Write-Host (Get-WebAppPoolState $appPoolName).Value
# To restart the app pool ...
Start-WebAppPool -Name $appPoolName
Get-WebAppPoolState -Name $appPoolName

The approach we took was to deploy an _app_offline.htm (App Offline) file with the application. That way we get a nice message explaining why the site is down.
Then when it is time for deployment we use Mircrosofts Webdeploy to rename the it to app_offline.htm. We put the code for the rename in a powershell script that runs as the first step of our Octopus Deployment.
write-host "Website: $WebSiteName"
# Take Website Offline
$path = "$WebDeployPath";
$path
$verb = "-verb:sync";
$verb
# Take root Website offline
$src = "-source:contentPath=```"$WebSiteName/_app_offline.htm```"";
$src
$dest = "-dest:contentPath=```"$WebSiteName/app_offline.htm```"";
$dest
Invoke-Expression "&'$path' $verb $src $dest";
# Take Sub Website 1 offline
$src = "-source:contentPath=```"$WebSiteName/WebApp1/_app_offline.htm```"";
$dest = "-dest:contentPath=```"$WebSiteName/WebApp1/app_offline.htm```"";
Invoke-Expression "&'$path' $verb $src $dest";
$WebSiteName is usually "Default Web Site". Also note that the ` are not single quotes but actually the backtick character (usually found with the tilda on your keyboard).
Now if octopus is deploying your web site to a new location, your web site will come back online automatically. If you don't want that, you can deploy the new website with the app_offline file allready in place. Then you can use the following script to remove it.
write-host $WebSiteName
# & "c:\Program Files (x86)\IIS\Microsoft Web Deploy V2\msdeploy.exe" -verb:delete -dest:contentPath="$WebSiteName/app_offline.htm"
# those arn't QUOTES!!!!, they are the back accent thing.
write-host "Website: $WebSiteName"
# Put Web app Online.
$path = "$WebDeployPath";
$path
$verb = "-verb:delete";
$verb
$dest = "-dest:contentPath=```"$WebSiteName/app_offline.htm```"";
$dest
Invoke-Expression "&'$path' $verb $dest";
# Put Sub Website Online
$dest = "-dest:contentPath=```"$WebSiteName/WebApp1/app_offline.htm```"";
Invoke-Expression "&'$path' $verb $dest";

Stopping apppool and/or setting App_Offline file is not enough for me. Both didn't give proper explanation to clients why site is down. Especially App_Offline. I need to clean up bin folder and this causes YSOD (http://blog.kurtschindler.net/more-app_offline-htm-woes/).
My solution:
First task redirects deployed site to different folder containing only index.html with proper message. Last task brings back original folder.

A better solution would be to use a network load balancer such as the f5 LTM. You can set up multiple servers to receive traffic for your site and then, when you are deploying, you can just disable the one node in the NLB so all the other traffic goes to the other machine.
I like the f5 because it is very scriptable. When we deploy to our websites we take no outage whatsoever. all traffic to the site is just pointed to the server that is not currently being upgraded.
There are caveats:
You must script the downing disable of the pool member in the NLM so that it works with your site. If your site requires sessions (such as depending on session state or shared objects) then you have to bleed the traffic from the NLB nodes. in f5 you can just disable them and then watch for the connection count to go to zero (also scriptable).
You must enforce a policy with your deveopers / dbas that states that all database changes MUST NOT cause degradation or failure in the existing code. This means that you have to be very careful with the databases and configurations. That way you can do your database updates before you even start deploying to the first pool of your website.

Related

Preserve Active Directory trust relationships securely in AWS between VM's that start and stop

Developing Active Directory for a scalable and hackable student environment and I cannot manage to preserve the Domain Trust Relationship after the VM's restart. On first launch everything works, but after stopping/starting the AD Set, the trust relationship is broken.
Configuration Basics.
Three machines built and running in AWS (Windows Server 2012)
Domain Controller (Pre-Built Forest, Domains, Users, Computers, GPO's, etc)
Two "Targets" with varying permissions.
AMIs are built and domian joined before imaging.
Prior to imaging, Target Boxes are REMOVED from the domain, leaving a single DC and two un-joined Windows Server 2012 boxes.
Boxes are stopped without SysPrep to preserve SIDs and other variables like admin passwords, and an image is taken. User data is enabled
At this point, I can relaunch these boxes from AMI, re-join the domain, and I have no issues after restarting.
Here are the next steps.
The AMI's are run through a code pipeline that applies user data to the boxes to Domain Join and set the DNS to the IP of the DC.
Code exists to prevent the targets from crating until the DC is listening so they can join the domain.
On creation, things work flawlessly again.
After stopping and restarting, however, I start getting "NO_LOGON_SERVER" errors with tools, and cannot login with a domain user.
There are obvious solutions, but nearly all of them manual, and this must be automated. Furthermore, I must configure this in a way that no passwords are exposed on the box or retained as tickets or hashes in lsass that could ruin the exploint path.
If it helps, here is the User-Data that domain joins the targets.
<powershell>
# Checking for response from DC
do {
Start-Sleep 30
} until(Test-NetConnection -InformationLevel Quiet '{DC_IP_ADDR}')
# "true" if Domain Joined
$dCheck = (Get-WmiObject -Class Win32_ComputerSystem).PartOfDomain
# Join domain if not already, skip if joined.
if ($dCheck -eq 'True') {
echo "I was domain joined after restarting." >> C:\Windows\Temp\log.txt
}
else {
# Allows 'rdesktop' by disabling NLA as a requirement
Set-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\' -Name "fDenyTSConnections" -Value 0
# Set DNS to DC IP address via custom variable
$adapterIndex = Get-NetAdapter | % { Process { If ( $_.Status -eq "up" ) { $_.ifIndex } }}
Set-DNSClientServerAddress –interfaceIndex $adapterIndex –ServerAddresses ('{DC_IP_ADDR}','8.8.8.8')
#Set DA Credential object and join domain
$username = 'MYDOMAIN\ADMINISTRATOR'; $password = ConvertTo-SecureString -AsPlainText 'NotTheActualPasswordButReallySecure' -Force; $Credentials = New-Object System.Management.Automation.PSCredential $Username,$Password
Add-Computer -Domain 'MYDOMAIN.LOCAL' -Credential $Credentials -Force -Restart
}
</powershell>
<persist>true</persist>
And here is the Domain Controller. It is scheduled to change it's DA Password after 4 minutes so that the password exposed in the user data above is no longer valid
<powershell>
# Enable Network Discovery
netsh advfirewall firewall set rule group="Network Discovery" new enable=Yes
# Allows 'rdesktop' by disabling NLA as a requirement
Set-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\' -Name "fDenyTSConnections" -Value 0
Start-Sleep -Seconds 240
# Recycle DA Password
powershell.exe -C "C:\Windows\Temp\recycle.ps1"
echo "Done" > "C:\Users\Administrator\Desktop\done.txt"
</powershell>

Issue with Set-CMTaskSequenceDeployment

Seems that New-CMTaskSequenceDeployment / Set-CMTaskSequenceDeployment cmdlet option -DeploymentOption does not work as expected.
I'm trying to automate a Task Sequence Deployment via Powershell. I use New-CMTaskSequenceDeployment cmdlet to create the deployment. The content of the TS should be downloaded before the start of the TS.
Works ok, but the -DeploymentOption DownloadAllContentLocallyBeforeStartingTaskSequence seems not to have any effect, when I check the deployment after the script ran, the option "pre-download content for this task sequence" isn't checked.
Same issue when I try Set-CMTaskSequenceDeployment.
Any hint from the community what I'm doing wrong?
...
# Create deployment for all waves now
foreach ($StrCollectionName in $ArrCollectionName)
{
$SchedulePhase2 = New-CMSchedule -Nonrecurring -Start $DateScheduleStartPhase2
Try {
$Deployment = New-CMTaskSequenceDeployment -CollectionName $StrCollectionName -TaskSequencePackageId $StrTaskSequenceID -DeployPurpose Required -AvailableDateTime $DateAvailablePhase1 -DeploymentOption DownloadAllContentLocallyBeforeStartingTaskSequence -SoftwareInstallation $False -SystemRestart $False -Schedule $SchedulePhase2 -RerunBehavior RerunIfFailedPreviousAttempt -AllowUsersRunIndependently $True -SendWakeUpPacket $True
Write-Host "Success - Deployment $Deployment created!"
}
Catch {
Write-Host "Error - Exception caught in creating deployment : $error[0]"
Exit
}
}
...
Looks like unfortunately (and unexpected) the pre-download behavior is different for package/program deployment and task sequence deployment.
In case of a package/program deployment the content download will start after start time in case the deployment has a mandatory time defined.
A TS deployment behaves different. It will start download when mandatory time (schedule) has been reached. Start time will be ignored.
This difference is independently from how the deployment has been started (Console or powershell cmdlet) therefore it is not an issue of the cmdlet.
First of all, you can check the picture below to make sure not to confuse these two options.
Difference between Preload content checkbox and Download all content locally before starting TS
Once done Here is my proposition :
Just by clicking, try to retrieve the property of you TSDeployment before and after you clicked the checkbox. You will see that one property changed. AdvertFlags
PS MUZ:\> (Get-CMTaskSequenceDeployment -DeploymentID MUZ200C5).AdvertFlags
[Convert]::ToString((Get-CMTaskSequenceDeployment -DeploymentID MUZ200C5).AdvertFlags,2)
Output :
34275328
10000010110000000000000000
From there, you can read from the MS doc : https://learn.microsoft.com/en-us/configmgr/develop/reference/core/servers/configure/sms_advertisement-server-wmi-class
From this, I learn that I need to change the 12th bit like this :
$advertflag = Get-CMTaskSequenceDeployment -DeploymentID MUZ200C5
$advertflag.AdvertFlags = $advertflag.AdvertFlags -bor "0x00001000"
$advertflag.put()
I hope it will help someone someday :)

Not able to spawn off email message from windows search in powershell

I have a powershell script that is searching the Windows Search index directly for emails and files. I have the following code:
$searchme="my thing to find"
$sql="SELECT System.FileName, System.ItemPathDisplay, System.DateCreated, System.DateModified, system.itemurl, system.itemtypetext FROM SYSTEMINDEX WHERE Contains(System.FileName, '"+$searchme+"') OR Contains('"+$searchme+"')"
$adapter = new-object system.data.oledb.oleDBDataadapter -argumentlist $sql, "Provider=Search.CollatorDSO;Extended Properties=’Application=Windows’;"
$ds = new-object system.data.dataset
$adapter.Fill($ds)
foreach ($record in $ds.Tables[0].Rows)
{
$exeparams = $record[4]
write-host $exeparams
write-host $exeparams.split(":")[0]
if ($exeparams.split(":")[0] -eq "mapi15")
{
$exeparams2="mapi://" + $exeparams.substring(8)
}
write-host $exeparams
write-host "start"
$exe="start"
$exe+" "+$exeparams | Out-File 'file.txt' -encoding Unicode
write-host "start-process"
Start-Process $exeparams
Start-Process $exeparams2
write-host "andpersand process"
&$exe $exeparams
&$exe $exeparams2
write-host "dotnet"
$proc = [Diagnostics.Process]::Start($exeparams)
$proc.WaitForExit()
$proc = [Diagnostics.Process]::Start($exeparams2)
$proc.WaitForExit()
}
There are several "shell" calls because I was trying to figure out if it was a process spawning issue. Files work with no issue. Emails however fail with either No such interface if i leave mapi15, or Unspecified error if i change mapi15 to mapi. I believe that Open mails in outlook from java using the protocol "mapi://" may be the solution, but if it is I am not sure how to apply this in powershell. Thank you for your help.
Ok, that took more work than I expected, and I blame Office 2013 for it. Here's the short answer:
$exeparams2 = $exeparams -replace "^mapi15", "mapi"
& start $exeparams2
That is the code that now opens an email for me. That code did not do that yesterday, all it did is tell me:
Either there is no default mail client or the current mail client cannot fulfill the messaging request. Please run Microsoft Outlook and set it as the default mail client.
Infuriating is what this was, because I did have Outlook, in fact it was running, and was the default mail application for everything email related. This annoyed me, and sent me on a quest to figure out WTF was wrong, and if I could fix it. The answers to that are "I'm not real sure WTF was wrong, except maybe a naming change on MS's part", and "yes, yes I can fix it".
I finally found the fix that worked for me (and I believe that this is probably Office 2013/Office 365 specific) was found at the bottom of this thread:
https://social.technet.microsoft.com/Forums/office/en-US/64c0bedf-2bcd-40aa-bb9c-ad5de20c84be/cannot-send-email-with-microsoft-outlook-2010-64-bit-from-adobe-outlook-not-recognised-as?forum=outlook
The process is simple. Change 2 Registry Entries, then re-set your default mail application. Registry entries:
HKLM:\Software\Clients\Mail(default)
HKLM:\Software\Clients\Mail\Microsoft Outlook(default)
Change the value of (Default) from "Microsoft Outlook" to simply "Outlook".
After that I set Outlook to be the default for everything it could be ( in Win8 that's Control Panel > All Control Panel Items > Default Programs > Set Default Programs then select Outlook, and choose the first option to make it default for all extensions that it is registered for). Once that was done I was able to run the modified code above to launch an email that I had search the index for.

Best Way to Trigger an Event When a Specific Error is Thrown in Windows 2003 Server?

I have a Windows 2003 Server that uses IIS to host a legacy ASP.NET web service that connects to a database on a remote Oracle database server that I have no control over. The problem is that the database server goes down every week or two, but then comes back up after about 5 minutes. I have to then restart IIS to remove any corrupt connections before my web service works again.
What is the best way to trigger an event (i.e. email myself and/or reset IIS) when a specific error code occurs (in this case it will be an ORA- type error, but I can get the Windows error code)?
IIS Setting?
Task Scheduler? (limited to scheduled tasks only I believe on Windows 2003 server, eg. per day/week/month etc)
Powershell Script?
Other options?
I know in Windows 2008 Server that you can configure the Task Scheduler to trigger an event when the server experiences certain error codes in its Error Log... but I can't find anything like this in the Task Scheduler of Windows 2003 Server.
Thanks.
I wrote a powershell shell script to monitor the sql server errorlog and report on specific errors. I stored where I last stop reading and then continued on from that point the next time I ran the script. This is the part that actually reads the log. Then you just need to store the position in some temp file and run this as a scheduled task. And send an email if the error occurs or even restart some service.
$path = $logs.file
Write-Host $path
if($currentLog.lastpos -ne $null){$pos = $currentLog.lastpos}
else{$pos = 0}
if($logs.enc -eq $null){$br = New-Object System.IO.BinaryReader([System.IO.File]::Open($path, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::ReadWrite))}
else{
$encoding = $logs.enc.toUpper().Replace('-','')
if($encoding -eq 'UTF16'){$encoding = 'Unicode'}
$br = New-Object System.IO.BinaryReader([System.IO.File]::Open($path, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::ReadWrite), [System.Text.Encoding]::$encoding)
}
$required = $br.BaseStream.Length - $pos
if($required -lt 0){
$pos = 0
$required = $br.BaseStream.Length
}
if($required -eq 0){$br.close(); return $null}
$br.BaseStream.Seek($pos, [System.IO.SeekOrigin]::Begin)|Out-Null
$bytes = $br.ReadBytes($required)
$result = [System.Text.Encoding]::Unicode.GetString($bytes)
$split = $result.Split("`n")
foreach($s in $split)
{
if($s.contains(" Error:"))
{
#Filter events here
}
}
$currentLog.lastpos = $br.BaseStream.Position
$br.close()

PowerShell Script Running as a Service Behaves Strangely

PowerShell Script Running as a Service Behaves Strangely
The Project:
Create a background process that determines if the on board network card is connected. If it is connected, disable the wireless network card. When the onboard network card is not connected, re-enable the wireless card.
Why:
Users hot-dock all the time, getting funky routing tables OR get bound to the wrong DNS servers. When they attempt to access a local resource, say printers, they aren’t able to and then are in my cube (they would file a ticket, but that too would be a local resource). Trying to convince users to disable their own wireless (via switch on laptop) or not hot dock has met with limited success.
The Problem:
The PowerShell script below does run, and does work under my testing conditions. Likely under most testing conditions as the code and wmi queries are pretty generic. Running the script manually yields the expected results, HOWEVER running the script as a service via the only method I could find, srvany.exe, yielded unexpected results and “broke stuff”.
Details:
Running the script as a service, via srvany.exe, works ONCE. When the loop comes back around to test the network connection or tries the method to enable or disable it. The errors indicate that “get-wmiobject” is not a proper Cmdlet. Huh? It works, manually, it works once, but a second time after it disabled the wireless network card it does not. Worse yet MY shell , outside of the service, suddenly can’t do a get-wmiobject, until…. until you go into Device Manager and re-enable the wireless network card yourself.
Debugging attempts:
I rewrote the script and cleaned it up a little to allow for it to get the objects outside of the Do While loop. Nothing changed, but I left the script that way as it seems cleaner anyhow. I enabled “Interact with Desktop” in the service properties and sure enough you can see the script trying to work and getting the before mentioned errors.
Please help. Again the object here is to run a background process, one with enough privileges in Vista or 7 to disable and enable the wireless network card.
#***********************************************************************
# "switch-wifi-srv.ps1"
# This script attempts to identify if a wired network card is in use if
# one is, the Wireless network card is disabled, until the wired network
# card is no longer in use.
#
# Written by Aaron Wurthmann - aaron (AT) wurthmann (DOT) com
#
# 2010.02.10 ver 2 (Service Version)
# If you edit please keep my name or at the very least original author's.
# As of this writing I am unsure if script will work with all scenarios,
# however it has worked for me on Dell laptops running Windows 7 x64.
# -----------------------------------------------------------------------
# This script comes with ABSOLUTELY NO WARRANTY.
# You may redistribute copies of the script under
# the terms of the GNU General Public License.
# -----------------------------------------------------------------------
# Service Installation:
# Aquire and install the Windows 2003 Resource Kit OR the srvany.exe.
# Use sc.exe and srvany.exe to create a service....
# sc create SwitchWifiAuto binPath= "C:\Program Files (x86)\Windows Resource Kits\Tools\srvany.exe" DisplayName= "Switch Wifi Automatically"
# Edit registry entry for SwitchWifiAuto, add a key and a string value...
# HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\SwitchWifiAuto\Parameters]
# "Application"="C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe -ExecutionPolicy RemoteSigned -File C:\\SwitchWifiAuto\\switch-wifi-srv.ps1"
#************************************************************************
$state=""
$wireStatus=""
$wifiStatus=""
# Get Wired and Wireless Card Objects
$objWire=get-wmiobject -class win32_networkadapter -namespace root\CIMV2 | Where-Object {$_.Name -notmatch "Wireless" -and $_.Name -notmatch "Virtual" -and $_.PhysicalAdapter -eq "True"}
$objWifi=get-wmiobject -class win32_networkadapter -namespace root\CIMV2 | where-object {$_.Name -match "Wireless"}
# Get Name of Service to be Used in totally useless Do While Loop
$objService=get-service -display "Switch Wifi Automatically"
# Begin Do While Loop
Do {
# Get status of wired network card. If enabled and connected set $state to Disable (which will later Disable the Wifi network card)
[string]$wireStatus=$objWire | % {$_.NetEnabled}
if($wireStatus -eq "True") {
$state="Disable"
}
# Get Status of wireless card.
if($objWifi){
[string]$wifiStatus=$objWifi | % {$_.NetEnabled}
# If $state is not set to disable and if the wireless card is currently disabled, enable it.
if($state -ne "Disable") {
if($wifiStatus -eq "False") {
Out-Null -InputOject ($objWifi | % {$_.Enable()})
}
# If $state is set to Disable and if wireless card is currently enabled, disable it.
} else {
if($wifiStatus -eq "True") {
Out-Null -InputOject ($objWifi | % {$_.Disable()})
}
}
}
# Reset Checked Variables for good measure
$state=""
$wireStatus=""
$wifiStatus=""
# Sleep for 120 seconds (two minutes)
Start-Sleep -s 120
# Continuing looping (do while) until the service is not running.
# This is of course technically useless as when the service is not running neither is the script to check if the service is not running.
# I made it this way however because I don't like infinite loops and I thought it would be funny to it this way instead of while $var=0
} while ($objService.Status -eq "Running")
Try to remove any output. Service don't have stdout stream. And when the buffer is full strange thing happens. Just a guess ( I never used powershell ).
Debugging attempts: I rewrote the script and cleaned it up a little to
allow for it to get the objects outside of the Do While loop.
You need to include these within the loop or you will not get updated values and the loop will do nothing.

Resources