Under what conditions does RmGetList return 2 for the lpdwRebootReasons output parameter? - winapi

Question

Background
I am designing an Inno Setup installer to install a Cygwin service, and I am puzzled by the behavior I'm seeing from the Windows Restart Manager APIs.
Specifically, when the service is running (started using the cygrunsrv utility), the RmGetList API function returns 2 (RmRebootReasonSessionMismatch) for its lpdwRebootReasons output parameter. This output parameter is an enumeration of type RM_REBOOT_REASON, and the description on MSDN for the RmRebootReasonSessionMismatch value is:
One or more processes are running in another Terminal Services session.
The Inno Setup log file contains lines like the following:
RestartManager found an application using one of our files: <executable name>
RestartManager found an application using one of our files: <service name>
Can use RestartManager to avoid reboot? No (2: Session Mismatch)
Inno Setup then proceeds trying to replace in-use files, as if Restart Manager was not used at all.
I am puzzled by this output value, because on two different machines I tested (Windows 10 1909 x64 and Windows Server 2012 R2), no terminal server/remote desktop users are logged on.
If I stop the service and start another executable (in the set of files to be replaced by the installer), RmGetList returns 0 (RmRebootReasonNone) for lpdwRebootReasons, and Inno Setup displays the normal dialog for in-use files and allows the user to select to automatically close them.
Process Explorer shows both processes (cygrunsrv.exe and the process it starts) running in session 0 and at System integrity level. Both are console subsystem executables.
Questions
Under what conditions does RmGetList return 2 (RmRebootReasonSessionMismatch) for its lpdwRebootReasons output parameter? (I'm trying to understand why this happens when the service is running.)
Does this value cause the entire Restart Manager session to fail, or can Restart Manager proceed even though it thinks applications are running in one or more different sessions?

Answer 1

For question 2, in the document RM_PROCESS_INFO
bRestartable
TRUE if the application can be restarted by the Restart Manager;
otherwise, FALSE. This member is always TRUE if the process is a
service. This member is always FALSE if the process is a critical
system process.
This value indicates if the application can be restarted by the Restart Manager.
For question 1, Note that services are running in session 0. If the process occupying the resource (registered in RmRegisterResources) is a service A, the RmGetList function which also running in the service process B will return lpdwRebootReasons = RmRebootReasonNone, bRestartable = TRUE.
But if A is not a service, then A & B are running in different sessions, lpdwRebootReasons = RmRebootReasonSessionMismatch and bRestartable = FALSE
other results:(B run with elevated privileges)
A & B is a console and in the same session：lpdwRebootReasons = RmRebootReasonNone, bRestartable = TRUE, ApplicationType = RmConsole.
A & B is a console and in the different session：lpdwRebootReasons = RmRebootReasonSessionMismatch, bRestartable = FALSE, ApplicationType = RmConsole.
A: service, B: console: lpdwRebootReasons = RmRebootReasonNone, bRestartable = TRUE, ApplicationType = RmService
(B not run with elevated privileges):
A & B is a console and in the different session：lpdwRebootReasons = RmRebootReasonCriticalProcess, bRestartable = FALSE, ApplicationType = RmCritical.
A: service, B: console: lpdwRebootReasons = RmRebootReasonPermissionDenied, bRestartable = FALSE, ApplicationType = RmCritical
According the document bRestartable depends on ApplicationType. And then, we can see that if the bRestartable = TRUE, then the lpdwRebootReasons = RmRebootReasonNone. But when bRestartable = FALSE, It depends on the other members RM_PROCESS_INFO.

Answer 2

A Clue from the RestartManager PowerShell Module
The RestartManager PowerShell module (special thanks to Heath Stewart) provides a simple PowerShell interface for the Restart Manager. My commands are as follows:
Set-Location <path to Cygwin root directory>
Start-RestartManagerSession
Get-ChildItem . -File -Include *.exe,*.dll -Recurse | RegisterRestartManagerResource
Get-RestartManagerProcess
Stop-RestartManagerProcess
These commands produce the following output:
Id : <process ID>
StartTime : <process start time>
Description : <executable started by cygrunsrv>
ServiceName :
ApplicationType : Console
ApplicationStatus : Running
IsRestartable : False
RebootReason : SessionMismatch
Id : <cygrunsrv process id>
StartTime : <cygrunsrv process start time>
Description : <description of service>
ServiceName : <service name>
ApplicationType : Service
ApplicationStatus : Running
IsRestartable : True
RebootReason : SessionMismatch
For some reason, Restart Manager sees the cygrunsrv.exe service process as restartable, but the executable it spawns as not restartable. (I am still curious about why this happens in the first place.)
An Imperfect Workaround Attempt
Based on this observed behavior, I first attempted the following workaround:
In the Inno Setup script's [Setup] section, set the following:
CloseApplications=yes
CloseApplicationsFilter=*.chm,*.pdf
RestartApplications=yes
The CloseApplicationsFilter directive specifies what files get registered with the Restart Manager. Note I do not specify *.exe or *.dll here; I want to manually specify only certain .exe files in the [Code] section.
Call the Inno Setup RegisterExtraCloseApplicationsResource function once for each .exe file in the setup that will NOT be spawned by cygrunsrv and put them in the RegisterExtraCloseApplicationsResources event procedure. Example:
[Code]
procedure RegisterExtraCloseApplicationsResources();
begin
RegisterExtraCloseApplicationsResource(false, ExpandConstant('{app}\bin\cygrunsrv.exe'));
end;
It's important not to register any executable spawned by cygrunsrv.exe or any of the Cygwin DLL files, because this will prevent the Restart Manager from taking effect in Inno Setup.
This solution is far from perfect, because executables normally started by cygrunsrv, if started separately, are not detected by Restart Manager (e.g., sshd.exe). For example, new SSH sessions are spawned in executables that the Restart Manager are not restartable.
A Better Solution
I've decided that a better solution is to detect any running executables from code and prompt the user apart from the Restart Manager functionality (which, simply put, doesn't work for Cygwin services).