VB6: Single-instance application across all user sessions - vb6

Question

I have an application that needs to be a single-instance app across all user sessions on a Windows PC. My research thus far has centered around using a mutex to accomplish this, but I am having an issue that I am not sure is really an issue, this is really a best-practice question I believe.
Here's the code first of all:
Private Const AppVer = "Global\UNIQUENAME" ' This is not what i am using but the name is unique
Public Sub Main()
Dim mutexValue As Long
mutexValue = CreateMutex(ByVal 0&, 1, AppVer)
If (Err.LastDllError = ERROR_ALREADY_EXISTS) Then
SaveTitle$ = App.Title
App.Title = "... duplicate instance."
MsgBox "A duplicate instance of this program exists."
CloseHandle mutexValue
Exit Sub
End If
' Else keep on truckin'
Now, based on this article I believe I understand that by passing the NULL pointer to the CreateMutex function as I am above I'm basically assigning whatever security descriptor is associated with the currently logged in user.
If that means what I think it does (I may need more guidance here) that tells me that other users who log in will not be able to "see" the mutex created under the original user's session, nor will they be able to create a mutex with the same name.
Now, emperical evidence seems to back this up. I used a message box to pop the "LastDLLError" I was receiving, and when another user attempted to launch the application (while it was already running under another user account) I would receive an ERROR_ACCESS_DENIED code. I am OK with testing against this along with the ERROR_ALREADY_EXISTS code and just exiting on either/or. However, this feels sort of hackish and I'm wondering if someone can suggest an alternative. The "right" thing to do seems to be to pass the proper pointer to the CreateMutex function such that any user has the proper permissions to view any existing mutexes (mutices?), but I'm not so sure this is possible without the currently logged in user being an admin (which is unacceptible). Any assistance/guidance is greatly appreciated. Thanks in advance!

Answer 1

You don't need admin priveleges to set security on you own mutexes. Here is a simple demo app that basicly gives Everyone/Full control to the mutex.
Option Explicit
Private Const STANDARD_RIGHTS_REQUIRED As Long = &HF0000
Private Const SYNCHRONIZE As Long = &H100000
Private Const MUTANT_QUERY_STATE As Long = &H1
Private Const MUTANT_ALL_ACCESS As Long = (STANDARD_RIGHTS_REQUIRED Or SYNCHRONIZE Or MUTANT_QUERY_STATE)
Private Const SECURITY_DESCRIPTOR_REVISION As Long = 1
Private Const DACL_SECURITY_INFORMATION As Long = 4
Private Declare Function CreateMutex Lib "kernel32" Alias "CreateMutexA" (lpMutexAttributes As Any, ByVal bInitialOwner As Long, ByVal lpName As String) As Long
Private Declare Function OpenMutex Lib "kernel32" Alias "OpenMutexA" (ByVal dwDesiredAccess As Long, ByVal bInheritHandle As Long, ByVal lpName As String) As Long
Private Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long
Private Declare Function InitializeSecurityDescriptor Lib "advapi32.dll" (pSecurityDescriptor As Any, ByVal dwRevision As Long) As Long
Private Declare Function SetSecurityDescriptorDacl Lib "advapi32.dll" (pSecurityDescriptor As Any, ByVal bDaclPresent As Long, pDacl As Any, ByVal bDaclDefaulted As Long) As Long
Private Declare Function SetKernelObjectSecurity Lib "advapi32.dll" (ByVal Handle As Long, ByVal SecurityInformation As Long, pSecurityDescriptor As SECURITY_DESCRIPTOR) As Long
Private Type SECURITY_DESCRIPTOR
Revision As Byte
Sbz1 As Byte
Control As Long
Owner As Long
Group As Long
pSacl As Long
pDacl As Long
End Type
Private Const MUTEX_NAME As String = "Global\20b70e57-1c2e-4de9-99e5-20f3961e6812"
Private m_hCurrentMutex As Long
Private Sub Form_Load()
Dim hMutex As Long
Dim uSec As SECURITY_DESCRIPTOR
hMutex = OpenMutex(MUTANT_ALL_ACCESS, 0, MUTEX_NAME)
If hMutex <> 0 Then
Call CloseHandle(hMutex)
MsgBox "Already running", vbExclamation
Unload Me
Exit Sub
End If
m_hCurrentMutex = CreateMutex(ByVal 0&, 1, MUTEX_NAME)
Call InitializeSecurityDescriptor(uSec, SECURITY_DESCRIPTOR_REVISION)
Call SetSecurityDescriptorDacl(uSec, 1, ByVal 0, 0)
Call SetKernelObjectSecurity(m_hCurrentMutex, DACL_SECURITY_INFORMATION, uSec)
End Sub
Private Sub Form_Unload(Cancel As Integer)
If m_hCurrentMutex <> 0 Then
Call CloseHandle(m_hCurrentMutex)
m_hCurrentMutex = 0
End If
End Sub

Answer 2

I was looking for a similar solution in VB6 late last year. At the time I was unable to find any examples of VB6 apps communicating across the user boundary, so I had to write my own.
See: Interprocess Communication via Semaphores
You can use the class to create and check for a global semaphore which will tell you if your app is already running under any user. I didn't look at the Mutex APIs but their usage is very similar. The GetSecurityDescriptor function is what you'll want to transpose if you've already got some Mutex code written.

Answer 3

I think your instincts are exactly right. I don't know any reason why it wouldn't be safe to infer from ERROR_ACCESS_DENIED that some other process has the mutex, so effectively it's the same as ERROR_ALREADY_EXISTS (in this context.) But at the same time, it doesn't feel quite right.
As you suggest, setting a proper security descriptor is indeed the right way to do it. MSDN says that granting MUTEX_ALL_ACCESS privileges increases the risk that the user will have to be an admin, and I think you do need MUTEX_ALL_ACCESS. But in my experience it works fine for non-admins.
Your question intrigued me enough do a quick test. That means I have some source code, and so here it is:
int wmain(int argc, wchar_t* argv[])
{
ACL *existing_dacl = NULL;
ACL *new_dacl = NULL;
PSECURITY_DESCRIPTOR security_descriptor = NULL;
bool owner = false;
HANDLE mutex = CreateMutex(NULL,FALSE,L"Global\\blah");
if(mutex == NULL)
wprintf(L"CreateMutex failed: 0x%08x\r\n",GetLastError());
if(GetLastError() == ERROR_ALREADY_EXISTS)
wprintf(L"Got handle to existing mutex\r\n");
else
{
wprintf(L"Created new mutex\r\n");
owner = true;
}
if(owner)
{
// Get the DACL on the mutex
HRESULT hr = GetSecurityInfo(mutex,SE_KERNEL_OBJECT,
DACL_SECURITY_INFORMATION,NULL,NULL,
&existing_dacl,NULL,
&security_descriptor);
if(hr != S_OK)
wprintf(L"GetSecurityInfo failed: 0x%08x\r\n",hr);
// Add an ACE to the ACL
EXPLICIT_ACCESSW ace;
memset(&ace,0,sizeof(ace));
ace.grfAccessPermissions = MUTEX_ALL_ACCESS;
ace.grfAccessMode = GRANT_ACCESS;
ace.grfInheritance = NO_INHERITANCE;
ace.Trustee.pMultipleTrustee = NULL;
ace.Trustee.MultipleTrusteeOperation = NO_MULTIPLE_TRUSTEE;
ace.Trustee.TrusteeForm = TRUSTEE_IS_NAME;
ace.Trustee.TrusteeType = TRUSTEE_IS_WELL_KNOWN_GROUP;
ace.Trustee.ptstrName = L"EVERYONE";
hr = SetEntriesInAcl(1,&ace,existing_dacl,&new_dacl);
if(hr != S_OK)
wprintf(L"SetEntriesInAcl failed: 0x%08x\r\n",hr);
// Set the modified DACL on the mutex
hr = SetSecurityInfo(mutex,SE_KERNEL_OBJECT,
DACL_SECURITY_INFORMATION,NULL,NULL,new_dacl,NULL);
if(hr != S_OK)
wprintf(L"SetSecurityInfo failed: 0x%08x\r\n",hr);
else
wprintf(L"Changed ACL\r\n");
LocalFree(existing_dacl);
LocalFree(new_dacl);
LocalFree(security_descriptor);
}
wprintf(L"Press any key...");
_getch();
CloseHandle(mutex);
return 0;
}