Windows 2008: application crashes when __COMPAT_LAYER=ElevateCreateProcess - winapi

There is application that can be started properly by Windows 2008, lets call it App1.
When the same application is started by another application (Launcher), App1 crashes in secur32.dll.
Research indicated environment in those 2 cases was different.
When application crashes, it's environment has one extra variable __COMPAT_LAYER=ElevateCreateProcess.
Setting __COMPAT_LAYER=asInvoker on a global system level didn't resolve the issue.
Adding App1 and Launcher to registry at HKLM\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers also doesn't resolve it.
Both programs have the following permission in their manifests: requestedExecutionLevel level="asInvoker" uiAccess="false"
UAC is turned off. Windows updates are up to date.
The only solution that helps is injecting another program in between that clears the __COMPAT_LAYER or sets it to asInvoker.
What can possibly be the cause of ElevateCreateProcess and how to avoid it? Thank you.

The cause of this behavior might be the naming of your Launcher process. Windows augments every process matching the naming pattern "*launch*.exe" with the ElevateCreateProcess Flag. This awkward behavior is specified in the windows\apppatch\sysmain.sdb shim database. You can view this by installing and running the application compatibility Toolkit. Under "System Database (32-bit)/Applications/Autorun" you can find this entry. Although disabling might be risky as well, rightclick -> Disable Entry does not work in my case either.

As it turns out the issue was prompted by calling GetUserName during dll init which crashes on Vista. Related post is https://groups.google.com/group/microsoft.public.win32.programmer.kernel/browse_thread/thread/ae1b1e4ecbb6b123/1e9fefd4c8b12452

Related

VB6 Deployment not working for Second user

I have a VB6 Application that I am deploying using Microsoft Development Environment.
The install package is working for Windows 7/10 32/64 bit.
The problem I am having is if a second user logs into windows and runs the installed app some of the controls are not working.
Forms that contain MSCOMCT2.OCX, TABCTL32.OCX and maybe others fail saying:
"Component ... or one of is dependencies not correctly registered: a file is missing or invalid"
But once again for the user that installed the app all is good.
I see in other installs there is a choice to install for only me or all users. I don't see a way to implement that in "Microsoft Development Environment", is this my problem?
What changes if you install an app for All users?
This may have to do with registry virtualization. The application may be running in some kind of compatibility mode so that the controls are not really being installed system-wide.
I'm not too familiar with those settings but it might be possible to adjust them so that the DLLs/OCX files are genuinely registered for all users and the application runs accordingly. There is a risk that if the compatibility settings were set intentionally, this could break something else.
Hopefully this is a useful starting point.
I will add that IMO installing the application not to require registration whatsoever is a much more robust solution. See How can you force VB6 to use the DLLs and OCXs from the app directory?

DllRegisterServer failed for comct332.ocx

I have a VB6 application I am trying to get working on a Windows 7 environment, however every time I start the application, I get the error:
"Component 'ComCt332.ocx' or one of its dependencies not correctly registered: a file is missing or invalid".
To resolve, I have tried to register the comct332.ocx file by running the regsvr32 in the Command Prompt in Administrator Mode but then I get the error:
"The module "comct332.ocx" was loaded but the call to DllRegisterServer failed with error code 0x80004005"
Other things I have tried include:
Deleting all parent nodes in the registry where 'comct332.ocx' exists
and running regsvr32 again in Admin Mode. Same result.
I granted admin permission to another user on the PC and I could register the
file successfully, and the application starts and runs successfully!
However when I log in as the previous user again, it fails miserably.
Any help, thoughts, other-things-to-try will be much appreicated. Thanks
If you have been keeping up on things as you must if you are to continue using VB6 successfully there are a number of things you'll be aware of.
One of these is the impact of UAC and per-user registry
virtualization.
Another is the impact of SysWOW registry redirection on 64-bit
systems.
You will understand that proper installation packages are more important than ever before. Windows has many auto-remediations for legacy software but some of them will not result in applications having all of the originally intended behaviors. Most of them will only be applied when your application "follows the right path" from installation to second run.
Here we have a case that is intended to be handled through use of a proper Windows Installer package, or at least a legacy setup recognized as such through Windows' "legacy installer detection heuristics." In general legacy scripted setups are deprecated but as long as they stay on the path Windows makes efforts to ensure they succeed.
Manually deploying by just copying over a bunch of files and randomly running regsvr32 on some of them has a reduced chance of success. This was never an approved method of deployment anyway.
You have most likely run afoul of some combination of registry virtualization and redirection.
The regsvr32 utility is a development tool, not a deployment tool. If you insist on trying to use it for deployment you must follow the same rules a developer must follow:
Run the correct version. On a 64-bit system there are both 64- and
32-bit versions of this utility. The 32-bit version which you must
use is located in the SysWOW64 folder.
Run it from an elevated command prompt. An easy
way to start one is to type <Winkey>cmd.exe<Ctrl-Shift-Enter> then
approve the UAC prompt or provide over-the-shoulder admin credentials
as needed.
There are many other things you need to know and handle in order to be successful. If you have ignored those most of them will only become apparent to you after your program gets installed and will run. A lot of them stem from filesystem virtualization.

How to automatically run a VB6 compiled software as Administrator and in Compatibility mode

We have an ERP application which is built in VB6 and it was running just fine till a few of our customer upgraded all their systems to Win7.
This software is programmed in such a way that it registers a few plugins (found in plugin folder) which are COM based at runtime. Now when the software is run by user without setting Compatibility Mode and Run as Administrator setting it fails and crashes.
I know we can manually set both Compatibility Mode as well as Run a Administrator by right clicking on program executable and then going to its properties. But this looks very unprofessional.
I think there will be some way to tell Win 7 to automatically run a software in Compatibility Mode and Run as Administrator.
Please help me.
Compatibility is an administrative function, not a development or deployment function. It is better to fix the application where possible, especially to remove any requirement for elevation.
There are plenty of tools for investigating the issues so you can correct them. However globally registering "plug ins" at runtime is a nasty one. VB6 component self-registration is always global unless registry virtualization can redirect it. Why not create installers for the plug-ins that can run elevated once during installation?
There are ways to set compatibility less manually, even as part of installation - though Microsoft discourages this. Maybe take a look at:
Compatibility Fix Database Management Strategies and Deployment
However the effort required might be better spent on remediating the problem. Support costs will be less over time.
As the other answers have said, you shouldn't need to run all the time elevated.
If you want to register the plugins after its started (as a normal user), you can use ShellExecute() with the "runas" verb to run regsvr32.exe, or use COM elevation which has been discussed many times before.
You can indicate that an application must run as admin by specifying it in the Application Manifest, which in an xml file that you can either embed or deploy with your application.
Once your application is running with admin rights it should be able to register and load the plugins. You should not need to run in compatibility mode to access the COM plugins.

VB6 application no longer opens on Vista computer

I have a VB6 app that formerly worked perfectly on a Vista machine as a scheduled task, but it will no longer open on the same machine. The app generates export files in a specified folder with no direct output on the screen. I get no errors, no missing references, just absolutely nothing.
The machine is running Vista Business 32-bit, UAC is disabled with a single administrator account, and automatic updates are turned off. The app resides in a non-protected folder, and the export files are put in a folder on the desktop. The client swears that the only change they made to that computer since I installed this app was installing Norton Antivirus, which has never caused problems before with our software.
In addition to the normal VB6 references, the app references Microsoft Scripting Runtime (scrrun.dll), and Microsoft DAO 3.6 (dao360.dll). Both of these files are present and registered on the target machine, along with all the other VB6 dependencies. I added MsgBox statements at the beginning of Sub Main() just to see if anything is being executed, and its not. Disabling Norton yielded no results, nor did reinstalling VB6 runtime to rule out any corrupted libraries. Not once did I get any messages, error or otherwise from my app.
I've never had an issue like this before and I'm completely stumped. Is there anything else that could be causing this?
Edit - The app does not run even when I run it manually, so the part about it being a scheduled task is irrelevant to my problem, sorry for including it.
The user has full administrator credentials, no compatibility mode was needed on the initial test which at the time, was done on this very machine I am having the problem on. For grins I tried compatibility mode for XP and 2000, still nothing.
Try to inspect - if you can access them - the Event Viewer messages. Maybe you will find some tell-tell signs in there...
You could try running the program in Windbg, a free standalone debugger from Microsoft. Compile your VB6 EXE into native code with symbols (create PDB files) and you will be able to debug your application in Windbg.
I would guess one of two things will happen.
Windbg will fail to load the EXE. Presumably with an error message that will identify your problem.
Windbg will load the EXE, and you can single-step through to see what happens.
Here's a 2006 blog post by a Microsoft guy about using Windbg with VB6, and 2004 blog post by another Microsoft VB guy with a brief introduction to Windbg.
Has the user changed their password? That will cause the scheduled task to fail until they re-enter the password on the task.
Have you tried running the process directly, instead of as a scheduled task? I'm far from an expert, but it may be that any errors being generated are not showing up because the program is running as a task.

Why do some installers need to restart the computer before you run the program?

I'm fairly new to deploying desktop applications, so this is the first time I'm building an installer for my software. Currently, the first time I run my software after installing it, it crashes. After that, it runs fine. I'm still in the process of debugging this issue, but I've noticed that a reboot immediately after installing and before running the software seems to fix this crash.
What are common reasons that some installers ask you to reboot before running the software?
Also, why do some installers need to restart before finishing the installation?
Thanks in advance!
Generally, reboots between install and running are because you're replacing system DLLs that are locked. The installer sets up a run-once registry entry that copies the DLLs during boot before they're locked by Windows, and then your application can use the new DLL that you laid down.
In general, this is highly discouraged. There should be no reason anymore for anyone other than Windows itself to need to reboot after install anymore.
I'd suggest going back into a debugger to find out more details about why your application is crashing. Hopefully you have enough tracing available in your code to allow that type of debugging.
In general, a reboot is only required if the installer needs to replace files that are currently in use (and locked) by the OS. Sometimes installers will err on the side of reboot-just-in-case, but this is rarer than it used to be.
We've got a case with an installer I maintain where we need to force a reboot for vista only due to UAC and privilege elevation. Once the setup is complete we need to trigger a program as the current user context, not the elevated Vista context.
In your case, I suspect that some files haven't been correctly registered and you just need to stick with the debugging ;)

Resources