Windows Installer fails on Win 10 but not Win 7 using WIX - windows

Question

I've recently been assigned to update our projects installer to run on Windows 10 and I'm at a bit of a loss on how to get it to work. I have no experience doing installers and am in the process of familiarizing myself not only with the process in general but also how our project handles it.
As of now, the installer works perfectly running on our Windows 7 VM, but when running it on our Windows 10 VM it fails near the end and begins to rollback. I've gotten it to spit out some log files and I'm digging through them but am fairly loss.
I've tracked down this bit:
MSI (s) (B0:F4) [17:39:02:883]: Note: 1: 1708
MSI (s) (B0:F4) [17:39:02:883]: Note: 1: 2205 2: 3: Error
MSI (s) (B0:F4) [17:39:02:883]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1708
MSI (s) (B0:F4) [17:39:02:883]: Note: 1: 2205 2: 3: Error
MSI (s) (B0:F4) [17:39:02:883]: Note: 1: 2228 2: 3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1709
MSI (s) (B0:F4) [17:39:02:883]: Product: GPEP -- Installation failed.
Near the end which seems to occur at or around when the installer seems to fail.
The next line has this at the end:
Installation success or error status: 1603.
I've looked into the error and found this: Error 1603
I'm looking into the solutions on that page but all of that should be in order. We're running the installer in the same way with the same permissions on the Win 10 and Win 7 VM's.
I doubt this will be enough information to get any concrete responses, so more than anything I'm looking for constructive advice how where to look and how to figure this out. I have more details I could post, but it's such a large volume of information and I don't know how to pick out what is genuinely relevant.

Answer 1

Adding this as an answer - it is too long as a comment, and I think it is the correct answer as well. If you read this answer, please also see my comment above for a very good MSI log file debugging tip from Rob Mensching (creator of WiX) - and how to ensure all entries make it into the log file by enabling "flush to log" for crashing custom actions.
The Answer
Dependency on a missing runtime like Powershell would certainly trigger the rollback. MSI hosts its own runtime for certain script custom actions (active scripting). For example VBScript and JavaScript. For reliable deployment, it is recommended that all custom actions used be either self-contained minimum-dependency C++ dlls or executables (win32) or (less desirable) VBScript or JavaScript (or even Installscript if you use Installshield - see details below).
Disputed opinion of mine: The worst custom actions to use for reliability and robustness are .NET binaries requiring a specific version of the .NET framework. This also applies to PowerShell - which is not only managed code, but also a script. I am very tempted to say that these technologies shouldn't be used for deployment, but if you need to use PowerShell you must at a minimum add a "verify PowerShell installed" custom action at the start of your setup, and exit gracefully with a proper error message displayed (and/or logged) if PowerShell is not available.
That is the end of the real answer :-). Below are some "verbose musing" in case you are making a package for general distribution (and not just a package for your own company's internal deployment). If I were you I would read it even if you only deploy internally, PowerShell custom actions could be a brewing deployment problem of caliber.
Both managed code and scripts are problematic. PowerShell is effectively both at the same time. Here is Rob Mensching's blog on why script custom actions are bad. Essentially: scripts are fragile, they lack language features, they are hard to debug and anti-virus products often block them. Read the blog. And here is Aaron Stebner's blog on why managed code is bad. Essentially you are not guaranteed a proper runtime environment when you depend on the presence of the .NET framework.
Verbose Musings
I am not sure what is installed as standard on Win7 and Win10. If your are deploying as an "internal package" to your company, I think it should be OK to just add a reliable check for the presence of PowerShell, and then to abort with a meaningful error message if PowerShell is not found. Opinion Warning: But overall .NET binaries and PowerShell scripts are the worst custom actions for reliability. I would never use them for setups targeting diverse computers.
If you are making an MSI for general distribution to any computer anywhere, I would take the time to convert the PowerShell script to something else. Preferably a C++ dll - which I find most reliable. There are no dependencies to speak of or layers to depend on. Even InstallScript is acceptable if you would have been using Installshield (it can run without a pre-installed runtime at this point - which has significantly improved its reliability and usefulness - it is an obtuse language though with rather archaic syntax. In fairness, not to be underestimated - it does the job, and is simpler than C++).
JavaScript and VBScript custom actions are possible to use even for MSIs that are for general distribution to any computer, but still not recommended. I tend to use them only for "internal company deployment" packages. These can be standardized and crucially scripts are transparent to other system administrators and packagers. They can see and inspect what is being done as part of the installation. This is generally desirable and one of the key benefits of MSI for corporate deployment, but sometimes you need a compiled binary to hide implementation details (for example when you validate a license key). Then scripts of any kind can't be used - obviously. By being transparent and also embedded in the MSI (so the full, running source is always available), it helps different application packagers to be able to pick up someone else's work when need be. And in a deployment team there is always someone available to debug scripts - but few may know proper C++. In corporations where developers of internal applications make their own MSI files without much deployment knowledge, scripting can go completely astray and cause very difficult deployment problems. Very often what is needed is small changes to the application itself to allow more reliable deployment. An application should do its own startup configuration for example - none of this should be done in setup scripts, but many developers do this.
Using script custom actions is controversial. If you ask 2 development experts you will get 4 opinions. In my view "white box" custom actions (scripts) are good for corporate use if they do something specific that isn't common so people can see what is going on. For stuff that is needed all the time, a corporation should make a compiled C++ dll driven by custom tables in the MSI file with full QA and rollback support - something that is generally always missing for all script custom actions (it isn't trivial to implement). A "data driven" (custom tables) C++ custom action has minimal dependencies as its biggest strength, and it is also transparent (what will happen is transparent, but the actual implementation is compiled and hidden - which can also improve security). The WiX toolkit provides such a custom action dll with rollback support written in C++. It should solve most custom tasks required for corporate deployment. All of this is way beyond your question though - just a digression :-).
If I were to guess I would say that Windows Installer might be updated to be able to host its own runtime for Powershell - but this is just speculation. I am not sure of the technical details - it would seem the whole .NET runtime would be needed? If you ask me, I would still prefer a JavaScript to a PowerShell script, but I realize you are probably committed to PowerShell as a company standard? Also, always prefer JavaScript over VB Script since it has something that looks like exception handling (which VB Script lacks entirely). UPDATE: real-world testing indicates that VBScript is actually better to use with MSI than Javascript. For example: I have seen obscure problems when accessing the MSI API with Javascript. MSI itself was probably tested more with VBScript than with Javascript when it was created. Let's be honest: both "languages" have severe limitations and both are hard to debug.
Rob Mensching, Chris Painter, Phil Wilson, Bob Arnson and probably others too (I am not sure of Stefan Kruger's position on scripts, or Robert Dickau's view) - will kill me for this, but here is a template for a JavaScript custom action (untested by me, but looks OK): How to debug an MSI Custom Action that is implemented in Javascript? . If I can just blurt it out: anything is better than PowerShell at the present time - even JavaScript.
Rest assured, I have wasted a lot of time debugging extremely poor VB Script custom actions. Probably the most incompetent and deprived language ever used for deployment. On Error Resume Next for error handling? It can't get much worse. I generally only use scripts for read-only operations and set property actions.
Maybe we will see VB Script deprecated and PowerShell added as a viable MSI scripting option in due time? I wouldn't judge this as safe until all operating systems in use would have at least a baseline version of the .NET framework installed - and even then I believe policies could lock specific versions of .NET from being used. Do you want a package that suddenly can't uninstall because the target version of the .NET framework is no longer operational? Fixing such an issue could be an incredible amount of work - especially for a corporation with a large package estate (thousands of packages, thousands of machines).
Recommended Custom Action Implementation
I wrote up a summary of "recommendations" for custom action implementation. It became pages long without saying much - I deleted it. Instead, here is a list of my custom action implementation preference (in order of decreasing robustness and reliability): 
UPDATE May,2018: no longer recommending Javascript over VBScript.
C++ dll
Installscript (InstallShield only)
VB Script
JavaScript
C# DTF
PowerShell
Summary:
For me PowerShell is at the time of writing absolutely the worst choice. It is both managed code (unreliable runtime) and a script custom action (poor debugging).
I would like to write C# / DTF custom actions like Chris does for simplicity, but I don't believe the time is ripe - the runtime environment cannot be guaranteed. In the real world you don't throw out a working C++ dll in favor of a C# dll. It is a huge reliability downgrade.
C++ dll and Installscript are the only choices for making a professional, vendor setup targeting diverse computers (not standardized desktops in managed environments - corporations, but computers anywhere in the world in all their heterogeneous states, in different languages and diverse hardware and software configurations).
A C++ custom action dll is significantly harder to set up and configure than other custom actions with its exports, build settings and outputs, but it is no magical impossibility. In return you get a lot: full debugging capability, advanced language features and error handling. And the big one: minimum dependencies (make sure you enable static linking to eliminate all possible dependencies). For debugging you can simply attach the Visual Studio debugger to a message box displayed by your custom action, and then you can step through code. This works for both user and system context custom actions. Full control. This actually makes debugging a C++ custom action easier than a script custom action, and certainly more reliable.
JavaScript I would generally avoid. It just isn't a complete language. I still think it is more reliable than managed code though - in terms of runtime dependencies and reliability (fewer runtime dependency pitfalls).
VB Script is acceptable for "internal corporate use" in a managed environment. I would never use it for a vendor setup for general distribution. But to distribute packages on a corporate network it can be used. Both for developers packaging their own applications, and for application packagers tweaking third party setups for corporate deployment. The primary advantages and disadvantages of VBScript actions:
As stated above, scripts should only be used in rare cases, and a win32 C++ dll or WiX's custom action dll should be used for all common scripting tasks that people tend to re-use. Scripts are only to be used when needed to get the job done.
VBScript custom actions are, like all script custom actions, in general hard to debug, vulnerable to anti-virus interference and lacking in language features needed to implement advanced coding constructs. You just don't have the language features and flexibility available with C++ (now even C++ custom actions can be blocked by security software - but it is not as common, but could that change as security is tightened?)
Scripts are transparent for everyone (both purpose and implementation) and can be debugged and maintained easily by several team members with work handed off between them. All can see what is going on and everyone can pick up someone else's work quickly.
The source embedded in the MSI is the right source, you don't need to maintain source files separately in a repository to compile it like you need for managed code (C#). For application packaging source control is rarely set up is my experience (it should be though).
Corporate packages target a standard operating environment (SOE). All the workstations are similar or the same, with the same anti-virus solution. This obviously means that the target computers are in a much more uniform state than what is normal. Any anti-virus issues will be detected and can be managed. Personally I haven't seen any major anti-virus interference problems with simple scripts for such package deployment.
There tends to be a lot of expertise in script debugging in packaging teams, but very little C++ knowledge (many know some C# and PowerShell though). Developers would likely prefer C#, but can easily handle scripts.
One thing that I am certain of, is that the availability of managed code custom actions will cause people to do way too many things in their setups that should never be done in a setup (rich API, relatively easy coding). This is all because coding is easier and faster, and the developer in question may lack an understanding of how proper deployment should be done. This inevitably leads to overuse of custom actions of all kinds, and in turn major deployment problems as the complexity of custom actions trigger unexpected errors.