Setting Registry Value in Wix 3.11 conditionally using VBScript - vbscript

I am hoping to write a very small VBScript in Wix 3.11 that checks if a particular registry entry has a value.
Basically, if HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Shell = "drive:\path\to\thisAppName.exe", we want to replace it with ours. We only need to know that thisAppName.exe is used as the current shell. (aka, a substring...) The Condition XML statement doesn't seem to support substring functionality.
When debugging the VBScript, the 'value' is always empty, and the session property value is empty. as well. This suggests to me that something is running out of sequence.
Due to small amount of scripting code, it makes sense to keep it in the Wix code. I'm hoping not to move this into a custom module, as this seems overkill for what is needed.
<Wix>
<Product>
...
<Property Id="SHELL_KEY">
<RegistrySearch Id="REG_SHELL_KEY" Root="HKLM"
Key="SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon"
Name="Shell"/>
</Property>
<CustomAction Id="REGISTRYKEYCHECK" Script="vbscript" Execute="immediate" Return="ignore">
<![CDATA[
Dim value
value = Session.Property("SHELL_KEY")
IF INSTR(value, "thisAppName.exe") > 0 THEN
Session.Property("RESET_SHELL_KEY") = "1"
END IF
]]>
</CustomAction>
<InstallExecuteSequence>
<Custom Action="REGISTRYKEYCHECK" Before="CostFinalize">1</Custom>
</InstallExecuteSequence>
...
<Directory>
...
<Component Id="ResetShellExe" Guid="{....}"
<Condition><![CDATA[RESET_SHELL_KEY]]</Condition>
<RegistryKey Root="HKLM" Key="SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon">
<RegistryValue Type="string" Name="Shell" Value="$(var.MyProgram.TargetFileName)" KeyPath="no"/>
</RegistryKey>
</Component>
<Feature Id="Complete" Level="1">
...
<ComponentRef Id="ResetShellExe"/>
</Feature>
...
</Product>
</Wix>

VBScript isn't necessary because Windows Installer does support substrings.

Related

WiX setup creating a Registry key doesn't work when also using CustomActions

I have issues understanding the order of operation of the WiX setup.
When trying to create a Registry key to add a menu entry to Windows Explorer context menu and simultaniously
using CustomActions the Registry key will not be added.
If I however only try to register the key, it works (any CustomAction code is commented out).
In my Product.wxs I have set elevated priviliges with
<Package InstallerVersion="200" Compressed="yes" InstallScope="perMachine" InstallPrivileges="elevated"/>.
In my <Feature> I have
<ComponentRef Id="RegistryEntries"/> referenced.
This is the code for creating the registry key
<Fragment>
<Directory Id="TARGETDIR" Name="SourceDir">
<Component Id="RegistryEntries" Guid="*">
<RegistryKey Root="HKCR"
Key="Excel.CSV\shell\Use MyConverter\command"
ForceCreateOnInstall="yes"
ForceDeleteOnUninstall="yes">
<RegistryValue Type="string" Value="[INSTALLLOCATION]$(var.SolutionName).exe %1"
KeyPath="yes"/>
</RegistryKey>
</Component>
<Directory Id="ProgramFilesFolder">
<Directory Id="HSZLG" Name="MyConverter">
<Directory Id="INSTALLLOCATION" Name="$(var.SolutionName)" />
</Directory>
</Directory>
<!--<Directory Id="ProgramMenuFolder">
<Directory Id="Shortcuts" Name="MyConverter" />
</Directory>-->
</Directory>
</Fragment>
Now Im also using the following Custom Actions:
<CustomAction Id="UnregisterImportFormat" BinaryKey="WixCustomAction" DllEntry="UnregisterImportDefinition" Execute="deferred" Impersonate="no" Return="check" />
<CustomAction Id="PropertiesForUnregisterImportFormat" Property="UnregisterImportFormat" Return="check"
Value="app=AB;key=10000P1000" />
And call them in the <InstallSequence> like this:
<InstallExecuteSequence>
<Custom Action="PropertiesForRegisterImportFormat" Before="RegisterImportFormat" />
<Custom Action="RegisterImportFormat" Before="InstallFinalize">(NOT Installed) OR REINSTALL</Custom>
<Custom Action="PropertiesForUnregisterImportFormat" Before="UnregisterImportFormat" />
<Custom Action="UnregisterImportFormat" Before="InstallFinalize">REMOVE</Custom>
</InstallExecuteSequence>
It'd be gladly appreciated if someone can point out what I am doing wrong here.
There are difficulties with (deferred) custom actions running without impersonation as well as the HKCR key because HKCR isn't an actual registry location - it's a merge of the interactive user and HKEY_LOCAL_MACHINE\Software\Classes. This goes into detail:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms724475(v=vs.85).aspx
and the custom action code is running with the system account (deferred, no impersonation) which is adding more confusion about which registry key you're creating. As that MSDN note says:
"To change the settings for the interactive user, store the changes under HKEY_CURRENT_USER\Software\Classes rather than HKEY_CLASSES_ROOT"
and I assume your code is attempting to use HKCR. Windows Installer is preferred for writing these registry entries because "it just works". Code running with the system account writing to HKCR is unreliable, so if you really must use code (and you shouldn't) then try HKEY_LOCAL_MACHINE\Software\Classes.

How do you add multiple entries to the `PATH` Envirnoment Variable using WiX?

I'm creating a MSI installer and need to add more than one entry to the PATH environment variable. Per the MSDN documentation:
Each row can contain only one value. For example, the entry Value;Value;[~] is more than one value and should not be used because it causes unpredictable results. The entry Value;[~] is just one value.
My installer source code looks like this currently (note, this is a per-machine installation), which is a violation of the above documentation:
<!-- NOTE: These two features are mutually exclusive -->
<Feature Id="Feature1" Level="1000" Absent="allow" AllowAdvertise="no" InstallDefault="local" TypicalDefault="install">
<Component Directory="INSTALLFOLDER">
<RegistryValue Action="write" Type="int" Root="HKLM" Key="SOFTWARE\MyProduct" Name="MyPathEntry1" Value="1" KeyPath="yes" />
<Environment Id="AddPathEntry1" Name="PATH" Value="[INSTALLFOLDER]SubDir1" Action="set" Permanent="yes" Part="last" System="yes" />
</Component>
</Feature>
<Feature Id="Feature2" Level="1000" Absent="allow" AllowAdvertise="no" InstallDefault="local" TypicalDefault="install">
<Component Directory="INSTALLFOLDER">
<RegistryValue Action="write" Type="int" Root="HKLM" Key="SOFTWARE\MyProduct" Name="MyPathEntry2" Value="1" KeyPath="yes" />
<Environment Id="AddPathEntry2" Name="PATH" Value="[INSTALLFOLDER]SubDir1;[INSTALLFOLDER]SubDir2;[INSTALLFOLDER]SubDir3" Action="set" Permanent="yes" Part="last" System="yes" />
</Component>
</Feature>
Now even though the above is "technically" a violation according to the MSDN documentation, it seems to work. I've tested fresh installations, modifying an installation, and upgrading. All seem to work with no hitch. But one thing I've learned with MSI is whenever possible, it's best to follow the rules to avoid messing up people's machines.
The natural solution of adding independent (i.e. not mutually exclusive) features containing only individual path components won't work because with MSI, you cannot guarantee the order in which features and/or components are installed. However, in this case, the order in which path components are added to the PATH environment variable is important due to how the PATH variable is used when finding unqualified executables.
The other question that may come to mind is, why am I using features? I want to give installers of the product the option to change their installation choice at a later date in time via the standard Add/Remove Programs or Program and Features Control Panel applet.
So, how can I add more than one path entry to the PATH environment variable in a deterministic order while following the recommended guidance from the MSDN? Or is the guidance from MSDN outdated and what I'm currently doing is perfectly fine?
I figured a way to do this which does not violate the guidance at MSDN for my particular situation.
The exact situation is I have a main program executable that installers may or may not want to be able to run from a Windows Command Shell. Additionally, there are other utilities that also installers may or may not wish to run from a Windows Command Shell.
The way this was originally modeled was as a set of 3 mutually exclusive features using radio buttons to decide how to update the system PATH environment variable. The 3 features boiled down to:
Don't modify the system's PATH environment variable (i.e. the program and optional tools can't be run from a Windows Command Shell by simply typing the name of the executable at any directory location).
Update the system's PATH environment variable so that only the main executable can be run from a Windows Command Shell at any given path.
Update the system's PATH environment variable to also add additional utility tool paths.
What I came to realize is that there are really only 2 features here, and they're not even mutually exclusive, but one is dependent on the other. The first radio button listed above is not really a feature--it's a no-op; it doesn't install anything! The second radio button represents the "main" feature in that the primary executable's directory is added to the system's PATH. The 3rd radio button depends upon the second: if the main executable is added to the PATH, then you can additionally add other utilities to the PATH as well.
This results in a much simpler UI. Instead of radio buttons, use checkboxes, and the second checkbox is disabled unless the first checkbox is checked. This also results in a much simpler implementaion for ensuring that the combination of features selected (say, for a command-line installation) is valid.
In a nutshell, here's what the WiX XML will boil down to in my particular case. I'm sure that this can be generalized (or modified) to suit other similar scenarios.
<Feature Id="AddExeToPath" Level="1" Absent="allow" AllowAdvertise="no" InstallDefault="local" TypicalDefault="install">
<Component Id="ExePath" Guid="{YOURGUID-HERE-0000-0000-00000000}" Directory="INSTALLFOLDER">
<CreateFolder />
<Environment Id="ExePath" Name="MY_PATHS" Value="[INSTALLFOLDER]SubDir1" Action="set" Part="first" System="yes" />
</Component>
<Component Directory="INSTALLFOLDER">
<RegistryValue Action="write" Type="integer" Root="HKLM" Key="SOFTWARE\MyProduct" Name="AddExeToPath" Value="1" KeyPath="yes" />
<Environment Id="AddToPath" Name="PATH" Value="%MY_PATHS%" Action="set" Part="last" System="yes" />
</Component>
<Component Id="RemoveExeFromPathOnUninstall" Guid="{YOURGUID-HERE-0000-0000-00000000}" Directory="INSTALLFOLDER">
<Condition>REMOVE><AddExeToPath OR REMOVE="ALL"</Condition>
<Environment Id="RemoveFromPath" Name="PATH" Value="%MY_PATHS%" Action="remove" />
</Component>
</Feature>
<Feature Id="AddToolsToPath" Level="1000" Absent="allow" AllowAdvertise="no" InstallDefaut="local" TypicalDefault="install">
<Component Id="SubDir2" Guid="{YOURGUID-HERE-0000-0000-00000000}" Directory="INSTALLFOLDER">
<CreateFolder />
<Environment Id="SubDir2" Name="MY_TOOLS" Value="[INSTALLFOLDER]SubDir2" Action="set" Part="first" System="yes" />
</Component>
<Component Id="SubDir3" Guid="{YOURGUID-HERE-0000-0000-00000000}" Directory="INSTALLFOLDER">
<CreateFolder />
<Environment Id="SubDir3" Name="MY_TOOLS" Value="[INSTALLFOLDER]SubDir3" Action="set" Part="last" System="yes" />
</Component>
<Component Directory="INSTALLFOLDER">
<RegistryValue Action="write" Type="integer" Root="HKLM" Key="SOFTWARE\MyProduct" Name="AddToolsToPath" Value="1" KeyPath="yes" />
<Environment Id="AddToolsToPath" Name="MY_PATHS" Value="%MY_TOOLS%" Action="set" Part="last" System="yes" />
</Component>
</Feature>
<CustomAction Id="InvalidPathFeatureSelection" Error="25000" Execute="firstSequence" />
<InstallExecuteSequence>
<Custom Action="InvalidPathFeatureSelection" Before="InstallValidate">
<![CDATA[NOT (REMOVE="ALL" OR (&AddToolsToPath >= 3 IMP &AddExeToPath >= 3))]]>
</Custom>
</InstallExecuteSequence>
This set of features and components results in the following:
If the feature AddExeToPath is selected for installation, the following components of the feature result in:
An environment variable named MY_PATHS is created and contains the path to the main executable.
The environment variable PATH is updated, placing %MY_PATHS% at the end its current value.
If the feature AddToolsToPath is selected, 3 components are installed:
The environment variable MY_TOOLS is created/set, and [INSTALLFOLDER]SubDir2 is put at the front of the variable's existing value (if any).
The environment variable MY_TOOLS is created/set and [INSTALLFOLDER]SubDir3 is put at the end of the variable's existing value (if any).
Note
These two components defined the way that they are ensure that the paths added to MY_TOOLS are added in the appropriate order.
The environment variable MY_PATHS is updated, placing %MY_TOOLS% at the end of the exisiting variable; again, this preserves the correct ordering of paths.
And so what you end up with is:
MY_TOOLS = [INSTALLFOLDER]SubDir2;[INSTALLFOLDER]SubDir3
MY_PATHS = [INSTALLFOLDER]SubDir1;%MY_TOOLS%
PATH = <existing_PATH_value>;%MY_PATHS%

How to conditionally set a file extension using Wix?

I'd like to conditionally set a file extension during install. As I understand it, in order to do anything conditionally in Wix, it should be a self-contained component. So for each file type association I would like to allow the user to set, I've got a component similar to the following:
<Component Id="FileAssocComponent_PS" Guid="DAFE9461-2DF0-934A-F204-6B28CEA23C01">
<Condition>FILE_ASSOC_PS</Condition>
<RegistryValue Root="HKLM" Key="SOFTWARE\PrinterApp\Capabilities\FileAssociations" Name=".prn" Value="PrinterApp.ps" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\PrinterApp\Capabilities\MIMEAssociations" Name="application/postscript" Value="PrinterApp.ps" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Classes\PrinterApp.ps" Name="FriendlyTypeName" Value="PostScript File" Type="string" />
<ProgId Id="PrinterApp.ps" Description="PostScript File" Icon="PrinterApp.ico" Advertise="yes">
<Extension Id="ps">
<Verb Id="open" Command="Open" Argument=""%1""/>
</Extension>
</ProgId>
</Component>
But this gives me the following error:
error LGHT0204: ICE19: Extension: 'ps' advertises component: 'FileAssocComponent_PS'. This component cannot be advertised because the KeyPath type disallows it.
I've tried setting KeyPath="yes" on one of the registry entries, but that doesn't work - and from what I've been able to find it's expecting a file KeyPath. But this is a component that doesn't contain any files!
How do I get around this error, or am I going about this the wrong way?
Advertised components require a keyfile so here are some ways around the error.
1)
Give the component a fake file ( printermimeinstalled.txt ) that won't harm the system.
2)
Author PrinterAppMime.ps as the keyfile of this component. Use the CopyFile element to get the file copied to PrinterApp.ps
Author PrinterAppNoMime.ps (same contents) as the keyfile of another component. Also use the CopyFile element to get the file copied to PrinterApp.ps. Give this component a mutually exclusive component condition so that only 1 component ever gets instaleld.
3)
Change the design of your app a little bit. Have PrinterApp.ps always installed and PrinterAppMimeServer.ps conditionally installed.
4)
Eliminate this custom action and use a custom action to author MSI temp table rows at installtime to defind the MIME stuff if the checkbox is selected.
Each of these 4 approaches have pro's and con's and I personally would choose #3.
If you set Advertise="no" you should be able to use the code you wrote. Here's an example I posted a couple of years ago here using a separate component for optional file associations.
<Component ....>
<ProgId Id="AcmeFoobar.Document" hDescription="ACME XYZ Document">
<Extension Id="pdf" ContentType="application/xyz">
<Verb Id="open" Command="Open" TargetFile="[APPLICATIONFOLDER]AcmeFoobar.exe" Argument="%1" />
</Extension>
</ProgId>
<Condition><![CDATA[DEFAULTVIEWER=1]]></Condition>
</Component>
I found a solution that worked for me. The issue I had was that I had a condition on the association of the extension to the exe. If the extension was unchecked to not associate, I needed the exe component to get installed but without the progid. Problem was that if a put a condition on the component, the progid wouldn't be created but the exe didn't get installed as well. The solution I found was two create two components. One with the condition and one with mutually exclusive condition. This is basically option 2 from Christopher Painters post.
See below:
<Component Id="My.exe" Guid="{D9CF6FDD-1234-4E90-85A1-3BF1F912C1E3}">
<Condition>NOT FILES_ASSOCIATIONS_ABC</Condition>
<File Id="My.exe.without_assoc" Name="My.exe" KeyPath="yes" Vital="yes" Compressed="yes" DiskId="1" Source=".\SourceDir\My.exe" />
</Component>
<Component Id="My.exe_assoc" Guid="{07F96643-5D74-1234-9DAE-CDEB5AC2D11E}">
<File Id="My.exe.with_assoc" Name="My.exe" KeyPath="yes" Vital="yes" Compressed="yes" DiskId="1" Source=".\SourceDir\My.exe" />
<Condition>FILES_ASSOCIATIONS_ABC</Condition>
<ProgId Id="My.Document" Description="My exe" Icon="MyIcon" Advertise="yes">
<Extension Id="abc">
<Verb Id="open" Command="My Exe" Argument=""%1"" />
<MIME Advertise="yes" ContentType="application/abc" Default="yes" />
</Extension>
</ProgId>
</Component>

Launch after install, with no UI?

How do I launch my application after install with no UI (or in quiet mode)? Thanks!
I had a installer with UI which has an option to run after install. Now I want my application to updates itself by downloading and running the new version of installer in quiet mode, but after updating done, it won't launch again.
From the msdn topic on sequencing custom actions:
As in the case of standard actions,
custom actions that are scheduled in
the InstallUISequence or
AdminUISequence run only if the
internal user interface is set to the
full level.
So I guess your custom action is scheduled in a UI sequence, not in InstallExecuteSequence. Try scheduling your custom action in the InstallExecuteSequence like this:
<InstallExecuteSequence>
<Custom Action='LaunchApplication' After='InstallFiles'/>
</InstallExecuteSequence>
where "LaunchApplication" should be replaced by the Id of your CustomAction element.
edit: I looked at the instructions that you followed, and I don't see the custom action for launching the application being scheduled in any sequence. It is only triggered from a UI action (clicking the Finish button). This explains why it is never executed during a silent install.
edit: full sample (it's a bit sloppy as it also tries to execute the custom action on uninstall, repair etc. but for some reason I couldn't get the "NOT Installed" condition to work)
<?xml version='1.0' encoding='utf-8'?>
<Wix xmlns='http://schemas.microsoft.com/wix/2006/wi'>
<Product
Name='ProductName'
Id='*'
Language='1033'
Version='0.0.1'
Manufacturer='ManufacturerName' >
<Package
Keywords='Installer'
Description='Launch application demo'
Manufacturer='ManufactererName'
InstallerVersion='100'
Languages='1033'
Compressed='yes'
SummaryCodepage='1252'/>
<Media Id='1' Cabinet='test.cab' EmbedCab='yes'/>
<Directory Id='TARGETDIR' Name="SourceDir">
<Directory Id='ProgramFilesFolder'>
<Directory Id='TestFolder' Name='Test' >
<Component Id="ExeComponent" Guid="*">
<File Id="ExeFile" Source="c:\windows\notepad.exe" />
</Component>
</Directory>
</Directory>
</Directory>
<Feature Id='Complete'
Display='expand'
Level='1'
Title='Test'
Description='Test'>
<ComponentRef Id="ExeComponent" />
</Feature>
<InstallExecuteSequence>
<Custom Action='LaunchInstalledExe' After='InstallFinalize'/>
</InstallExecuteSequence>
<CustomAction Id="LaunchInstalledExe"
FileKey="ExeFile"
ExeCommand=""
Execute="immediate"
Impersonate="yes"
Return="asyncNoWait" />
</Product>
</Wix>
In my final solution I used two properties, one for UI (LAUNCH_APP_ON_EXIT), one for command line arguments (UPDATING_AUTOMATICALLY).
I have to do this because if I run the CustomAction after InstallFinalize in full UI mode, the application would start before you click the "Finish" button.
Now I can call setup.exe /qn UPDATING_AUTOMATICALLY=1 in my program to update.
Here is it all:
<Property Id="LAUNCH_APP_ON_EXIT" Value="1" />
<Property Id="UPDATING_AUTOMATICALLY" Value ="0" />
<CustomAction Id="LaunchApplication" FileKey="mainExecutableFile" ExeCommand="" Execute="immediate" Impersonate="yes" Return="asyncNoWait" />
<UI>
<!-- explainations: http://www.dizzymonkeydesign.com/blog/misc/adding-and-customizing-dlgs-in-wix-3/ -->
<UIRef Id="MyWixUI_InstallDir" />
<UIRef Id="WixUI_ErrorProgressText"/>
<Publish Dialog="MyExitDialog" Control="Finish" Order="1" Event="DoAction" Value="LaunchApplication">LAUNCH_APP_ON_EXIT</Publish>
</UI>
<InstallExecuteSequence>
<Custom Action='LaunchApplication' After='InstallFinalize'>UPDATING_AUTOMATICALLY = 1</Custom>
</InstallExecuteSequence>
I would assume that you are launching your app from a custom action, which is triggered through a property bound to the checkbox. If that is the case, you can try specifying that property as a command line argument to setup.exe. Say, if your custom action is bound to the MSI property LAUNCH_NEW_VERSION, you can call setup.exe like this:
setup.exe /q LAUNCH_NEW_VERSION=1
The standard setup bootstrapper should pass that property/value to the MSI engine. If it doesn't, you might consider invoking the .msi directly instead of calling the bootstrapper exe to run your installer.
This is the approach I took.
<Property Id="WixShellExecTarget" Value="[#(the id of your exe here)]" />
<CustomAction Id="LaunchApplication" BinaryKey="WixCA" DllEntry="WixShellExec" Impersonate="yes" />
This will execute which ever file id you enter in the Value. The [# ] is needed. I used this and ran it via the UI but you should be able to call this custom action anywhere and it work.

How to register file types/extensions with a WiX installer?

I didn't find an explicit answer to this question in the WiX Documentation (or Google, for that matter). Of course I could just write the appropriate registry keys in HKCR, but it makes me feel dirty and I'd expect this to be a standard task which should have a nice default solution.
For bonus points, I'd like to know how to make it "safe", i.e. don't overwrite existing registrations for the file type and remove the registration on uninstall only if it has been registered during installation and is unchanged.
Unfortunately there's no way to do a "safe" association with Windows Installer.
We just write everything out to the registry and then have a separate component that takes over the system-wide default and is only installed if no other application has already registered itself as the default.
With Vista there's the new "default programs" interface, again you write everything out to the registry. Here's a complete example that we're using in our installer. (WiX 3.0)
Update: 12 months have passed since my original answer and I have a better understanding of file associations. Rather than writing everything manually I'm now using proper ProgId definitions which improves handling for advertised packages. See the updated code posted in response to this question.
<Component ....>
<RegistryValue Root="HKLM" Key="SOFTWARE\AcmeFoobar\Capabilities" Name="ApplicationDescription" Value="ACME Foobar XYZ Editor" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\AcmeFoobar\Capabilities" Name="ApplicationIcon" Value="[APPLICATIONFOLDER]AcmeFoobar.exe,0" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\AcmeFoobar\Capabilities" Name="ApplicationName" Value="ACME Foobar" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\AcmeFoobar\Capabilities\DefaultIcon" Value="[APPLICATIONFOLDER]AcmeFoobar.exe,1" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\AcmeFoobar\Capabilities\FileAssociations" Name=".xyz" Value="AcmeFoobar.Document" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\AcmeFoobar\Capabilities\MIMEAssociations" Name="application/xyz" Value="AcmeFoobar.Document" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\AcmeFoobar\Capabilities\shell\Open\command" Value=""[APPLICATIONFOLDER]AcmeFoobar.exe" "%1"" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\RegisteredApplications" Name="Acme Foobar" Value="SOFTWARE\AcmeFoobar\Capabilities" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Classes\.xyz" Name="Content Type" Value="application/xyz" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Classes\.xyz\AcmeFoobar.Document\ShellNew" Value="" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Classes\.xyz\OpenWithList\AcmeFoobar.exe" Value="" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Classes\.xyz\OpenWithProgids" Name="AcmeFoobar.Document" Value="" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Classes\Applications\AcmeFoobar.exe\SupportedTypes" Name=".xyz" Value="" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Classes\Applications\AcmeFoobar.exe\shell\open" Name="FriendlyAppName" Value="ACME Foobar" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\AcmeFoobar.exe" Value="[!AcmeFoobar.exe]" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\AcmeFoobar.exe" Name="Path" Value="[APPLICATIONFOLDER]" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Classes\SystemFileAssociations\.xyz\shell\edit.AcmeFoobar.exe" Value="Edit with ACME Foobar" Type="string" />
<RegistryValue Root="HKLM" Key="SOFTWARE\Classes\SystemFileAssociations\.xyz\shell\edit.AcmeFoobar.exe\command" Value=""[APPLICATIONFOLDER]AcmeFoobar.exe" "%1"" Type="string" />
</Component>
<Component ....>
<ProgId Id="AcmeFoobar.Document" Description="ACME XYZ Document">
<Extension Id="pdf" ContentType="application/xyz">
<Verb Id="open" Command="Open" TargetFile="[APPLICATIONFOLDER]AcmeFoobar.exe" Argument="%1" />
</Extension>
</ProgId>
<Condition><![CDATA[DEFAULTVIEWER=1]]></Condition>
</Component>
After some additional research, I found a partial answer to this question in the WiX Tutorial. It shows an advertised solution and does not work with WiX 3.0, but given that information, I figured it out. Add a ProgId element to the component containing your executable, like the following:
<ProgId Id="MyApplication.MyFile" Description="My file type">
<Extension Id="myext" ContentType="application/whatever">
<Verb Id="open" Command="open" TargetFile="MyApplication.exe" Argument=""%1""/>
</Extension>
</ProgId>
myext is the file extension without the dot, and MyApplication.exe is the file id (not name) of the executable file (i.e. the Id attribute of the File element).
This will register the file type with your executable and will supply a default icon (a white page with the application icon on it), which is sufficient for my needs. If you want to specify a dedicated icon, it seems you still have to do this yourself, like the following (code from the linked tutorial):
<Registry Id='FooIcon1' Root='HKCR' Key='.xyz' Action='write' Type='string' Value='AcmeFoobar.xyzfile' />
<Registry Id='FooIcon2' Root='HKCR' Key='AcmeFoobar.xyzfile' Action='write' Type='string' Value='Acme Foobar data file' />
<Registry Id='FooIcon3' Root='HKCR' Key='AcmeFoobar.xyzfile\DefaultIcon' Action='write' Type='string' Value='[INSTALLDIR]Foobar.exe,1' />
I didn't find a good solution for my bonus question though.
Edit: I started writing this before the previous answer came. However, my solution actually works, in contrast to the previous answer.
"If your application handles its own file data type, you will need to register a file association for it. Put a ProgId inside your component. FileId should refer to the Id attribute of the File element describing the file meant to handle the files of this extension. Note the exclamation mark: it will return the short path of the file instead of the long one:"
<ProgId Id='AcmeFoobar.xyzfile' Description='Acme Foobar data file'>
<Extension Id='xyz' ContentType='application/xyz'>
<Verb Id='open' Sequence='10' Command='Open' Target='[!FileId]' Argument='"%1"' />
</Extension>
</ProgId>
Reference: https://www.firegiant.com/wix/tutorial/getting-started/beyond-files/

Resources