MSIX randomly not installing Windows service - windows

Question

I have a Windows desktop application build using Maui and installed using an MSIX installer. A key part of the desktop application is a Windows service which should be installed by the MSIX installer.
I've found that randomly however it does not install the service. The installer also does not throw any error causing end users to open up the app only for it to crash (because of a missing service).
What I've found:
On a system where the app has not been installed before the service gets installed without problems
At any point when updating the app with a newer version or simply reinstalling the app the service might be missing afterwards
Once you get in a state where the service is missing no amount of reboots or reinstalls can fix the problem with one exception: installing a completely new version can fix the problem (for example: upgrading from 1.6.0 to 1.7.0 can fix the problem, same for downgrading).
Windows event viewer does not report any errors or problems as far as I've found (checked the AppXDeployment-Server logs)
The installer does prompt for elevated rights
I'm sure I probably made a mistake in my appxmanifest but I don't see it. Any help would be appreciated.
Our appxmanifest looks like this
<?xml version="1.0" encoding="utf-8"?>
<Package
xmlns="http://schemas.microsoft.com/appx/manifest/foundation/windows10"
xmlns:uap="http://schemas.microsoft.com/appx/manifest/uap/windows10"
xmlns:rescap="http://schemas.microsoft.com/appx/manifest/foundation/windows10/restrictedcapabilities"
xmlns:desktop="http://schemas.microsoft.com/appx/manifest/desktop/windows10"
xmlns:desktop6="http://schemas.microsoft.com/appx/manifest/desktop/windows10/6"
IgnorableNamespaces="uap rescap">
<Identity Name="NAME" Publisher="PUBLISHER" Version="0.0.0.0" />
<Properties>
<DisplayName>NAME</DisplayName>
<PublisherDisplayName>NAME</PublisherDisplayName>
<Logo>$placeholder$.png</Logo>
</Properties>
<Dependencies>
<TargetDeviceFamily Name="Windows.Universal" MinVersion="10.0.17763.0" MaxVersionTested="10.0.19041.0" />
<TargetDeviceFamily Name="Windows.Desktop" MinVersion="10.0.17763.0" MaxVersionTested="10.0.19041.0" />
<PackageDependency Name="Microsoft.VCLibs.140.00.UWPDesktop" MinVersion="14.0.24217.0" Publisher="CN=Microsoft Corporation, O=Microsoft Corporation, L=Redmond, S=Washington, C=US" />
</Dependencies>
<Resources>
<Resource Language="x-generate" />
</Resources>
<Applications>
<Application Id="App" Executable="$targetnametoken$.exe" EntryPoint="$targetentrypoint$">
<uap:VisualElements
DisplayName="$placeholder$"
Description="$placeholder$"
Square150x150Logo="$placeholder$.png"
Square44x44Logo="$placeholder$.png"
BackgroundColor="transparent">
<uap:DefaultTile Square71x71Logo="$placeholder$.png" Wide310x150Logo="$placeholder$.png" Square310x310Logo="$placeholder$.png" />
<uap:SplashScreen Image="$placeholder$.png" />
</uap:VisualElements>
<Extensions>
<desktop:Extension Category="windows.startupTask" Executable="NAME.exe" EntryPoint="MauiProgram.CreateMauiApp">
<desktop:StartupTask TaskId="NAME" Enabled="true" DisplayName="NAME" />
</desktop:Extension>
<desktop6:Extension
Category="windows.service"
Executable="SERVICE_EXECUTABLE.exe"
EntryPoint="SERVICE_NAMESPACE.Program">
<desktop6:Service
Name="SERVICE_NAME"
StartupType="auto"
StartAccount="localSystem"
Arguments="/manage"
>
</desktop6:Service>
</desktop6:Extension>
</Extensions>
</Application>
</Applications>
<Capabilities>
<rescap:Capability Name="runFullTrust" />
<rescap:Capability Name="allowElevation" />
<rescap:Capability Name="packagedServices" />
<rescap:Capability Name="packageManagement" />
<rescap:Capability Name="localSystemServices" />
</Capabilities>
</Package>