Xamarin UWP MediaPlayer streaming audio from URL starts in middle of file, or doesn't start at all - xamarin

Question

I'm writing a music player app to stream music from Onedrive in Xamarin Forms, which compiles to both Android and UWP. To play a audio file, I pass the URL of the audio file to the native function "Play" which is called via an interface in both the Android and UWP code, which streams the audio file. The Android implementation works perfectly, so I know there is nothing wrong the the audio files or URLs. But for the UWP implementation, larger files sometimes start in the middle of the file, or in one case don't play at all. Again, these all work in Android. Some files work fine though. I'm using MP3 and Flac files, I think I've only had problems with Flac, I assume because they are larger. No errors seem to be thrown as far as I can tell.
The goal here is a to start the audio stream as fast as possible from the URL.
I've tried the most common implementation to stream audio to UWP MediaPlayer that I've found, which is in the code below, as well as using a BackgroundDownloader as a source for MediaPlayer. I've also tried downloading the file beforehand, which takes forever and I'd like to avoid.
[assembly: Dependency(typeof(CloudPlayer.UWP.PlayMusic_UWP))]
namespace CloudPlayer.UWP
{
public class PlayMusic_UWP : PlayMusic
{
MediaPlayer mediaPlayer { get; set; }
public PlayMusic_UWP()
{
mediaPlayer = new MediaPlayer();
}
public void Play(string filePath)
{
mediaPlayer.Source = MediaSource.CreateFromUri(new Uri(filePath));
mediaPlayer.Play();
}
}
}