Poor HttpWebRequest performance when downloading a file - performance

Question

After reading many threads, I don't seem to find an answer to my issue - so here goes.
Using HttpWebRequest to simply download a file, and its performance is very slow, and seems capped around 1.5 - 2 Mbps.
WebClient.DownloadFile on the other hand performs nicely, and so does download via a browser.
I am trying to wrap my head around whats going on, and what I'm missing. I am writing the code in Powershell, so there might be something there.
I want to use HttpWebRequest in order to maintain control through the download, and keep track of progress, since I'm going to handle some very big files 30-70+ GB
Very simply put - I have the following code snippet - for simplicity I write to a Memory stream, uses a commonly available 111MB Nvidia download, and have included some performance measurement logic:
$Buffer = New-Object -TypeName "Byte[]" -ArgumentList 65536
$FilePath = "e:\temp\test.exe"
$Request = [System.Net.HttpWebRequest]::Create("https://download.nvidia.com/gfnpc/GeForceNOW-release.exe")
$Request.set_Timeout(15000)
$Response = $Request.GetResponse()
$TotalLength = $Response.get_ContentLength()
$TotalLengthKB = [System.Math]::Floor($Response.get_ContentLength()/1024)
$ResponseStream = $Response.GetResponseStream()
$TargetStream = New-Object System.IO.MemoryStream
$TargetStream.SetLength($TotalLength)
$DownloadBytes = 0
$DownloadKB = 0
$ObservedCounts = #{}
$StopWatch = New-Object System.Diagnostics.Stopwatch
$StopWatch.Start()
Try {
Do {
# Fill download buffer for this request
$Count = $ResponseStream.Read($Buffer, 0, $Buffer.Length)
If ($ObservedCounts.ContainsKey($Count)) {
$ObservedCounts[$Count]++
} Else {
$ObservedCounts[$Count] = [Int32]1
}
# Update downloaded bytes and progress for this request
$DownloadedBytes = $DownloadedBytes + $Count
$DownloadedKB = [System.Math]::Floor($DownloadedBytes/1024)
$Progress = [System.Math]::Floor($DownloadedKB / $TotalLengthKB * 100)
Write-Progress -Activity "Downloading" -Id 1 -PercentComplete $Progress -Status "Downloaded $($DownloadedKB) KB of $($TotalLengthKB) KB"
# Write data to the file for this request.
$TargetStream.Write($Buffer, 0, $Count)
} Until ($Count -eq 0 -or $StopWatch.Elapsed.Seconds -ge 15)
Write-Host "Done"
} Catch {
# There was an error during processing of this request
Throw $_
} Finally {
if ($TargetStream.CanWrite) {
$TargetStream.Flush()
$TargetStream.Close()
}
$TargetStream.Dispose()
$ResponseStream.Dispose()
$StopWatch.Stop()
Write-Host ('Stopped after {0}' -f $StopWatch.Elapsed.ToString())
Write-Host ('Downloaded: {0} KB' -f $DownloadedKB)
Write-Host "Observed Counts:"
#($ObservedCounts)|Sort-Object Value -Descending|Format-Table|Out-String -Stream
}
After 15 seconds the download stops, and displays the stats:
PS E:\temp> .\test.ps1
Done
Stopped after 00:00:15.0142467
Downloaded: 17407 KB
Observed Counts:
Name Value
---- -----
16383 1088
16 67
3 2
PS E:\temp>
Same file with WebClient.DownloadFile():
PS E:\temp> Measure-Command {$WebClient.DownloadFile("https://download.nvidia.com/gfnpc/GeForceNOW-release.exe", "e:\temp\blah.exe")}
Days : 0
Hours : 0
Minutes : 0
Seconds : 7
Where a simple WebClient.DownloadFile() Completes the 111MB download in about 7 seconds, in 15 seconds with HttpWebRequest I barely managed 17MB.
I am suspecting that the .Read from the ResponseStream seems to be ignoring the buffer size, as the most of the time it returns 16383 bytes - and it really doesn't care what size I set the buffer to, unless I lower it to less than 16383 bytes.
I tried putting a BufferedStream in between the Read/Write streams, but doesn't really change anything.
I do quite a bit of Powershell scripting, and often use .NET classes, but are by no means skilled into C# or other .NET "real" languages, so I may very well be doing something newbish I simply cant see, so hopefully a friendly soul can help me discover the errors of my ways.
It must clearly be something in my code, as WebClient uses the same classes under the hood, and works fine enough.
Thanks in advance.

Answer 1

I would most definitely use BITS for these applications.
Is reliable and you can control the throttle.
Give a sneak peak on:
https://learn.microsoft.com/en-us/windows/win32/bits/using-windows-powershell-to-create-bits-transfer-jobs