Powershell - Sequential foreach loop zipping files - windows

Question

I have a number of IIS servers, each with a number of sites on them and I want to zip all IIS logs regularly.
I cobbled together the following powershell script with the help of this site and google:
$files = Get-ChildItem "D:\logfiles\IIS-Logs\*.log" -Recurse | Where-Object {($_.LastWriteTime -lt (Get-Date).AddDays(-7))}
foreach ($file in $files) {& 'C:\Program Files\WinRAR\winrar.exe' a -tl -df -m5 "$file.rar" $File}
The problem with this script is that if there are.. say... 2,000 total log files it tried to launch 2,000 simultaneous copies of Winrar and the server will crash. This was unexpected. I expected it to zip the files one at a time. Sequentially.
Does anyone have any ideas to make this work like I want?
I'd really like to use Winrar vs the native Compress-Archive option because:
I want the file dates to reflect the zipped file, not the date it was zipped.
I want the utility to delete the files after archiving because the utility will not delete the file if the archiving failed.
I'm not married to Winrar if I can achieve this another way.

Answer 1

For testing purpose, i created a LogFolder with 3 separate log folders, Log1, Log2 and Log3. Each of these Log folders have 2000 files with 2MB data each. This is the command i ran to Compress each folder `seperately
You can also run these in serial if the performance is too slow (when reading from same disk and writing to same as well).
$ElementsInLog = (Get-ChildItem C:\temp\LogFolder\*.txt -Recurse).Length
$ElementsInLog1 = (Get-ChildItem C:\temp\LogFolder\Log1\*.txt -Recurse).Length
$ElementsInLog2 = (Get-ChildItem C:\temp\LogFolder\Log2\*.txt -Recurse).Length
$ElementsInLog3 = (Get-ChildItem C:\temp\LogFolder\Log3\*.txt -Recurse).Length
Write-Output "Main: $ElementsInLog`nLog1: $ElementsInLog1`nLog2: $elementsInLog2`nLog3: $elementsInLog3"
Write-output "Total File Size: $((Get-ChildItem C:\temp\LogFolder\Log1\*.txt -Recurse | Measure-Object length -Sum).Sum / 1024 / 1024 / 1024) GB"
Write-Output "Starting Tasks..."
$job1 = Start-Job -ScriptBlock {
Write-Output "Log1: $(Get-Date -Format G)"
Get-ChildItem C:\temp\LogFolder\Log1 -Recurse | Compress-Archive -DestinationPath C:\temp\log1.zip -CompressionLevel Fastest
Write-Output "Finished: $(Get-Date -Format G)"
}
$job2 = Start-Job -ScriptBlock {
Write-Output "Log2: $(Get-Date -Format G)"
Get-ChildItem C:\temp\LogFolder\Log3 -Recurse | Compress-Archive -DestinationPath C:\temp\log3.zip -CompressionLevel Fastest
Write-Output "Finished: $(Get-Date -Format G)"
}
$job3 = Start-Job -ScriptBlock {
Write-Output "Log2: $(Get-Date -Format G)"
Get-ChildItem C:\temp\LogFolder\Log4 | Compress-Archive -DestinationPath C:\temp\log4.zip -CompressionLevel Fastest
Write-Output "Finished: $(Get-Date -Format G)"
}
while($job1.State -eq "Running" -or $job2.State -eq "Running" -or $job3.State -eq "Running") {
Start-Sleep 5
}
Receive-Job $job1
Receive-Job $job2
Receive-Job $job3
Output Received
Main: 8000
Log1: 2000
Log2: 2000
Log3: 2000
Total File Size: 4.12791967391968 GB
Starting Tasks...
Log1: 2/10/2020 8:36:22 PM
Finished: 2/10/2020 8:37:30 PM
Log2: 2/10/2020 8:36:22 PM
Finished: 2/10/2020 8:37:27 PM
Log3: 2/10/2020 8:36:22 PM
Finished: 2/10/2020 8:37:28 PM

Answer 2

You probably want to use "Start-Process -Wait" instead of using the &. Using the -Wait flag on Start-Process forces it to wait for completion, and will then cause your program to run sequentially. Check this article on how to use Start-Process: A Better PowerShell Start Process You might also want to use the cmdlet Compress-Archive instead of the command line program Winrar, which might be better integrated and give you better feedback in your scripting.
Something like this with WinRAR?
$files = Get-ChildItem "D:\logfiles\IIS-Logs\*.log" -Recurse | Where-Object {($_.LastWriteTime -lt (Get-Date).AddDays(-7))}
foreach ($file in $files) {
$args = 'a -tl -df -m5 "' + $file.rar + '" ' + $File
Start-Process -Wait -filepath 'C:\Program Files\WinRAR\winrar.exe' -ArgumentList $args
}
Or this with Compress-Archive
$files = Get-ChildItem "D:\logfiles\IIS-Logs\*.log" -Recurse | Where-Object {($_.LastWriteTime -lt (Get-Date).AddDays(-7))}
foreach ($file in $files) {
Compress-Archive -Path $file.FullName -DestinationPath "$($File.FullName).rar"
{
The above are untested, but should work unless I made a typo.