powershell sorting really large collection of objects - sorting

Question

I am trying to generate histograms from a very large collection of objects (-gt 250k). I need to sort the collection on a property of each object. My line of script looks like:
$ch = $ch | sort TotalCount -descending <br>
where $ch[x].totalcount would be some integer.
The script works but it takes over an hour to sort and consumes 6GB of memory. How do I speed up the process?
I've done some searching for a solution and several web sites suggest using [array]::sort as it is much quicker. As this is a collection of objects, I'm not sure how I would use a static System.Array sort method. Even if I could, I don't see how to make the array descending (although reversing the result should be pretty simple).
Any suggestions on how to sort really large collections with powershell?

Answer 1

Let's create an array with 2500 elements. Each element of the array is an object containing the property totalCount and we assign an integer to it.
$array = #()
1..2500 | % {
$array += New-Object pscustomobject -Property #{
totalCount = $_;
}
}
Now let's sort this array and measure the total time for executing the command.
We start with the classic Sort-Object using the -descending parameter:
(Measure-Command {
$array = $array | Sort-Object TotalCount -descending
}).TotalSeconds
Total time in seconds is : 0.1217965
Let's now use the method Reverse of the class System.Array : [Array]::Reverse()
(Measure-Command {
[Array]::Reverse([array]$array)
}).TotalSeconds
Total time in seconds is : 0.0002594
Quite a difference!
Let's now see other possibilities, lets create a System.Collections.ArrayList
$array = New-Object System.Collections.ArrayList
1..2500 | % {
$o = New-Object pscustomobject -Property #{
totalCount = $_;
}
[Void] $array.Add($o)
}
And we rince and repeat. We first use the Reverse method of the class System.Collections.ArrayList, then we pass the collection to the Reverse method of System.Array.
(Measure-Command {
$array.reverse()
}).TotalSeconds
Total time in seconds is : 0.0002459
Slight improvement, but pretty similar overall.
Now we typecast the system collection and use [Array]::Reverse()
(Measure-Command {
[Array]::Reverse([array]$array)
}).TotalSeconds
Total time in seconds is : 0.0008172
Over twice as much time. This clearly shows it wasn't a good idea, so we scrap it.
Conclusion:
A System.Array with [Array]::Reverse() is definitely faster than Sort-Object, however keep in mind that System.Array is immutable, so if building the array is part of the performance issue, I'd definitely recommend using System.Collections.ArrayList since it's mutable.

Answer 2

[array]::reverse() is NOT sorting the array in any way.