Parallelise bubble sort using CUDA - algorithm

Question

I was given an assignment, to parallelize Bubble Sort and implement it using CUDA.
I don't see how bubble sort could possibly be parallelized. I think its inherently sequential. Since, it compares two consecutive elements and swaps them after a conditional branch.
Thoughts, anyone?

Answer 1

To be completely honest, I had trouble thinking about a way to parallelize bubble sort as well. I initially thought of a hybrid sort where you could tile, bubble sort each tile, and then merge (probably would still improve performance if you could make it work). However, I browsed for "Parallel Bubble Sort", and found this page. If you scroll down you'll find the following parallel bubble sort algorithm:
For k = 0 to n-2
If k is even then
for i = 0 to (n/2)-1 do in parallel
If A[2i] > A[2i+1] then
Exchange A[2i] ↔ A[2i+1]
Else
for i = 0 to (n/2)-2 do in parallel
If A[2i+1] > A[2i+2] then
Exchange A[2i+1] ↔ A[2i+2]
Next k
You could run the for-loop in the CPU and then use a kernel for each of the do in parallels. This seems efficient for large arrays, but might be too much overhead with small arrays. Large arrays are assumed if you're writing a CUDA implementation. Since the swaps within these kernels are with adjacent pairs of elements, you should be able to tile accordingly. I've searched for generic, non-gpu-specific parallel bubble sorts and this was the only one I could find.
I did find a (very slightly) helpful visualization here, which can be seen below. I'd love to discuss this more in the comments.
EDIT: I found another parallel version of bubble sort called Cocktail Shaker Sort. Here's the pseudocode:
procedure cocktailShakerSort( A : list of sortable items ) defined as:
do
swapped := false
for each i in 0 to length( A ) - 2 do:
if A[ i ] > A[ i + 1 ] then // test whether the two elements are in the wrong order
swap( A[ i ], A[ i + 1 ] ) // let the two elements change places
swapped := true
end if
end for
if not swapped then
// we can exit the outer loop here if no swaps occurred.
break do-while loop
end if
swapped := false
for each i in length( A ) - 2 to 0 do:
if A[ i ] > A[ i + 1 ] then
swap( A[ i ], A[ i + 1 ] )
swapped := true
end if
end for
while swapped // if no elements have been swapped, then the list is sorted
end procedure
It looks like this also has two for-loops comparing adjacent elements bubbly.. These algorithms look kind of like similar opposites, since the first algorithm (which I now learned is called odd-even sort) assumes sorted and lets the for-loops specify false, while cocktail shaker sort conditionally checks sorted in each loop.
The code included in this post for the odd-even sort seems to just run the while loop enough times to guarantee sorted, where the wikipedia pseudocode checks. A potential first-pass could be to implement the algorithm of this post and then optimize with the check, although the check may actually be slower with CUDA.
Regardless the sort will be slow. Here's a related SO question fyi, but there isn't much help. They agree it's not effective for small arrays, and really emphasize its failure.
Are you looking for specific CUDA code or was this enough? It seems like you wanted an overview of possible options and understand CUDA implementation.

Answer 2

TL;DR;
For a complete implementation of a Generic Parallel Bubble Sort, take a look at generic-bubble-sort.cu. *Generic" in here means the algorithm sorts any kind of elements as long as you provide your comparator.
At best
With a number linearly proportional to N threads (say N/2) you can get a Parallel Bubble Sort of O(N) time complexity (where N is the size of the array you want to sort).
A hint
It might not be trivial but when you look closely, you'd realize that all what the Sequential Bubble Sort does is swapping pairs of elements if not ordered correctly One pair at the time!.
Since pairs can be sorted independently, a Parallel Bubble Sort could take advantage of the property that ordering pairs is independent.
An approach
Let's say we want to sort the following array ascendantly :
# [7][1][3][2][0]
We'd first take our initial unsorted array and consider every two elements array[i] and array [i+1] to be an indepent pair. For this first iteration i is going to be an EVEN index, so our paris are { {array[0], array[1]} , {array[2],array[3]}, ...}.
# [7][1][3][2][0] <-- Unsorted array of 5 elements
# [7][1] [3][2] [0] <-- A set of independent pairs.
Then, we'd swap every two elements of each pair if they are not in the desired order.
# [7][1] [3][2] [0] --┑ Sorting first set of pairs
# |
# [1][7] [2][3] [0] <-┛ starting from an even idx
Our array after this first iteration would look like this :
# [1][7][2][3][0] <-- Result after first iteration
We are now going to reiterate for a second time, but unlike previously, we will now sort pairs starting from an ODD index { {array[1], array[2]} , {array[3],array[4]}, ...}. It's worth mentioning that elements with no peers won't be considered.
# [1][7][2][3][0] <-- Result after first iteration
# [1] [7][2] [3][0] --┑ Sorting second set of pairs
|
# [1] [2][7] [0][3] <-┛ starting from an odd index
# [1][2][7][0][3] <-- Result after second iteration
After N EVEN/ODD pair sorting iterations we'd have a sorted array.
# [1][2] [7][0] [3] --┑
# [1][2] [0][7] [3] |
# |
# [1][2][0][7][3] |
# | The whole parallel sorting
# [1] [2][0] [7][3] | will converge after N iterations
# [1] [0][2] [3][7] | So we keep sorting pairs for 3 more
# | iterations.
# [1][0][2][3][7] |
# |
# [1][0] [2][3] [7] |
# [0][1] [2][3] [7] <-┛
#
# [0][1][2][3][7] <-- Sorted array!
Parallel Bubble Sort with CUDA
A straightforward implementation of a CUDA program for the approach above would be done as follows:
each thread would be responsible for sorting an individual pair
you would need N/2 threads
since warp divergence is a thing we'd need to care about synchronizing our threads
USING A SINGLE BLOCK: if our threads fit into a single block we'd only use __synchronize() after each iteration and we'd be able to take advantage of the shared memory by having all our array there.
USING MORE THAN ONE BLOCK: we would have to ensure thread synchronization for all the threads in our kernel. We'd only be able to perform one iteration per kernel launch, and launch our kernel N times. The bad news is that we can only use the global memory for processing our array since the shared memory has just kernel lifetime.
Some code
Here's a simple implementation of what's explained above considering only one block. The whole code is available in this repo.
template<typename T>
__global__
void bubbleSort(T* v, const unsigned int n, ShouldSwap<T> shouldSwap) {
const unsigned int tIdx = threadIdx.x;
for (unsigned int i = 0; i < n; i++) {
unsigned int offset = i % 2;
unsigned int leftIndex = 2 * tIdx + offset;
unsigned int rightIndex = leftIndex + 1;
if (rightIndex < n) {
if (shouldSwap(v[leftIndex ], v[rightIndex ])) {
swap<T>(&v[leftIndex ], &v[rightIndex ]);
}
}
__syncthreads();
}
}
If you're wondering about ShouldSwap and swap implementations here's the code:
swap
A device function for swapping elements.
template<typename T>
__host__ __device__ __inline__
void swap (T* a, T* b) {
T tmp = *a;
*a = *b;
*b = tmp;
}
ShouldSwap
A C++ Functor used as a generic comparator.
template<typename T>
__host__ __device__
bool ShouldSwap<T>::operator() (const T left, const T right) const {
return left > right;
}