Shuffling pandas DataFrame rows if the values "overlap" between two columns - sorting

Question

I have the following pandas dataframe
import numpy as np
import pandas as pd
df = pd.DataFrame({"first_element":[20, 125, 156, 211, 227, 220, 230, 472, 4765], "second_element":[35, 145, 178, 233, 321, 234, 231, 498, 8971], "next":[0.32, 0.04, 0.59, 0.103, 0.37, 0.92, 0.81, 0.24, 0.77]})
df = df[["first_element", "second_element", "next"]]
print(df)
### print(df) outputs:
first_element second_element next
0 20 35 0.320
1 125 145 0.040
2 156 178 0.590
3 211 233 0.103
4 227 321 0.370
5 220 234 0.920
6 230 231 0.810
7 472 498 0.240
8 4765 8971 0.770
In this DataFrame, each row is considered an "interval" along a real line, [first_element, second_element], e.g. 20 to 35, 125 to 145.
If I wished to sort df based on both columns, I would use .sort_values(), i.e.
sorted_df = df.sort_values(["first_element", "second_element"], ascending=[True, False])
which outputs
print(sorted_df)
first_element second_element next
0 20 35 0.320
1 125 145 0.040
2 156 178 0.590
3 211 233 0.103
5 220 234 0.920
4 227 321 0.370
6 230 231 0.810
7 472 498 0.240
8 4765 8971 0.770
There are several intervals which intersect/overlap, namely [211, 233], [220, 234], [227, 321], [230, 231]. Because [230, 231] is a subset of [211, 233], there are several ways to order these two.
My goal is (1) write a function that finds all overlapping "intervals" (the values in the two columns first_element and second_element) and (2) randomly shuffle these intervals.
The goal (2) sounds very tricky, because one would need to separately shuffle/re-order multiple "groups" of overlapping intervals. For example, let's say our dataframe was larger, and had the following overlapping intervals:
[211, 233], [220, 234], [227, 321], [230, 231], [5550, 5879], [5400, 5454]
I would want to separately re-shuffle [211, 233], [220, 234], [227, 321], [230, 231] and [5550, 5879], [5400, 5454], not mix up the subsets of overlapping intervals.
There are several ways to shuffle rows with pandas, e.g. shuffle by the index
def shuffle_by_index(df):
index = list(df.index)
random.shuffle(index)
df = df.ix[index]
df.reset_index()
return df
or use sklearn
import sklearn.utils
shuffled = sklearn.utils.shuffle(df)
df = df.reset_index(drop=True)
but (1) how does one search for all overlapping intervals in a pythonic/pandas way and (2) how do I select these subsets of overlapping intervals and only shuffle those individually?

Answer 1

This is not the best way to solve it, but it gives your desired results. I have left second part for you.
import numpy as np
import pandas as pd
df = pd.DataFrame({"first_element":[20, 125, 156, 211, 227, 220, 230, 472, 4765], "second_element":[35, 145, 178, 233, 321, 234, 231, 498, 8971], "next":[0.32, 0.04, 0.59, 0.103, 0.37, 0.92, 0.81, 0.24, 0.77]})
df = df[["first_element", "second_element", "next"]]
sorted_df = df.sort_values(["first_element", "second_element"], ascending=[True, False])
sorted_df.reset_index(0, inplace = True)
prev_min = sorted_df.first_element.iloc[0]
prev_max = sorted_df.second_element.iloc[0]
labels = []
label_counter = 1
labels.append(label_counter)
for rowIndex in xrange(1, sorted_df.shape[0]):
row = sorted_df.iloc[rowIndex]
if row.first_element > prev_max:
# totally different interval, may be overlapping interval
prev_min = row.first_element
prev_max = row.second_element
label_counter += 1
labels.append(label_counter)
elif row.first_element >= prev_min:
prev_max = max(prev_max, row.second_element)
labels.append(label_counter)
sorted_df['overlapping_index'] = labels
# group sorted_df by overlapping index, and randomly select the save interval group