How to groupby with custom function in python cuDF? - rapids

Question

I am new to using GPU for data manipulations, and have been struggling to replicate some of the functions in cuDF. For instance, I want to get a mode value for each group in the dataset. In Pandas it is easily done with custom functions:
df = pd.DataFrame({'group': [1, 2, 2, 1, 3, 1, 2],
'value': [10, 10, 30, 20, 20, 10, 30]}
| group | value |
| ----- | ----- |
| 1 | 10 |
| 2 | 10 |
| 2 | 30 |
| 1 | 20 |
| 3 | 20 |
| 1 | 10 |
| 2 | 30 |
def get_mode(customer):
freq = {}
for category in customer:
freq[category] = freq.get(category, 0) + 1
key = max(freq, key=freq.get)
return [key, freq[key]]
df.groupby('group').agg(get_mode)
| group | value |
| ----- | ----- |
| 1 | 10 |
| 2 | 30 |
| 3 | 20 |
However, I just can't seem to be able to replicate the same functionality in cuDF. Even though there seems to be a way to do it, of which I have found some examples, but it somehow doesn't work in my case. For example, the following is the function I tried to use for cuDF:
def get_mode(group, mode):
print(group)
freq = {}
for i in range(cuda.threadIdx.x, len(group), cuda.blockDim.x):
category = group[i]
freq[category] = freq.get(category, 0) + 1
mode = max(freq, key=freq.get)
max_freq = freq[mode]
df.groupby('group').apply_grouped(get_mode, incols=['group'],
outcols=dict((mode=np.float64))
Can someone please help me understand what is going wrong here, and how to fix it? Attempting to run the code above throws the following error (hopefully I managed to put it under the spoiler):
Error code
TypingError: Failed in cuda mode pipeline (step: nopython frontend)
Failed in cuda mode pipeline (step: nopython frontend)
- Resolution failure for literal arguments:
No implementation of function Function(<function impl_get at 0x7fa8f0500710>) found for signature:
>>> impl_get(DictType[undefined,undefined]<iv={}>, int64, Literal[int](0))
There are 2 candidate implementations:
- Of which 1 did not match due to:
Overload in function 'impl_get': File: numba/typed/dictobject.py: Line 710.
With argument(s): '(DictType[undefined,undefined]<iv=None>, int64, int64)':
Rejected as the implementation raised a specific error:
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type DictType[undefined,undefined]<iv=None>
During: typing of argument at /opt/conda/lib/python3.7/site-packages/numba/typed/dictobject.py (719)
File "../../opt/conda/lib/python3.7/site-packages/numba/typed/dictobject.py", line 719:
def impl(dct, key, default=None):
castedkey = _cast(key, keyty)
^
raised from /opt/conda/lib/python3.7/site-packages/numba/core/typeinfer.py:1086
- Of which 1 did not match due to:
Overload in function 'impl_get': File: numba/typed/dictobject.py: Line 710.
With argument(s): '(DictType[undefined,undefined]<iv={}>, int64, Literal[int](0))':
Rejected as the implementation raised a specific error:
TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type DictType[undefined,undefined]<iv={}>
During: typing of argument at /opt/conda/lib/python3.7/site-packages/numba/typed/dictobject.py (719)
File "../../opt/conda/lib/python3.7/site-packages/numba/typed/dictobject.py", line 719:
def impl(dct, key, default=None):
castedkey = _cast(key, keyty)
During: resolving callee type: BoundFunction((<class 'numba.core.types.containers.DictType'>, 'get') for DictType[undefined,undefined]<iv={}>)
During: typing of call at /tmp/ipykernel_33/2595976848.py (6)
File "../../tmp/ipykernel_33/2595976848.py", line 6:
<source missing, REPL/exec in use?>
During: resolving callee type: type(<numba.cuda.compiler.Dispatcher object at 0x7fa8afe49520>)
During: typing of call at <string> (10)
File "<string>", line 10:
<source missing, REPL/exec in use?>

Answer 1

cuDF builds on top of Numba's CUDA target to enable UDFs. This doesn't support using a dictionary in the UDF, but you your use case can expressed with built-in operations with pandas or cuDF by combining value_counts and drop_duplicates.
import pandas as pd
​
df = pd.DataFrame(
{
'group': [1, 2, 2, 1, 3, 1, 2],
'value': [10, 10, 30, 20, 20, 10, 30]
}
)
​
out = (
df
.value_counts()
.reset_index(name="count")
.sort_values(["group", "count"], ascending=False)
.drop_duplicates(subset="group", keep="first")
)
print(out[["group", "value"]])
group value
4 3 20
1 2 30
0 1 10

Answer 2

This is probably not the answer you are looking for, but I found a workaround for mode. It isn't the best way, doesn't use GPU, and can be quite slow.
import pandas as pd
import cudf
df = cudf.DataFrame({'group': [1, 2, 2, 1, 3, 1, 2],
'value': [10, 10, 30, 20, 20, 10, 30]}
df.to_pandas().groupby('group').agg({'value':pd.Series.mode})