How to write a UDF in Tajo [closed] - etl

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am wondering if I can write a UDF in Tajo especially in Python. My use case is for ETL where I want to group log records by some ID (browser ID), then sort the records in the same group by timestamp and then finally use my UDF to go over the sorted records in each group.

Answer 1

I'm a Tajo PMC member. Thank you for your interest in Tajo.
Currently, Tajo does not support Python UDF yet. But, Tajo was designed to have multiple UDF function implementations for each single function signature. Python function UDF feature can be added to Tajo easily.
Now, you should use Java-based one for your own custom functions. There are two ways to implement your custom UDFs. Of course, both ways are very easy.
The first way is to use a legacy Tajo UDF interface. In this manner, each function implementation should be a class which inherits GeneralFunction class. eval() is a body of each UDF implementation. An example is as follow:
#Description(
functionName = "pow",
description = "x raised to the power of y",
example = "> SELECT pow(9.0, 3.0)\n"
+ "729",
returnType = FLOAT8,
paramTypes = {
#ParamTypes(paramTypes = {FLOAT8, FLOAT8})
}
)
public class Pow extends GeneralFunction {
public Pow() {
super(new Column[] {
new Column("x", FLOAT8),
new Column("y", FLOAT8)
});
}
#Override
public Datum eval(Tuple params) {
Datum value1Datum = params.get(0);
Datum value2Datum = params.get(1);
if(value1Datum instanceof NullDatum || value2Datum instanceof NullDatum) {
return NullDatum.get();
}
return DatumFactory.createFloat8(Math.pow(value1Datum.asFloat8(), value2Datum.asFloat8()));
}
}
Another way is to use new function interface. In this manner, you need to just implement a static method for each UDF. Each static method for a function can have one or more Java primitive type or Java object type parameters.
According to whether each parameter is primitive or object type, each parameter has an implicitly different effect. A primitive type argument does not allow NULL value. In this case, if the parameter takes NULL value, this function will automatically return NULL value without actually invoking the function. With SQL's three-value logic, NULL handling like this is very common in SQL function. So, basically, Tajo provides this feature.
An example of a function with primitive parameters:
#ScalarFunction(name = "pow", returnType = FLOAT8, paramTypes = {FLOAT8, FLOAT8})
public static double pow(double x, double y) {
return Math.pow(x, y);
}
In contrast, Object type parameter allows NULL value. In this case, each function must explicitly handle NULL value as follow:
#ScalarFunction(name = "pow", returnType = FLOAT8, paramTypes = {FLOAT8, FLOAT8})
public static Double pow(Double x, Double y) {
if (x == null || y == null) {
return null;
}
return Math.pow(x, y);
}
In order to add your own UDFs to a Tajo cluster, you need to make a jar file, including your own function classes. Currently, Tajo does not allow to users to add custom jars in runtime. So, you should copy the custom jars to ${TAJO_HOME}/lib and then restart Tajo cluster.
Also, Tajo will support CREATE FUNCTION feature to allow users to add functions in runtime soon.