I am encountering a problem when trying to aggregate a KGroupedStream< String, TsdbObject >
where TsdbObject is a POJO that has a method Double getValue(). The following statements
show the groupBy and attempted aggregation:
KGroupedStream< String, TsdbObject > assets_grouped_by_parents =
kstream.groupBy( group_by_parent_mapper, Serialized.with( Serdes.String(), tsdb_object_serde ) );
KTable< String, Double > sums_of_groups_by_parents =
assets_grouped_by_parents.aggregate( new SummerInitializer(), new SummerAggregator() );
The aggregation is done by the following classes:
private class SummerAggregator implements Aggregator< String, TsdbObject, Double > {
#Override
public Double apply(String key, TsdbObject value, Double aggregate) {
System.out.println( "SummerAggregator.apply: key is " + key + ", value is " + value +
", aggregate is " + aggregate );
return aggregate + value.getValue();
}
}
private class SummerInitializer implements Initializer< Double > {
#Override
public Double apply() {
// TODO Auto-generated method stub
System.out.println( "SummerInitializer" );
return 0.0;
}
}
When I execute the application, I get the following exception:
Encountered the following error during processing:
java.lang.ClassCastException: [B cannot be cast to java.lang.Double
at com.ui.kafka.experiments.metrics.TsdbObjectRollUp$SummerAggregator.apply(TsdbObjectRollUp.java:1)
at org.apache.kafka.streams.kstream.internals.KStreamAggregate$KStreamAggregateProcessor.process(KStreamAggregate.java:79)
The referenced line inKStreamAggregate is:
// try to add the new new value
if (value != null) {
newAgg = aggregator.apply(key, value, newAgg);
}
The strange thing is that the value of newAgg, which is supposed to be a Double, is:
[0, 0, 0, 0, 0, 0, 0, 21]
which certainly isn't castable to a Double. Where did this weird value come from?
You need to pass in an DoubleSerde for the result value type of assets_grouped_by_parents.aggregate(...) using the optional parameter Materialized.withValueSerde():
KTable<String, Double> sums_of_groups_by_parents =
assets_grouped_by_parents.aggregate(
new SummerInitializer(),
new SummerAggregator(),
Materialized.withValueSerde(Serdes.DoubleSerde()));
You might also need to specify the StringSerde for the key, if it's not set as default serde in the config.
Related
In the icCube Builder ETL, I want to group the data on more than one field. Also, as aggregation function, I would like to make use of MAX and MIN.
Example data:
(same data in text)
groupId phase startDate endDate
100 start 1-May-2018 5-May-2018
100 start 4-May-2018 7-May-2018
100 start 28-Apr-2018 1-May-2018
100 middle 4-May-2018 11-May-2018
100 middle 1-May-2018 10-May-2018
100 end 12-May-2018 15-May-2018
100 end 11-May-2018 13-May-2018
100 end 13-May-2018 14-May-2018
100 end 9-May-2018 12-May-2018
200 start 4-Apr-2018 2-May-2018
200 middle 18-Apr-2018 3-May-2018
200 middle 1-May-2018 1-May-2018
300 end 21-Apr-2018 24-Apr-2018
I would like to group this data on groupId and phase and get the minimum startDate and the maximum endDate:
How to best do that in the icCube ETL?
We're adding a new version of groupBy View in the ETL layer to support this. However you can create a Java view to perform the groupBy.
Something like :
package iccube.pub;
import java.util.*;
import java.lang.*;
import org.joda.time.*;
import crazydev.iccube.pub.view.*;
public class CustomJavaView implements IOlapBuilderViewLogic
{
private Map<List<Comparable>,List<Agg>> cached;
public CustomJavaView()
{
}
public void onInitMainTable(Map<String, IOlapCachedTable> cachedTables, IOlapDataTableDef mainTable)
{
cached = new HashMap();
}
public boolean onNewRow(IOlapViewContext context, Map<String, IOlapCachedTable> cachedTables, IOlapDataTableDef mainTable, IOlapReadOnlyDataRow mainTableRow)
{
// create the groupby key (list of values)
final List<Comparable> groupBy = Arrays.asList(mainTableRow.get("phase"), mainTableRow.get("groupId"));
// get the aggregators for values for the keys, build them if not already there
final List<Agg> aggs = cached.computeIfAbsent(groupBy, key -> Arrays.asList(new Agg(true), new Agg(false)));
// add values
aggs.get(0).add(mainTableRow.getAsDateTime("startDate"));
aggs.get(1).add(mainTableRow.getAsDateTime("endDate"));
return true; // false to stop
}
public void onProcessingCompleted(IOlapViewContext context, Map<String, IOlapCachedTable> cachedTables)
{
// now we can fire rows
for (Map.Entry<List<Comparable>, List<Agg>> entry : cached.entrySet())
{
final List<Comparable> groupByKey = entry.getKey();
final List<Agg> aggs = entry.getValue();
// create empty row
final IOlapDataTableRow row = context.newRow();
row.set("phase",groupByKey.get(0));
row.set("groupId",groupByKey.get(1));
row.set("startDate",aggs.get(0).date);
row.set("endDate",aggs.get(1).date);
context.fireRow(row);
}
}
// this is the Aggregator, you could implement something more complicated
static class Agg
{
final int isMin;
LocalDateTime date;
Agg(boolean isMin)
{
this.isMin = isMin ? -1 : 1;
}
void add(LocalDateTime ndate)
{
if (ndate != null)
{
date = ( date!= null && ((date.compareTo(ndate) * isMin) > 0)) ? date : ndate;
}
}
}
}
Initial data:
public class Stats {
int passesNumber;
int tacklesNumber;
public Stats(int passesNumber, int tacklesNumber) {
this.passesNumber = passesNumber;
this.tacklesNumber = tacklesNumber;
}
public int getPassesNumber() {
return passesNumber;
}
public void setPassesNumber(int passesNumber) {
this.passesNumber = passesNumber;
}
public int getTacklesNumber() {
return tacklesNumber;
}
public void setTacklesNumber(int tacklesNumber) {
this.tacklesNumber = tacklesNumber;
}
}
Map<String, List<Stats>> statsByPosition = new HashMap<>();
statsByPosition.put("Defender", Arrays.asList(new Stats(10, 50), new Stats(15, 60), new Stats(12, 100)));
statsByPosition.put("Attacker", Arrays.asList(new Stats(80, 5), new Stats(90, 10)));
I need to calculate an average of Stats by position. So result should be a map with the same keys, however values should be aggregated to single Stats object (List should be reduced to single Stats object)
{
"Defender" => Stats((10 + 15 + 12) / 3, (50 + 60 + 100) / 3),
"Attacker" => Stats((80 + 90) / 2, (5 + 10) / 2)
}
I don't think there's anything new in Java8 that could really help in solving this problem, at least not efficiently.
If you look carefully at all new APIs, then you will see that majority of them are aimed at providing more powerful primitives for working on single values and their sequences - that is, on sequences of double, int, ? extends Object, etc.
For example, to compute an average on sequence on double, JDK introduces a new class - DoubleSummaryStatistics which does an obvious thing - collects a summary over arbitrary sequence of double values.
I would actually suggest that you yourself go for similar approach: make your own StatsSummary class that would look along the lines of this:
// assuming this is what your Stats class look like:
class Stats {
public final double a ,b; //the two stats
public Stats(double a, double b) {
this.a = a; this.b = b;
}
}
// summary will go along the lines of:
class StatsSummary implements Consumer<Stats> {
DoubleSummaryStatistics a, b; // summary of stats collected so far
StatsSummary() {
a = new DoubleSummaryStatistics();
b = new DoubleSummaryStatistics();
}
// this is how we collect it:
#Override public void accept(Stats stat) {
a.accept(stat.a); b.accept(stat.b);
}
public void combine(StatsSummary other) {
a.combine(other.a); b.combine(other.b);
}
// now for actual methods that return stuff. I will implement only average and min
// but rest of them are not hard
public Stats average() {
return new Stats(a.getAverage(), b.getAverage());
}
public Stats min() {
return new Stats(a.getMin(), b.getMin());
}
}
Now, above implementation will actually allow you to express your proper intents when using Streams and such: by building a rigid API and using classes available in JDK as building blocks, you get less errors overall.
However, if you only want to compute average once somewhere and don't need anything else, coding this class is a little overkill, and here's a quick-and-dirty solution:
Map<String, Stats> computeAverage(Map<String, List<Stats>> statsByPosition) {
Map<String, Stats> averaged = new HashMap<>();
statsByPosition.forEach((position, statsList) -> {
averaged.put(position, averageStats(statsList));
});
return averaged;
}
Stats averageStats(Collection<Stats> stats) {
double a, b;
int len = stats.size();
for(Stats stat : stats) {
a += stat.a;
b += stat.b;
}
return len == 0d? new Stats(0,0) : new Stats(a/len, b/len);
}
There is probably a cleaner solution with Java 8, but this works well and isn't too complex:
Map<String, Stats> newMap = new HashMap<>();
statsByPosition.forEach((key, statsList) -> {
newMap.put(key, new Stats(
(int) statsList.stream().mapToInt(Stats::getPassesNumber).average().orElse(0),
(int) statsList.stream().mapToInt(Stats::getTacklesNumber).average().orElse(0))
);
});
The functional forEach method lets you iterate over every key value pair of your given map.
You just put a new entry in your map for the averaged values. There you take the key you have already in your given map. The new value is a new Stats, where the arguments for the constructor are calculated directly.
Just take the value of your old map, which is the statsList in the forEach function, map the values from the given stats to Integer value with mapToInt and use the average function.
This function returns an OptionalDouble which is nearly the same as Optional<Double>. Preventing that anything didn't work, you use its orElse() method and pass a default value (like 0). Since the average values are double you have to cast the value to int.
As mentioned, there doubld probably be a even shorter version, using reduce.
You might as well use custom collector. Let's add the following methods to Stats class:
public Stats() {
}
public void accumulate(Stats stats) {
passesNumber += stats.passesNumber;
tacklesNumber += stats.tacklesNumber;
}
public Stats combine(Stats acc) {
passesNumber += acc.passesNumber;
tacklesNumber += acc.tacklesNumber;
return this;
}
#Override
public String toString() {
return "Stats{" +
"passesNumber=" + passesNumber +
", tacklesNumber=" + tacklesNumber +
'}';
}
Now we can use Stats in collect method:
System.out.println(statsByPosition.entrySet().stream().collect(
Collectors.toMap(
entity -> entity.getKey(),
entity -> {
Stats entryStats = entity.getValue().stream().collect(
Collector.of(Stats::new, Stats::accumulate, Stats::combine)
); // get stats for each map key.
// get average
entryStats.setPassesNumber(entryStats.getPassesNumber() / entity.getValue().size());
// get average
entryStats.setTacklesNumber(entryStats.getTacklesNumber() / entity.getValue().size());
return entryStats;
}
))); // {Attacker=Stats{passesNumber=85, tacklesNumber=7}, Defender=Stats{passesNumber=12, tacklesNumber=70}}
If java-9 is available and StreamEx, you could do :
public static Map<String, Stats> third(Map<String, List<Stats>> statsByPosition) {
return statsByPosition.entrySet().stream()
.collect(Collectors.groupingBy(e -> e.getKey(),
Collectors.flatMapping(e -> e.getValue().stream(),
MoreCollectors.pairing(
Collectors.averagingDouble(Stats::getPassesNumber),
Collectors.averagingDouble(Stats::getTacklesNumber),
(a, b) -> new Stats(a, b)))));
}
Why toString doesn´t work in my code? The output should be all elements that are in the idChild[].
Error:
child[Ljava.lang.String;#15db9742
public String[] onePointCrossover(int father, int mother) {
String linha1 = individualID.get(father);
idFather = linha1.split(" ");
String linha2 = individualDep.get(father);
depenFather= linha2.split(" ");
String linha3 = individualHour.get(father);
hourFather = linha3.split(" ");
String linhaA = individualID.get(mother);
idMother = linha1.split(" ");
String linhaB = individualDep.get(mother);
depenMother= linha2.split(" ");
String linhaC = individualHour.get(mother);
hourMother = linha3.split(" ");
String [] idChild = new String [idFather.length];
int crossPoint = (int) (Math.random()*idFather.length);
for(int i=0; i<idFather.length; i++)
{
if (i<crossPoint)
idChild[i] = idFather[i];
else
idChild [i] = idMother[i];
}
System.out.println("child" + idChild.toString());
return idChild;
}
If you want to loop through all childs in your array, then you need to loop through it, other wise you are attempting to read an array of objects as a string!
Try:
foreach (string s in idChild)
{
System.out.println(s);
}
This is the way toString() works (documentation here): the default implementation of the Object class (and of all arrays) shows the class name, the # symbol and the hexadecimal representation of the hash code of the object:
public String toString() {
return getClass().getName() + "#" + Integer.toHexString(hashCode());
}
The documentation says:
Returns a string representation of the object. In general, the toString method returns a string that "textually represents" this object.
So it's really up to the programmer to choose what "textually represents" means.
If you want to print the String representation of all the items in an array you have to iterate over it.
If I pass an anonymous function as an argument, like e.g. in this code sample:
val someMap = someData.map(line => (line.split("\\|")(0), // key
line.split("\\|")(1) + "|" + // value as string concat
line.split("\\|")(4) + "|" +
line.split("\\|")(9)))
I could catch, e.g. an ArrayIndexOutOfBoundsException like this:
try {
val someMap = someData.map(line => (line.split("\\|")(0), // key
line.split("\\|")(1) + "|" + // value as string concat
line.split("\\|")(4) + "|" +
line.split("\\|")(9)))
} catch {
case e1: ArrayIndexOutOfBoundsException => println("exception in line " )
}
The problem with this is that I do not have access to the inner function's scope. In this case I would like to print the line (from the anonymous function) which caused the exception.
How can I do this? Is there some way of catching an exception within an anonymous function? Is there a way to access the scope of an anonymous function from the outside for debugging purposes?
edit: I'm using Scala 2.9.3
You could use Either
val result =
someData.map {
line =>
try {
val values = (line.split("\\|")(0), // key
line.split("\\|")(1) + "|" + // value as string concat
line.split("\\|")(4) + "|" +
line.split("\\|")(9))
Right(values)
} catch {
case e1: ArrayIndexOutOfBoundsException =>
Left(s"exception in line $line")
}
}
result.foreach {
case (Right(values)) => println(values)
case (Left(msg)) => println(msg)
}
But if you are importing data from a text file, I would try to do it without exceptions (because it's not really exceptional to get invalid data in that case):
val result =
someData.map {
line =>
val fields = line.split("\\|")
if (fields.length < 9) {
Left(s"Error in line $line")
} else {
val values = (fields(0), Seq(fields(1), fields(4), fields(9)))
Right(values)
}
}
result.foreach {
case (Right((key, values))) => println(s"$key -> ${values.mkString("|")}")
case (Left(msg)) => println(msg)
}
Perhaps this will give you some ideas:
try {
val someMap = someData.map { line =>
try {
(line.split("\\|")(0), // key
line.split("\\|")(1) + "|" + // value as string concat
line.split("\\|")(4) + "|" +
line.split("\\|")(9)))
} catch {
case inner: ArrayIndexOutOfBoundsException => {
println("exception in " + line)
throw inner;
}
}
}
} catch {
case outer: ArrayIndexOutOfBoundsException => ...
}
The other answers give nice functional solutions using Either etc. If you were using Scala 2.10, you could also use Try as
val lines = List("abc", "ef");
println(lines.map(line => Try(line(3))));
to get List[Try[Char]], where you can examine each element if it succeeded or failed. (I haven't tried to compile this.)
If for any reasons you prefer exceptions, you need to catch the exception inside the mapping function and rethrow it with information about the line. For example:
// Your own exception class holding a line that failed:
case class LineException(line: String, nested: Exception)
extends Exception(nested);
// Computes something on a line and throw a proper `LineException`
// if the processing fails:
def lineWorker[A](worker: String => A)(line: String): A =
try {
worker(line)
} catch {
case (e: Exception) => throw LineException(line, e);
}
def getNth(lines: List[String], i: Int): List[Char]
= lines.map(lineWorker(_.apply(i)));
val lines = List("abc", "ef");
println(getNth(lines, 1));
println(getNth(lines, 2));
You can also express it using Catch from scala.util.control.Exception:
case class LineException(line: String, nested: Throwable)
extends Exception(nested); // we need Throwable here ^^
import scala.util.control.Exception._
// Returns a `Catch` that wraps any exception to a proper `LineException`.
def lineExceptionCatch[T](line: String): Catch[T]
= handling[T](classOf[Exception]).by(e => throw LineException(line, e));
def lineWorker[A](worker: String => A)(line: String): A =
lineExceptionCatch[A](line)(worker(line))
// ...
First your outer try/catch is useless. If you List (or other structure) is empty, map function won't do anything => no ArrayIndexOutOfBoundsException will be thrown.
As for the inner loop, i would sugest another solution with Scalaz Either:
import scalaz._
import EitherT._
import Id.Id
val someMap = someData.map { line =>
fromTryCatch[Id, (String, String)] {
(line.split("\\|")(0), // key
line.split("\\|")(1) + "|" + // value as string concat
line.split("\\|")(4) + "|" +
line.split("\\|")(9))
}
}
and then chain you operations on List[EitherT[...]]
I like to generate multiple tuples from a single tuple. What I mean is:
I have file with following data in it.
>> cat data
ID | ColumnName1:Value1 | ColumnName2:Value2
so I load it by the following command
grunt >> A = load '$data' using PigStorage('|');
grunt >> dump A;
(ID,ColumnName1:Value1,ColumnName2:Value2)
Now I want to split this tuple into two tuples.
(ID, ColumnName1, Value1)
(ID, ColumnName2, Value2)
Can I use UDF along with foreach and generate. Some thing like the following?
grunt >> foreach A generate SOMEUDF(A)
EDIT:
input tuple : (id1,column1,column2)
output : two tuples (id1,column1) and (id2,column2) so it is List or should I return a Bag?
public class SPLITTUPPLE extends EvalFunc <List<Tuple>>
{
public List<Tuple> exec(Tuple input) throws IOException {
if (input == null || input.size() == 0)
return null;
try{
// not sure how whether I can create tuples on my own. Looks like I should use TupleFactory.
// return list of tuples.
}catch(Exception e){
throw WrappedIOException.wrap("Caught exception processing input row ", e);
}
}
}
Is this approach correct?
You could write a UDF or use a PIG script with built-in functions.
For example:
-- data should be chararray, PigStorage('|') return bytearray which will not work for this example
inpt = load '/pig_fun/input/single_tuple_to_multiple.txt' as (line:chararray);
-- split by | and create a row so we can dereference it later
splt = foreach inpt generate FLATTEN(STRSPLIT($0, '\\|')) ;
-- first column is id, rest is converted into a bag and flatten it to make rows
id_vals = foreach splt generate $0 as id, FLATTEN(TOBAG(*)) as value;
-- there will be records with (id, id), but id should not have ':'
id_vals = foreach id_vals generate id, INDEXOF(value, ':') as p, STRSPLIT(value, ':', 2) as vals;
final = foreach (filter id_vals by p != -1) generate id, FLATTEN(vals) as (col, val);
dump final;
Test INPUT:
1|c1:11:33|c2:12
234|c1:21|c2:22
33|c1:31|c2:32
345|c1:41|c2:42
OUTPUT
(1,c1,11:33)
(1,c2,12)
(234,c1,21)
(234,c2,22)
(33,c1,31)
(33,c2,32)
(345,c1,41)
(345,c2,42)
I hope it helps.
Cheers.
Here is the UDF version. I prefer to return a BAG:
import java.io.IOException;
import org.apache.pig.EvalFunc;
import org.apache.pig.backend.executionengine.ExecException;
import org.apache.pig.data.BagFactory;
import org.apache.pig.data.DataBag;
import org.apache.pig.data.DataType;
import org.apache.pig.data.Tuple;
import org.apache.pig.data.TupleFactory;
import org.apache.pig.impl.logicalLayer.FrontendException;
import org.apache.pig.impl.logicalLayer.schema.Schema;
/**
* Converts input chararray "ID|ColumnName1:Value1|ColumnName2:Value2|.." into a bag
* {(ID, ColumnName1, Value1), (ID, ColumnName2, Value2), ...}
*
* Default rows separator is '|' and key value separator is ':'.
* In this implementation white spaces around separator characters are not removed.
* ID can be made of any character (including sequence of white spaces).
* #author
*
*/
public class TupleToBagColumnValuePairs extends EvalFunc<DataBag> {
private static final TupleFactory tupleFactory = TupleFactory.getInstance();
private static final BagFactory bagFactory = BagFactory.getInstance();
//Row separator character. Default is '|'.
private String rowsSeparator;
//Column value separator character. Default i
private String columnValueSeparator;
public TupleToBagColumnValuePairs() {
this.rowsSeparator = "\\|";
this.columnValueSeparator = ":";
}
public TupleToBagColumnValuePairs(String rowsSeparator, String keyValueSeparator) {
this.rowsSeparator = rowsSeparator;
this.columnValueSeparator = keyValueSeparator;
}
/**
* Creates a tuple with 3 fields (id:chararray, column:chararray, value:chararray)
* #param outputBag Output tuples (id, column, value) are added to this bag
* #param id
* #param column
* #param value
* #throws ExecException
*/
protected void addTuple(DataBag outputBag, String id, String column, String value) throws ExecException {
Tuple outputTuple = tupleFactory.newTuple();
outputTuple.append(id);
outputTuple.append(column);
outputTuple.append( value);
outputBag.add(outputTuple);
}
/**
* Takes column{separator}value from splitInputLine, splits id into column value and adds them to the outputBag as (id, column, value)
* #param outputBag Output tuples (id, column, value) should be added to this bag
* #param id
* #param splitInputLine format column{separator}value, which start from index 1
* #throws ExecException
*/
protected void parseColumnValues(DataBag outputBag, String id,
String[] splitInputLine) throws ExecException {
for (int i = 1; i < splitInputLine.length; i++) {
if (splitInputLine[i] != null) {
int columnValueSplitIndex = splitInputLine[i].indexOf(this.columnValueSeparator);
if (columnValueSplitIndex != -1) {
String column = splitInputLine[i].substring(0, columnValueSplitIndex);
String value = null;
if (columnValueSplitIndex + 1 < splitInputLine[i].length()) {
value = splitInputLine[i].substring(columnValueSplitIndex + 1);
}
this.addTuple(outputBag, id, column, value);
} else {
String column = splitInputLine[i];
this.addTuple(outputBag, id, column, null);
}
}
}
}
/**
* input - contains only one field of type chararray, which will be split by '|'
* All inputs that are: null or of length 0 are ignored.
*/
#Override
public DataBag exec(Tuple input) throws IOException {
if (input == null || input.size() != 1 || input.isNull(0)) {
return null;
}
String inputLine = (String)input.get(0);
String[] splitInputLine = inputLine.split(this.rowsSeparator, -1);
if (splitInputLine.length > 1 && splitInputLine[0].length() > 0) {
String id = splitInputLine[0];
DataBag outputBag = bagFactory.newDefaultBag();
if (splitInputLine.length == 1) { // there is just an id in the line
this.addTuple(outputBag, id, null, null);
} else {
this.parseColumnValues(outputBag, id, splitInputLine);
}
return outputBag;
}
return null;
}
#Override
public Schema outputSchema(Schema input) {
try {
if (input.size() != 1) {
throw new RuntimeException("Expected input to have only one field");
}
Schema.FieldSchema inputFieldSchema = input.getField(0);
if (inputFieldSchema.type != DataType.CHARARRAY) {
throw new RuntimeException("Expected a CHARARRAY as input");
}
Schema tupleSchema = new Schema();
tupleSchema.add(new Schema.FieldSchema("id", DataType.CHARARRAY));
tupleSchema.add(new Schema.FieldSchema("column", DataType.CHARARRAY));
tupleSchema.add(new Schema.FieldSchema("value", DataType.CHARARRAY));
return new Schema(new Schema.FieldSchema(getSchemaName(this.getClass().getName().toLowerCase(), input), tupleSchema, DataType.BAG));
} catch (FrontendException exx) {
throw new RuntimeException(exx);
}
}
}
Here is how it is used in PIG:
register 'path to the jar';
define IdColumnValue myPackage.TupleToBagColumnValuePairs();
inpt = load '/pig_fun/input/single_tuple_to_multiple.txt' as (line:chararray);
result = foreach inpt generate FLATTEN(IdColumnValue($0)) as (id1, c2, v2);
dump result;
A good inspiration for writing UDFs with bags see DataFu source code by LinkedIn
You could use TransposeTupleToBag (UDF from DataFu lib) on the output of STRSPLIT to get the bag, and then FLATTEN the bag to create separate row per original column.