Using Reduce in Apache Spark - hadoop

Question

I am trying to use Apache spark to load up a file, and distribute the file to several nodes in my cluster and then aggregate the results and obtain them. I don't quite understand how to do this.
From my understanding the reduce action enables Spark to combine the results from different nodes and aggregate them together. Am I understanding this correctly?
From a programming perspective, I don't understand how I would code this reduce function.
How exactly do I partition the main dataset into N pieces and ask them to be parallel processed by using a list of transformations?
reduce is supposed to take in two elements and a function for combining them. Are these 2 elements supposed to be RDDs from the context of Spark or can they be any type of element? Also, if you have N different partitions running parallel, how would reduce aggregate all their results into one final result(since the reduce function aggregates only 2 elements)?
Also, I don't understand this example. The example from the spark website uses reduce, but I don't see the data being processed in parallel. So, what is the point of the reduce? If I could get a detailed explanation of the loop in this example, I think that would clear up most of my questions.
class ComputeGradient extends Function<DataPoint, Vector> {
private Vector w;
ComputeGradient(Vector w) { this.w = w; }
public Vector call(DataPoint p) {
return p.x.times(p.y * (1 / (1 + Math.exp(w.dot(p.x))) - 1));
}
}
JavaRDD<DataPoint> points = spark.textFile(...).map(new ParsePoint()).cache();
Vector w = Vector.random(D); // current separating plane
for (int i = 0; i < ITERATIONS; i++) {
Vector gradient = points.map(new ComputeGradient(w)).reduce(new AddVectors());
w = w.subtract(gradient);
}
System.out.println("Final separating plane: " + w);
Also, I have been trying to find the source code for reduce from the Apache Spark Github, but the source is pretty huge and I haven't been able to pinpoint it. Could someone please direct me towards which file I could find it in?

Answer 1

That is a lot of questions. In the future, you should break this up into multiple. I will give a high level that should answer them for you.
First, here is the file with reduce.
Second, most of your problems come from trying to micromanage too much (only necessary if you need to performance tune). You need to first understand the core of what Spark is about and what an RDD is. It is a collection that is parallelized under the hood. From your programming perspective it is just another collection. And reduce is just a function on that collection, a common one in functional programming. All it does is run an operator against all of your collection turning it into one result like below:
((item1 op item2) op item3) op ....
Last, in the example, the code is merely running an iterative algorithm over the data to converge on some point. This is a common task for machine learning algorithms.
Again, I wouldn't focus on the details until you get a better understanding of the high level of distributed programming. Spark is just an abstraction on top to turn this type of programming back into regular code :)