What is the difference between clustering and matching?
For example: There's a pool of four elements and in the one scenario I want to generate pairs. What I do is I measure the distance of each element to each other which yields a 2x2 matrix. Then the matching algorithm finds the two pairings with the lowest or highest weighted sum.
What is a clustering algorithm doing? When I demand a cluster number of two then the result is the same, or not?
Specifying the number of elements in a cluster (pairs for example) doesn't make much sense. If you have been looking at k-means (k-medoids), the k actually indicates how many clusters will be created in total. So, if you have 4 elements and use k = 2, you can get one cluster with 1 element and another cluster with 3 elements, depending on the data you have. Anyway, clustering on 4 elements doesn't make sense.
Here I'm kinda stuck with this quiz problem. It asks for the recurrence function for number of ways that a set of n points can be clustered into k non-empty clusters.
My initial thought is that it should be S(n,k) = nS(n, k-1) since for every increase in the number of clusters by one, there should be n more ways to add a cluster to existing clusters of k-1 in size.
The picture attached is the actual question. Thanks a lot!
enter image description here
You can get k non-empty clusters, containing n objects:
by adding n-th object to any existing cluster (there are k of them, so k*S(n-1,k) variants)
or making new cluster containing single n-th object in addition to (k-1) existing clusters (S(n-1,k-1) variants)
I have a big sequence file storing the tfidf values for documents. Each line represents line and the columns are the value of tfidfs for each term (the row is a sparse vector). I'd like to pick the top-k words for each document using Hadoop. The naive solution is to loop through all the columns for each row in the mapper and pick the top-k but as the file becomes bigger and bigger I don't think this is a good solution. Is there a better way to do that in Hadoop?
1. In every map calculate TopK (this is local top K for each map)
2. Spawn a signle reduce , now top K from all mappers will flow to this reducer and hence global Top K will be evaluated.
Think of the problem as
1. You have been given the results of X number of horse races.
2. You need to find Top N fastest horse.
Can someone example the computation of median/quantiles in map reduce?
My understanding of Datafu's median is that the 'n' mappers sort the
data and send the data to "1" reducer which is responsible for sorting
all the data from n mappers and finding the median(middle value)
Is my understanding correct?,
if so, does this approach scale for
massive amounts of data as i can clearly see the one single reducer
struggling to do the final task.
Thanks
Trying to find the median (middle number) in a series is going to require that 1 reducer is passed the entire range of numbers to determine which is the 'middle' value.
Depending on the range and uniqueness of values in your input set, you could introduce a combiner to output the frequency of each value - reducing the number of map outputs sent to your single reducer. Your reducer can then consume the sort value / frequency pairs to identify the median.
Another way you could scale this (again if you know the range and rough distribution of values) is to use a custom partitioner that distributes the keys by range buckets (0-99 go to reducer 0, 100-199 to reducer 2, and so on). This will however require some secondary job to examine the reducer outputs and perform the final median calculation (knowing for example the number of keys in each reducer, you can calculate which reducer output will contain the median, and at which offset)
Do you really need the exact median and quantiles?
A lot of the time, you are better off with just getting approximate values, and working with them, in particular if you use this for e.g. data partitioning.
In fact, you can use the approximate quantiles to speed up finding the exact quantiles (actually in O(n/p) time), here is a rough outline of the strategy:
Have a mapper for each partition compute the desired quantiles, and output them to a new data set. This data set should be several order of magnitues smaller (unless you ask for too many quantiles!)
Within this data set, compute the quantiles again, similar to "median of medians". These are your initial estimates.
Repartition the data according to these quantiles (or even additional partitions obtained this way). The goal is that in the end, the true quantile is guaranteed to be in one partition, and there should be at most one of the desired quantiles in each partition
Within each of the partitions, perform a QuickSelect (in O(n)) to find the true quantile.
Each of the steps is in linear time. The most costly step is part 3, as it will require the whole data set to be redistributed, so it generates O(n) network traffic.
You can probably optimize the process by choosing "alternate" quantiles for the first iteration. Say, you want to find the global median. You can't find it in a linear process easily, but you can probably narrow it down to 1/kth of the data set, when it is split into k partitions. So instead of having each node report its median, have each node additionally report the objects at (k-1)/(2k) and (k+1)/(2k). This should allow you to narrow down the range of values where the true median must lie signficantly. So in the next step, you can each node send those objects that are within the desired range to a single master node, and choose the median within this range only.
O((n log n)/p) to sort it then O(1) to get the median.
Yes... you can get O(n/p) but you can't use the out of the box sort functionality in Hadoop. I would just sort and get the center item unless you can justify the 2-20 hours of development time to code the parallel kth largest algorithm.
In many real-world scenarios, the cardinality of values in a dataset will be relatively small. In such cases, the problem can be efficiently solved with two MapReduce jobs:
Calculate frequencies of values in your dataset (Word Count job, basically)
Identity mapper + a reducer which calculates median based on < value - frequency> pairs
Job 1. will drastically reduce the amount of data and can be executed fully in parallel. Reducer of job 2. will only have to process n (n = cardinality of your value set) items instead of all values, as with the naive approach.
Below, an example reducer of the job 2. It's is python script that could be used directly in Hadoop streaming. Assumes values in your dataset are ints, but can be easily adopted for doubles
import sys
item_to_index_range = []
total_count = 0
# Store in memory a mapping of a value to the range of indexes it has in a sorted list of all values
for line in sys.stdin:
item, count = line.strip().split("\t", 1)
new_total_count = total_count + int(count)
item_to_index_range.append((item, (total_count + 1, new_total_count + 1)))
total_count = new_total_count
# Calculate index(es) of middle items
middle_items_indexes = [(total_count / 2) + 1]
if total_count % 2 == 0:
middle_items_indexes += [total_count / 2]
# Retrieve middle item(s)
middle_items = []
for i in middle_items_indexes:
for item, index_range in item_to_index_range:
if i in range(*index_range):
middle_items.append(item)
continue
print sum(middle_items) / float(len(middle_items))
This answer builds up on top of a suggestion initially coming from the answer of Chris White. The answer suggests using a combiner as a mean to calculate frequencies of values. However, in MapReduce, combiners are not guaranteed to be always executed. This has some side effects:
reducer will first have to compute final < value - frequency > pairs and then calculate median.
In the worst case scenario, combiners will never be executed and the reducer will still have to struggle with processing all individual values
I'm working on a clustering algorithm to group similar ranges of real numbers. After I group them, I have to create one range for that cluster, i.e., cluster centroid. For example, if one cluster contains values <1,6>, <0,7> and <0,6>, that means that this cluster is for all those with values <0,7>. The question is how to create such a resulting range. I was thinking to take the min and max value of all values in the cluster, but that would mean that the algorithm is very sensitive on noise. I should do it somehow weighted, but I'm not sure how. Any hints? Thanks.
Perhaps you can convert all ranges to their midpoints before running your clustering algorithm. That way you convert your problem into clustering points on a line. Previously, the centroid range could 'grow' and in the next iteration consume more ranges that perhaps should belong to another cluster.
midpoints = []
for range in ranges
midpoints[range] = range.min + (range.max - range.min) / 2
end
After the algorithm is finished you can do as you previously suggested and take the min and max values of all the ranges in the cluster to create the range for that centroid.