I have a custom class MW. MW gets 2 matrices-(ke matrix and val matrix).
And I am trying to sumup all the matrices coming into reducer.
So I need to first parse my string and I stored them into 2 double array.
I am geting all the ke matrix and val matrix in reducer.
But I am not able to sumup.
Any suggestion.
Inorder to get the sum outside the forloop,i declared them as static.
public class Reducer extends
Reducer<IntWritable, MW, Text, Text> {
static double[][] key;
static double[][] value;
public void reduce(IntWritable keys,
Iterable<MW> values, Context context)
throws IOException, InterruptedException {
for (MW c : values)
{
String data = c.toString();
data = data.trim();
String[] parts = data.split("#");
String part1 = parts[0];
String part2 = parts[1];
/*
* Parse key
*/
String[] keyrows = part1.split(",");
String[][] keymatrix = new String[keyrows.length][];
int keyr = 0;
for (String keyrow : keyrows) {
keymatrix[keyr++] = keyrow.split("\\|");
}
double[][] ke = new double[keymatrix.length][keymatrix[0].length];
for (int i = 0; i<keymatrix.length; i++) {
for (int j = 0; j<keymatrix[0].length; j++) {
ke[i][j] = Double.valueOf(keymatrix[i][j]);
}
}
key = new double[ke.length][ke[0].length];
for(int sumi = 0;sumi<ke.length;sumi++){
for(int sumj=0;sumj<ke[0].length;sumj++){
key[sumi][sumj] += ke[sumi][sumj];
}
}
/*Parsing value
*/
String[] valuerows = part2.split(",");
String[][] valuematrix = new String[valuerows.length][];
int valr = 0;
for (String valuerow : valuerows) {
valuematrix[valr++] = valuerow.split("\\|");
}
double[][] val = new double[valuematrix.length][valuematrix[0].length];
for (int i = 0; i<valuematrix.length; i++) {
for (int j = 0; j<valuematrix[0].length; j++) {
val[i][j] = Double.valueOf(valuematrix[i][j]);
}
}
//calculating sum for value
value = new double[val.length][val[0].length];
for(int sumi = 0;sumi<val.length;sumi++){
for(int sumj=0;sumj<val[0].length;sumj++){
value[sumi][sumj] += val[sumi][sumj];
}
}
}
System.out.println("sum 1");
for(int diai=0;diai<key.length;diai++){
for(int diaj=0;diaj<key[0].length;diaj++){
System.out.print(key[diai][diaj]+"\t");
}
System.out.println("");
}
System.out.println("sum 2");
for(int diai=0;diai<value.length;diai++){
for(int diaj=0;diaj<value[0].length;diaj++){
System.out.print(value[diai][diaj]+"\t");
}
System.out.println("");
}
UPDATE
I think the problem is with in line
key = new double[ke.length][ke[0].length];
and
value = new double[val.length][val[0].length];
before summing I am rebuilding the matrix key and value inside the loop.
It should build it once before the loop and then add to it.
But to do that I should do
double[][] key = new double[ke.length][ke[0].length];
double[][] value = new double[val.length][val[0].length];
before
for (MW c : values)
{
but
How will I get the dimensions outside the for loop?
yes i solved the problem .
i emitted the dimensions as key to reducer. It worked.
Related
in this methode there is only one instruction that not working:
private static byte[] encodePassword(String password,String salt) throws UnsupportedEncodingException
{
String mergedPasswordAndSalt =mergePasswordAndSalt(password, salt);
SHA512Digest digester =new SHA512Digest();
byte[] hash = new byte[digester.getDigestSize()];
digester.update(hash, 0, mergedPasswordAndSalt.length());
digester.doFinal(hash, 0);
System.out.println("init hash= "+Base64.encode(hash));
for (int i = 1; i < ITERATIONS; ++i) {
digester.update(hash, 0, mergedPasswordAndSalt.length());
digester.doFinal(Bytes.concat(hash, mergedPasswordAndSalt.getBytes("UTF-8")),0);
}
System.out.println("FINAL hash= "+Base64.encode(hash));
return hash;
}
that instructions: is the equivalent of this one in the java api that is way:
for (int i = 1; i < ITERATIONS; ++i) {
hash = digester.digest(Bytes.concat(hash, mergedPasswordAndSalt.getBytes("UTF-8")));
}
I have the solution:
To concat bytes i used this method:
public static byte[] concat(byte[]... arrays) {
int length = 0;
byte[][] arr$ = arrays;
int pos = arrays.length;
for(int i$ = 0; i$ < pos; ++i$) {
byte[] array = arr$[i$];
length += array.length;
}
byte[] result = new byte[length];
pos = 0;
byte[][] arr$$=arrays;
arr$=arr$$;
int len$ = arrays.length;
for(int i$ = 0; i$ < len$; ++i$) {
byte[] array = arr$[i$];
System.arraycopy(array, 0, result, pos, array.length);
pos += array.length;
}
return result;
}
To do 4999 iteration on the digest we need a method that takes the hash after every iteration and works in the current digest concatenated with the first diegest(generate out of loop):
private static byte[] encodePassword(String password,String salt) throws UnsupportedEncodingException
{
String mergedPasswordAndSalt =mergePasswordAndSalt(password, salt);
byte[] hash = new byte[88];
hash=digestt(mergedPasswordAndSalt.getBytes("UTF-8"));
for (int i = 1; i < ITERATIONS; ++i) {
hash=digestt(concat(hash,mergedPasswordAndSalt.getBytes("UTF-8")));
}
return hash;
}
public static byte[] digestt(byte[] bytes) {
Digest digest = new SHA512Digest();
byte[] resBuf = new byte[digest.getDigestSize()];
digest.update(bytes, 0, bytes.length);
digest.doFinal(resBuf, 0);
return resBuf;
}
thank you very much i asked many questions and you are always there for help.
I want to classify sentences as positive, negative or neutral. I have got the code written in java but i am not understanding how to use the code for my file.In the above code where i should use my file and the results for each sentence should come as positive, negative and neutral. Please guide me for this.
public class SWN3 {
private String pathToSWN = "data"+File.separator+"SentiWordNet_3.0.0.txt";
private HashMap<String, String> _dict;
public SWN3(){
_dict = new HashMap<String, String>();
HashMap<String, Vector<Double>> _temp = new HashMap<String, Vector<Double>>();
try{
BufferedReader csv = new BufferedReader(new FileReader(pathToSWN));
String line = "";
while((line = csv.readLine()) != null)
{
String[] data = line.split("\t");
Double score = Double.parseDouble(data[2])- Double.parseDouble(data[3]);
String[] words = data[4].split(" ");
for(String w:words)
{
String[] w_n = w.split("#");
w_n[0] += "#"+data[0];
int index = Integer.parseInt(w_n[1])-1;
if(_temp.containsKey(w_n[0]))
{
Vector<Double> v = _temp.get(w_n[0]);
if(index>v.size())
for(int i = v.size();i<index; i++)
v.add(0.0);
v.add(index, score);
_temp.put(w_n[0], v);
}
else
{
Vector<Double> v = new Vector<Double>();
for(int i = 0;i<index; i++)
v.add(0.0);
v.add(index, score);
_temp.put(w_n[0], v);
}
}
}
Set<String> temp = _temp.keySet();
for (Iterator<String> iterator = temp.iterator(); iterator.hasNext();) {
String word = (String) iterator.next();
Vector<Double> v = _temp.get(word);
double score = 0.0;
double sum = 0.0;
for(int i = 0; i < v.size(); i++)
score += ((double)1/(double)(i+1))*v.get(i);
for(int i = 1; i<=v.size(); i++)
sum += (double)1/(double)i;
score /= sum;
String sent = "";
if(score>=0.75)
sent = "strong_positive";
else
if(score > 0.25 && score<=0.5)
sent = "positive";
else
if(score > 0 && score>=0.25)
sent = "weak_positive";
else
if(score < 0 && score>=-0.25)
sent = "weak_negative";
else
if(score < -0.25 && score>=-0.5)
sent = "negative";
else
if(score<=-0.75)
sent = "strong_negative";
_dict.put(word, sent);
}
}
catch(Exception e){e.printStackTrace();}
}
public String extract(String word, String pos)
{
return _dict.get(word+"#"+pos);
}
}
I'm working on this code to manage and save data coming from the Microsoft kinect, the data are stored in the int array int[] depthValues, what I'd like to do is to store and save an average of more frames (let's say 10), in order to get smoother data, leaving the remaining part of the code as it is.
Here's the code:
import java.io.File;
import SimpleOpenNI.*;
import java.util.*;
SimpleOpenNI kinect;
void setup()
{
size(640, 480);
kinect = new SimpleOpenNI(this);
kinect.enableDepth();
}
int precedente = millis();
void draw()
{
kinect.update();
PImage depthImage = kinect.depthImage();
image(depthImage, 0, 0);
int[] depthValues = kinect.depthMap();
StringBuilder sb = new StringBuilder();
Deque<Integer> row = new LinkedList<Integer>();
int kinectheight = 770; // kinect distance from the baselevel [mm]
int scaleFactor = 1;
int pixelsPerRow = 640;
int pixelsToSkip = 40;
int rowNum = 0;
for (int i = 0; i < depthValues.length; i++) {
if (i > 0 && i == (rowNum + 1) * pixelsPerRow) {
fillStringBuilder(sb, row);
rowNum++;
sb.append("\n");
row = new LinkedList<Integer>();
}
if (i >= (rowNum * pixelsPerRow) + pixelsToSkip) {
row.addFirst((kinectheight - depthValues[i]) * scaleFactor);
}
}
fillStringBuilder(sb, row);
String kinectDEM = sb.toString();
final String[] txt= new String[1]; //creates a string array of 2 elements
int savingtimestep = 15000; // time step in millisec between each saving
if (millis() > precedente + savingtimestep) {
txt[0] = "ncols 600\nnrows 480\nxllcorner 0\nyllcorner 0\ncellsize 91.6667\nNODATA_value 10\n" +kinectDEM;
saveStrings("kinectDEM0.tmp", txt);
precedente = millis();
// delete the old .txt file, from kinectDEM1 to kinectDEMtrash
File f = new File (sketchPath("kinectDEM1.txt"));
boolean success = f.delete();
// rename the old .txt file, from kinectDEM0 to kinectDEM1
File oldName1 = new File(sketchPath("kinectDEM0.txt"));
File newName1 = new File(sketchPath("kinectDEM1.txt"));
oldName1.renameTo(newName1);
// rename kinectDEM0.tmp file to kinectDEM0.txt
File oldName2 = new File(sketchPath("kinectDEM0.tmp"));
File newName2 = new File(sketchPath("kinectDEM0.txt"));
oldName2.renameTo(newName2);
}
}
void fillStringBuilder(StringBuilder sb, Deque<Integer> row) {
boolean emptyRow = false;
while (!emptyRow) {
Integer val = row.pollFirst();
if (val == null) {
emptyRow = true;
} else {
sb.append(val);
val = row.peekFirst();
if (val != null) {
sb.append(" ");
}
}
}
}
You have an int[] array variable named depthValues. You can use this value just like you can use any other value. In other words, you can create an array or ArrayList that holds multiple int[] values.
Here's how you might use an ArrayList to hold previous values:
ArrayList<int[]> previousDepthValues = new ArrayList<int[]>();
void draw(){
//add current depth map to ArrayList
previousDepthValues.add(kinect.depthMap());
//limit the ArrayList to hold 10 values
if(previousDepthValues.size() == 11){
previousDepthValues.remove(0);
}
//create an array to hold the averaged values
int[] averageDepthValues = new int[previousDepthValues.get(0).length];
//loop over the 10 previous depth values in the ArrayList
for(int[] depthValue : previousDepthValues){
//loop over the ints in each previous depth values array
for(int i = 0; i < averageDepthValues.length; i++){
//add all of the values up
averageDepthValues[i] += depthValue[i];
}
}
//divide each number to get the average
for(int i = 0; i < averageDepthValues.length; i++){
averageDepthValues[i] /= averageDepthValues.length;
}
//averageDepthValues now holds the average of the last 10 frames
}
I have to set up a queue class that implements from a deque class. I need to use this to set up two deck cards with a random order. I have the code below, it works when the first deck is created but for some reason it does not work with the second deck, its the same code that im reusing.
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 5
at prog.pkg4.Deque.insertOnBack(Prog4.java:93)
at prog.pkg4.Queue.insert(Prog4.java:153)
at prog.pkg4.Prog4.createDeck(Prog4.java:465)
at prog.pkg4.Prog4.topTrump(Prog4.java:444)
at prog.pkg4.Prog4.main(Prog4.java:287)
initiates the two decks
Queue player = new Queue();
Queue computer = new Queue();
player = createDeck(player, cards);
computer = createDeck(computer, cards);
code to create random deck
public static Queue createDeck(Queue queue, GreekHero[] cards){
Random rand = new Random();
int temp = 0;
int r;
for(int i = 0; i < 30; i++){
r = rand.nextInt(30);
cards[temp] = cards[i];
cards[i] = cards[r];
cards[r] = cards[temp];
}
for(int i = 0; i < 29; i++){
queue.insert(cards[i]);
System.out.println(queue.insertions());
System.out.println(queue);
}
return queue;
}
class Queue{
private Deque queue;
public Queue(){
queue = new Deque();
}
public void insert(Object o){
queue.insertOnBack(o);
}
public Object delete(){
return queue.deleteFromFront();
}
public boolean isEmpty(){
return queue.isEmpty();
}
public String toString(){
return queue.toString();
}
public int insertions(){
return queue.getInsertions();
}
}
i've tested out the deque code several times i know it works, as demonstrated by the first deck that is created, im just not sure what could be causing the problem for the second deck.
EDIT: I've added the Deque class code below, the way i have this set up is that if the number of insertions equals the size of the array, it should double in size, as mentioned before it works with the first deque but on the second deque it stops at size of array - 1. I've increased the size to test out and I could make it bigger to satisfy this project but I need to create a deque with an increasing array.
class Deque{
private Object[] arrayObject;
private int beggining; //tracks first element in array
private int insertions; //counts the items in the array
private static int SIZE = 30; //size of array
public Deque(){
arrayObject = new Object[SIZE];
beggining = 0;
insertions = 0;
}
// displays position of first element in circular array
public Object getBeggining(){
int temp = beggining + 1;
if(temp == SIZE)
temp = 0;
return temp;
}
public int getInsertions(){
return insertions;
}
public Object indexOne(){
int temp = beggining + 1;
if(temp == SIZE)
temp = 0;
return arrayObject[temp];
}
public String toString(){
if(isEmpty())
return "Empty";
int temp = beggining + 1;
if( temp >= SIZE)
temp = 0;
String s = "Current Index:\n[("+arrayObject[temp]+")";
int loops = 0;
for(int i = temp + 1; loops < insertions - 1; i++){
if(i >= SIZE)
i = 0;
s += ", ("+arrayObject[i]+")";
loops++;
}
s += "]";
return s;
}
public String toStore(){
String s = "Store Index:\n[(1: "+arrayObject[1]+")";
for(int i = 1; i <= SIZE - 1; i++)
s += ", ("+(i+1)+": "+arrayObject[i]+")";
s += "]";
return s;
}
public void insertOnFront(Object o){
if(insertions == SIZE)
arrayObject = increaseArray();
arrayObject[beggining] = o;
beggining--;
if(beggining < 0)
beggining = SIZE - 1;
insertions++;
}
public Object deleteFromFront(){
if(isEmpty())
return null;
int count = beggining + 1;
if(count >= SIZE)
count = 0;
Object temp = arrayObject[count];
beggining += 1;
insertions--;
if(insertions > 0)
insertions = 0;
return temp;
}
public void insertOnBack(Object o){
int temp = beggining + insertions + 1;
if(insertions == SIZE - 1)
arrayObject = increaseArray();
if(temp >= SIZE)
temp = 0 + (temp - SIZE);
arrayObject[temp] = o;
insertions++;
}
public Object deleteFromBack(){
if(isEmpty())
return null;
int count = beggining + insertions;
Object temp = arrayObject[count];
insertions--;
if(insertions >= 0)
insertions = 0;
return temp;
}
public boolean isEmpty(){
if(insertions > 0)
return false;
else
return true;
}
public Object[] increaseArray(){
SIZE *= 2;
int loops = 0;
int j = beggining;
Object[] newArray = new Object[SIZE];
for(int i = j; loops <= SIZE/2; i++){
if(j >= SIZE/2)
j = 0;
newArray[i] = arrayObject[j];
loops++;
j++;
}
return newArray;
}
}
I solved the issue by moving the SIZE variable as an instance variable of the class and removed static from it. I don't know why the issue popped in on the second iteration rather than on the first try, ill look it up later, if anyone knows please post it here.
Based on the current implementation, I will get an arraylist which contains some 1000 unique names in the alphabetically sorted order(A-Z or Z-A) from some source.
I need to find the index of the first word starting with a given alphabet.
So to be more precise, when I select an alphabet, for eg. "M", it should give me the index of the first occurrence of the word starting in "M" form the sorted list.
And that way I should be able to find the index of all the first words starting in each of the 26 alphabets.
Please help me find a solution which doesn't compromise on the speed.
UPDATE:
Actually after getting the 1000 unique names, the sorting is also done by one of my logics.
If this can be done while doing the sorting itself, I can avoid the reiteration on the list after sorting to find the indices for the alphabets.
Is that possible?
Thanks,
Sen
I hope this little piece of code will help you. I guessed the question is related to Java, because you mentioned ArrayList.
String[] unsorted = {"eve", "bob", "adam", "mike", "monica", "Mia", "marta", "pete", "Sandra"};
ArrayList<String> names = new ArrayList<String>(Arrays.asList(unsorted));
String letter = "M"; // find index of this
class MyComp implements Comparator<String>{
String first = "";
String letter;
MyComp(String letter){
this.letter = letter.toUpperCase();
}
public String getFirst(){
return first;
}
#Override
public int compare(String s0, String s1) {
if(s0.toUpperCase().startsWith(letter)){
if(s0.compareTo(first) == -1 || first.equals("")){
first = s0;
}
}
return s0.toUpperCase().compareTo(s1.toUpperCase());
}
};
MyComp mc = new MyComp(letter);
Collections.sort(names, mc);
int index = names.indexOf(mc.getFirst()); // the index of first name starting with letter
I'm not sure if it's possible to also store the index of the first name in the comparator without much overhead. Anyway, if you implement your own version of sorting algorithm e.g. quicksort, you should know about the index of the elements and could calculate the index while sorting. This depends on your chosen sorting algorithm and implementation. In fact if I know how your sorting is implemented, we could insert the index calculation.
So I came up with my own solution for this.
package test.binarySearch;
import java.util.Random;
/**
*
* Binary search to find the index of the first starting in an alphabet
*
* #author Navaneeth Sen <navaneeth.sen#multichoice.co.za>
*/
class SortedWordArray
{
private final String[] a; // ref to array a
private int nElems; // number of data items
public SortedWordArray(int max) // constructor
{
a = new String[max]; // create array
nElems = 0;
}
public int size()
{
return nElems;
}
public int find(String searchKey)
{
return recFind(searchKey, 0, nElems - 1);
}
String array = null;
int arrayIndex = 0;
private int recFind(String searchKey, int lowerBound,
int upperBound)
{
int curIn;
curIn = (lowerBound + upperBound) / 2;
if (a[curIn].startsWith(searchKey))
{
array = a[curIn];
if ((curIn == 0) || !a[curIn - 1].startsWith(searchKey))
{
return curIn; // found it
}
else
{
return recFind(searchKey, lowerBound, curIn - 1);
}
}
else if (lowerBound > upperBound)
{
return -1; // can't find it
}
else // divide range
{
if (a[curIn].compareTo(searchKey) < 0)
{
return recFind(searchKey, curIn + 1, upperBound);
}
else // it's in lower half
{
return recFind(searchKey, lowerBound, curIn - 1);
}
} // end else divide range
} // end recFind()
public void insert(String value) // put element into array
{
int j;
for (j = 0; j < nElems; j++) // find where it goes
{
if (a[j].compareTo(value) > 0) // (linear search)
{
break;
}
}
for (int k = nElems; k > j; k--) // move bigger ones up
{
a[k] = a[k - 1];
}
a[j] = value; // insert it
nElems++; // increment size
} // end insert()
public void display() // displays array contents
{
for (int j = 0; j < nElems; j++) // for each element,
{
System.out.print(a[j] + " "); // display it
}
System.out.println("");
}
} // end class OrdArray
class BinarySearchWordApp
{
static final String AB = "12345aqwertyjklzxcvbnm";
static Random rnd = new Random();
public static String randomString(int len)
{
StringBuilder sb = new StringBuilder(len);
for (int i = 0; i < len; i++)
{
sb.append(AB.charAt(rnd.nextInt(AB.length())));
}
return sb.toString();
}
public static void main(String[] args)
{
int maxSize = 100000; // array size
SortedWordArray arr; // reference to array
int[] indices = new int[27];
arr = new SortedWordArray(maxSize); // create the array
for (int i = 0; i < 100000; i++)
{
arr.insert(randomString(10)); //insert it into the array
}
arr.display(); // display array
String searchKey;
for (int i = 97; i < 124; i++)
{
searchKey = (i == 123)?"1":Character.toString((char) i);
long time_1 = System.currentTimeMillis();
int result = arr.find(searchKey);
long time_2 = System.currentTimeMillis() - time_1;
if (result != -1)
{
indices[i - 97] = result;
System.out.println("Found " + result + "in "+ time_2 +" ms");
}
else
{
if (!(i == 97))
{
indices[i - 97] = indices[i - 97 - 1];
}
System.out.println("Can't find " + searchKey);
}
}
for (int i = 0; i < indices.length; i++)
{
System.out.println("Index [" + i + "][" + (char)(i+97)+"] = " + indices[i]);
}
} // end main()
}
All comments welcome.