public class StreamStringCompression {
private Iterator iterator;
public StreamStringCompression(Iterator iterator) {
this.iterator = iterator;
}
public String encode(String str) {
System.out.println(str);
return str;
}
public static void main(String[] args) {
String str = "aaaabbcaaa";
StreamStringCompression encoder = new StreamStringCompression(Collections.singletonList(str).iterator());
str.chars().mapToObj(i -> (char)i).map(i -> encoder.encode(""+i));
}
}
I have the above Java example, and I want to print the stream of characters such a way that it should print like below.
1a
2a
3a
4a
1b
2b
c
1a
2a
3a
How can I do this with java8 streams by maintaining a state?
Related
I'm completely new in Hadoop Framework and I want to write a "MapReduce" program (HadoopJoin.java) that joins on x attribute between two tables R and S. The structure of the two tables is :
R (tag : char, x : int, y : varchar(30))
and
S (tag : char, x : int, z : varchar(30))
For example we have for R table :
r 10 r-10-0
r 11 r-11-0
r 12 r-12-0
r 21 r-21-0
And for S table :
s 11 s-11-0
s 21 s-41-0
s 21 s-41-1
s 12 s-31-0
s 11 s-31-1
The result should look like :
r 11 r-11-0 s 11 s-11-0
etc.
Can anyone help me please ?
It will be very difficult to describe join in mapreduce for someone who is new to this Framework but here I provide a working implementation for your situation and I definitely recommend you to read section 9 of Hadoop The Definitive Guide 4th Eddition. It has described how to implement Join in mapreduce very well.
First of all you might consider using higher level frameworks such as Pig, Hive and Spark because they provide join operation in their core part of implementation.
Secondly There are many ways to implement mapreduce depending of the nature of your data. This ways include map-side join and reduce-side join. In this answer I have implemented the reduce-side join:
Implementation:
First of all we should have two different mapper for two different datset notice that in your case same mapper can be used for two dataset but in many situation you need different mappers for different dataset and because of that I have defined two mappers to make this solution more general:
I have used TextPair that have two attributes, one of them is the key that is used to join data and the other one is a tag that specify which dataset this record belongs to. If it belongs to the first dataset this tag will be 0. otherwise it will be 1.
I have implemented TextPair.FirstComparator to ensure that for each key(join by key) the record of the first dataset is the first key which is received by reducer. And all the other records in second dataset with that id are received after that. Actually this line of code will do the trick for us:
job.setGroupingComparatorClass(TextPair.FirstComparator.class);
So in reducer the first record that we will receive is the record from dataset1 and after that we receive record from dataset2. The only thing that should be done is that we have to write those records.
Mapper for dataset1:
public class JoinDataSet1Mapper
extends Mapper<LongWritable, Text, TextPair, Text> {
#Override
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String[] data = value.toString().split(" ");
context.write(new TextPair(data[1], "0"), value);
}
}
Mapper for DataSet2:
public class JoinDataSet2Mapper
extends Mapper<LongWritable, Text, TextPair, Text> {
#Override
protected void map(LongWritable key, Text value, Context context)
throws IOException, InterruptedException {
String[] data = value.toString().split(" ");
context.write(new TextPair(data[1], "1"), value);
}
}
Reducer:
public class JoinReducer extends Reducer<TextPair, Text, NullWritable, Text> {
public static class KeyPartitioner extends Partitioner<TextPair, Text> {
#Override
public int getPartition(TextPair key, Text value, int numPartitions) {
return (key.getFirst().hashCode() & Integer.MAX_VALUE) % numPartitions;
}
}
#Override
protected void reduce(TextPair key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
Iterator<Text> iter = values.iterator();
Text stationName = new Text(iter.next());
while (iter.hasNext()) {
Text record = iter.next();
Text outValue = new Text(stationName.toString() + "\t" + record.toString());
context.write(NullWritable.get(), outValue);
}
}
}
Custom key:
public class TextPair implements WritableComparable<TextPair> {
private Text first;
private Text second;
public TextPair() {
set(new Text(), new Text());
}
public TextPair(String first, String second) {
set(new Text(first), new Text(second));
}
public TextPair(Text first, Text second) {
set(first, second);
}
public void set(Text first, Text second) {
this.first = first;
this.second = second;
}
public Text getFirst() {
return first;
}
public Text getSecond() {
return second;
}
#Override
public void write(DataOutput out) throws IOException {
first.write(out);
second.write(out);
}
#Override
public void readFields(DataInput in) throws IOException {
first.readFields(in);
second.readFields(in);
}
#Override
public int hashCode() {
return first.hashCode() * 163 + second.hashCode();
}
#Override
public boolean equals(Object o) {
if (o instanceof TextPair) {
TextPair tp = (TextPair) o;
return first.equals(tp.first) && second.equals(tp.second);
}
return false;
}
#Override
public String toString() {
return first + "\t" + second;
}
#Override
public int compareTo(TextPair tp) {
int cmp = first.compareTo(tp.first);
if (cmp != 0) {
return cmp;
}
return second.compareTo(tp.second);
}
public static class FirstComparator extends WritableComparator {
private static final Text.Comparator TEXT_COMPARATOR = new Text.Comparator();
public FirstComparator() {
super(TextPair.class);
}
#Override
public int compare(byte[] b1, int s1, int l1,
byte[] b2, int s2, int l2) {
try {
int firstL1 = WritableUtils.decodeVIntSize(b1[s1]) + readVInt(b1, s1);
int firstL2 = WritableUtils.decodeVIntSize(b2[s2]) + readVInt(b2, s2);
return TEXT_COMPARATOR.compare(b1, s1, firstL1, b2, s2, firstL2);
} catch (IOException e) {
throw new IllegalArgumentException(e);
}
}
#Override
public int compare(WritableComparable a, WritableComparable b) {
if (a instanceof TextPair && b instanceof TextPair) {
return ((TextPair) a).first.compareTo(((TextPair) b).first);
}
return super.compare(a, b);
}
}
}
JobDriver:
public class JoinJob extends Configured implements Tool {
#Override
public int run(String[] args) throws Exception {
Job job = Job.getInstance(getConf(), "Join two DataSet");
job.setJarByClass(getClass());
Path ncdcInputPath = new Path(getConf().get("job.input1.path"));
Path stationInputPath = new Path(getConf().get("job.input2.path"));
Path outputPath = new Path(getConf().get("job.output.path"));
MultipleInputs.addInputPath(job, ncdcInputPath,
TextInputFormat.class, JoinDataSet1Mapper.class);
MultipleInputs.addInputPath(job, stationInputPath,
TextInputFormat.class, JoinDataSet2Mapper.class);
FileOutputFormat.setOutputPath(job, outputPath);
job.setPartitionerClass(JoinReducer.KeyPartitioner.class);
job.setGroupingComparatorClass(TextPair.FirstComparator.class);
job.setMapOutputKeyClass(TextPair.class);
job.setReducerClass(JoinReducer.class);
job.setOutputKeyClass(Text.class);
return job.waitForCompletion(true) ? 0 : 1;
}
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new JoinJob(), args);
System.exit(exitCode);
}
}
I am trying to concat two string using Lambda expression. I have created functional interface StringFormatter which has format method which will take two strings as argument. it should return String.
The ans should be Returns "Lambda Expression" or Returns "Lambda – Expression" depending on lambda expression.
public class Main {
public static void main(String[] args) {
String s1="Lambda";
String s2="Expression";
new Main().print(s1,s2);
}
public void print(String st1, String st2) {
String result= (st1, st2) -> { return st1+" "+ st2;}; /*error- The target type of this expression must be a functional interface*/
System.out.println(result);
}
}
#java.lang.FunctionalInterface
interface StringFormatter {
abstract String format(String s1, String s2);
}
I think you want something like that:
public class Main {
public static void main(String[] args) {
String s1="Lambda";
String s2="Expression";
// format 1
new Main().print(s1,s2, (str1, str2) -> str1 + " " + str2);
// format 2
new Main().print(s1,s2, (str1, str2) -> str1 + "-" + str2);
}
void print(String str1, String str2, StringFormatter formatter) {
String result = formatter.format(str1, str2);
System.out.println(result);
}
}
#java.lang.FunctionalInterface
interface StringFormatter {
String format(String s1, String s2);
}
Much simpler solution:
import java.util.function.BinaryOperator;
// Define the printing function with this lambda
BinaryOperator<String> printFunction = (string1, string2) -> string1 + " " + string2;
// Call it and get the result
System.out.println(print(st1, st2));
The BinaryOperator interface is available in java.util.function package. It can be used to perform a binary operation(eg: concatenation) on two values of T type and returns a T type result.
public void print(String st1, String st2) {
BinaryOperator<String> concat = (a, b) -> a + " " + b;
System.out.println(concat.apply(st1, st2));
}
I'm trying to understand/learn Consumer/BiConsumer in Java8.
test1 and test2 methods are working fine.
But if I tried to use the old fashion way by implementing BiConsumer in a class in test3 method.
And then override accept method in the class, str.substring method cannot resolve the method substring.
Can't I use the old fashion way in #FunctionalInterface or did I do something wrong in the code?
public class BiConsumerTest {
static void test1(String name, Integer count) {
method1(name, count, (str, i) -> {
System.out.println(str.substring(i));
});
}
static void test2(String name, Integer count) {
BiConsumer<String, Integer> consumer = (str, i) -> {
System.out.println(str.substring(i));
};
method1(name, count, consumer);
}
private static void method1(String name, Integer count, BiConsumer<String, Integer> consumer) {
consumer.accept(name, count);
}
private void test3(String name, Integer count) {
BiConsumer<String, Integer> consumer = new ConsumerImpl<String, Integer>();
consumer.accept(name, count);
}
class ConsumerImpl<String, Integer> implements BiConsumer<String, Integer> {
#Override
public void accept(String str, Integer count) {
str.substring(count); // str cannot find substring method !!!
}
}
public static void main(String[] args) {
String name = "aaa bbb ccc";
Integer count = 6;
test1(name, count);
test2(name, count);
}
}
You cannot define a class with known types as type-parameters (so this is incorrect - ConsumerImpl<String, Integer>). Plus there were few other syntactical mistakes. Below works -
import java.util.function.BiConsumer;
public class TestClass {
static void test1(String name, Integer count) {
method1(name, count, (str, i) -> {
System.out.println(str.substring(i));
});
}
static void test2(String name, Integer count) {
BiConsumer<String, Integer> consumer = (str, i) -> {
System.out.println(str.substring(i));
};
method1(name, count, consumer);
}
private static void method1(String name, Integer count, BiConsumer<String, Integer> consumer) {
consumer.accept(name, count);
}
static void test3(String name, Integer count) {
BiConsumer<String, Integer> consumer = new ConsumerImpl();
consumer.accept(name, count);
}
static class ConsumerImpl implements BiConsumer<String, Integer> {
#Override
public void accept(String str, Integer count) {
System.out.println(str.substring(count)); // str cannot find substring method !!!
}
}
public static void main(String[] args) {
String name = "aaa bbb ccc";
Integer count = 6;
test1(name, count);
test2(name, count);
test3(name, count);
}
}
I'd like to be able to update the value of a TextView in a fragment created using the FragmentGridPagerAdapter class, but I haven't really an idea of how I could achieve that.
Here is where I call my class to create the fragments.
final GridViewPager pager = (GridViewPager) findViewById(R.id.pager);
pager.setAdapter(new SGridPagerAdapter(this, getFragmentManager(), data));
DotsPageIndicator dotsPageIndicator = (DotsPageIndicator) findViewById(R.id.page_indicator);
dotsPageIndicator.setPager(pager);
And here is my class that extends the FragmentGridPagerAdapter.
public class SGridPagerAdapter extends FragmentGridPagerAdapter {
String[] mData;
private final Context mContext;
private static final String TAG = SGridPagerAdapter.class.getSimpleName();
public SGridPagerAdapter(Context ctx, FragmentManager fm, String[] data) {
super(fm);
mData = data;
mContext = ctx;
}
#Override
public Fragment getFragment(int row, int column) {
Log.d(TAG, "row : " + row + "/columns : " + column);
Fragment f = null;
switch(column) {
case 0 :
f = cardFragment(R.string.welcome_title, R.string.welcome_text);
break;
case 1 :
f = SelectionFragment.newInstance("secondFragment");
break;
}
return f;
}
private Fragment cardFragment(int titleRes, int textRes) {
Resources res = mContext.getResources();
CardFragment fragment = CardFragment.create(res.getText(titleRes), res.getText(textRes));
// Add some extra bottom margin to leave room for the page indicator
//fragment.setCardMarginBottom(
// res.getDimensionPixelSize(R.dimen.card_margin_bottom));
return fragment;
}
#Override
public int getRowCount() {
return 1;//mData.length;
}
#Override
public int getColumnCount(int row) {
return mData.length;
}
}
Well,
I'd like to know how can I change the sort order of my simple WordCount program after the reduce task? I've already made another map to order by value instead by keys, but it still ordered in ascending order.
Is there an easy simple method to do this (change the sort order)?!
Thanks
Vellozo
If you are using the older API (mapred.*), then set the OutputKeyComparatorClass in the job conf:
jobConf.setOutputKeyComparatorClass(ReverseComparator.class);
ReverseComparator can be something like this:
static class ReverseComparator extends WritableComparator {
private static final Text.Comparator TEXT_COMPARATOR = new Text.Comparator();
public ReverseComparator() {
super(Text.class);
}
#Override
public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
try {
return (-1)* TEXT_COMPARATOR
.compare(b1, s1, l1, b2, s2, l2);
} catch (IOException e) {
throw new IllegalArgumentException(e);
}
}
#Override
public int compare(WritableComparable a, WritableComparable b) {
if (a instanceof Text && b instanceof Text) {
return (-1)*(((Text) a)
.compareTo((Text) b)));
}
return super.compare(a, b);
}
}
In the new API (mapreduce.*), I think you need to use the Job.setSortComparator() method.
This one is almost the same as above, just looks a bit simpler
class MyKeyComparator extends WritableComparator {
protected DescendingKeyComparator() {
super(Text.class, true);
}
#SuppressWarnings("rawtypes")
#Override
public int compare(WritableComparable w1, WritableComparable w2) {
Text key1 = (Text) w1;
Text key2 = (Text) w2;
return -1 * key1.compareTo(key2);
}
}
Then add it it to the job
job.setSortComparatorClass(MyKeyComparator.class);
Text key1 = (Text) w1;
Text key2 = (Text) w2;
you can change the above text type as per ur use.