How do I achieve this using java=8
I have a CSV in below format and from this i want to populate Map<String, Map<String, String>
where the outer map will have key scriptId and transationType as these are the distinct Type and inner map for scriptId key should contain first 5 values stating from position 2 as key and 3 as value.
<scriptId<
<TATA,TATA Moters>
<REL,Reliance Industries Ltd>
<LNT, L&T>
<SBI, State Bank of India>>
<transactionType,<
<P,B>
<S,S>>
Content of CSV File
Type,ArcesiumValue,GICValue
scriptId,TATA,TATA Moters
scriptId,REL,Reliance Industries Ltd
scriptId,LNT,L&T
scriptId,SBI,State Bank of India
transactionType,P,B
transactionType,S,S
How do i generate this using Java8
public void loadReferenceData() throws IOException {
List<Map<String, Map<String, String>>> cache = Files.lines(Paths.get("data/referenceDataMapping.csv")).skip(1)
.map(mapRefereneData).collect(Collectors.toList());
System.out.println(cache);
}
public static Function<String, Map<String, Map<String, String>>> mapRefereneData = (line) -> {
String[] sp = line.split(",");
Map<String, Map<String, String>> cache = new HashMap<String, Map<String, String>>();
try {
if (cache.containsKey(sp[0])) {
cache.get(sp[0]).put(sp[1], sp[2]);
} else {
Map<String, String> map = new HashMap<String, String>();
map.put(sp[1], sp[2]);
cache.put(sp[0], map);
}
} catch (NumberFormatException e) {
e.printStackTrace();
}
return cache;
};
Well it is much simpler to use two Collectors:
Map<String, Map<String, String>> groupCSV = Files.lines(Paths.get("..."))
.skip(1L).map(l -> l.split(","))
.collect(Collectors.groupingBy(a -> a[0], Collectors.toMap(a -> a[1], a -> a[2])));
Related
I want to read mulitpart/formdata, one part is application/JSON. I can't get them to Map<String,String>, Is there any way to parse Part to String?
private Map<String, String> getFormData(String path, MultiValueMap<String, Part> partMultiValueMap) {
if (partMultiValueMap != null) {
Map<String, String> formData = new HashMap<>();
Map<String, Part> multiPartMap = partMultiValueMap.toSingleValueMap();
for (Map.Entry<String, Part> partEntry : multiPartMap.entrySet()) {
Part part = partEntry.getValue();
if (part instanceof FormFieldPart) {
formData.put(partEntry.getKey(), ((FormFieldPart) part).value());
} else {
String bodyString = bufferToStr(part.content());
formData.put(partEntry.getKey(), bodyString);
}
}
return formData;
}
return null;
}
extra Flux
private String bufferToStr(Flux<DataBuffer> content){
AtomicReference<String> res = new AtomicReference<>();
content.subscribe(buffer -> {
byte[] bytes = new byte[buffer.readableByteCount()];
buffer.read(bytes);
DataBufferUtils.release(buffer);
res.set(new String(bytes, StandardCharsets.UTF_8));
});
return res.get();
}
Subscribe is async; bufferToStr value may be null?
You could do it in non-blocking way with StringDecoder
Basically you could write your code to return Mono<Map<>>
Note: I'm using Pair class here to return key-value and later collect them to Map
Pair I'm using here is from package org.springframework.data.util.Pair
public Mono<Map<String, String>> getFormData(MultiValueMap<String, Part> partMultiValueMap) {
Map<String, Part> multiPartMap = partMultiValueMap.toSingleValueMap();
return Flux.fromIterable(multiPartMap.entrySet())
.flatMap(entry -> {
Part part = entry.getValue();
if (part instanceof FormFieldPart) {
return Mono.just(
Pair.of(entry.getKey(), ((FormFieldPart) part).value()) // return Pair
);
} else {
return decodePartToString(part.content()) // decoding DataBuffers to string
.flatMap(decodedString ->
Mono.just(Pair.of(entry.getKey(), decodedString))); // return Pair
}
})
.collectMap(Pair::getFirst, Pair::getSecond); // map and collect pairs to Map<>
}
private Mono<String> decodePartToString(Flux<DataBuffer> dataBufferFlux) {
StringDecoder stringDecoder = StringDecoder.textPlainOnly();
return stringDecoder.decodeToMono(dataBufferFlux,
ResolvableType.NONE,
MimeTypeUtils.TEXT_PLAIN,
Collections.emptyMap()
);
}
I was going through How to remove a key from HashMap while iterating over it?, but my requirement is bit different.
class Main {
public static void main(String[] args) {
Map<String, String> hashMap = new HashMap<>();
hashMap.put("RED", "#FF0000");
hashMap.put("BLACK", null);
hashMap.put("BLUE", "#0000FF");
hashMap.put("GREEN", "#008000");
hashMap.put("WHITE", null);
// I wan't result like below - get All keys whose value is null
List<String> collect = hashMap.values()
.stream()
.filter(e -> e == null)
.collect(Collectors.toList());
System.out.println(collect);
// Result - BLACK, WHITE in list
}
}
Try this:
import java.util.*;
import java.util.stream.*;
class Main {
public static void main(String[] args) {
Map<String, String> hashMap = new HashMap<>();
hashMap.put("RED", "#FF0000");
hashMap.put("BLACK", null);
hashMap.put("BLUE", "#0000FF");
hashMap.put("GREEN", "#008000");
hashMap.put("WHITE", null);
// I wan't result like below - get All keys whose value is null
List<String> collect = hashMap.keySet()
.stream()
.filter(e -> Objects.isNull(hashMap.get(e)))
.collect(Collectors.toList());
System.out.println(collect);
// Result - BLACK, WHITE in list
}
}
As pointed out in the comments, you can try this as well:
import java.util.*;
import java.util.stream.*;
class Main {
public static void main(String[] args) {
Map<String, String> hashMap = new HashMap<>();
hashMap.put("RED", "#FF0000");
hashMap.put("BLACK", null);
hashMap.put("BLUE", "#0000FF");
hashMap.put("GREEN", "#008000");
hashMap.put("WHITE", null);
// I wan't result like below - get All keys whose value is null
List<String> collect = hashMap.entrySet()
.stream()
.filter(e -> Objects.isNull(e.getValue()))
.map(e -> e.getKey())
.collect(Collectors.toList());
System.out.println(collect);
// Result - BLACK, WHITE in list
}
}
This is more optimized, as compared to the first solution.
I'd like to fetch sonar.timemachine.period1 via wsclient.
Seeing that it doesn't have one, I decided to bake one for myself
private Map<String, String> retrievePeriodProperties(final WsClient wsClient, int requestedPeriod) {
if (requestedPeriod > 0) {
final WsRequest propertiesWsRequestPeriod =
new GetRequest("api/properties/sonar.timemachine.period" + requestedPeriod);
final WsResponse propertiesWsResponsePeriod =
wsClient.wsConnector().call(propertiesWsRequestPeriod);
if (propertiesWsResponsePeriod.isSuccessful()) {
String resp = propertiesWsResponsePeriod.content();
Map<String, String> map = new HashMap<>();
map.put(Integer.toString(requestedPeriod), resp);
return map;
}
}
return new HashMap<>();
}
but it always return an empty Map<>
Any lead where I can go from this direction?
You can use org.sonar.api.config.Settings to fetch properties defined in SonarQube.
Would someone help me with getting the array of the map values with stream and filtering?
public class TheMap extends HashMap<String, String> {
public TheMap(String name, String title) {
super.put("name", name);
super.put("title", title);
}
public static void main(final String[] args) {
Map<Long, Map<String, String>>map = new HashMap<>();
map.put(0L, null);
map.put(1L, new TheMap("jane", "engineer"));
map.put(2L, new TheMap("john", "engineer"));
map.put(3L, new TheMap(null, "manager"));
map.put(4L, new TheMap("who", null));
map.put(5L, new TheMap(null, null));
}
}
The result that I am looking for is an ArrayList<TheMap> with only these two entries:
TheMap("jane", "engineer")
TheMap("john", "engineer")
Basically, retrieve TheMap with none-null name and title.
List<Map<String, String>> list =
map.values().stream().filter(v ->
v != null &&
!v.entrySet().isEmpty() &&
!v.containsValue(null)).
collect(Collectors.toList());
If you need an arrayList of TheMap, try the following way:
ArrayList<TheMap> as = map.values()
.stream()
.filter(v -> v != null && v.get("name") != null && v.get("title") != null)
.map(m -> (TheMap)m)
.collect(Collectors.toCollection(ArrayList::new)));
I am trying to Streaming the data from Kafka to Spark
JavaPairInputDStream<String, String> directKafkaStream = KafkaUtils.createDirectStream(ssc,
String.class,
String.class,
StringDecoder.class,
StringDecoder.class,
kafkaParams, topics);
Here i am iterating over the JavaPairInputDStream to process the RDD's.
directKafkaStream.foreachRDD(rdd ->{
rdd.foreachPartition(items ->{
while (items.hasNext()) {
String[] State = items.next()._2.split("\\,");
System.out.println(State[2]+","+State[3]+","+State[4]+"--");
};
});
});
I can able to fetch the data in foreachRDD and my requirement is have to access State Array globally. When i am trying to access the State Array globally i am getting Exception
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
Any suggestions ? Thanks.
This is more of a joining your lookup table with streaming RDD to get all the items that have a matching 'code' and 'violationCode' fields.
The flow should be like this.
Create an RDD of Hive lookup table => lookupRdd
Create DStream from kafka stream
For each RDD in Dstream, join lookupRDD with streamRdd, process the joined items(calculate sum of amount...) and save this processed result.
Note Below code is incomplete. Please complete all the TODO comments.
JavaPairDStream<String, String> streamPair = directKafkaStream.mapToPair(new PairFunction<Tuple2<String, String>, String, String>() {
#Override
public Tuple2<String, String> call(Tuple2<String, String> tuple2) throws Exception {
System.out.println("Tuple2 Message is----------" + tuple2._2());
String[] state = tuple2._2.split("\\,");
return new Tuple2<>(state[4], tuple2._2()); //pair <ViolationCode, data>
}
});
streamPair.foreachRDD(new Function<JavaPairRDD<String, String>, Void>() {
JavaPairRDD<String, String> hivePairRdd = null;
#Override
public Void call(JavaPairRDD<String, String> stringStringJavaPairRDD) throws Exception {
if (hivePairRdd == null) {
hivePairRdd = initHiveRdd();
}
JavaPairRDD<String, Tuple2<String, String>> joinedRdd = stringStringJavaPairRDD.join(hivePairRdd);
System.out.println(joinedRdd.take(10));
//todo process joinedRdd here and save the results.
joinedRdd.count(); //to trigger an action
return null;
}
});
}
public static JavaPairRDD<String, String> initHiveRdd() {
JavaRDD<String> hiveTableRDD = null; //todo code to create RDD from hive table
JavaPairRDD<String, String> hivePairRdd = hiveTableRDD.mapToPair(new PairFunction<String, String, String>() {
#Override
public Tuple2<String, String> call(String row) throws Exception {
String code = null; //TODO process 'row' and get 'code' field
return new Tuple2<>(code, row);
}
});
return hivePairRdd;
}