// given a set of Item objects, group them by the managers of creator and owners
Map<String, List<Item>> managersItems =
itemSet.parallelStream().flatMap(item -> {
// get the list of the creator and owners
List<String> users = new ArrayList();
users.add(item.getCreator());
users.addAll(item.getOwners());
return Stream.of(users.toArray(new String[] {})).map(user -> {
LdapUserInfo ldapUser = LdapUserInfoFactory.create(user);
String manager = ldapUser.getManager();
return new AbstractMap.SimpleImmutableEntry<String, Item(manager, item);
});
}).collect(
Collectors.groupingBy(Map.Entry::getKey, Collectors.mapping(Map.Entry::getValue, Collectors.toList())));
This code compiles fine in Eclipse Mars, but gets the following eror in Eclipse Luna:
Type mismatch: cannot convert from Map<Object,List<Object>> to Map<String,List<WeblabInfo>>
If I do not assign the returned to a Map with Map<String, List<Item>> managersItem = in Eclipse Luna, the error is at Map.Entry::getKey and Map.Entry::getValue statement with message:
The type Map.Entry does not define getKey(Object) that is applicable here".
What did I do wrong?
You didn't do anything wrong. Eclipse compiler has problems with type inference that causes these issues. If Luna compatibility is important, you will have to add explicit types to lambda expressions. Try, for example, Map.Entry::<String,Item>getKey
On another note, it's not necessary to convert a List to array to stream it. You can directly call users.stream(). But even creating the List isn't necessary. You can use Stream.concat(Stream.of(item.getCreator()), item.getOwners().stream()) instead (granted, it's a bit unwieldy).
Finally (and most importantly), avoid using parallelStream with blocking code, such as looking up data in an external system. Parallel streams are designed to handle CPU-bound tasks.
I was able to come up with this solution from Misha's answer. This is working with Eclipse Luna Java compiler
Map<String, List<Item>> managersItems = itemSet
.stream()
.<Map.Entry<String, Item>> flatMap(item -> {
return Stream.concat(Stream.of(item.getCreatorLogin()), item.getOwners().stream()).map(
user -> {
LdapUserInfo ldapUser = LdapUserInfoFactory.create(user);
String manager = ldapUser.getManagerLoginName();
return new AbstractMap.SimpleEntry<String, Item>(manager, info);
});
})
.collect(Collectors.groupingBy(Map.Entry<String, Item>::getKey,
Collectors.mapping(Map.Entry<String, Item>::getValue,
Collectors.toList())));
Related
We are using java 8 parallel stream to process a task, and we are submitting the task through ForkJoinPool#submit. We are not using jvm wide ForkJoinPool.commonPool, instead we are creating our own custom pool to specify the parallelism and storing it as static variable.
We have validation framework, where we subject a list of tables to a List of Validators, and we submit this job through the custom ForkJoinPool as follows:
static ForkJoinPool forkJoinPool = new ForkJoinPool(4);
List<Table> tables = tableDAO.findAll();
ModelValidator<Table, ValidationResult> validator = ValidatorFactory
.getInstance().getTableValidator();
List<ValidationResult> result = forkJoinPool.submit(
() -> tables.stream()
.parallel()
.map(validator)
.filter(result -> result.getValidationMessages().size() > 0)
.collect(Collectors.toList())).get();
The problem we are having is, in the downstream components, the individual validators which run on separate threads from our static ForkJoinPool rely on tenant_id, which is different for every request and is stored in an InheritableThreadLocal variable. Since we are creating a static ForkJoinPool, the threads pooled by the ForkJoinPool will only inherit the value of the parent thread, when it is created first time. But these pooled threads will not know the new tenant_id for the current request. So for subsequent execution these pooled threads are using old tenant_id.
I tried creating a custom ForkJoinPool and specifying ForkJoinWorkerThreadFactory in the constructor and overriding the onStart method to feed the new tenant_id. But that doesnt work, since the onStart method is called only once at creation time and not during individual execution time.
Seems like we need something like the ThreadPoolExecutor#beforeExecute which is not available in case of ForkJoinPool. So what alternative do we have if we want to pass the current thread local value to the statically pooled threads?
One workaround would be to create the ForkJoinPool for each request, rather than make it static but we wouldn't want to do it, to avoid the expensive nature of thread creation.
What alternatives do we have?
I found the following solution that works without changing any underlying code. Basically, the map method takes a functional interface which I am representing as a lambda expression. This expression adds a preExecution hook to set the new tenantId in the current ThreadLocal and cleaning it up in postExecution.
forkJoinPool.submit(tables.stream()
.parallel()
.map((item) -> {
preExecution(tenantId);
try {
return validator.apply(item);
} finally {
postExecution();
}
}
)
.filter(validationResult ->
validationResult.getValidationMessages()
.size() > 0)
.collect(Collectors.toList())).get();
The best option in my view would be to get rid of the thread local and pass it as an argument instead. I understand that this could be a massive undertaking though. Another option would be to use a wrapper.
Assuming that your validator has a validate method you could do something like:
public class WrappingModelValidator implements ModelValidator<Table. ValidationResult> {
private final ModelValidator<Table. ValidationResult> v;
private final String tenantId;
public WrappingModelValidator(ModelValidator<Table. ValidationResult> v, String tenantId) {
this.v = v;
this.tenantId = tenantId;
}
public ValidationResult validate(Table t) {
String oldValue = YourThreadLocal.get();
YourThreadLocal.set(tenantId);
try {
return v.validate(t);
} finally {
YourThreadLocal.set(oldValue);
}
}
}
Then you simply wrap your old validator and it will set the thread local on entry and restore it when done.
I'm looking to create a gRPC response that returns a list of maps. Here is what I imagined the structure being:
message GetSettingsResponse {
repeated map<string, string> settings = 1;
}
Repeating maps is not supported, however, and I had to nest the map in a separate message to make it work:
message GetSettingsResponse {
repeated Setting settings = 1;
}
message Setting {
map<string, string> setting = 1;
}
This works, but it forces us to write some confusing code on both the client and server. Is there any way to avoid this solution and get closer to my desired structure?
No, basically. What you have is the closest you can do in protobuf.
I'm writing a basic application to test the Interactive Queries feature of Kafka Streams. Here is the code:
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KeyValueBytesStoreSupplier waypointsStoreSupplier = Stores.persistentKeyValueStore("test-store");
StoreBuilder waypointsStoreBuilder = Stores.keyValueStoreBuilder(waypointsStoreSupplier, Serdes.ByteArray(), Serdes.Integer());
final KStream<byte[], byte[]> waypointsStream = builder.stream("sample1");
final KStream<byte[], TruckDriverWaypoint> waypointsDeserialized = waypointsStream
.mapValues(CustomSerdes::deserializeTruckDriverWaypoint)
.filter((k,v) -> v.isPresent())
.mapValues(Optional::get);
waypointsDeserialized.groupByKey().aggregate(
() -> 1,
(aggKey, newWaypoint, aggValue) -> {
aggValue = aggValue + 1;
return aggValue;
}, Materialized.<byte[], Integer, KeyValueStore<Bytes, byte[]>>as("test-store").withKeySerde(Serdes.ByteArray()).withValueSerde(Serdes.Integer())
);
final KafkaStreams streams = new KafkaStreams(builder.build(), new StreamsConfig(createStreamsProperties()));
streams.cleanUp();
streams.start();
ReadOnlyKeyValueStore<byte[], Integer> keyValueStore = streams.store("test-store", QueryableStoreTypes.keyValueStore());
KeyValueIterator<byte[], Integer> range = keyValueStore.all();
while (range.hasNext()) {
KeyValue<byte[], Integer> next = range.next();
System.out.println(next.value);
}
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
protected static Properties createStreamsProperties() {
final Properties streamsConfiguration = new Properties();
streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "random167");
streamsConfiguration.put(StreamsConfig.CLIENT_ID_CONFIG, "client-id");
streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
streamsConfiguration.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, Serdes.String().getClass().getName());
streamsConfiguration.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, Serdes.Integer().getClass().getName());
//streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 10000);
return streamsConfiguration;
}
So my problem is, every time I run this I get this same error:
Exception in thread "main" org.apache.kafka.streams.errors.InvalidStateStoreException: the state store, test-store, may have migrated to another instance.
I'm running only 1 instance of the application, and the topic I'm consuming from has only 1 partition.
Any idea what I'm doing wrong ?
Looks like you have a race condition. From the kafka streams javadoc for KafkaStreams::start() it says:
Start the KafkaStreams instance by starting all its threads. This function is expected to be called only once during the life cycle of the client.
Because threads are started in the background, this method does not block.
https://kafka.apache.org/10/javadoc/index.html?org/apache/kafka/streams/KafkaStreams.html
You're calling streams.store() immediately after streams.start(), but I'd wager that you're in a state where it hasn't initialized fully yet.
Since this is code appears to be just for testing, add a Thread.sleep(5000) or something in there and give it a go. (This is not a solution for production) Depending on your input rate into the topic, that'll probably give a bit of time for the store to start filling up with events so that your KeyValueIterator actually has something to process/print.
Probably not applicable to OP but might help others:
In trying to retrieve a KTable's store, make sure the the KTable's topic exists first or you'll get this exception.
I failed to call Storebuilder before consuming the store.
Typically this happens for two reasons:
The local KafkaStreams instance is not yet ready (i.e., not yet in
runtime state RUNNING, see Run-time Status Information) and thus its
local state stores cannot be queried yet. The local KafkaStreams
instance is ready (e.g. in runtime state RUNNING), but the particular
state store was just migrated to another instance behind the scenes.
This may notably happen during the startup phase of a distributed
application or when you are adding/removing application instances.
https://docs.confluent.io/platform/current/streams/faq.html#handling-invalidstatestoreexception-the-state-store-may-have-migrated-to-another-instance
The simplest approach is to guard against InvalidStateStoreException when calling KafkaStreams#store():
// Example: Wait until the store of type T is queryable. When it is, return a reference to the store.
public static <T> T waitUntilStoreIsQueryable(final String storeName,
final QueryableStoreType<T> queryableStoreType,
final KafkaStreams streams) throws InterruptedException {
while (true) {
try {
return streams.store(storeName, queryableStoreType);
} catch (InvalidStateStoreException ignored) {
// store not yet ready for querying
Thread.sleep(100);
}
}
}
I'm doing some introspection and analysis of csproj files using the Microsoft.Build.Evaluation tools in a small C# console app. I want to locate the actual location of Reference items, using the same heuristics as MSBuild itself ie the locations described here. I'm heading towards auto conversion of build artifacts into packages, similar to what's outlined on the JetBrains blog here
The only examples I can find expect the HintPath to be correct, for example this project, and I know there are some HintPaths that are not currently correct, I don't want to trust them. This project very close what I'm trying to do, with the added complication that I want to use real resolution behaviour to find dependencies.
I have an instance of a Microsoft.Build.Evaluation.Project object for my csproj, and I can't see any methods available on it that could exersize the resolution for me. I think what I'm hoping for is a magic Resolve() method for a Reference or a ProjectItem, a bit like this method.
I can probably find an alternative by constraining my own search to a set of limited output paths used by this build system, but I'd like to hook into MSBuild if I can.
The reference resolution is one of the trickiest parts of MSBuild. The logic of how assemblies are located is implemented inside the a standard set of tasks:
ResolveAssemblyReference, ResolveNativeReference, etc. The logic is how this works is very complicated, you can see that just by looking at number of possible parameters to those tasks.
However you don't need to know the exact logic to find the location of referenced files. There are standard targets called "ResolveAssemblyReferences", "ResolveProjectReferences" and some others more specialized for native references, COM references. Those targets are executed as part of the normal build. If you just execute those targets separately, you can find out the return values, which is exactly what you need. The same mechanism is used by IDE to get location of refereces, for Intellisense, introspection, etc.
Here is how you can do it in code:
using Microsoft.Build.BuildEngine;
using Microsoft.Build.Execution;
using Microsoft.Build.Framework;
using System;
using System.Collections.Generic;
class Program
{
static int Main(string[] args)
{
if (args.Length < 1)
{
Console.WriteLine("Usage: GetReferences.exe <projectFileName>");
return -1;
}
string projectFileName = args[0];
ConsoleLogger logger = new ConsoleLogger(LoggerVerbosity.Normal);
BuildManager manager = BuildManager.DefaultBuildManager;
ProjectInstance projectInstance = new ProjectInstance(projectFileName);
var result = manager.Build(
new BuildParameters()
{
DetailedSummary = true,
Loggers = new List<ILogger>() { logger }
},
new BuildRequestData(projectInstance, new string[]
{
"ResolveProjectReferences",
"ResolveAssemblyReferences"
}));
PrintResultItems(result, "ResolveProjectReferences");
PrintResultItems(result, "ResolveAssemblyReferences");
return 0;
}
private static void PrintResultItems(BuildResult result, string targetName)
{
var buildResult = result.ResultsByTarget[targetName];
var buildResultItems = buildResult.Items;
if (buildResultItems.Length == 0)
{
Console.WriteLine("No refereces detected in target {0}.", targetName);
return;
}
foreach (var item in buildResultItems)
{
Console.WriteLine("{0} reference: {1}", targetName, item.ItemSpec);
}
}
}
Notice, the engine is called to invoke specific targets in the project. Your project usually does not build, but some targets might be invoked by pre-requisite targets.
Just compile it and will print a sub-set of all dependencies. There might be more dependencies if you use COM references or native dependencies for your project. It should be easy to modify the sample to get those as well.
What new features in java 7 is going to be implemented?
And what are they doing now?
Java SE 7 Features and Enhancements from JDK 7 Release Notes
This is the Java 7 new features summary from the OpenJDK 7 features page:
vm JSR 292: Support for dynamically-typed languages (InvokeDynamic)
Strict class-file checking
lang JSR 334: Small language enhancements (Project Coin)
core Upgrade class-loader architecture
Method to close a URLClassLoader
Concurrency and collections updates (jsr166y)
i18n Unicode 6.0
Locale enhancement
Separate user locale and user-interface locale
ionet JSR 203: More new I/O APIs for the Java platform (NIO.2)
NIO.2 filesystem provider for zip/jar archives
SCTP (Stream Control Transmission Protocol)
SDP (Sockets Direct Protocol)
Use the Windows Vista IPv6 stack
TLS 1.2
sec Elliptic-curve cryptography (ECC)
jdbc JDBC 4.1
client XRender pipeline for Java 2D
Create new platform APIs for 6u10 graphics features
Nimbus look-and-feel for Swing
Swing JLayer component
Gervill sound synthesizer [NEW]
web Update the XML stack
mgmt Enhanced MBeans [UPDATED]
Code examples for new features in Java 1.7
Try-with-resources statement
this:
BufferedReader br = new BufferedReader(new FileReader(path));
try {
return br.readLine();
} finally {
br.close();
}
becomes:
try (BufferedReader br = new BufferedReader(new FileReader(path)) {
return br.readLine();
}
You can declare more than one resource to close:
try (
InputStream in = new FileInputStream(src);
OutputStream out = new FileOutputStream(dest))
{
// code
}
Underscores in numeric literals
int one_million = 1_000_000;
Strings in switch
String s = ...
switch(s) {
case "quux":
processQuux(s);
// fall-through
case "foo":
case "bar":
processFooOrBar(s);
break;
case "baz":
processBaz(s);
// fall-through
default:
processDefault(s);
break;
}
Binary literals
int binary = 0b1001_1001;
Improved Type Inference for Generic Instance Creation
Map<String, List<String>> anagrams = new HashMap<String, List<String>>();
becomes:
Map<String, List<String>> anagrams = new HashMap<>();
Multiple exception catching
this:
} catch (FirstException ex) {
logger.error(ex);
throw ex;
} catch (SecondException ex) {
logger.error(ex);
throw ex;
}
becomes:
} catch (FirstException | SecondException ex) {
logger.error(ex);
throw ex;
}
SafeVarargs
this:
#SuppressWarnings({"unchecked", "varargs"})
public static void printAll(List<String>... lists){
for(List<String> list : lists){
System.out.println(list);
}
}
becomes:
#SafeVarargs
public static void printAll(List<String>... lists){
for(List<String> list : lists){
System.out.println(list);
}
}
New Feature of Java Standard Edition (JSE 7)
Decorate Components with the JLayer Class:
The JLayer class is a flexible and powerful decorator for Swing components. The JLayer class in Java SE 7 is similar in spirit to the JxLayer project project at java.net. The JLayer class was initially based on the JXLayer project, but its API evolved separately.
Strings in switch Statement:
In the JDK 7 , we can use a String object in the expression of a switch statement. The Java compiler generates generally more efficient bytecode from switch statements that use String objects than from chained if-then-else statements.
Type Inference for Generic Instance:
We can replace the type arguments required to invoke the constructor of a generic class with an empty set of type parameters (<>) as long as the compiler can infer the type arguments from the context. This pair of angle brackets is informally called the diamond.
Java SE 7 supports limited type inference for generic instance creation; you can only use type inference if the parameterized type of the constructor is obvious from the context. For example, the following example does not compile:
List<String> l = new ArrayList<>();
l.add("A");
l.addAll(new ArrayList<>());
In comparison, the following example compiles:
List<? extends String> list2 = new ArrayList<>();
l.addAll(list2);
Catching Multiple Exception Types and Rethrowing Exceptions with Improved Type Checking:
In Java SE 7 and later, a single catch block can handle more than one type of exception. This feature can reduce code duplication. Consider the following code, which contains duplicate code in each of the catch blocks:
catch (IOException e) {
logger.log(e);
throw e;
}
catch (SQLException e) {
logger.log(e);
throw e;
}
In releases prior to Java SE 7, it is difficult to create a common method to eliminate the duplicated code because the variable e has different types.
The following example, which is valid in Java SE 7 and later, eliminates the duplicated code:
catch (IOException|SQLException e) {
logger.log(e);
throw e;
}
The catch clause specifies the types of exceptions that the block can handle, and each exception type is separated with a vertical bar (|).
The java.nio.file package
The java.nio.file package and its related package, java.nio.file.attribute, provide comprehensive support for file I/O and for accessing the file system. A zip file system provider is also available in JDK 7.
Source: http://ohmjavaclasses.blogspot.com/
Java Programming Language Enhancements # Java7
Binary Literals
Strings in switch Statement
Try with Resources (How it works) or ARM (Automatic Resource Management)
Multiple Exception Handling
Suppressed Exceptions
underscore in literals
Type Inference for Generic Instance Creation using Diamond Syntax
Improved Compiler Warnings and Errors When Using Non-Reifiable Formal Parameters with Varargs Methods
Official reference
Official reference with java8
wiki reference
In addition to what John Skeet said, here's an overview of the Java 7 project. It includes a list and description of the features.
Note: JDK 7 was released on July 28, 2011, so you should now go to the official java SE site.
Language changes:
-Project Coin (small changes)
-switch on Strings
-try-with-resources
-diamond operator
Library changes:
-new abstracted file-system API (NIO.2) (with support for virtual filesystems)
-improved concurrency libraries
-elliptic curve encryption
-more incremental upgrades
Platform changes:
-support for dynamic languages
Below is the link explaining the newly added features of JAVA 7 , the explanation is crystal clear with the possible small examples for each features :
http://radar.oreilly.com/2011/09/java7-features.html
Using Diamond(<>) operator for generic instance creation
Map<String, List<Trade>> trades = new TreeMap <> ();
Using strings in switch statements
String status= “something”;
switch(statue){
case1:
case2:
default:
}
Underscore in numeric literals
int val 12_15;
long phoneNo = 01917_999_720L;
Using single catch statement for throwing multiple exception by using “|” operator
catch(IOException | NullPointerException ex){
ex.printStackTrace();
}
No need to close() resources because Java 7 provides try-with-resources statement
try(FileOutputStream fos = new FileOutputStream("movies.txt");
DataOutputStream dos = new DataOutputStream(fos)) {
dos.writeUTF("Java 7 Block Buster");
} catch(IOException e) {
// log the exception
}
binary literals with prefix “0b” or “0B”
I think ForkJoinPool and related enhancement to Executor Framework is an important addition in Java 7.
The following list contains links to the the enhancements pages in the Java SE 7.
Swing
IO and New IO
Networking
Security
Concurrency Utilities
Rich Internet Applications (RIA)/Deployment
Requesting and Customizing Applet Decoration in Dragg able Applets
Embedding JNLP File in Applet Tag
Deploying without Codebase
Handling Applet Initialization Status with Event Handlers
Java 2D
Java XML – JAXP, JAXB, and JAX-WS
Internationalization
java.lang Package
Multithreaded Custom Class Loaders in Java SE 7
Java Programming Language
Binary Literals
Strings in switch Statements
The try-with-resources Statement
Catching Multiple Exception Types and Rethrowing Exceptions with Improved Type Checking
Underscores in Numeric Literals
Type Inference for Generic Instance Creation
Improved Compiler Warnings and Errors When Using Non-Reifiable Formal Parameters with Varargs Methods
Java Virtual Machine (JVM)
Java Virtual Machine Support for Non-Java Languages
Garbage-First Collector
Java HotSpot Virtual Machine Performance Enhancements
JDBC
Reference 1 Reference 2