i am trying to perform nested loop of streams. while i collect the result in list it Throws "illegal state exception" stating that steam is closed
public void execute(Stream<Trade> trade, Stream<Order> order){
order.filter(o -> trade.anyMatch(t -> t.getInstrumentId() ==
o.getInstrumentId() && t.getGroupid() == o.getGroupId()))
.collect(Collectors.toList());// Illegal exception only when i include this line
}
Exception is: java.lang.IllegalStateException: stream has already been operated upon or closed
i know that we should not use stream more than once, but in this case i am just filtering and then collecting.. till filtering it doesnot show up an error but while collecting it throws this exception.
The order stream is used once
The trade steam is not
You write that for each element of the order stream, you'll need to check a condition based on the content of trade but you can iterate it only once, here is the problem.
You may pass the Collection from which comes the trade stream instead.
Related
I need to record failure reason in metrics for each failed http call when using Vert.x WebClient. This compiles:
.onFailure()
.retry()
.withBackOff(Duration.ofMillis(INITIAL_RETRY_DELAY_MS))
.until(retryTimeExpired(wrapper))
I'm recording metrics in retryTimeExpired method. But at runtime I get this:
Caused by: java.lang.IllegalArgumentException: Invalid retry configuration, `when` cannot be used with a back-off configuration
at io.smallrye.mutiny.groups.UniRetry.when(UniRetry.java:156)
at io.smallrye.mutiny.groups.UniRetry.until(UniRetry.java:137)
I could of course add sleep but this is reactive. It would be possible to block for a short time but I would hate to block the thread. Any ideas how to do this without sleep?
You could try using many sequential onFailures. As long as the first doesn't handle the exception (recoverWithItem, recoverWithNull, recoverWithUni) or throw its own the next should observe the same failure.
service.apiMethod(key)
.invoke(record -> logger.info("Succeeded to read record."))
.onFailure()
.invoke(exception -> logger.warn("Failed to read record."))
.onFailure()
.retry().withBackOff(delay).indefinitely();
Is there an equivalent of PublishSubject from RxJava in Kotlin Coroutines library?
Channels cannot be a replacement for PublishSubject since they do not publish values to multiple collectors (each value can be collected by a single collector only). Even MutableSharedFlow that supports multiple collectors, still does not allow emitting values without waiting for collectors to finish processing previous values. How can we create a flow with functionality similar to the PublishSubject?
The following code will create a Flow equivalent to the PublishSubject:
fun <T> publishFlow(): MutableSharedFlow<T> {
return MutableSharedFlow(
replay = 0,
extraBufferCapacity = Int.MAX_VALUE
)
}
The main attributes of the PublishSubject are that it does not replay old values to new observers, and still allows to publish new values/events without waiting for the observers to handle them. So this functionality can be achieved with MutableSharedFlow by specifying replay = 0 for preventing new collectors from collecting old values, and extraBufferCapacity = Int.MAX_VALUE to allow publishing new values without waiting for busy collectors to finish collecting previous values.
One can add the following forceEmit function to be called instead of tryEmit, to ensure that the value is actually emitted:
fun <T> MutableSharedFlow<T>.forceEmit(value: T) {
val emitted = tryEmit(value)
check(emitted){ "Failed to emit into shared flow." }
}
Since we have a buffer with MAX_VALUE capacity, this forceEmit function should never fail if we use it with our publishFlow. If the flow will be replaced somehow with a different flow that does not support emitting without suspending, we will get an exception and will know to handle the case where the buffer is full and one cannot emit without suspending.
Notice that having a buffer of MAX_VALUE capacity may cause high consumption of memory if the collection of values by the collectors takes a long time, so it is more suitable for cases where the collectors perform a short synchronous operation (similarly to RxJava observers).
In my batch job, I have a single step with reading from Database , processing the record and writing back the same record to same table.(ie updating record with processed values or error reason if processing failed).
I am using AsyncItemProcessor for multi thread processing. When I get error in ItemProcessor.process() method, I throw an exception and batch job ends with FAILED status. This failed status is a requirement.
Because, its AsyncItemProcessor, I am unable to access ItemProcessListener.onProcessError().
How do I write the errorMessage to Item Table when there is an error ?
This is a known limitation of using the AsyncItemProcessor which is mentioned in its Javadoc:
While not an exhaustive list, things like StepExecution.filterCount will not
reflect the number of filtered items and
ItemProcessListener.onProcessError(Object, Exception) will not be called.
There is an open issue to update the reference documentation as well.
How do I write the errorMessage to Item Table when there is an error ?
The AsyncItemProcessor submits a FutureTask to the task executor and the only way to know if an exception happened in the task is by unwrapping the future (the exception will be actually wrapped in a java.util.concurrent.ExecutionException when the FutureTask.get is called). Now since the future is unwrapped in the AsyncItemWriter, you can use an ItemWriteListener and react to processing errors. You can find a complete example here.
I m trying to read data from kafka and insert into cassandra using storm. I've configured the topology also, however I'm getting some issue and I don't have clue why that is happening.
Here is my submitter piece.
TopologyBuilder topologyBuilder = new TopologyBuilder();
topologyBuilder.setSpout("spout", new KafkaSpout(spoutConfig));
topologyBuilder.setBolt("checkingbolt", new CheckingBolt("cassandraBoltStream")).shuffleGrouping("spout");
topologyBuilder.setBolt("cassandrabolt", new CassandraInsertBolt()).shuffleGrouping("checkingbolt");
Here, if I comment the last line, I don't see any exceptions. With the last line, I'm getting the below error:
InvalidTopologyException(msg:Component: [cassandrabolt] subscribes from non-existent stream: [default] of component [checkingbolt])
Can someone please help me, what is wrong here?
Here is the outputFieldDeclarer in CheckingBolt
public void declareOutputFields(OutputFieldsDeclarer ofd) {
ofd.declareStream(cassandraBoltStream, new Fields(new String[]{"jsonFields"}));
}
I don't have anything in declareOutputFields method for CassandraInsertBolt as that bolt doesn't emit any values.
TIA
The problem here is that you're mixing up stream names and component (i.e. spout/bolt) names. Component names are used for referring to different bolts, while stream names are used to refer to different streams coming out of the same bolt. For example, if you have a bolt named "evenOrOddBolt", it might emit two streams, an "even" stream and and "odd" stream. In many cases though, you only have one stream coming out of a bolt, which is why Storm has some convenience methods for using a default stream name.
When you do .shuffleGrouping("checkingbolt"), you are using one of these convenience methods, effectively saying "I want this bolt to consume the default stream coming out of the checkingbolt". There is an overloaded version of this method you can use if you want to explicitly name the stream, but it's only useful if you have multiple streams coming out of the same bolt.
When you do ofd.declareStream(cassandraBoltStream, new Fields(new String[]{"jsonFields"}));, you are saying the bolt will emit on a stream named "cassandraBoltStream". This is probably not what you want to do, you want to declare that it will emit on the default stream. You do this by using the ofd.declare method instead.
Refer to the documentation for more details.
The following java 8 stream doesn't have any terminal operation. Isn't the following block supposed to be lazy, since I only have intermediate operation and hasn't been operated yet by a terminal operation. I get "stream has already been operated upon or closed" when I run this block. See https://ideone.com/naR7GB
Stream<String> s = Stream.of("A", "B");
s.map(String::toUpperCase);
s.map(String::toLowerCase);
Stack Trace:
java.lang.IllegalStateException: stream has already been operated upon or closed
at java.util.stream.AbstractPipeline.<init>(AbstractPipeline.java:203)
at java.util.stream.ReferencePipeline.<init>(ReferencePipeline.java:94)
at java.util.stream.ReferencePipeline$StatelessOp.<init>(ReferencePipeline.java:618)
at java.util.stream.ReferencePipeline$3.<init>(ReferencePipeline.java:187)
at java.util.stream.ReferencePipeline.map(ReferencePipeline.java:186)
You need to apply the second map() to the mapped instance:
s.map(String::toUpperCase).map(String::toLowerCase);
Or
Stream<String> s = Stream.of("A", "B");
Stream<String> s2 = s.map(String::toUpperCase);
Stream<String> s3 = s2.map(String::toLowerCase);
As you can only do 1 operation on the same stream instance.
But remember, you can only consume s once! So either you consume s2 or s3, you can not consume both. That's why we usually write a chained call, since it does not make sense to keep the intermediate objects.