I want to read the last message written to a SingleChronicleQueue instance.
While "chronicle.createTailer().direction(TailerDirection.BACKWARD).toEnd()" works while we are on the same cycle as the last written message, as soon as we are in one of the future cycles (compared to the last written message), tailer.readDocument(...) always returns "false".
I have implemented a test to reproduce the issue based on the SingleChronicleQueueTest.testForwardFollowedBackBackwardTailer test:
#Test
public void testForwardFollowedBackBackwardTailer() {
File tmpDir = getTmpDir();
// when "forwardToFuture" flag is set, go one day to the future
AtomicBoolean forwardToFuture = new AtomicBoolean(false);
TimeProvider timeProvider = () -> forwardToFuture.get()
? System.currentTimeMillis() + Duration.ofDays(1).toMillis()
: System.currentTimeMillis();
try (RollingChronicleQueue chronicle = SingleChronicleQueueBuilder.binary(tmpDir)
.rollCycle(TEST2_DAILY)
.wireType(this.wireType)
.timeProvider(timeProvider)
.build()) {
ExcerptAppender appender = chronicle.acquireAppender();
int entries = chronicle.rollCycle().defaultIndexSpacing() + 2;
for (int i = 0; i < entries; i++) {
int finalI = i;
appender.writeDocument(w -> w.writeEventName("hello").text("world" + finalI));
}
// go to the future (and to the next roll cycle)
forwardToFuture.set(true);
for (int i = 0; i < 3; i++) {
readForward(chronicle, entries);
readBackward(chronicle, entries);
}
}
}
After these changes to the "testForwardFollowedBackBackwardTailer" method,
the test fails at assertTrue(documentContext.isPresent()) line in the "readBackward" method.
Is there any way to reliably read the last message from SingleChronicleQueue instance, no matter how far in the past the last message is? (without reading through the whole chronicle instance from the start)
Which version are you using as this should work. If you have the latest version, it is a bug.
Can you submit a pull request with this unit test?
Related
This is related to the previous question I have posted. I think that while it is related, it might be different enough to warrant its own question.
The code used is:
public static void main(String[] args){
ChronicleQueue QUEUE = SingleChronicleQueueBuilder.single("./chronicle/roll")
.rollCycle(RollCycles.MINUTELY).build();
ExcerptTailer TAILER = QUEUE.createTailer();
ArrayList<Long> seqNums = new ArrayList<>();
//this reads all roll cycles starting from first and carries on to next rollcycle.
//busy spinner that spins non-stop trying to read from queue
int currentCycle = TAILER.cycle();
System.out.println(TAILER.cycle());
while(true){
//if it moves over to new cycle, start over the sequencing (fresh start for next day)
int cycleCheck = TAILER.cycle();
long indexCheck = TAILER.index();
System.out.println(cycleCheck);
System.out.println("idx: "+indexCheck);
if (currentCycle != cycleCheck){
LOGGER.warn("Changing to new roll cycle, from: "+currentCycle+" to: "+cycleCheck+". Clearing list of size "+seqNums.size());
seqNums.clear(); // this may cause a memory issue see: https://stackoverflow.com/a/6961397/16034206
currentCycle = cycleCheck;
TAILER.moveToCycle(currentCycle);
cycleCheck = TAILER.cycle();
indexCheck = TAILER.index();
System.out.println("cycle: "+cycleCheck);
System.out.println("idx: "+indexCheck);
}
//TODO:2nd option, on starting the chronicle runner, always move to end, and wait for next day's cycle to start
if (TAILER.readDocument(w -> w.read("packet").marshallable(
m -> {
long seqNum = m.read("seqNum").readLong();
int size = seqNums.size();
if (size > 0){
int idx;
if ((idx = seqNums.indexOf(seqNum)) >= 0){
LOGGER.warn("Duplicate seqNum: "+seqNum+" at idx: "+idx);
}else{
long previous = seqNums.get(size-1);
long gap = seqNum - previous;
if (Math.abs(gap) > 1L){
LOGGER.error("sequence gap at seqNum: "+previous+" and "+seqNum+"! Gap of "+gap);
}
}
}
seqNums.add(seqNum);
System.out.println(m.read("moldUdpHeader").text());
}
))){ ; }else { TAILER.close(); break; }
//breaks out from spinner if nothing to be read.
//a named tailer could be used to pick up from where is left off.
}
}
At this point, I have 2 roll cycle files, one ends in a sequence Number of 1001, then the next file starts with seqNum of 0. Using the while loop, it would read both files, but there is an if statement to check that the cycle has changed or not and reset accordingly.
The output is as follows:
The output when .moveToCycle() is commented:
As you can see, the first index of the next file is read as part of previous file, but when I use the TAILER.moveToCycle(currentCycle) it moves to start of the next file again, but it has a different index this time. If you comment this line of code out, it will not re-read the entry with seqNum of 0.
Alright, I tested the following and it works just fine. How it works is that it reads the value (I am assuming the internal workings would only shift the index and cycle after it reads an incoming value), then tests for cycle change (from testing before reading to testing after reading). This is probably how one should iterate over multiple roll cycle files, while keeping track of when it roll overs.
Also, note that previously it prints cycle and index before printing the object, now it prints object before printing cycle and index, so its likely that you may misread it and assume it doesn't work if you try to test the following code.
public static void main(String[] args){
ChronicleQueue QUEUE = SingleChronicleQueueBuilder.single("./chronicle/roll")
.rollCycle(RollCycles.FIVE_MINUTELY).build();
ExcerptTailer TAILER = QUEUE.createTailer();
ArrayList<Long> seqNums = new ArrayList<>();
//this reads all roll cycles starting from first and carries on to next roll cycle.
//busy spinner that spins non-stop trying to read from queue
int currentCycle = TAILER.cycle();
System.out.println(TAILER.cycle());
AtomicLong seqNum = new AtomicLong();
while(true){
if (TAILER.readDocument(w -> w.read("packet").marshallable(
m -> {
long val = m.read("seqNum").readLong();
seqNum.set(val);
System.out.println(m.read("moldUdpHeader").text());
}
))){
//if it moves over to new cycle, start over the sequencing (fresh start for next day)
int cycleCheck = TAILER.cycle();
long indexCheck = TAILER.index();
System.out.println("cycle: "+cycleCheck);
System.out.println("idx: "+indexCheck);
if (currentCycle != cycleCheck){
LOGGER.warn("Changing to new roll cycle, from: "+currentCycle+" to: "+cycleCheck+". Clearing list of size "+seqNums.size());
seqNums.clear(); // this may cause a memory issue see: https://stackoverflow.com/a/6961397/16034206
currentCycle = cycleCheck;
}
int size = seqNums.size();
long val = seqNum.get();
if (size > 0){
int idx;
if ((idx = seqNums.indexOf(seqNum)) >= 0){
LOGGER.warn("Duplicate seqNum: "+seqNum+" at idx: "+idx);
}else{
long previous = seqNums.get(size-1);
long gap = val - previous;
if (Math.abs(gap) > 1L){
LOGGER.error("sequence gap at seqNum: "+previous+" and "+seqNum+"! Gap of "+gap);
}
}
}
seqNums.add(val);
} else { TAILER.close(); break; }
//breaks out from spinner if nothing to be read.
//a named tailer could be used to pick up from where is left off.
}
}
sometime based on some condition it may want to jump (or move forward) a few steps inside the for loop,
how to do it is kolin?
a simplified use case:
val datArray = arrayOf(1, 2, 3......)
/**
* start from the index to process some data, return how many data has been
consumed
*/
fun processData_1(startIndex: Int) : Int {
// process dataArray starting from the index of startIndex
// return the data it has processed
}
fun processData_2(startIndex: Int) : Int {
// process dataArray starting from the index of startIndex
// return the data it has processed
}
in Java it could be:
for (int i=0; i<datArray.lenght-1; i++) {
int processed = processData_1(i);
i += processed; // jump a few steps for those have been processed, then start 2nd process
if (i<datArray.lenght-1) {
processed = processData_2(i);
i += processed;
}
}
How to do it in kotlin?
for(i in array.indices){
val processed = processData(i);
// todo
}
With while:
var i = 0
while (i < datArray.length - 1) {
var processed = processData_1(i)
i += processed // jump a few steps for those have been processed, then start 2nd process
if (i < datArray.length - 1) {
processed = processData_2(i)
i += processed
}
i++
}
You can do that with continue as stated in the Kotlin docs here: https://kotlinlang.org/docs/reference/returns.html
Example:
val names = arrayOf("james", "john", "jim", "jacob", "johan")
for (name in names) {
if(name.length <= 4) continue
println(name)
}
This would only print names longer than 4 characters (as it skips names with a length of 4 and below)
Edit: this only skips one iteration at a time. So if you want to skip multiple, you could store the process state somewhere else and check the status for each iteration.
I have a List of a custom CallRecord objects
public class CallRecord {
private String callId;
private String aNum;
private String bNum;
private int seqNum;
private byte causeForOutput;
private int duration;
private RecordType recordType;
.
.
.
}
There are two logical conditions and the output of each is:
Highest seqNum, sum(duration)
Highest seqNum, sum(duration), highest causeForOutput
As per my understanding, Stream.max(), Collectors.summarizingInt() and so on will either require several iterations for the above result. I also came across a thread suggesting custom collector but I am unsure.
Below is the simple, pre-Java 8 code that is serving the purpose:
if (...) {
for (CallRecord currentRecord : completeCallRecords) {
highestSeqNum = currentRecord.getSeqNum() > highestSeqNum ? currentRecord.getSeqNum() : highestSeqNum;
sumOfDuration += currentRecord.getDuration();
}
} else {
byte highestCauseForOutput = 0;
for (CallRecord currentRecord : completeCallRecords) {
highestSeqNum = currentRecord.getSeqNum() > highestSeqNum ? currentRecord.getSeqNum() : highestSeqNum;
sumOfDuration += currentRecord.getDuration();
highestCauseForOutput = currentRecord.getCauseForOutput() > highestCauseForOutput ? currentRecord.getCauseForOutput() : highestCauseForOutput;
}
}
Your desire to do everything in a single iteration is irrational. You should strive for simplicity first, performance if necessary, but insisting on a single iteration is neither.
The performance depends on too many factors to make a prediction in advance. The process of iterating (over a plain collection) itself is not necessarily an expensive operation and may even benefit from a simpler loop body in a way that makes multiple traversals with a straight-forward operation more efficient than a single traversal trying to do everything at once. The only way to find out, is to measure using the actual operations.
Converting the operation to Stream operations may simplify the code, if you use it straight-forwardly, i.e.
int highestSeqNum=
completeCallRecords.stream().mapToInt(CallRecord::getSeqNum).max().orElse(-1);
int sumOfDuration=
completeCallRecords.stream().mapToInt(CallRecord::getDuration).sum();
if(!condition) {
byte highestCauseForOutput = (byte)
completeCallRecords.stream().mapToInt(CallRecord::getCauseForOutput).max().orElse(0);
}
If you still feel uncomfortable with the fact that there are multiple iterations, you could try to write a custom collector performing all operations at once, but the result will not be better than your loop, neither in terms of readability nor efficiency.
Still, I’d prefer avoiding code duplication over trying to do everything in one loop, i.e.
for(CallRecord currentRecord : completeCallRecords) {
int nextSeqNum = currentRecord.getSeqNum();
highestSeqNum = nextSeqNum > highestSeqNum ? nextSeqNum : highestSeqNum;
sumOfDuration += currentRecord.getDuration();
}
if(!condition) {
byte highestCauseForOutput = 0;
for(CallRecord currentRecord : completeCallRecords) {
byte next = currentRecord.getCauseForOutput();
highestCauseForOutput = next > highestCauseForOutput? next: highestCauseForOutput;
}
}
With Java-8 you can resolved it with a Collector with no redudant iteration.
Normally, we can use the factory methods from Collectors, but in your case you need to implement a custom Collector, that reduces a Stream<CallRecord> to an instance of SummarizingCallRecord which cotains the attributes you require.
Mutable accumulation/result type:
class SummarizingCallRecord {
private int highestSeqNum = 0;
private int sumDuration = 0;
// getters/setters ...
}
Custom collector:
BiConsumer<SummarizingCallRecord, CallRecord> myAccumulator = (a, callRecord) -> {
a.setHighestSeqNum(Math.max(a.getHighestSeqNum(), callRecord.getSeqNum()));
a.setSumDuration(a.getSumDuration() + callRecord.getDuration());
};
BinaryOperator<SummarizingCallRecord> myCombiner = (a1, a2) -> {
a1.setHighestSeqNum(Math.max(a1.getHighestSeqNum(), a2.getHighestSeqNum()));
a1.setSumDuration(a1.getSumDuration() + a2.getSumDuration());
return a1;
};
Collector<CallRecord, SummarizingCallRecord, SummarizingCallRecord> myCollector =
Collector.of(
() -> new SummarizinCallRecord(),
myAccumulator,
myCombiner,
// Collector.Characteristics.CONCURRENT/IDENTITY_FINISH/UNORDERED
);
Execution example:
List<CallRecord> callRecords = new ArrayList<>();
callRecords.add(new CallRecord(1, 100));
callRecords.add(new CallRecord(5, 50));
callRecords.add(new CallRecord(3, 1000));
SummarizingCallRecord summarizingCallRecord = callRecords.stream()
.collect(myCollector);
// Result:
// summarizingCallRecord.highestSeqNum = 5
// summarizingCallRecord.sumDuration = 1150
You don't need and should not implement the logic by Stream API because the tradition for-loop is simple enough and the Java 8 Stream API can't make it simpler:
int highestSeqNum = 0;
long sumOfDuration = 0;
byte highestCauseForOutput = 0; // just get it even if it may not be used. there is no performance hurt.
for(CallRecord currentRecord : completeCallRecords) {
highestSeqNum = Math.max(highestSeqNum, currentRecord.getSeqNum());
sumOfDuration += currentRecord.getDuration();
highestCauseForOutput = Math.max(highestCauseForOutput, currentRecord.getCauseForOutput());
}
// Do something with or without highestCauseForOutput.
I am trying to instrument java synchronized block using ASM. The problem is that after instrumenting, the execution time of the synchronized block takes more time. Here it increases from 2 msecs to 200 msecs on Linux box.
I am implementing this by identifying the MonitorEnter and MonitorExit opcode.
I try to instrument at three level 1. just before the MonitorEnter 2. after MonitorEnter 3. Before MonitorExit.
1 and 3 together works fine, but when i do 2, the execution time increase dramatically.
Even if we instrument another single SOP statement, which is intended to be executed just once, it give higher values.
Here the sample code (prime number, 10 loops):
for(int w=0;w<10;w++){
synchronized(s){
long t1 = System.currentTimeMillis();
long num = 2000;
for (long i = 1; i < num; i++) {
long p = i;
int j;
for (j = 2; j < p; j++) {
long n = p % i;
}
}
long t2 = System.currentTimeMillis();
System.out.println("Time>>>>>>>>>>>> " + (t2-t1) );
}
Here the code for instrumention (here System.currentMilliSeconds() gives the time at which instrumention happened, its no the measure of execution time, the excecution time is from obove SOP statement):
public void visitInsn(int opcode)
{
switch(opcode)
{
// Scenario 1
case 194:
visitFieldInsn(Opcodes.GETSTATIC, "java/lang/System", "out", "Ljava/io /PrintStream;");
visitLdcInsn("TIME Arrive: "+System.currentTimeMillis());
visitMethodInsn(Opcodes.INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V");
break;
// scenario 3
case 195:
visitFieldInsn(Opcodes.GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;");
visitLdcInsn("TIME exit : "+System.currentTimeMillis());
visitMethodInsn(Opcodes.INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V");
break;
}
super.visitInsn(opcode);
// scenario 2
if(opcode==194)
{
visitFieldInsn(Opcodes.GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;");
visitLdcInsn("TIME enter: "+System.currentTimeMillis());
visitMethodInsn(Opcodes.INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V");
}
}
I am not able to find the reason why it is happening and how t correct it.
Thanks in advance.
The reason lies in the internals of the JVM that you were using for running the code. I assume that this was a HotSpot JVM but the answers below are equally right for most other implementations.
If you trigger the following code:
int result = 0;
for(int i = 0; i < 1000; i++) {
result += i;
}
This will be translated directly into Java byte code by the Java compiler but at run time the JVM will easily see that this code is not doing anything. Executing this code will have no effect on the outside (application) world, so why should the JVM execute it? This consideration is exactly what compiler optimization does for you.
If you however trigger the following code:
int result = 0;
for(int i = 0; i < 1000; i++) {
System.out.println(result);
}
the Java runtime cannot optimize away your code anymore. The whole loop must always run since the System.out.println(int) method is always doing something real such that your code will run slower.
Now let's look at your example. In your first example, you basically write this code:
synchronized(s) {
// do nothing useful
}
This entire code block can easily be removed by the Java run time. This means: There will be no synchronization! In the second example, you are writing this instead:
synchronized(s) {
long t1 = System.currentTimeMillis();
// do nothing useful
long t2 = System.currentTimeMillis();
System.out.println("Time>>>>>>>>>>>> " + (t2-t1));
}
This means that the effective code might be look like this:
synchronized(s) {
long t1 = System.currentTimeMillis();
long t2 = System.currentTimeMillis();
System.out.println("Time>>>>>>>>>>>> " + (t2-t1));
}
What is important here is that this optimized code will be effectively synchronized what is an important difference with respect to execution time. Basically, you are measuring the time it costs to synchronize something (and even that might be optimized away after a couple of runs if the JVM realized that the s is not locked elsewhere in your code (buzzword: temporary optimization with the possibility of deoptimization if loaded code in the future will also synchronize on s).
You should really read this:
http://www.ibm.com/developerworks/java/library/j-jtp02225/
http://www.ibm.com/developerworks/library/j-jtp12214/
Your test for example misses a warm-up, such that you are also measuring how much time the JVM will use for byte code to machine code optimization.
On a side note: Synchronizing on a String is almost always a bad idea. Your strings might be or might not be interned what means that you cannot be absolutely sure about their identity. This means, that synchronization might or might not work and you might even inflict synchronization of other parts of your code.
I can't seem to find a great way to express the following in Xtend without resorting to a while loop:
for(int i = 0; i < 3; i++){
println("row ");
}
println("your boat");
So, I guess my question has two parts:
Is there a better way to do the above? I didn't see anything promising in their documenation
A large portion of the features the language has are just Xtend library extensions (and they're great!). Is there range() functionality à la Python that I don't know about?
I ended up rolling my own and got something like the following:
class LanguageUtil {
def static Iterable<Integer> range(int stop) {
range(0, stop)
}
def static Iterable<Integer> range(int start, int stop) {
new RangeIterable(start, stop, 1)
}
def static Iterable<Integer> range(int start, int stop, int step) {
new RangeIterable(start, stop, step)
}
}
// implements Iterator and Iterable which is bad form.
class RangeIterable implements Iterator<Integer>, Iterable<Integer> {
val int start
val int stop
val int step
var int current
new(int start, int stop, int step) {
this.start = start;
this.stop = stop;
this.step = step
this.current = start
}
override hasNext() {
current < stop
}
override next() {
val ret = current
current = current + step
ret
}
override remove() {
throw new UnsupportedOperationException("Auto-generated function stub")
}
/**
* This is bad form. We could return a
* new RangeIterable here, but that seems worse.
*/
override iterator() {
this
}
}
The exact replacement for
for(int i = 0; i < 3; i++){
println("row ");
}
is
for (i : 0 ..< 3) {
println("row ")
}
Notice the exclusive range operator: ..<
Also you can doing it more idiomatically with
(1..3).forEach[println("row")]
Very new to Xtend but man it makes programming on the jvm awesome.
To me a range-based forEach implies the range is somehow meaningful. For looping a specific number of times with no iteration variable, I find Ruby's times loop expresses the intent more clearly:
3.times [|println("row")]
Sadly it's not a part of IntegerExtensions, but the implementation is trivial:
def static times(int iterations, Runnable runnable) {
if (iterations < 0) throw new IllegalArgumentException(
'''Can't iterate negative («iterations») times.''')
for (i: 0 ..< iterations) runnable.run()
}
Heh, I found the answer a little while later:
for(i: 1..3) {
println("row ")
}
Since Xtend 2.6, we also support the "traditional" for-loop, just like in Java.
There is actually a version of forEach() that accepts a lambda with two parameters.
It is useful if you need to access the iteration index within the loop.
(10..12).forEach[ x, i | println('''element=«x» index=«i»''')]
prints:
element=10 index=0
element=11 index=1
element=12 index=2