jBcrypt: BCrypt.checkpw suddenly takes ~30 times as long - bcrypt
In our web-application, we use jBcrypt for hashing passwords. We use 13 log_rounds when hashing the password.
Normally, BCrypt.checkpw() takes about 1 second. But from time to time (after a few days), it suddenly starts getting slow and takes almost 30 seconds from that time on and does not recover to normal speed.Restarting Tomcat is the only things that helps here.
I wouldn't wonder if that happened from time to time, for example if there is a high CPU load or a GC is running. But that's not the case, it just suddenly starts getting slow. Only the login-process is affected, the rest of the application is still fast.
We also do not have any determinable memory leaks or other performance issues. It's just BCrypt.checkpw() that is slow.
A thread-dump shows that the time is consumed by BCrypt.checkpw and subsequent method calls, especially BCrypt.encipher:
Thread 8597: (state = IN_JAVA)
- org.mindrot.jbcrypt.BCrypt.encipher(int[], int) #bci=0, line=490 (Interpreted frame)
- org.mindrot.jbcrypt.BCrypt.key(byte[]) #bci=122, line=562 (Interpreted frame)
- org.mindrot.jbcrypt.BCrypt.crypt_raw(byte[], byte[], int) #bci=89, line=629 (Compiled frame)
- org.mindrot.jbcrypt.BCrypt.hashpw(java.lang.String, java.lang.String) #bci=226, line=692 (Interpreted frame)
- org.mindrot.jbcrypt.BCrypt.checkpw(java.lang.String, java.lang.String) #bci=3, line=763 (Interpreted frame)
I've only found one similar issue here on SO, but multiple Classloaders can not be an issue in our case:
Variable and degrading performance when using jbcrypt
Does anyone have an idea what's happening here?
Related
Tensorflow-GPU performance decreases each epoch on Windows machine
I'm seeing decreasing performance with each subsequent epoch running Tensorflow-GPU with Keras on Windows. I'm training a 16-layer CNN and load my data using tf.data. The first epoch performs as well as expected: ~1hr training time. The CPU and GPU CUDA load is at 85-90%. Temperatures are reasonable (CPU between 65-70C and GPU at 70C) -- no thermal throttling occurs. But by the second epoch, GPU CUDA load inexplicably drops to 33-50%. CPU load appears reduced as well, to around 65-75%. I don't see any big change in disk I/O speeds (the data is being loaded from an NVMe SSD). I don't think this is a hardware issue. I'm using an RTX 3090 and a 4-core i5-6500 CPU and as mentioned, the performance on the first epoch is quite good without enough thermal headroom to continue. My tf.data pipeline looks like this: # Construct tf.data.Dataset paths_ds = tf.data.Dataset.from_tensor_slices(image_paths) if shuffle: paths_ds = paths_ds.shuffle(buffer_size=len(image_paths), reshuffle_each_iteration=True) # reshuffle each epoch dataset = paths_ds.map( lambda path: (self.load_image(path, scale_range, crop_size, augment), self.get_onehot_label(path, tf_class_index_by_image, self.num_classes)), num_parallel_calls=tf.data.experimental.AUTOTUNE ) dataset = dataset.batch(batch_size) dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE) A map and a prefetch, both using AUTOTUNE. I suspect maybe AUTOTUNE is the problem but I'm not sure how to debug this and determine how the number of threads and prefetch buffer size might be evolving over time. There is no documentation on how AUTOTUNE works. EDIT: I did some profiling and have discovered that from epoch 2 onwards, Iterator::MapAndBatch is consuming significant time (it hard to even see in Epoch 1 because the data it collects is always ready). The cause appears to be some sort of perplexing slowness in my "load_image" function. Each operation it performs (read file, decode JPEG, convert to float, resize) is followed by a pause in Epoch 2 onwards and this becomes slower over time. For example, Epoch 1 with everything functioning correctly: Note the read is immediately followed by a decode, etc. Now in epoch 2: What could be causing this? Attempting to elevate the priority of the process using Process Explorer has no effect.
WinDbg runaway command output explained
I have a production CPU issue, after days of regular activity suddenly the CPU starts to peak. I've saved the dump file and run the !runaway command to get the list of highest CPU time consuming threads. the output is below: User Mode Time Thread Time 21:110 0 days 10:51:39.781 19:f84 0 days 10:41:59.671 5:cc4 0 days 0:53:25.343 48:74 0 days 0:34:20.140 47:1670 0 days 0:34:09.812 13:460 0 days 0:32:57.640 8:14d4 0 days 0:19:30.546 7:d90 0 days 0:03:15.000 23:1520 0 days 0:02:21.984 22:ca0 0 days 0:02:08.375 24:72c 0 days 0:02:01.640 29:10ac 0 days 0:01:58.671 27:1088 0 days 0:01:44.390 As you can see, the output shows I've 2 threads: 21 & 19, that consumes more than 20 hours of CPU time combined ,I was able to track the callstack of 1 of those threads like so: ~21s !CLRStack the output doesn't matter at the moment, let's call it the "X callstack" What I would like, is an explanation about the !runaway command output. from what I understand, a dump file is a snapshot of the current state of the application. so my questions are: How can the runaway command shows 10:51 hours value for thread 21, when the dumping process only took a few seconds? Does it mean that the specific "instance" of the X callstack I've found with the !CLRStack command is hang more than 10 hours? or it's the total time the 21 thread executed his whole X callstacks executions? If so, it seems strange that the 21 thread responsible for so many executions of the X callstacks. As I know the origin is a web request (the runtime should assign a random thread for each call) I've a speculation that may answer those 2 questions: Maybe the windbg calculate the time by taking the thread callstack actual time and dividing it by the scope of the dumping process, so if for example the specific execution of the X callstack took 1 second and the whole dumping process took 3 seconds (33%), while the process was running for total of 24 hours the output will show: 8 hours (33% of 24 hours) Am I right, or completely got it wrong?
This answer is intended to be comprehensible for the OP. It's not intended to be correct into all bits and bytes. [...] and dividing it by the scope of the dumping process [...] This understanding is probably the root of all evil: dumping a process only gives you the state of the process at a certain point in time. The duration of dumping the process is 0.0 seconds, since all threads are suspended during the operation. (so, relative time for your process, nothing has changed and time is standing still; of course wall clock time changes) You are thinking of dumping a process as monitoring it over a longer period of time, which is not the case. Dumping a process just takes time because it involves disk activity etc. So no, there is no "scope" and thus you cannot (it's really hard) measure performance issues with crash dumps. How can the runaway command shows 10:51 hours value for thread 21, [...] How can your C# program know how long the program is running if you only have a timer event that fires every second? The answer is: it uses a variable and increases the value. That's roughly how Windows does it. Windows is responsible for thread scheduling and each time it re-schedules threads, it updates a variable that contains the thread time. When writing the crash dump, the information that was collected by the OS long time ago already, is included in the crash dump. [...] when the dumping process only took a few seconds? Since the crash dump is taken by a thread of WinDbg, the time for that is accounted on that thread. You would need to debug WinDbg and do !runaway on a WinDbg thread to see how much CPU time that took. Potentially a nice exercise and the .dbgdbg (debug the debugger) command may be new to you; other than that, this particular case is not really helpful. Does it mean that the specific "instance" of the X callstack I've found with the !CLRStack command is hang more than 10 hours? No. It means that at the point in time when you created the crash dump, that specific method was executed. Not more, not less. This information is unrelated to !runaway, because the thread may have been doing something totally different for a long time, but that ended just a moment ago. or it's the total time the 21 thread executed his whole X callstacks executions? No. A crash dump does not contain such detailed performance data. You need a performance profiler like JetBrains dotTrace do get that information. A profiler will look at callstacks very often, then aggregate identical call stacks and derive CPU time per call stack.
Python 3 multiprocessing: optimal chunk size
How do I find the optimal chunk size for multiprocessing.Pool instances? I used this before to create a generator of n sudoku objects: processes = multiprocessing.cpu_count() worker_pool = multiprocessing.Pool(processes) sudokus = worker_pool.imap_unordered(create_sudoku, range(n), n // processes + 1) To measure the time, I use time.time() before the snippet above, then I initialize the pool as described, then I convert the generator into a list (list(sudokus)) to trigger generating the items (only for time measurement, I know this is nonsense in the final program), then I take the time using time.time() again and output the difference. I observed that the chunk size of n // processes + 1 results in times of around 0.425 ms per object. But I also observed that the CPU is only fully loaded the first half of the process, in the end the usage goes down to 25% (on an i3 with 2 cores and hyper-threading). If I use a smaller chunk size of int(l // (processes**2) + 1) instead, I get times of around 0.355 ms instead and the CPU load is much better distributed. It just has some small spikes down to ca. 75%, but stays high for much longer part of the process time before it goes down to 25%. Is there an even better formula to calculate the chunk size or a otherwise better method to use the CPU most effective? Please help me to improve this multiprocessing pool's effectiveness.
This answer provides a high level overview. Going into detais, each worker is sent a chunk of chunksize tasks at a time for processing. Every time a worker completes that chunk, it needs to ask for more input via some type of inter-process communication (IPC), such as queue.Queue. Each IPC request requires a system call; due to the context switch it costs anywhere in the range of 1-10 μs, let's say 10 μs. Due to shared caching, a context switch may hurt (to a limited extent) all cores. So extremely pessimistically let's estimate the maximum possible cost of an IPC request at 100 μs. You want the IPC overhead to be immaterial, let's say <1%. You can ensure that by making chunk processing time >10 ms if my numbers are right. So if each task takes say 1 μs to process, you'd want chunksize of at least 10000. The main reason not to make chunksize arbitrarily large is that at the very end of the execution, one of the workers might still be running while everyone else has finished -- obviously unnecessarily increasing time to completion. I suppose in most cases a delay of 10 ms is a not a big deal, so my recommendation of targeting 10 ms chunk processing time seems safe. Another reason a large chunksize might cause problems is that preparing the input may take time, wasting workers capacity in the meantime. Presumably input preparation is faster than processing (otherwise it should be parallelized as well, using something like RxPY). So again targeting the processing time of ~10 ms seems safe (assuming you don't mind startup delay of under 10 ms). Note: the context switches happen every ~1-20 ms or so for non-real-time processes on modern Linux/Windows - unless of course the process makes a system call earlier. So the overhead of context switches is no more than ~1% without system calls. Whatever overhead you're creating due to IPC is in addition to that.
Nothing will replace the actual time measurements. I wouldn't bother with a formula and try a constant such as 1, 10, 100, 1000, 10000 instead and see what works best in your case.
Testing Erlang function performance with timer
I'm testing the performance of a function in a tight loop (say 5000 iterations) using timer:tc/3:
{Duration_us, _Result} = timer:tc(M, F, [A])
This returns both the duration (in microseconds) and the result of the function. For argument's sake the duration is N microseconds.
I then perform a simple average calculation on the results of the iterations.
If I place a timer:sleep(1) function call before the timer:tc/3 call, the average duration for all the iterations is always > the average without the sleep:
timer:sleep(1),
timer:tc(M, F, [A]).
This doesn't make much sense to me as the timer:tc/3 function should be atomic and not care about anything that happened before it.
Can anyone explain this strange functionality? Is it somehow related to scheduling and reductions?
Do you mean like this:
4> foo:foo(10000).
Where:
-module(foo).
-export([foo/1, baz/1]).
foo(N) -> TL = bar(N), {TL,sum(TL)/N} .
bar(0) -> [];
bar(N) ->
timer:sleep(1),
{D,_} = timer:tc(?MODULE, baz, [1000]),
[D|bar(N-1)]
.
baz(0) -> ok;
baz(N) -> baz(N-1).
sum([]) -> 0;
sum([H|T]) -> H + sum(T).
I tried this, and it's interesting. With the sleep statement the mean time returned by timer:tc/3 is 19 to 22 microseconds, and with the sleep commented out, the average drops to 4 to 6 microseconds. Quite dramatic!
I notice there are artefacts in the timings, so events like this (these numbers being the individual microsecond timings returned by timer:tc/3) are not uncommon:
---- snip ----
5,5,5,6,5,5,5,6,5,5,5,6,5,5,5,5,4,5,5,5,5,5,4,5,5,5,5,6,5,5,
5,6,5,5,5,5,5,6,5,5,5,5,5,6,5,5,5,6,5,5,5,5,5,5,5,5,5,5,4,5,
5,5,5,6,5,5,5,6,5,5,7,8,7,8,5,6,5,5,5,6,5,5,5,5,4,5,5,5,5,
14,4,5,5,4,5,5,4,5,4,5,5,5,4,5,5,4,5,5,4,5,4,5,5,5,4,5,5,4,
5,5,4,5,4,5,5,4,4,5,5,4,5,5,4,4,4,4,4,5,4,5,5,4,5,5,5,4,5,5,
4,5,5,4,5,4,5,5,5,4,5,5,4,5,5,4,5,4,5,4,5,4,5,5,4,4,4,4,5,4,
5,5,54,22,26,21,22,22,24,24,32,31,36,31,33,27,25,21,22,21,
24,21,22,22,24,21,22,21,24,21,22,22,24,21,22,21,24,21,22,21,
23,27,22,21,24,21,22,21,24,22,22,21,23,22,22,21,24,22,22,21,
24,21,22,22,24,22,22,21,24,22,22,22,24,22,22,22,24,22,22,22,
24,22,22,22,24,22,22,21,24,22,22,21,24,21,22,22,24,22,22,21,
24,21,23,21,24,22,23,21,24,21,22,22,24,21,22,22,24,21,22,22,
24,22,23,21,24,21,23,21,23,21,21,21,23,21,25,22,24,21,22,21,
24,21,22,21,24,22,21,24,22,22,21,24,22,23,21,23,21,22,21,23,
21,22,21,23,21,23,21,24,22,22,22,24,22,22,41,36,30,33,30,35,
21,23,21,25,21,23,21,24,22,22,21,23,21,22,21,24,22,22,22,24,
22,22,21,24,22,22,22,24,22,22,21,24,22,22,21,24,22,22,21,24,
22,22,21,24,21,22,22,27,22,23,21,23,21,21,21,23,21,21,21,24,
21,22,21,24,21,22,22,24,22,22,22,24,21,22,22,24,21,22,21,24,
21,23,21,23,21,22,21,23,21,23,22,24,22,22,21,24,21,22,22,24,
21,23,21,24,21,22,22,24,21,22,22,24,21,22,21,24,21,22,22,24,
22,22,22,24,22,22,21,24,22,21,21,24,21,22,22,24,21,22,22,24,
24,23,21,24,21,22,24,21,22,21,23,21,22,21,24,21,22,21,32,31,
32,21,25,21,22,22,24,46,5,5,5,5,5,4,5,5,5,5,6,5,5,5,5,5,5,4,
6,5,5,5,6,5,5,5,5,5,5,5,6,5,5,5,5,4,5,4,5,5,5,5,6,5,5,5,5,5,
5,5,6,5,5,5,5,5,5,5,6,5,5,5,5,4,6,4,6,5,5,5,5,5,5,4,6,5,5,5,
5,4,5,5,5,5,5,5,6,5,5,5,5,4,5,5,5,5,5,5,6,5,5,5,5,5,5,5,6,5,
5,5,5,4,5,5,6,5,5,5,6,5,5,5,5,5,5,5,6,5,5,5,6,5,5,5,5,5,5,5,
6,5,5,5,5,4,5,4,5,5,5,5,6,5,5,5,5,5,5,4,5,4,5,5,5,5,5,6,5,5,
5,5,4,5,4,5,5,5,5,6,5,5,5,5,5,5,5,6,5,5,5,5,5,5,5,6,5,5,5,5,
---- snip ----
I assume this is the effect you are referring to, though when you say always > N, is it always, or just mostly? Not always for me anyway.
The above results extract was without the sleep. Typically when using sleep timer:tc/3 returns low times like 4 or 5 most of the time without the sleep, but sometimes big times like 22, and with the sleep in place it's usually big times like 22, with occasional batches of low times.
It's certainly not obvious why this would happen, since sleep really just means yield. I wonder if all this is not down to the CPU cache. After all, especially on a machine that's not busy, one might expect the case without the sleep to execute most of the code all in one go without it getting moved to another core, without doing so much else with the core, thus making the most out of the caches... but when you sleep, and thus yield, and come back later, the chances of cache hits might be considerably less.
Measuring performance is a complex task especially on new HW and in modern OS. There are many things which can fiddle with your result. First thing, you are not alone. It is when you measure on your desktop or notebook, there can be other processes which can interfere with your measurement including system ones. Second thing, there is HW itself. Moder CPUs have many cool features which control performance and power consumption. They can boost performance for a short time before overheat, they can boost performance when there is not work on other CPUs on the same chip or other hyper thread on the same CPU. On another hand, they can enter power saving mode when there is not enough work and CPU doesn't react fast enough to the sudden change. It is hard to tell if it is your case, but it is naive to thing previous work or lack of it can't affect your measurement. You should always take care to measure in steady state for long enough time (seconds at least) and remove as much as possible other things which could affect your measurement. (And do not forget GC in Erlang as well.)
Java - Repeated function call reduces execution time
I have this following code
public class BenchMark {
public static void main(String args[]) {
doLinear();
doLinear();
doLinear();
doLinear();
}
private static void doParallel() {
IntStream range = IntStream.range(1, 6).parallel();
long startTime = System.nanoTime();
int reduce = range
.reduce((a, item) -> a * item).getAsInt();
long endTime = System.nanoTime();
System.out.println("parallel: " +reduce + " -- Time: " + (endTime - startTime));
}
private static void doLinear() {
IntStream range = IntStream.range(1, 6);
long startTime = System.nanoTime();
int reduce = range
.reduce((a, item) -> a * item).getAsInt();
long endTime = System.nanoTime();
System.out.println("linear: " +reduce + " -- Time: " + (endTime - startTime));
}
}
I was trying to benchmark streams but came through this execution time steadily decreasing upon calling the same function again and again
Output:
linear: 120 -- Time: 57008226
linear: 120 -- Time: 23202
linear: 120 -- Time: 17192
linear: 120 -- Time: 17802
Process finished with exit code 0
There is a huge difference between first and second execution time.
I'm sure JVM might be doing some tricks behind the scenes but can anybody help me understand whats really going on there ?
Is there anyway to avoid this optimization so I can benchmark true execution time ?
I'm sure JVM might be doing some tricks behind the scenes but can anybody help me understand whats really going on there? The massive latency of the first invocation is due to the initialization of the complete lambda runtime subsystem. You pay this only once for the whole application. The first time your code reaches any given lambda expression, you pay for the linkage of that lambda (initialization of the invokedynamic call site). After some iterations you'll see additional speedup due to the JIT compiler optimizing your reduction code. Is there anyway to avoid this optimization so I can benchmark true execution time? You are asking for a contradiction here: the "true" execution time is the one you get after warmup, when all optimizations have been applied. This is the runtime an actual application would experience. The latency of the first few runs is not relevant to the wider picture, unless you are interested in single-shot performance. For the sake of exploration you can see how your code behaves with JIT compilation disabled: pass -Xint to the java command. There are many more flags which disable various aspects of optimization.
UPDATE: Refer #Marko's answer for an explanation of the initial latency due to lambda linkage. The higher execution time for the first call is probably a result of the JIT effect. In short, the JIT compilation of the byte codes into native machine code occurs during the first time your method is called. The JVM then attempts further optimization by identifying frequently-called (hot) methods, and re-generate their codes for higher performance. Is there anyway to avoid this optimization so I can benchmark true execution time ? You can certainly account for the JVM initial warm-up by excluding the first few result. Then increase the number of repeated calls to your method in a loop of tens of thousands of iterations, and average the results. There are a few more options that you might want to consider adding to your execution to help reduce noises as discussed in this post. There are also some good tips from this post too.
true execution time There's no thing like "true execution time". If you need to solve this task only once, the true execution time would be the time of the first test (along with time to startup the JVM itself). In general the time spent for execution of given piece of code depends on many things: Whether this piece of code is interpreted, JIT-compiled by C1 or C2 compiler. Note that there are not just three options. If you call one method from another, one of them might be interpreted and another might be C2-compiled. For C2 compiler: how this code was executed previously, so what's in branch and type profile. The polluted type profile can drastically reduce the performance. Garbage collector state: whether it interrupts the execution or not Compilation queue: whether JIT-compiler compiles other code simultaneously (which may slow down the execution of current code) The memory layout: how objects located in the memory, how many cache lines should be loaded to access all the necessary data. CPU branch predictor state which depends on the previous code execution and may increase or decrease number of branch mispredictions. And so on and so forth. So even if you measure something in the isolated benchmark, this does not mean that the speed of the same code in the production will be the same. It may differ in the order of magnitude. So before measuring something you should ask yourself why you want to measure this thing. Usually you don't care how long some part of your program is executed. What you usually care is the latency and the throughput of the whole program. So profile the whole program and optimize the slowest parts. Probably the thing you are measuring is not the slowest.
Java VM loads a class into memory first time the class is used. So the difference between 1st and 2nd run may be caused by class loading.