The issue is reproducible when running a JMeter test under specific machine. The number of users is 1. The JMeter test uses regex to exclude certain domains ^((?!somerealdomain\.com).)*$. The thread is closed almost immediately and the error in the jmeter log is:
2018-09-03 15:53:12,596 ERROR o.a.j.JMeter: Uncaught exception:
java.lang.StackOverflowError: null
at org.apache.oro.text.regex.Perl5Matcher.__match(Unknown Source) ~[oro-2.0.8.jar:?]
at org.apache.oro.text.regex.Perl5Matcher.__match(Unknown Source) ~[oro-2.0.8.jar:?]
at org.apache.oro.text.regex.Perl5Matcher.__match(Unknown Source) ~[oro-2.0.8.jar:?]
The Perl5Matcher error is present approximately 1000 times.
As the issue is reproducible only on one from our 3 performance agents, I am not sure what is causing it.
The heap is 1GB on all machines:
HEAP=-Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m
Java version is the 1.8.0_181 an all three agents.
Do you have ideas where to look?
Try settings Xss to same value everywhere:
-Xss256k
Also please show all the jmeter.log of:
jmeter.log
jmeter-server.log
If only one machine is affected it means that it is not the same as other machines therefore it can produce not the same load as you expect so I would start with identifying the differences.
Double check that your load generators are running the same OS
Double check that your load generators use the same JVM version (preferably from Oracle, preferably server JRE or JDK) vendor, architecture, build, etc.
Double check that your load generators use the same (preferably latest) JMeter version
Double check that your load generators have the same configuration, the best way is setting up one node according to JMeter Best Practices and copy the installation over to all slave machines
Save response data for failing request and verify that it is the same for all slaves, you can do this by adding the next 2 lines to user.properties file:
jmeter.save.saveservice.output_format=xml
meter.save.saveservice.response_data=true
Try amending your regular expression to exclude the domain to something like:
^((?!somerealdomain).)*$
or
^((?!somerealdomain.com).)*$
I found the solution. I added to the start of the PATH C:\Program Files\Java\jre1.8.0_181\bin; and then restarted the computer as explained here.
What lead me to this was that after I have installed latest Java JRE 64-bit version and I run java -version, the message was "Java is not recognized as an internal or external command". It was strange that the JMeter test has even started its execution on this machine.
There was no need to do this on the other two agents. Thank you all for your kind suggestions.
Related
When I execute ApacheJmeter then UI is coming blank.Please help me how I can get user interface of jmeter.I also installed java development kit v 17.0.3.
Check jmeter.log file for any suspicious entries. If it doesn't contain anything meaningful - increase JMeter logging verbosity to DEBUG level
Try earlier version of Java, the minimum you can run JMeter 5.4.3 with is Java 8
Try nightly build of JMeter
Try another look and feel, i.e. run JMeter as:
jmeter -Jjmeter.laf=CrossPlatform
While running the script in non-gui mode for jmeter tests, test gets completed but it does shows min: 0, max: 0...
note:
JMeter 5.4 is installed without any 3rd party plugin.
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
Please help
It means that none of Samplers were executed and the reasons could be in:
The number of threads in Thread Group is 0
The test contains CSV Data Set Config and the .csv file is not there
The test relies on a JMeter Plugin which is not installed
etc.
The reason should be in jmeter.log file, check it and I believe you will find either the cause or at least a clue there.
Also be aware that according to 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article you should always be using the latest version of JMeter so consider upgrading to JMeter 5.4.1 (or whatever is the latest stable version available at JMeter Downloads page) on next available opportunity
I guess you probably found the answer - but maybe this response will be useful to someone.
I've been battling a similar issue -> locally all was working fine (in GUI and non-GUI mode), but as soon as I tried to run the jmx on the server I could see it would start threads, all logs seemed normal (standard info logs within jmeter.log at start) but still no requests were actually made (confirmed by observing logs from the target server). I tried running trace logs but still nothing out of the ordinary - I could see proper HTTP config, threads were as if normal... But the generated jtl file only contained a header and no requests actually were made.
After some hair pulling I finally found my issue - I'm using variables for threads, rampup and loop which I invoke using ${__P(threads,)} syntax and when running the non-GUI load test I pass those with -J to the command line (f.ex. jmeter -Jthreads=50 ...)
I didn't pass the loop parameter since I figured it would just default to 1 (and that is also the impression I get when looking at the logs where I see all thread started ... entries) - turns out for some reason it will log, but won't execute anything actually till I pass a value for that loop.
So if you run a test plan and notice it only simulates doing something but you get no results in the target log file -> confirm you're setting ALL variables used for thread groups including those you suspect to default
During Distributed testing with Jmeter 3.3 in non gui mode i'm getting error as, how can I fix this :
I'm using same version of JMeter and JDK on Master as well as Slave machines.
The JVM should have exited but did not.
The following non-daemon threads are still running (DestroyJavaVM is OK):
Thread[main,5,main],
stackTrace:java.net.SocketInputStream#socketRead0
java.net.SocketInputStream#socketRead
java.net.SocketInputStream#read
java.net.SocketInputStream#read
java.io.BufferedInputStream#fill
java.io.BufferedInputStream#read
java.io.DataInputStream#readByte
sun.rmi.transport.StreamRemoteCall#executeCall
sun.rmi.server.UnicastRef#invoke
java.rmi.server.RemoteObjectInvocationHandler#invokeRemoteMethod
java.rmi.server.RemoteObjectInvocationHandler#invoke
com.sun.proxy.$Proxy19#rrunTest
org.apache.jmeter.engine.ClientJMeterEngine#runTest at line:149
org.apache.jmeter.engine.DistributedRunner#start at line:132
org.apache.jmeter.engine.DistributedRunner#start at line:149
org.apache.jmeter.JMeter#runNonGui at line:1005
org.apache.jmeter.JMeter#startNonGui at line:910
org.apache.jmeter.JMeter#start at line:538
sun.reflect.NativeMethodAccessorImpl#invoke0
sun.reflect.NativeMethodAccessorImpl#invoke
sun.reflect.DelegatingMethodAccessorImpl#invoke
java.lang.reflect.Method#invoke
org.apache.jmeter.NewDriver#main at line:248
I strongly recommend using this jmeter property:
jmeterengine.force.system.exit=true
documented here. These Chinese language web pages link link tipped me off.
You can add -Jjmeterengine.force.system.exit=true on the command line when launching JMeter, or add jmeterengine.force.system.exit=true to JMETER_HOME/bin/jmeter.properties.
How I Confirmed This Fix
With JMeter 5.1 and java version "1.8.0_231" on MS-Win10, we're using a customized version of this JMeter InfluxDB backend Listener.
After my 60 second test run from the command line (jmeter.bat -n -t plan.jtl), the command line hung after displaying this output (very similar to op):
Tidying up ... # Wed Jan 29 14:41:04 CST 2020 (1580330464874)
... end of run
The JVM should have exited but did not.
The following non-daemon threads are still running (DestroyJavaVM is OK):
Thread[DestroyJavaVM,5,main], stackTrace:
Thread[pool-2-thread-3,5,main], stackTrace:sun.misc.Unsafe#park
java.util.concurrent.locks.LockSupport#parkNanos
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#awaitNanos
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#take
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#take
java.util.concurrent.ThreadPoolExecutor#getTask
java.util.concurrent.ThreadPoolExecutor#runWorker
java.util.concurrent.ThreadPoolExecutor$Worker#run
java.lang.Thread#run
Thread[pool-2-thread-4,5,main], stackTrace:sun.misc.Unsafe#park
java.util.concurrent.locks.LockSupport#parkNanos
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#awaitNanos
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#take
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#take
java.util.concurrent.ThreadPoolExecutor#getTask
java.util.concurrent.ThreadPoolExecutor#runWorker
java.util.concurrent.ThreadPoolExecutor$Worker#run
java.lang.Thread#run
Thread[pool-2-thread-1,5,main], stackTrace:sun.misc.Unsafe#park
java.util.concurrent.locks.LockSupport#parkNanos
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#awaitNanos
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#take
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#take
java.util.concurrent.ThreadPoolExecutor#getTask
java.util.concurrent.ThreadPoolExecutor#runWorker
java.util.concurrent.ThreadPoolExecutor$Worker#run
java.lang.Thread#run
After modifying my command line as follows, jmeter.bat cleanly exited instead of hanging and all the ugly stack trace went away too:
jmeter.bat -n -Jjmeterengine.force.system.exit=true -t plan.jtl
To confirm that the problem was caused by our customized JMeter InfluxDB backend Listener, I removed it from the .jmx and I also removed the jmeterengine.force.system.exit=true. No hang, no ugly stacktrace (I actually love stacktraces).
I have not taken the next step to discover whether the problem is with the official JMeter InfluxDB backend Listener or with our customized variant, which is not (and will never be) available publicly.
Should mention one gap in this story. I feel this test conclusively points to our customized backend listener (or jmeter's). However, its odd that none of the threads in the above thread dump seem to belong to the backend listener. So I applaud that JMeter did the right thing by dumping the stack trace -- few other apps go to the extent of auto-dumping when appropriate for troubleshooting. But in this case, perhaps that jmeter auto-dump code needs to be enhanced, because it did not point to the culprit backend listener code. Anyone over there at Apache listening in on this?
Good luck.
Most probably your JMeter engine(s) is(are) overloaded therefore cannot gracefully shut down running threads when you request them to do so.
Make sure you follow JMeter Best Practices
The very first "best practice" states Always use latest version of JMeter so consider migrating to JMeter 5.0 or whatever latest version is available at JMeter Downloads page
Make sure your JMeter instances have enough headroom to operate in terms of CPU, RAM and so on. You can use JMeter PerfMon Plugin for this if you don't have other monitoring software in place/in mind.
Take a thread dump and examine it - this way you will know where exactly your test is stuck
Introduce reasonable timeout values in HTTP Request Defaults so in case when server fails to respond JMeter wouldn't wait infinitely but rather fail with an error
And finally (however I wouldn't recommend this) you can suppress this check by adding the next line to user.properties file:
jmeter.exit.check.pause=-1
if you go for this keep in mind that you may run into a situation when JMeter slaves will still be trying to execute something even after your test ends so you will need to kill and restart the processes manually or using a script.
My goal is to run a load test using 4 Azure servers as load generators and 1 Azure server to initiate the test and gather results. I had the distributed test running and I was getting good data. But today when I remote start the test 3 of the 4 load generators fail with all the http transactions erroring. The failed transactions log the following error:
Non HTTP response message: java.lang.ClassNotFoundException: org.apache.commons.logging.impl.Log4jFactory (Caused by java.lang.ClassNotFoundException: org.apache.commons.logging.impl.Log4jFactory)
I confirmed the presence of commons-logging-1.2.jar in the jmeter\lib folder on each machine.
To try to narrow down the issue I set up one Azure server to both initiate the load and run JMeter-server but this fails too. However, if I start the test from the JMeter UI on that same server the test runs OK. I think this rules out a problem in the script or a problem with the Azure machines talking to each other.
I also simplified my test plan down to where it only runs one simple http transaction and this still fails.
I've gone through all the basics: reinstalled jmeter, updated java to the latest version (1.8.0_111), updated the JAVA_HOME environment variable and backed out the most recent Microsoft Security update on the server. Any advice on how to pick this problem apart would be greatly appreciated.
I'm using JMeter 3.0r1743807 and Java 1.8
The Azure servers are running Windows Server 2008 R2
I did get a resolution to this problem. It turned out to be a conflict between some extraneous code in a jar file and a component of JMeter. It was “spooky” because something influenced the load order of referenced jar files and JMeter components.
I had included a jar file in my JMeter script using the “Add directory or jar to classpath” function in the Test Plan. This jar file has a piece of code I needed for my test along with many other components and one of those components, probably a similar logging function, conflicted with a logging function in JMeter. The problem was spooky; the test ran fine for months but started failing at the maximally inconvenient time. The problem was revealed by creating a very simple JMeter test that would load and run just fine. If I opened the simple test in JMeter then, without closing JMeter, opened my problem test, my problem test would not fail. If I reversed the order, opening the problem test followed by the simple test then the simple test would fail too. Given that the problem followed the order in which things loaded I started looking at the jar files and found my suspect.
When I built the script I left the jar file alone thinking that the functions I need might have dependencies to other pieces within the jar. Now that things are broken I need to find out if that is true and happily it is not. So, to fix the problem I changed the extension on my jar file to zip then edited it in 7-zip. I removed all the code except what I needed. I kept all the folders in the path to my needed code, I did this for two reasons; I did not have to update my code that called the functions and when I tried changing the path the functions did not work.
Next I changed the extension on the file back to jar and changed the reference in JMeter’s “Add directory or jar to classpath” function to point to the revised jar. I haven’t seen the failure since.
Many thanks to the folks who looked at this. I hope the resolution will help someone out.
I am performing remote distributed testing through JMeter on my windows machine as master (Windows 10) and amazon Linux as the slave. I have configured jmeter.properties with client.rmi.localport and server.rmi.localport on both.
All configurations are fine as I can connect and remote start my JMeter server.
But I am not able to see the test results on my master machine.
I could see the results on Windows 7 with same configurations in jmeter.properties.
Also, I saw one more process JMeterMenuBar: setRunning(true,127.0.0.1:****) & JMeterMenuBar: setRunning(false,127.0.0.1:****) on Windows 7 but can't see these on Windows 10.
Please resolve for this.
Being not enough telepathic unfortunately I cannot identify and fix your issue remotely. However here are few possible workarounds:
Copy JMeter and Java folders from Windows 7 to Windows 10 machine.
Enable extra debug logging on Windows 7, Windows 10 and remote JMeter engines by adding the next line to user.properties file:
log_level.jmeter=DEBUG
and compare the output in both cases
You can also try adding the next line to system.properties file
java.net.preferIPv4Stack=true
as Windows 10 might be using IPv6 addresses by default or it may be something due to dual stack clash
And finally, JMeter automatically removes response data from successful samplers, it means you won't be able to see anything apart pass/fail flag and some timings. If you need to see the response data as well - add the next line to user.properties file on remote engines:
mode=Standard
JMeter restart is required to pick any property change up
See Apache JMeter Properties Customization Guide for more information on JMeter properties and ways of working with them
Thanks for your help.
I had the logs enabled and after posting the question I found one of the java class which JMeter is invoking is corrupted and hence I am unable to see the response on Master.
I re-installed JAVA JDK and it worked fine.