There is a log showing the stuck thread marked by Weblogic:
<Apr 23, 2013 7:48:25 AM CST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '276' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "668" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl#23ba221b[
GET /XXX/saveInfo.do?fx=duration&info=25159,0,0,0,0,0,0,25153 HTTP/1.1
In my case, we observed if there are too many stuck threads, our server will be getting slower response time and eat more and more memory.
I want to make the count of stuck thread be a health index for my auto-reporting robot. How to count it other than log files ? Does any command or api help me to count stuck threads ?
summarized the solution (wlst sample code) from #viccari :
from tempfile import mktemp
connect('your_account', 'your_account_pass', 'localhost:7001')
# dump thread details to a temp file
file = mktemp()
threadDump(writeToFile="true", serverName="your_server_name", fileName=file)
# count the string token "[STUCK]" by line
count = 0
f = open(file, "r")
for line in f.readlines():
if line.find("STUCK") > 0:
count = count + 1
print "NUM_OF_STUCK_THREADS: ", count
You can access the thread pool health details using WLST (WebLogic Scripting Tools) scripts (if you are familiar with Python, it should not be a problem), or by accessing JMX counters using Java.
This post contains a sample script that sends an alert email whenever stuck threads exist, and in the comments section you will find some samples of Java code acessing JMX counters. There are more examples of both methods on the web.
Related
I'm executing the script for 1 hour but it is running for 10 minutes , i also check loop forever, test data is also proper, all the script is running properly without any error , I run the script thrice validate all the things but im not getting why it is happening
How to overwrite the issues and how to run a the script for 1 hour
Normally you can find the reason for termination of a thread or test in jmeter.log file. If it is not there or it's vague - you can increase JMeter logging level to something more verbose
The most common reasons for premature end of the test are:
Not enough loops to cover the anticipated duration of the test in the Thread Group
Not enough test data if CSV Data Set Config is configured to stop thread on EOF
Thread group is configured to stop the thread/test on a sampler error
There is Flow Control Action sampler somewhere configured to stop thread/test
There is a runtime error like OutOfMemoryError or StackOverFlowError
I am trying to get the Thread Group number of the thread group executing a request so I can correlate it with a property (or maybe a CSV line), but I am having a hard time getting the number alone.
I am able to get the full thread name (including the number) with thread.getThreadName(), but I would need to isolate the number from the name and I would like to skip that part if possible.
Methods that I have used:
ctx.getThreadGroup().getName() returns Thread Group
ctx.getThreadGroup().getThreadName() returns null for some reason
ctx.getThreadNum() returns 0 as it is the first thread to run inside the Thread Group
ctx.getThread().getThreadName() returns Thread Group 1-1
ctx.getThread().getThreadNum() returns 0
I know the number has to be stored somewhere as the log prints this line:
2022-07-10 13:31:10,313 INFO o.a.j.e.StandardJMeterEngine: Starting ThreadGroup: 1 : Thread Group
Where can I find this information?
Thank you.
Regards.
I think you're looking for ctx.getThreadGroup().getNumThreads() function
More information: Top 8 JMeter Java Classes You Should Be Using with Groovy
i set up a 16-hour soak test in jmeter. use a csv file for different users for 10000 users. use ultimate thread group with these settings:
one row
start threads count: 20
initial delay: 0
start up time : 60 sec
hold for load : 57600 sec
shut down time : 10
i run the test 3 times. all of them ended in 8-9 hours.
on the server pc, i look for some settings of iis but didnt see anything.
the second day, in view result tree there was a response such that:
sampler result page
response message: non http response message: en established comnection was aborted by the software in your host machine.
does jmeter any limitation or iis have some timeout value?
JMeter doesn't have any internal "timeouts" for the test, it might be the case you test has.
For example I can think of 2 possible options:
Your CSV Data Set Config has either Recycle on EOF set to True or Stop thread on EOF set to False so when all 10000 lines are used the test stops
Your Ultimate Thread Group is configured to Stop Thread or Stop Test on the error and your test terminates when error occurs
The exact reason for stopping the test can be found in the jmeter.log file
Objective:
I have csv files with unique data e.g say 200 lines. I want to run endurance test, which executed for duration e.g 10min and avail a unique data from csv.
Setup:
Conisdering this, i did stepping thread setup as below with "shared mode: All threads". And i was expecting each min/60sec, 20 new threads will add and so on till 10min (expecting 200 threads at the end). With Stop thread EOF- True ( want to process data once no dup) and Recycle of EOF= False.
However,API executed in one min only and csv data read by few threads < 20 and done (EOF encounter), though jmeter honoring next 9min without executing my API's.
Expectation : 200 threads will read 200 lines i.e each thread 1 line and occurance of thread will be 20 threads in 60sec. Same like diagram, without doing Stop thread EOF=false. In nutshell, each thread execute once and then stop like kind of stuff.
Any sugeestion, how could i acheive this? Thanks!!
Given your setup each thread will read next line from the CSV file on each iteration
For example:
Thread 1 starts
Thread 1 reads line 1 from CSV
Thread 1 executes 1st sampler, response time is 1 second
Thread 1 executes 2nd sampler, response time is 2 seconds
Thread 1 starts 2nd iteration
Thread 1 reads line 2 from CSV
etc.
So you either need to switch to the Thread Group implementation which has Threads iterations limit setting like Concurrency Thread Group or Arrivals Thread Group or Free-Form Arrivals Thread Group
If you want to continue with your Stepping Thread Group - you need to slow down your threads using Timers
We frequently get a series of Stuck threads on our Weblogic servers. I've analyzed this over a period of time.
What I'd like to understand is whether this stuck thread block indicates it is still reading data from the open socket to the database since the queries are simple SELECT stuff?
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:239)
at oracle.net.ns.DataPacket.receive(DataPacket.java:92)
We've run netstat and other commands, the sockets from the Weblogic app server to the Database match the number of connections in the pool.
Any ideas what else we should be investigating here?
Stack trace of thread dump:
"[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x61a5b000 nid=0x25f runnable [0x6147b000..0x6147eeb0]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:239)
at oracle.net.ns.DataPacket.receive(DataPacket.java:92)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:172)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:117)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:92)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:77)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1023)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:999)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:584)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:183)
at oracle.jdbc.driver.T4CStatement.fetch(T4CStatement.java:1000)
at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:314)
- locked <0x774546e0> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:228)
- locked <0x774546e0> (a oracle.jdbc.driver.T4CConnection)
at weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
The bit starting from weblogic.work.ExecuteThread.run to here has been omitted. We have 8 sets of thread dumps - and each show the thread waiting on the same line, and the same object locked
at oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:314)
- locked <0x774546e0> (a oracle.jdbc.driver.T4CConnection)
At the time the stack was printed, it seems blocked waiting for more data from the server
at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:228)
Maybe it is just the query which is taking more than StuckThreadMaxTimeand WL issues a Warning.
If possible I would try:
Find which query or queries are getting the threads stuck and check execution time
Use Wireshark to analyze communication with database
Have a look at the driver source code (JD comes to mind) to understand stack trace
if you use weblogic debug flag -Dweblogic.debug.DebugJDBCSQL you will be able to trace the SQL which is actually being executed