We have one application which use Tibco EMS as message middleware, however we found that some threads which related with tibco keep increasing and now it is over 5000, and we need to restart our server after several days as we received thread threshold alert.
"TIBCO EMS TCPLink Reader (Server-800893673)" #108119 daemon prio=5
os_prio=0 tid=0x00007f04e239e000 nid=0x6da2 runnable
[0x00007f0152baa000] java.lang.Thread.State: RUNNABLE at
java.net.SocketInputStream.socketRead0(Native Method) at
java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at
java.net.SocketInputStream.read(SocketInputStream.java:171) at
java.net.SocketInputStream.read(SocketInputStream.java:141) at
java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at
java.io.BufferedInputStream.read(BufferedInputStream.java:265)
locked <0x000000073b113240> (a java.io.BufferedInputStream) at java.io.DataInputStream.readInt(DataInputStream.java:387) at
com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:625)
at
com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at
com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
"TIBCO EMS Pinger for connection Server-800893673" #108118 daemon
prio=5 os_prio=0 tid=0x00007f01986fb800 nid=0x6da1 waiting on
condition [0x00007f01530af000] java.lang.Thread.State:
TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at
com.tibco.tibjms.TibjmsxLinkTcp$LocalPinger.run(TibjmsxLinkTcp.java:113)
at java.lang.Thread.run(Thread.java:748)
Related
I have a very simple Jmeter test plan with a thread group and supported by basic config elements.
My HTTP Request Sampler just points to the home page of the application which gives facility for login authentication.
Few facts to share
When I access the page in chrome and view the loading time in chrome developer, the same is completed in 5 seconds with embedded resources downloaded in parallel.
When I record the same through jmeter and start re-play it takes 12 seconds
Next I disabled all the non-html resources and marked the home page to download embedded resources.
When I enable the checkbox Download all embedded resources with parallel download check box and the value set at 6, jmeter goes to a hang state.
I thought that might be the issue with Jmeter setting, then i tried www.w3schools.com site, and I was able to refer on the home page and during replay all the embedded resources were downloaded.
Please help me to understand if the page i am trying to script has any security involved, where we cannot download parallel resources through script or what can be the possible correction.
The application is accessible through https protocol.
Heap Dump
"Image Fetcher 0" Id=5734 TIMED_WAITING on java.util.Vector#567840bf
at java.lang.Object.wait(Native Method)
- waiting on java.util.Vector#567840bf
at sun.awt.image.ImageFetcher.nextImage(Unknown Source)
at sun.awt.image.ImageFetcher.fetchloop(Unknown Source)
at sun.awt.image.ImageFetcher.run(Unknown Source)
"ResDownload-Thread-5659" Id=5709 TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack#c26b878
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack#c26b878
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
...
"ResDownload-Thread-5521" Id=5704 TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack#c26b878
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack#c26b878
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
...
"ResDownload-Thread-5545" Id=5703 TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack#c26b878
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack#c26b878
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
...
"ResDownload-Thread-5555" Id=5702 TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack#c26b878
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.SynchronousQueue$TransferStack#c26b878
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)
at java.util.concurrent.SynchronousQueue.poll(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
Make sure to tick Disable cache box in the browser developer tools:
as it might be the case your browser is not making the actual requests and returning the embedded resources from disk or memory cache
Add HTTP Cache Manager to your test plan in order to mimic the browser cache.
JMeter might fail to recognise certain resources like images embedded in CSS files so it might be the case you're suffering from a JMeter bug. So you can consider using i.e. Parallel Controller as the workaround for edge cases.
If you're doing your test from enterprise and thus have a enterprise proxy, maybe the website you're hitting requires some rules to allow all resources to be downloaded, so you may need to adjust command line options:
https://jmeter.apache.org/usermanual/get-started.html#proxy_server
Also, be aware that you should not be load testing third-party websites (Google Analytics , trackers ...) that your application may be using, so use the exclude pattern to ignore this.
You say that JMeter hangs, to debug, use Help > Create a Thread Dump and show content of output file in your question.
My guess is that it's trying to establish connection , since it has no access it hangs, so you ca set in Advanced tab of HTTP Request:
Connect timeout
Response timeout
I tested website performance with Jmeter 2.13, I have never used Jmeter before, When I set the number of thread is 500, it is OK, ans when set 750, there are some errors shows below. During the tests, my CPU ,bandwidth, memory,IO are not full. I have tried other ways ,such as changing maxThread of tomcat7 and so on, but they did not work, any help is appreciated!
org.apache.http.conn.HttpHostConnectException: Connection to http://10.0.0.160:8080 refused
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.open(MeasuringConnectionManager.java:107)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:517)
at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:331)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1146)
at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1135)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:434)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
... 14 more
There are two possibilities here.
Your server is not able handle the number of incoming connections from the same IP address. Check the size of connection pool and maximum incoming connections. Use some java tool like java melody to check total number of connections. Also check the thread-pool size and max thread settings.
Or It could be JMeter client issue. Try to run the test from two different load generators and split the load of 750. Client may not be able to establish socket connections for all the 750 users.
Long running Stand-alone Spring app on a weekly schedule (not on any App server) usually takes few days to complete but failed due to java.net.SocketException: Broken pipe.
Here's stack trace:
Caused by: java.sql.SQLRecoverableException: IO Error: Broken pipe
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:876)
at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1153)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1275)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3576)
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3620)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:93)
at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:703)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:644)
... 28 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at oracle.net.ns.DataPacket.send(DataPacket.java:199)
at oracle.net.ns.NetOutputStream.flush(NetOutputStream.java:211)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:227)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1122)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:863)
The java.net.SocketException: Broken pipe exception suggests that, you are writing while the other end has already closed the connection. As the other end isn't reading it. As I also said, if this is happening there is something wrong with your application protocol specification or implementation, most probably that you don't even have one.
If application is dealing with heavy amount of data, think about JVM memory issues. Another possibility is that other end of connection does not wait long enough then it closes the connection, because application is taking too long to prepare the data and send. And connection times out.
Read this article to know more about it.
Recently I started getting the following problem which results in messages not being delivered to:
"PollThread" prio=10 tid=0x00007f0a2cf86000 nid=0x76b8 in Object.wait() [0x00007f09eb6bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.checkUsable(RemoteHconn.java:2121)
- locked <0x000000048f040a10> (a com.ibm.mq.jmqi.remote.api.RemoteHconn$ReconnectMutex)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.enterCall(RemoteHconn.java:1787)
at com.ibm.mq.jmqi.remote.api.RemoteHconn.enterCall(RemoteHconn.java:1764)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiPutMessageWithProps(RemoteFAP.java:7804)
at com.ibm.mq.jmqi.remote.api.RemoteFAP.jmqiPut(RemoteFAP.java:7254)
at com.ibm.mq.ese.jmqi.InterceptedJmqiImpl.jmqiPut(InterceptedJmqiImpl.java:496)
at com.ibm.mq.ese.jmqi.ESEJMQI.jmqiPut(ESEJMQI.java:385)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer$SpiIdentifiedProducerShadow.sendInternal(WMQMessageProducer.java:812)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer$ProducerShadow.send(WMQMessageProducer.java:531)
at com.ibm.msg.client.wmq.internal.WMQMessageProducer.send(WMQMessageProducer.java:1178)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.sendMessage(JmsMessageProducerImpl.java:927)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.send_(JmsMessageProducerImpl.java:783)
at com.ibm.msg.client.jms.internal.JmsMessageProducerImpl.send(JmsMessageProducerImpl.java:446)
Entire client application is unresponsive. How do I troubleshoot this type of problems with IBM MQ classes for JMS? Is it connection setup problems? There are no error in MQ log file. MQ version is 7.5.2. Thanks in advance for any help.
The lock is being held on the ReconnectMutex meaning that a reconnection to the queue manager is currently in progress and so waits until it is notified that the reconnection has been successful. Is there another thread that looks like it is attempting to reconnect to the queue manager and is not moving? Do you know if the queue manager is up and running at this time?
I'm using c3p0 ComboPooledDataSource to pool database connections (to an Oracle 10g database). I'm having problems with handling database connectivity outages.
If there is no connection to the database when the first connection is being acquired, the checkout timeout fires and it fails as expected.
However, if a connection outage occurs after one or more connections have been acquired and are already in the connection pool, calling getConnection() just hangs. No exception is thrown. I'm guessing this is because it's trying to use a pooled connection but that connection is no longer alive.
Is there a way to check whether the connection is valid before trying to use it? I tried setting testConnectionOnCheckout=true but it doesn't seem to have any effect.
This is the thread dump
C3P0PooledConnectionPoolManager[identityToken->2rvy8f8x1oujxrx1majv5s|be41d5]-
HelperThread-#2" daemon prio=6 tid=0x0307a800 nid=0x840 in Object.wait() [0x03d1f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method) - waiting on <0x28387f88> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)
at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:635)
- locked <0x28387f88> (a com.mchange.v2.async.ThreadPoolAsynchronousRunner)
Locked ownable synchronizers:
- None
I encountered the same issue. In my case, it was caused by the JDBC driver not being setup to timeout on socket failures. I made the following additions to my C3P0 ComboPooledDataSource configuration:
cpds = new ComboPooledDataSource();
...
//--------------------------------------------------------------------------------------
// NOTE: Once you decide to use cpds.setProperties() to set some connection properties,
// all properties must be set, including user/password, otherwise an exception
// will be thrown
Properties prop = new Properties();
prop.setProperty("oracle.net.CONNECT_TIMEOUT",
Integer.toString(JDBC_CONNECTION_TIMEOUT_IN_MILLISECONDS));
prop.setProperty("oracle.jdbc.ReadTimeout",
Integer.toString(JDBC_SOCKET_TIMEOUT_IN_MILLISECONDS));
prop.setProperty("user", username);
prop.setProperty("password", password);
cpds.setProperties(prop);
//--------------------------------------------------------------------------------------
...
The Oracle driver properties are applied when C3P0 creates the Connection object. These two properties in particular will cause an exception to be thrown if the socket connection is inactive for more than 30 seconds.
If you're not connecting to an Oracle database, there are similar properties for other JDBC drivers for other database vendors. Some of them are shown near the bottom of this page.