Recently we started in production with a new application hosted in several WebSphere Aplication Server and would be nice to have them monitored/graphed with more or less the same parameters we monitor/graph our Jboss servers.
Right now I managed to monitor several points with wsadmin using jython scripts:
Java HEAP
Issue a "Test connection" to all datasources
However I'm not able to find the way to monitor JDBC Connection Pools, to check PoolSize, WaitingThreadCount, and FreePoolSize values. I can monitor them on RealTime using Tivoli Performance Viewer included in the WAS DMGR:
But I didn't find the way to get it through wsadmin (or any other way) so I can obtain the values and add to Cacti/RRD to obtain graphs like we already have with Jboss:
Does anyone managed to get this data from websphere with wsadmin or any other tools?
if you really insist on wsadmin
A number of WAS MBeans exposes stats attribute. This attribute represents runtime statistics of the component. In your case, the MBean type would be JDBCProvider and its stats object implements javax.management.j2ee.statistics.JDBCStats interface defined in JSR-77.
Having that stats attribute value at hand, you'll be able to extract all other data.
One important note: in wsadmin you'll need to use getAttribute_jmx function of AdminControl, not just getAttribute.
Advertisement mode Working with wsadmin and MBeans can be tough, especially when it comes to accessing complex attributes. You may find this process easier with WDR.
Other options
Starting wsadmin process periodically only to query one or two attributes seem add too much overhead to me. An alternative is to install some code in your WSAS that could expose those stats in easily consumable way. One such tool is Jolokia. Jolokia is a web application exposing MBeans via HTTTP, using XML or JSON formats. Having Jolokia running in your WSAS, you can simply query it periodically from any programming language and then feed your time series choice.
Obviously, WSAS has its own specifics: extra MBeanServer, security, hence you'll need extra descriptors and code. Also, by default Jolokia can't serialize JSR-77 objects, so you'll need to provide those serializers yourself. I've been using Jolokia with WSAS in the past and all of those missing parts can be found in a clone of Jolokia repo. Roland Huss (the author of Jolokia) implemented some of those (excluding EAR and WSAS descriptors) in Jolokia-Extra project.
Finally I managed to obtain the stats following these addresses https://www.ibm.com/developerworks/websphere/techjournal/1112_guillemenot/1112_guillemenot.html?ca=drs-
Basically I've created a function to get the statistics about JDBC connection pools I need:
def getStats(server,driver,datasource):
perfStr = AdminControl.queryNames( 'type=Perf,process=' + server + ',*')
if perfStr == "":
print "Sorry I can't find server " + server
sys.exit(1)
perfObj = AdminControl.makeObjectName( perfStr)
srvrStr = AdminControl.queryNames( 'type=Server,process=' + server + ',*')
srvrObj = AdminControl.makeObjectName( srvrStr)
stats = AdminControl.invoke_jmx( perfObj, 'getStatsObject', [ srvrObj, java.lang.Boolean('true')], ['javax.management.ObjectName', 'java.lang.Boolean'])
try:
waitingThreads=stats.getStats('connectionPoolModule').getStats(driver).getStats(datasource).getStatistic('WaitingThreadCount').getCurrent()
poolSize=stats.getStats('connectionPoolModule').getStats(driver).getStats(datasource).getStatistic('PoolSize').getCurrent()
freePoolSize=stats.getStats('connectionPoolModule').getStats(driver).getStats(datasource).getStatistic('FreePoolSize').getCurrent()
percentUsed=stats.getStats('connectionPoolModule').getStats(driver).getStats(datasource).getStatistic('PercentUsed').getCurrent()
print "WaitingThreadCount=" + str(waitingThreads) + ", PoolSize=" + str(poolSize) + ", FreePoolSize=" + str(freePoolSize) + ", PercentUsed=" + str(percentUsed)
except:
print "Ooops, something went wrong :("
raise
And in order to identify server, driver and datasource variables I also added a function to list them:
def listServers():
"""List the servers Database Connection Pools"""
servers = AdminControl.queryNames( 'type=Perf,*').split("\n")
for i in range(0, len(servers)):
srvName = servers[i].split(",")[1].split("=")[1]
if srvName == "nodeagent":
continue
print "Server: " + srvName
perfStr = AdminControl.queryNames( 'type=Perf,process=' + srvName +',*')
perfObj = AdminControl.makeObjectName( perfStr)
srvrStr = AdminControl.queryNames( 'type=Server,process=' + srvName +',*')
srvrObj = AdminControl.makeObjectName( srvrStr)
stats = AdminControl.invoke_jmx( perfObj, 'getStatsObject', [ srvrObj, java.lang.Boolean('true')], ['javax.management.ObjectName', 'java.lang.Boolean'])
for driver in stats.getStats('connectionPoolModule').subCollections():
print "\tDriver Name: " + driver.getName()
for datasource in stats.getStats('connectionPoolModule').getStats(driver.getName()).subCollections():
print "\t\tDatasource: " + datasource.getName()
Which will output something like:
Server: APP_CLUSTER_APP01
Driver Name: Oracle JDBC Driver (XA)
Datasource: jdbc/AppEngine
Datasource: jdbc/AppEngineH
Server: APP_CLUSTER_APP02
Driver Name: Oracle JDBC Driver (XA)
Datasource: jdbc/AppEngine
Datasource: jdbc/AppEngineH
Server: SOL_CLUSTER_SOL01
Driver Name: Oracle JDBC Driver (XA)
Datasource: jdbc/dict1ds
Datasource: jdbc/dict2ds
Datasource: jdbc/dict3ds
Server: SOL_CLUSTER_SOL02
Driver Name: Oracle JDBC Driver (XA)
Datasource: jdbc/dict1ds
Datasource: jdbc/dict2ds
Datasource: jdbc/dict3ds
Related
I am performing this query with my Ruby mongo driver:
begin
User.collection.find({}).count()
rescue => e
Rails.logger.error e.to_s
end
I would like to catch all situations where this operation fails. The main reason it would fail is if the server was unavailable.
For example, one of the errors I see occasionally is:
Mongo::Error::NoServerAvailable (No server is available matching preference: #<Mongo::ServerSelector::Primary:0x70302744731080 tag_sets=[] max_staleness=nil> using server_selection_timeout=30 and local_threshold=0.015)
I want to catch errors after just 6 seconds.
From the docs I see that there are a few different timeout options (connect_timeout, server_selection_timeout, socket_timeout). But I am not sure which to pass, and how to pass them.
You're on the right track. server_selection_timeout is the correct option in this case -- it tells the driver how long to wait to find a suitable server before timing out. You can set that option on a new client in the Mongoid configuration file (config/mongoid.yml).
You want your configuration file to look something like this:
development: # (or production or whatever environment you're using)
clients:
default:
# your default client here
new_client: # the name of your new client with different options
database: mongoid
hosts:
- localhost:27017
options:
server_selection_timeout: 6
...
Read the Mongoid Configuration documentation to learn more about setting up config files.
Then, you want to use the new client you defined to perform your query, and rescue any Mongo::Error::NoServerAvailable errors.
begin
User.with(client: 'new_client').collection.find({}).count()
rescue Mongo::Error::NoServerAvailable => e
# perform error handling here
end
Note that this starts up a new Mongo::Client instance, which is an expensive operation if done repeatedly. I would recommend that you close the extra client once you are done using it, like so:
Mongoid::Clients.with_name('new_client').close
Jmeter My JDBC connection is paramterize using Property file and I am reading value using ${__P(propertyname)} but it is reading only first value and not reading second value from property file.
Created a custom property file
Server1=10:1433;Instance=SQL2014;DatabaseName=Repository15;domain=ABC
Server2=20:1433;Instance=SQL2008R2;DatabaseName=Repository14;domain=ABC
In Jmeter Test Plan i have two threads. Under each thread i have added JDBC Connection Configuration
First Thread Database URL = ${__P(Server1)}
Second Thread Database URL = ${__P(Server2)}
when we run the test it is reading only first property file and ending the connection
You need to define different variable name for each JDBC Connection Configuration and connect the relevant JDBC Requests for each.
Each name must be different. If there are two configuration elements using the same name, only one will be saved. JMeter logs a message if a duplicate name is detected.
We have a website running in Windows Server 2008 + SQLServer 2008 + Ruby + Sinatra + Sequel/Puma
We've developed an API for our website.
When the access points are requested by many clients, at the same time, the clients start getting RequestTimeout exceptions.
I investigated a bit, and I noted that Puma is managing multi threading fine.
But Sequel (or any layer below Sequel) is processing one query at time, even if they came from different clients.
In fact, the RequestTimeout exceptions don't occur if I launch many web servers, each one listening one different port, and I assign one different port to each client.
I don't know yet if the problem is Sequel, ADO, ODBC, Windows, SQLServer or what.
The truth is that I cannot switch to any other technology (like TinyTDS)
Bellow is a little piece of code with screenshots that you can use to replicate the bug:
require 'sinatra'
require 'sequel'
CONNECTION_STRING =
"Driver={SQL Server};Server=.\\SQLEXPRESS;" +
"Trusted_Connection=no;" +
"Database=pulqui;Uid=;Pwd=;"
DB = Sequel.ado(:conn_string=>CONNECTION_STRING)
enable :sessions
configure { set :server, :puma }
set :public_folder, './public/'
set :bind, '0.0.0.0'
get '/delaybyquery.json' do
tid = params[:tid].to_s
begin
puts "(track-id=#{tid}).starting access point"
q = "select p1.* from liprofile p1, liprofile p2, liprofile p3, liprofile p4, liprofile p5"
DB[q].each { |row| # this query should takes a lot of time
puts row[:id]
}
puts "(track-id=#{tid}).done!"
rescue=>e
puts "(track-id=#{tid}).error:#{e.to_s}"
end
end
get '/delaybycode.json' do
tid = params[:tid].to_s
begin
puts "(track-id=#{tid}).starting access point"
sleep(30)
puts "(track-id=#{tid}).done!"
rescue=>e
puts "(track-id=#{tid}).error:#{e.to_s}"
end
end
There are 2 access points in the code above:
delaybyquery.json, that generates a delay by joining the same table 5
times. Note that the table must be about 1000 rows in order to get the
query working really slow; and
delaybycode.json, that generates a delay by just calling the ruby sleep
function.
Both access points receives a tid (tracking-id) parameter, and both write the
outout in the CMD, so you can follow the activity of both process in the same
window and check which access point is blocking incoming requests from other
browsers.
For testing I'm opening 2 tabs in the same chrome browser.
Below are the 2 testings that I'm performing.
Step #1: Run the webserver
c:\source\pulqui>ruby example.app.rb -p 81
I get the output below
Step #2: Testing Delay by Code
I called to this URL:
127.0.0.1:81/delaybycode.json?tid=123
and 5 seconds later I called this other URL
127.0.0.1:81/delaybycode.json?tid=456
Below is the output, where you can see that both calls are working in parallel
click here to see the screenshot
Step #3: Testing Delay by Query
I called to this URL:
127.0.0.1:81/delaybyquery.json?tid=123
and 5 seconds later I called this other URL
127.0.0.1:81/delaybyquery.json?tid=456
Below is the output, where you can see that calls are working 1 at time.
Each call to an access point is finishing with a query timeout exception.
click here to see the screenshot
This is almost assuredly due to win32ole (the driver that Sequel's ado adapter uses). It probably doesn't release the GVL during queries, which would cause the issues you are seeing.
If you cannot switch to TinyTDS or switch to JRuby, then your only option if you want concurrent queries is to run separate webserver processes, and have a reverse proxy server dispatch requests to them.
Here is our environment:
.Net version: 4.5
Database: Oracle 12.1.0.2 (odp.net)
We are using LLBL "Adapter" but I don't think that has anything to do with the issue
LLBLGen Pro version: 4.1
Llbl Gen Pro Runtime: 4.1.13.1213
When we do an Insert(always into different tables which we are using for the short period and then removing) we use the following code:
int numRecords = strings.Count();
var insertCmd = "insert into " + tableName + " (StringField) values (:StringField)";
var oracleCommand = new OracleCommand();
oracleCommand.CommandText = insertCmd;
oracleCommand.CommandType = CommandType.Text;
oracleCommand.BindByName = true;
oracleCommand.ArrayBindCount = numRecords;
oracleCommand.Parameters.Add(":StringField", OracleDbType.NVarchar2, strings.ToArray(), ParameterDirection.Input);
// this is an LLBL adapter. Like I said, I think the issue is below the LLBL layer.
this.adapter.ExecuteActionQuery(new ActionQuery(oracleCommand));
When the database is getting hit hard with multiple of these inserts in parallel, we get the following error and the insert call never returns from the database.
WG_6.Index_586.TVD: An exception was caught during the execution of an action query: ORA-24381: error(s) in array DML
ORA-12592: TNS:bad packet
ORA-12592: TNS:bad packet
ORA-12592: TNS:bad packet
ORA-12592: TNS:bad packet
ORA-03111: break received on communication channel
ORA-03111: break received on communication channel
ORA-03111: break received on communication channel
On the database, using Toad's session browser, I can see that the "Current Statement" is correct.
insert into schemaX.tableY(StringField) values(:Stringfield)
Under the Waits tab in Toad, there is the following message:
“Waiting for SQL*Net more data from client - waited X hundred seconds, so far” and the X keeps incrementing until we hit our database timeout.
We tried with batches of 1 million and this gave us the best performance for our scenario. However, this hanging issue arose. I then decrease the ArrayBindCount to 500K, 100K, 50K, 10K and then 5K. Only when I used 5K did it stop happening.
A couple of notes:
This happens more frequently when the database is on a different physical machine than the client. When using a local VM, it rarely happens. The network that we are using is generally very reliable with no other noted issues.
From the error message(ORA-12592: TNS:bad packet), it seems that the issue might be on the client and perhaps related to code in the "Oracle.DataAccess.Client"(ODAC) dll.
My next steps for troubleshooting are to use Reflector to debug the call from the ODAC code and also to get more reliable client side tracing while forcing this error to occur.
I had the same situation when trying to insert into an Oracle table using the ArrayBinding.
Using a small number for oracleCommand.ArrayBindCount seemed to improve the frequency of the errors (same like yours) but not completely.
The solution was to use the Managed data access. I suggest you get the latest ODP.NET, add a reference to ManagedDataAccess and change to:
using Oracle.ManagedDataAccess.Client;
using Oracle.ManagedDataAccess.Types;
This fixed problem in my case and with no need to change anything in the code.
I'd like to use the WSADMIN command that is part of WebSphere 7 to query the state of the queues on the system.
Can anyone help me out?
Thanks
For anyone interested, here's the jython version of jeff's answer.
qpoint = 'WebSphere:*,type=SIBQueuePoint'
queues = AdminControl.queryNames(qpoint).split()
for q in queues:
identifier = AdminControl.getAttribute(q, 'identifier')
size = AdminControl.getAttribute(q, 'depth')
print identifier + ' size: ' + size + ' messages'
print AdminControl.getAttributes(q)
So to find out the queue depths I've written this JACK script...
set qpoint "WebSphere:*,type=SIBQueuePoint"
set queues [$AdminControl queryNames $qpoint]
foreach q $queues {
set identifier [$AdminControl getAttribute $q identifier]
set size [$AdminControl getAttribute $q depth]
puts "$identifier size: $size messages"
puts [$AdminControl getAttributes $q]
Stuff it in a file on the box, jeff.jacl and call the command...
/opt/IBM/WebSphere/AppServer/bin # ./wsadmin.sh -profile jp.cmd.jacl
And what do you get? well you get a whole bag of awesomeness!
WASX7209I: Connected to process "server1" on node WRSNode using SOAP connector; The type of process is: UnManagedProcess
CHANGE_REQUEST size: 15 messages
{depth 15} {state ACTIVE} {id CFAC834BE6AF5D9A30451D01_QUEUE_51} {identifier CHANGE_REQUEST} {highMessageThreshold 50000} {sendAllowed true}
ETL_DEAD size: 378 messages
Next job is to see if I can all the java code that is used by JACL directly.
In order to retrieve the depth of a SIB queue using the WebSphere PMI, you will need to select the following two counters:
AvailableMessageCount and UnavailableMessageCount
Here is how: From the WebSphere Application Server Admin Console, go to the Performance Monitoring Infrastructure (PMI) panel of the application server where the messaging engine is hosted:
Application servers > your_app_server_name > Performance Monitoring Infrastructure (PMI)
You will be on the Configuration tab by default. You can choose to switch to the Runtime tab if you wish this monitoring to start without restarting the application server.
Once on the PMI panel, click on the link "Custom", the label of the last radio button. This should take you to the Custom monitoring level panel. From the left-hand navigation tree, select:
- SIB Service
- SIB Messaging Engines - *- Destinations- Queues Select both counters: AvailableMessageCount and UnavailableMessageCount and click the Enable button located at the top. Your setting should be saved at this point.