I want to start my WAS server(on windows) in debug mode.But my application is huge and server time outs while starting in debugging mode.I have increased the timeout value to 1000 .What should i do?
If you're running your server by using Rational Application Developer, it is possible that your server is basically waiting on a breakpoint. Sounds silly, but hey, happened to me a few times before.
If it's a regular installation:
If it's a Network Deployment topology, make sure that the deployment manager and the node agent are up and running. Look at their log; they're often overlooked.
If it's a single-server topology, then it's possible that your debug port (7777 by default) is in use. When that happens, the debugger process never quite "connects" to WebSphere.
To sum it up: it starts with the logs...
Try changing connection settings on your server from using RMI to SOAP. If you are behind a firewall and you application is timing out on startup then it may help you. Also do you get any other information when the timeout occurs?
Related
I have problem with my tomcat launched as windows service. There are few web applications running on it, and when I attempt to stop service (programmatically) it takes more than 30 seconds to stop it and SCM aborts operation due to the time limitation (default is 30 seconds). This can be configured in registry, but I am not able to change it (and seems to be ugly solution). How can I solve this? Is there some kill_timeout for tomcat? Or how should I stop my applications?
The Tomcat config GUI (AKA The Apache "Commons Daemon Service Manager") has a Shutdown tab (Tip: open tomcatw.exe in your installation path). Here you will see a Timeout value. This might solve your problem.
It is also explained in https://tomcat.apache.org/tomcat-7.0-doc/windows-service-howto.html#Command_line_parameters:
ParameterName Default Description
[...]
--StopTimeout No Timeout Defines the timeout in seconds that procrun waits for service to exit gracefully.
Alternatively, check this: Tomcat doesn't stop. How can I debug this?
I implemented a web application to start the Tomcat service works very quickly, but spending hours and when more users are entering is getting slow (up to 15 users approx.).
Checking RAM usage statistics (20%), CPU (25%)
Server Features:
RAM 8GB
Processor i7
Windows Server 2008 64bit
Tomcat 7
MySql 5.0
Struts2
-Xms1024m
-Xmx1024m
PermGen = 1024
MaxPernGen = 1024
I do not use Web server, we publish directly on Tomcat.
Entering midnight slowness is still maintained (only 1 user online)
The solution I have is to restart the Tomcat service and response time is again excellent.
Is there anyone who has experienced this issue? Any clue would be appreciated.
Not enough details provided. Need more information :(
Use htop or top to find memory and CPU usage per process & per thread.
CPU
A constant 25% CPU usage in a 4 cores system can indicate that a single-core application/thread is running 100% CPU on the only core it is able to use.
Which application is eating the CPU ?
Memory
20% memory is ~1.6GB. It is a bit more than I expect for an idle server running only tomcat + mysql. The -Xms1024 tells tomcat to preallocate 1GB memory so that explains it.
Change tomcat settings to -Xms512 and -Xmx2048. Watch tomcat memory usage while you throw some users at it. If it keeps growing until it reaches 2GB... then freezes, that can indicate a memory leak.
Disk
Use df -h to check disk usage. A full partition can make the issues you are experiencing.
Filesystem Size Used Avail Usage% Mounted on
/cygdrive/c 149G 149G 414M 100% /
(If you just discovered in this example that my laptop is running out of space. You're doing it right :D)
Logs
Logs are awesome. Yet they have a bad habit to fill up the disk. Check logs disk usage. Are logs being written/erased/rotated properly when new users connect ? Does erasing logs fix the issue ? (copy them somewhere for future analysis before you erase them)
If not. Logs are STILL awesome. They have the good habit to help you track bugs. Check tomcat logs. You may want to set logging level to debug. What happens last when the website die ? Any useful error message ? Do user connections are still received and accepted by tomcat ?
Application
I suppose that the 25% CPU goes to tomcat (and not mysql). Tomcat doesn't fail by itself. The application running on it must be failing. Try removing the application from tomcat (you can eventually put an hello world instead). Can tomcat keep working overnight without your application ? It probably can, in which case the fault is on the application.
Enable full debug logging in your application and try to track the issue. Run it straight from eclipse in debug mode and throw users at it. Does it fail consistently in the same way ?
If yes, hit "pause" in the eclipse debugger and check what the application is doing. Look at the piece of code each thread is currently running + its call stack. Repeat that a few times. If there is a deadlock, an infinite loop, or similar, you can find it this way.
You will have found the issue by now if you are lucky. If not, you're unfortunate and it's a tricky bug that might be deep inside the application. That can get tricky to trace. Determination will lead to success. Good luck =)
For performance related issue, we need to follow the given rules:
You can equalize and emphasize the size of xms and xmx for effectiveness.
-Xms2048m
-Xmx2048m
You can also enable the PermGen to be garbage collected.
-XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
If the page changes too frequently to make this option logical, try temporarily caching the dynamic content, so that it doesn't need to be regenerated over and over again. Any techniques you can use to cache work that's already been done instead of doing it again should be used - this is the key to achieving the best Tomcat performance.
If there any database related issue, then can follow sql query perfomance tuning
rotating the Catalina.out log file, without restarting Tomcat.
In details,There are two ways.
The first, which is more direct, is that you can rotate Catalina.out by adding a simple pipe to the log rotation tool of your choice in Catalina's startup shell script. This will look something like:
"$CATALINA_BASE"/logs/catalina.out WeaponOfChoice 2>&1 &
Simply replace "WeaponOfChoice" with your favorite log rotation tool.
The second way is less direct, but ultimately better. The best way to handle the rotation of Catalina.out is to make sure it never needs to rotate. Simply set the "swallowOutput" property to true for all Contexts in "server.xml".
This will route System.err and System.out to whatever Logging implementation you have configured, or JULI, if you haven't configured.
See more at: Tomcat Catalina Out
I experienced a very slow stock Tomcat dashboard on a clean Centos7 install and found the following cause and solution:
Slow start up times for Tomcat are often related to Java's
SecureRandom implementation. By default, it uses /dev/random as an
entropy source. This can be slow as it uses system events to gather
entropy (e.g. disk reads, key presses, etc). As the urandom manpage
states:
When the entropy pool is empty, reads from /dev/random will block until additional environmental noise is gathered.
Source: https://www.digitalocean.com/community/questions/tomcat-8-5-9-restart-is-really-slow-on-my-centos-7-2-droplet
Fix it by adding the following configuration option to your tomcat.conf or (preferred) a custom file into /tomcat/conf/conf.d/:
JAVA_OPTS="-Djava.security.egd=file:/dev/./urandom"
We encountered a similar problem, the cause was "catalina.out". It is the standard destination log file for "System.out" and "System.err". It's size kept on increasing thus slowing things down and ultimately tomcat crashed. This problem was solved by rotating "catalina.out". We were using redhat so we made a shell script to rotate "catalina.out".
Here are some links:-
Mulesoft article on catalina (also contains two methods of rotating):
Tomcat Catalina Introduction
If "catalina.out" is not the problem then try this instead:-
Mulesoft article on optimizing tomcat:
Tuning Tomcat Performance For Optimum Speed
We had a problem, which looks similar to yours. Tomcat was slow to respond, but access log showed just milliseconds for answer. The problem was streaming responses. One of our services returned real-time data that user could subscribe to. EPOLL were becoming bloated. Network requests couldn't get to the Tomcat. And whats more interesting, CPU was mostly idle (since no one could ask server to do anything) and acceptor/poller threads were sitting in WAIT, not RUNNING or IN_NATIVE.
At the time we just limited amount of such requests and everything became normal.
We have IBM Websphere 7.0 scheduled to run as windows service on Windows 7 / 2008 R2.
When this service is stopped, the java.exe process associated with this is not stopping. Hence it's a manual task to kill the process in Task manager.
Does anyone know how to make the java.exe also stop after the Websphere is stopped?
Thanks
Karthik
Take a thread dump of the Java process and see what it's doing. Perhaps it's waiting for some resource etc:
jps
jstack <pid>
Karthik,
Try switching on WASService Tracing (see pages 55 onwards in the following link)
http://blogs.hbr.org/schwartz/2012/03/the-magic-of-doing-one-thing-a.html
Check the server's logs. Is the server really stopped? Do you see a log line saying that the server is stopped?
If not, then it means that something is preventing your server instance from stopping. I have seen it happening before when JavaEE code used to spawn long-running threads without closing them.
You can configure WebSphere to start with debugging support, and then connect to it via, say, Eclipse. Using Eclipse, you can look at the active threads on the server, including each thread's stack; that might help you track the problem down.
If the server's logs show that the server is stopped... well... that's what PMR's are for, I guess.
I am experiencing an issue with delayed writes to the application logs for a Java EE web application running in IBM WebSphere v. 7.x. Logging statements taking up to an hour to appear in the application logs.
The problem doesn't appear related to heavy loads; WAS is responding to page requests almost instantly, and I am testing against a box that isn't used for performance testing, and on a holiday no less -- there is very little activitiy on the server.
My guess would be that the thread associated with logging has been configured with very low priority, but I cant figure out where that would be configured via the admin console or the configuration files.
Has anyone else experienced this sort of issue with WebSphere?
it's possible you don't even enough available threads in the thread pool. Its consistant with the page requests being fast, as they are controlled by the WebContainer threads.
Try increasing it:
Servers > Application Servers > Thread pools > ...
Not sure exactly which one to increase its max value. In worst case, increase'em all. Increase it heavily, so to be sure.
Other options:
make sure you enough disk space / try to connect with jConsole to inquire.
When I am debugging on my Windows 7 IIS7 machine, I get this error during a debug:
The web server process that was being
debugged has been terminated by IIS.
this can be avoided by configuring
application pool setting in IIS. see
help for further details.
What am I doing wrong?
When you are debugging, IIS will not service any other requests until you are done stepping through your code. That includes the "ping" request that IIS sends to itself. Since IIS doesn't hear back from itself, it decides to shut itself down, which promptly terminates your debugging.
The solution is to increase the Ping Maximum Response Time in the application pool settings from its default value of 90 seconds. Set it to something high enough that will give you enough time to debug your code (like maybe 300 seconds).
Microsoft has a long-winded write-up here.
Edit: Others have suggested setting "Ping Enabled" to false. There are several reasons why I prefer to keep it in place, just with a larger interval, but the most important is that you will (most likely) have worker processing pinging enabled on production, and you should strive to develop and debug under a configuration that is as close to production as possible. If you do NOT have ping enabled on production, then by all means disable it locally as well.
http://weblogs.asp.net/soever/archive/2009/06/18/debugging-sharepoint-asp-net-code-smart-key-codes-disable-timeout.aspx
Your App Pool -> Advanced Settings -> Ping Enabled to False
IIS has a health-checking feature which periodically checks to see if an IIS worker process is hung or otherwise unusuable. If a worker process is stopped in the debugger, it looks unhealthy from the perspective of IIS, and IIS kills it and spins up a new process.
To change this behavior (on your dev workstation-- don't want to disable this in production!) go to the IIS management tool, select the Application Pools node in the left pane, and right-click on the app pool that your app lives in, and choose "Advanced Settings". From there, in the "process model" section, set "Ping Enabled" to False. You may also want to set the idle timeout to be a very large number.
See this IIS.NET article for more discussion of this issue and a screenshot. See this TechNet article for how to set these settings via code/script outside the admin tool.
If you have microsoft's scom running and configured where you work (assuming this is not a for-fun project) and you are able to create a management pack for it or know someone who is, that may help you pinpoint what is causing the issue. I realize its a long shot, but if that does describe your scenario that is what I would do if no other solution is found.