I have a webapp deployed successfully in Jetty webserver.
The webserver responds to requests fine.
When I access the app it renders the home page.
Recently I noticed that when I don't use the app for certain period of time it breaks somehow. The period is somewhere around 2/3 weeks.
When I access the webapp after 2/3 weeks of idle I receive this output.
If I try to access any other link, i.e. the login page (/login.faces) I receive:
Problem accessing /error/not-found.faces. Reason:
/error/not-found.xhtml Not Found in ExternalContext as a Resource
which normally used to work before idling.
If I restart the webserver everything returns to normal and works fine. There are scheduled tasks set which make the app interact every day with database. (There is a scheduled task for fetching currency rates via webservice).
Therefore, my question is what would be the cause which breaks the site and makes it unavailable after idling? Is this a webserver (jetty) issue? Am I missing any setting which is crucial?
FYI, the project structure is: Java with Spring, Hibernate, JSF (PrimeFaces) and Jetty
This occurred due to permissions in CentOS.
If anyone faces the same issue make sure to check the logs have appropriate permissions to read and write
Related
I have a problem with a server that called server A:
Server A: Red Hat Enterprise Linux Server release 7.2 (Maipo)
Server B: Red Hat Enterprise Linux Server release 7.7 (Maipo)
jdk-8u231 installed on all of servers.
I have an Spring Boot application running on 2 servers.
Whenever i use Jmeter to send 100 concurrency request to application running on each servers, the application running on Server B have no problem.
But in Server A, the application will be not responding, that mean the Process (PID) still running but I can't visit actuator endpoint, cannot visit Swagger page, cannot send new request ... log file show nothing since that time.
Thread dump and heap dump have no significant difference.
Could anyone show me how to analysis that problem?
I still have no idea why the problem occur.
Well, I can only speculate here, but here some ideas that can help:
There are two possible sources of issue here Java Application and Linux (+its network policies, firewalls and so forth).
Since You don't know for sure, what happens, try working by "elimination".
Create a script that will run 100 concurrent requests. Place the script at the Server A (the problematic one) and run The script will run against "localhost" (obviously). If you see that it works, then the issue is not in Java at all. Probably some network policies or linux setup, who knows.
Place a log message in the controller of the java application and examine the log. The log should print the request number among other things, so that you'll be able to understand whether you get stuck after a well defined number of requests or its always a different number.
Check the configurations of Spring Boot application. Maybe there is a difference in the number of threads allocated to serve the request by the embedded web server that runs inside the spring boot application (assuming you're not using a reactive stack) and this number differs. In this case you won't be able to call rest endpoints, actuator, etc.
If JMX connection is available to the setup, connect via the JMX and check the MBean of Tomcat (again, assuming there is a tomcat under the hood) to check pretty much the same information as in 4.
You've mentioned thread dumps. Try to take more than one thread dump but one before you're running JMeter test, one during the running (when everything still works), one when everything is stuck.
In the thread dumps check the actual stacktraces, maybe all the threads are stuck working with Database or something and can't serve requests like I've explained in "4"
Examine GC logs, maybe GC works so hard that you can't really interact with the application.
I have deployed a simple Spring boot app in Google App Engine Flexible. The app. has two APIs, one to add the user data into the DB (xxx.appspot.com/add) the other to get all the user data from the DB (xxx.appspot.com/all).
I wanted to see how GAE scales for the load, hence used JMeter to create a load with 100 user concurrency ramped up in 10 seconds and calls these two APIs in half a second delay, forever. While it runs fine for sometime (with just one instance), it starts to fail after 30 seconds or so with a "java.net.SocketException" or "The server responded with a status of 502".
After this error, when I try to access the same API from the browser, it displays,
Error: Server Error
The server encountered a temporary error and could not complete your
request. Please try again in 30 seconds.
The service is back to normal after 30 mins or so, and whenever the load test happens it repeats the same behavior as mentioned above. I expect GAE to auto-scale based on the load coming in to handle it without any down time (using multiple instances), instead it just crashes or blocks the service (without any information in the log). My app.yaml configuration is,
runtime: java
env: flex
service: hello-service
automatic_scaling:
min_num_instances: 1
max_num_instances: 10
I am a bit stuck with this one, Any help would be greatly appreciated. Thanks in advance.
The solution was to increase the resource configuration, details below.
Given that I did not set a resource parameter, it defaulted to the pre-defined values for both CPU and Memory. In this case, the default
memory was set at 0.6GB. App Engine Flex instances uses about 0.4GB
for overhead processes. Given Java is known to consume higher memory, there is a
great likelihood that the overhead processes consumed more than the
approximate 0.4GB value. Now instances in App Engine are restarted due
to a variety of reasons including optimization due to memory use. This
explains why your instances went off and it shows Tomcat is starting
up (they got restarted) and ends up in 502 error due to the nginx is
not able to complete the request. Fixing the above may lessen if not completely eliminate the 502s.
After I have specified the resources attribute and increased the configuration in app.yaml 502 error seems to be gone.
I have a Digital Ocean droplet (512MB RAM, 20GB SSD Disk, Ubuntu 13.10 x64) on which
a MongoDB instance and
a Tomcat 7 server
run.
On the Tomcat server, following applications are installed
Apache CXF-based application, which takes processes web service requests, interacts with the database and executes scheduled jobs,
Vaadin application,
JSF (Primefaces) application and
Psi Probe.
When I
restart Tomcat,
use the Vaadin and/or JSF application,
then for several weeks do nothing on that machine (it basically is idle during that time),
then try to open the JSF and/or Vaadin application,
I find the site unresponsive (nothing is displayed after I enter the URL in the browser).
When I restart Tomcat (sudo service tomcat7 restart), everything works again. I don't see any obvious problems in the Tomcat logs.
How can I find out,
whether the problem is on the Tomcat side (one of the applications consumes too many resources even if idle) or on the OS side (nothing happens on the machine and therefore the OS puts itself into a "hibernating" mode) and
if the problem is with Tomcat, exactly which of the application is causing it?
Please start from top to bottom.
then try to open the JSF and/or Vaadin application,
I find the site unresponsive (nothing is displayed after I enter the
URL in the browser).
Check if the service is still running before restarting sudo service tomcat7 status and/or ps -ef | grep tomcat
Check with netstat -patune | grep <portnumber, e.g. 443> if the server is listening on the configured ports
Check your httpd/Apache/Tomcat access logs if the request reaches the server and if yes, check if there are errors or timeouts related to the requests
Check if the DB connection is still possible
To force some error logs, try to change your maxIdle, maxActive and maxWait attributes of your Tomcat's Connection Pool configuration. maxWait default is -1, connections created sometimes during these weeks will wait forever.
#Patrick provided some excellent basic tests.
I notice that you only have 512 MB of RAM. With some fairly beefy software such as tomcat, plus MongoDB on top of that, your machine may simply be overloaded.
Based on that, I would propose a couple additional things to check:
sudo free
Should tell you how much memory is being used, and how much swap space you use.
sudo top
Will tell you which process is using the most memory. You may want to sort the output of top by memory (default is usually by CPU utilization).
Most importantly, check the log files in /var/log (in particular /var/log/messages). You may find indications that the kernel killed one of your processes (probably tomcat).
I have tomcat application which uses webservice using apache axis. Everything works fine if single user is using application. Multiple user are also working fine except in one condition. I have webservice call which can take 30 sec. When more than one user call a controller which uses this particular method, both thread seems to go into deadlock condition.
Worst part is it hangs whole application. I am able to reproduce this condition in local environment many times, so i am very confident that this particular call block if requested by more than one user at same time. I also tried setting timeout in web service call using axis specific method ((Stub)service).setTimeout(1000*60); but its also not working. Is anyone face this condition before. I have no clue why single thread blocking hang whole tomcat. Tomcat is working but it stopped responding to any other request. I am using spring for my web application.
I have several tomcat instances running in physically independent machines.
I want to configure the tomcat to share sessions between this instances.
I have tried to configure org.apache.catalina.session.PersistentManager from http://tomcat.apache.org/tomcat-6.0-doc/config/manager.html. But I only see the session file when I shutdown the tomcat instances and I don't know if the instances are sharing this session. I think not. Because It doesn't make sense if tomcat writes down the session only on shutdown.
The other thing that I found is the cluster-howto but I can't do that couse the machines can't see eachother. They only shares a storage path to use.
Other thing that I think I can do is to implement a manager but it seems a litle bit tricky.
I have to add that I am using tomcat for deploying grails war files and I am using the grails session. I thing it has something to do with Spring
So, the question is: What is the best thing you think I can do to accomplish more effectibly the task? Or maybe I am missing something? Can you give me any pointer?
You have the F5 Big IP doing the load balancing in front of the tomcat servers, so it will handle the sessionID for you by sending you back to the correct Tomcat server. Use the sticky-round-robin algorithm.
As per the usecase in your comments -
What I am trying to do is to save some
data in session, then redirect to a
login server, who in a success
scenario it redirects to my servers.
And my concern is what happend if the
load balancer redirects the request to
the server that doesn't previously
saved the needed data in session.
Maybe sticky session is what I need.
So: can I configure sticky session in
a non tomcat-cluster enviromnent?
At the successful login - you redirect back first to the BigIP. It will pick up the sessionID from the browser. It will send you to the correct Tomcat and you should be able to retrieve the session data.
If not, looks like you need to store the "sessionID" itself against some "user Id" in a database but thats a bad design. I think the former should work