I have requirement to monitor some particular java processes on that server and display on web with color (red or green). Is there any such lib in spring world?
let's say
Process Name
Status
process1
RED
process2
GREEN
I don't want to use any monitoring toll such as grafan, promotheus, appdynamics, geneos..
if your status red mean process is stop, green mean process is running,there 2 ways.
if you are using spring cloud like eureka, it supplys some apis
each process just supply a simple request. if the process is running, the request will return success, if not, the request may be 404...
Related
Requirement:-
I have to write code to monitor all the running applications on the server and give their name as output if it's down.
Research:-
During my research I found that:-
There are several tools like azure and monito that themselves monitor all the applications but this does not match our requirements.
We can write code that can check all the running services on the local desktop or the server and from there we can also check the running status of the required applications and if the status is stopped or sleep then we can easily notify.
We can send requests to the deployed URL at some regular interval and if we get a response status rather than 200 then we can notify the user as something is wrong and this particular website is not working.
If anyone can through some light on this and can suggest some more methods or references from their experience, it will be highly appreciated.
Question:
Is there an option within spring or its embedded servlet container to open ports when spring is ready to handle traffic?
Situation:
In the current setup i use a spring boot application running in google cloud run.
Circumstances:
Cloud run does not support liveness/readyness probes, it considers an open port as "application ready".
Cloud run sends request to the container although spring is not ready to handle requests.
Spring start its servlet container, open its ports while still spinning up its beans.
Problem:
Traffic to an unready application will result in a lot of http 429 status codes.
This affects:
new deployments
scaling capabilities of cloud run
My desire:
Configure spring/servlet container to delay opening ports when application is actually ready
Delaying opening ports to the time the application is ready would ease much pain without interfering too much with the existing code base.
Any alternatives not causing too much pain?
Things i found and considered not viable
Using native-image is not an option as it is considered experimental and consumes more RAM at compile time than our deployment pipeline agents allow to allocate (max 8GB vs needed 13GB)
another answer i found: readiness check for google cloud run - how?
which i don't see how it could satisfy my needs, since spring-boot startup time is still slow. That's why my initial idea was to delay opening ports
I did not have time to test the following, but one thing i stumbled upon is
a blogpost about using multiple processes within a container. Though it is against the recommendation of containers principles, it seems viable for the time until cloud run supports probes of any type.
As you are well aware of the fact that “Cloud Run currently does not have a readiness/liveness check to avoid sending requests to unready applications” I would say there is not much that can be done on Cloud Run’s side except :
Try and optimise the Spring boot app as per the docs.
Make a heavier entrypoint in Cloud Run service that takes care of
more setup tasks. This stackoverflow thread mentions how “A
’heavier’ entrypoint will help post-deploy responsiveness, at the
cost of slower cold-starts” ( this is the most relevant solution
from a Cloud Run perspective and outlines the issue correctly)
Run multiple processes in a container in Cloud Run as you
mentioned.
This question seems more directed at Spring Boot specifically and I found an article with a similar requirement.
However, if you absolutely need the app ready to serve when requests come in, we have another alternative to Cloud Run, Google Kubernetes Engine (GKE) which makes use of readiness/liveness probes.
I have a problem with a server that called server A:
Server A: Red Hat Enterprise Linux Server release 7.2 (Maipo)
Server B: Red Hat Enterprise Linux Server release 7.7 (Maipo)
jdk-8u231 installed on all of servers.
I have an Spring Boot application running on 2 servers.
Whenever i use Jmeter to send 100 concurrency request to application running on each servers, the application running on Server B have no problem.
But in Server A, the application will be not responding, that mean the Process (PID) still running but I can't visit actuator endpoint, cannot visit Swagger page, cannot send new request ... log file show nothing since that time.
Thread dump and heap dump have no significant difference.
Could anyone show me how to analysis that problem?
I still have no idea why the problem occur.
Well, I can only speculate here, but here some ideas that can help:
There are two possible sources of issue here Java Application and Linux (+its network policies, firewalls and so forth).
Since You don't know for sure, what happens, try working by "elimination".
Create a script that will run 100 concurrent requests. Place the script at the Server A (the problematic one) and run The script will run against "localhost" (obviously). If you see that it works, then the issue is not in Java at all. Probably some network policies or linux setup, who knows.
Place a log message in the controller of the java application and examine the log. The log should print the request number among other things, so that you'll be able to understand whether you get stuck after a well defined number of requests or its always a different number.
Check the configurations of Spring Boot application. Maybe there is a difference in the number of threads allocated to serve the request by the embedded web server that runs inside the spring boot application (assuming you're not using a reactive stack) and this number differs. In this case you won't be able to call rest endpoints, actuator, etc.
If JMX connection is available to the setup, connect via the JMX and check the MBean of Tomcat (again, assuming there is a tomcat under the hood) to check pretty much the same information as in 4.
You've mentioned thread dumps. Try to take more than one thread dump but one before you're running JMeter test, one during the running (when everything still works), one when everything is stuck.
In the thread dumps check the actual stacktraces, maybe all the threads are stuck working with Database or something and can't serve requests like I've explained in "4"
Examine GC logs, maybe GC works so hard that you can't really interact with the application.
I'm looking for an option to stop my Liberty server if a specific application fails to start.
I can't find any option for this in the docs, the closest thing I've found to achieve this is health policies but they don't look to be a good fit.
You could write something (even as simple as a bash script) to check the logs for:
"CWWKZ0001I: Application {appName} started"
and if you don't see it in x time, then execute "/wlp/bin/server stop {serverName}"
Could do it all through mbean invocations via java or REST calls by checking the state of the app (WebSphere:service=com.ibm.websphere.application.ApplicationMBean,name=*) and if it's not 'Started' by x time, then invoke an api to stop it (for collective environment you could use WebSphere:feature=collectiveController,type=ServerCommands,name=ServerCommands, otherwise you could use the osgi framework api).
sc query state= all works as expected from the command line.
From within another Service, sc query state= all doesn't print anything to that sub-process' stdout (captured by the parent, of course).
Is there a permission/privilege that the Service needs in order to list/start/stop the other servies?
A little background: I am making a service that periodically restarts some misbehaving services.
Well, for one don't do that, at least not in a blocking manner. In order for your own service to respond to the SCM (Service Control Manager) in order to return its status, the service has to be able to execute its dispatcher code. This means that if you call this program and wait for it to exit you'll wait indefinitely. One way to mitigate this would be to put this into a separate thread so it's not blocking your dispatching and your service will continue to talk to the SCM.
Alternatively (and probably better) you could use the EnumServicesStatusEx function to talk to the SCM and inquire about the statuses of other services yourself. The function itself doesn't mention anything about being blocking, so you'd have to figure out yourself whether it is and then use a thread again to prevent your service from stopping to talk to the SCM.
One last note: if those misbehaving services are yours, you should more likely fix the respective code. I've had a share of legacy code and had one misbehaving service which got its own helper application as "fault action" (can be configured in service configuration as SERVICE_CONFIG_FAILURE_ACTIONS) that would go about and restart the service whenever it crashed. Once I took that code over, figured out the cause and fixed it, the service was stable again and that application isn't really needed anymore.