I'm looking for an option to stop my Liberty server if a specific application fails to start.
I can't find any option for this in the docs, the closest thing I've found to achieve this is health policies but they don't look to be a good fit.
You could write something (even as simple as a bash script) to check the logs for:
"CWWKZ0001I: Application {appName} started"
and if you don't see it in x time, then execute "/wlp/bin/server stop {serverName}"
Could do it all through mbean invocations via java or REST calls by checking the state of the app (WebSphere:service=com.ibm.websphere.application.ApplicationMBean,name=*) and if it's not 'Started' by x time, then invoke an api to stop it (for collective environment you could use WebSphere:feature=collectiveController,type=ServerCommands,name=ServerCommands, otherwise you could use the osgi framework api).
Related
Question:
Is there an option within spring or its embedded servlet container to open ports when spring is ready to handle traffic?
Situation:
In the current setup i use a spring boot application running in google cloud run.
Circumstances:
Cloud run does not support liveness/readyness probes, it considers an open port as "application ready".
Cloud run sends request to the container although spring is not ready to handle requests.
Spring start its servlet container, open its ports while still spinning up its beans.
Problem:
Traffic to an unready application will result in a lot of http 429 status codes.
This affects:
new deployments
scaling capabilities of cloud run
My desire:
Configure spring/servlet container to delay opening ports when application is actually ready
Delaying opening ports to the time the application is ready would ease much pain without interfering too much with the existing code base.
Any alternatives not causing too much pain?
Things i found and considered not viable
Using native-image is not an option as it is considered experimental and consumes more RAM at compile time than our deployment pipeline agents allow to allocate (max 8GB vs needed 13GB)
another answer i found: readiness check for google cloud run - how?
which i don't see how it could satisfy my needs, since spring-boot startup time is still slow. That's why my initial idea was to delay opening ports
I did not have time to test the following, but one thing i stumbled upon is
a blogpost about using multiple processes within a container. Though it is against the recommendation of containers principles, it seems viable for the time until cloud run supports probes of any type.
As you are well aware of the fact that “Cloud Run currently does not have a readiness/liveness check to avoid sending requests to unready applications” I would say there is not much that can be done on Cloud Run’s side except :
Try and optimise the Spring boot app as per the docs.
Make a heavier entrypoint in Cloud Run service that takes care of
more setup tasks. This stackoverflow thread mentions how “A
’heavier’ entrypoint will help post-deploy responsiveness, at the
cost of slower cold-starts” ( this is the most relevant solution
from a Cloud Run perspective and outlines the issue correctly)
Run multiple processes in a container in Cloud Run as you
mentioned.
This question seems more directed at Spring Boot specifically and I found an article with a similar requirement.
However, if you absolutely need the app ready to serve when requests come in, we have another alternative to Cloud Run, Google Kubernetes Engine (GKE) which makes use of readiness/liveness probes.
I have a problem with a server that called server A:
Server A: Red Hat Enterprise Linux Server release 7.2 (Maipo)
Server B: Red Hat Enterprise Linux Server release 7.7 (Maipo)
jdk-8u231 installed on all of servers.
I have an Spring Boot application running on 2 servers.
Whenever i use Jmeter to send 100 concurrency request to application running on each servers, the application running on Server B have no problem.
But in Server A, the application will be not responding, that mean the Process (PID) still running but I can't visit actuator endpoint, cannot visit Swagger page, cannot send new request ... log file show nothing since that time.
Thread dump and heap dump have no significant difference.
Could anyone show me how to analysis that problem?
I still have no idea why the problem occur.
Well, I can only speculate here, but here some ideas that can help:
There are two possible sources of issue here Java Application and Linux (+its network policies, firewalls and so forth).
Since You don't know for sure, what happens, try working by "elimination".
Create a script that will run 100 concurrent requests. Place the script at the Server A (the problematic one) and run The script will run against "localhost" (obviously). If you see that it works, then the issue is not in Java at all. Probably some network policies or linux setup, who knows.
Place a log message in the controller of the java application and examine the log. The log should print the request number among other things, so that you'll be able to understand whether you get stuck after a well defined number of requests or its always a different number.
Check the configurations of Spring Boot application. Maybe there is a difference in the number of threads allocated to serve the request by the embedded web server that runs inside the spring boot application (assuming you're not using a reactive stack) and this number differs. In this case you won't be able to call rest endpoints, actuator, etc.
If JMX connection is available to the setup, connect via the JMX and check the MBean of Tomcat (again, assuming there is a tomcat under the hood) to check pretty much the same information as in 4.
You've mentioned thread dumps. Try to take more than one thread dump but one before you're running JMeter test, one during the running (when everything still works), one when everything is stuck.
In the thread dumps check the actual stacktraces, maybe all the threads are stuck working with Database or something and can't serve requests like I've explained in "4"
Examine GC logs, maybe GC works so hard that you can't really interact with the application.
I like to write a shell script for a backend server using Spring Boot (v2.1.1) to start multiple microservices in a certain order - some services depend on other to be running.
What is the 'best practice'?
Of course i could run the .jars like this (original post):
#!/bin/bash
java -jar myjar1.jar &
java -jar myjar2.jar &
java -jar myjar3.jar &
But this would start the .jars simultaneously, afaik.
How can i ensure, that a certain service myjar1.jar started properly and after that, another service myjar2.jar is started. Because every service is a SpringBootApplication i assume that there are certain possibilities to do so?!
I read this SO solution but I don't want to create any symlinks, because i just need that for development purposes.
Well it is very specific to your service as to when it gets started.
At process level, as soon as you execute the command service is running, so you will need your service to share the state when its up.
One way i can think of is in your script start the service, expose health api and check if its up. if it is move to next one. You need to make use of curl and sleep command in your scripts.
But I would like to know why you want to do that. Specially for your microservices, your services should not depend on each other. They may need some data, but they should be resilient to the fact that services may come and go. You should have a very strong reason to do as you are doing, cause in real world env, it is very difficult to ensure order is maintained.
We are migrating some apps from WAS full profile to WAS Liberty profile.
Some apps have MDBs and need JMS Activation Specs definitions connected to MQ.
In order to enforce strict FIFO ordering of messages in a cluster, we set the "WAS_EndpointInitialState" property to "INACTIVE" on those Activation Specs to tell WAS full profile to not start the Activation Spec on startup. When the cluster starts, we start (ie "resume") the activation on one server only.
Q: How to achieve this with Liberty (v16.0.x) ?
I don't see an equivalent parameter within the "properties.wmqJms" stanza.
Thanks
Liberty doesn't have an equivalent parameter/capability for activation specs.
You can open a request for enhancement here:
https://www.ibm.com/developerworks/rfe/?PROD_ID=544
In case it helps during the meantime, a crude way of simulating the capability is to start the server with the jmsActivationSpec elements commented out, and make configuration updates to uncomment as you want them activated.
Unfortunately as-is (with v16.0.0.3 and the current beta version), it is not possible to deploy an application with MDBs in production due to a serious lack on functionalities in Liberty profile (JMS Activations).
When using the jmsActivationSpec+ properties.wmqJms stanzas, it is impossible:
to configure the activation to stop after x failed tentatives to consume the message. Liberty tries to consume the message forever without any notification!!
to start the activation in an inactive state on startup., so it is impossible to enforce the FIFO paradigm on a Q when deployed in a cluster (or collective or other form of cluster)
Those are already captured in the following RFE:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95885
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95794
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=88543
For us it's a clear no-go to move to WebSphere Liberty profile for those reasons
This is way too late for the OP, but in case someone comes here looking for a current answer.
Liberty / Open Liberty now offer (as of 18.0.0.1) such a function, which you can enable via the autoStart attribute, e.g.:
<jmsActivationSpec autoStart="false" id="myJMSActSpec"/>
See here for a quick example of how you would use the EndpointControl MBean and/or the server resume CLI command to start message delivery into the server.
I am using WebSphere 6.2 and my requirement is I admin have to get an alert when the application on the server stopped or server down .. How to achieve this ? In higher versions this feature is there ? please help me
Thanks in Advance ,
Raj
I'm assuming you mean WebSphere application server processes (rather than the physical server on which WebSphere is running) and individual applications running on those processes. I'm also assuming you mean when those elements have stopped unexpectedly rather than when somebody has deliberately stopped them.
If so, you're going to have to use external monitoring software to detect most of those conditions. We use a combination of scripts that scan for processes and specific error messages in logs and external site-monitoring software that checks for application responsiveness. Such scripts can be standalone, handwritten scripts, or run under generic monitoring tools from IBM (Tivoli) or 3rd-parties.
Alternatively, I think you should also be able to write something that uses JMX to read specific things about WebSphere state, and there is at least one sophisticated monitoring tool you could purchase, IBM Tivoli Composite Application Manager (ITCAM) for Application Diagnostics, which can monitor WebSphere internals.