Any idea why syslog:// protocol from bluemix app is not connecting to PaperTrail third party logging service?
Here is my process:
create user provided service
bind this service
restage
then on the papertrail nothing happens:
You can try it out. The testing app URL is:
http://658.eu-gb.mybluemix.net
Its just Hello world simple nodejs app.
Most likely the configuration is correct but you are not able to see anything on Papertrail because of a problem with Loggregator component; you should setup an application that iteratively writes on the log to see very few lines arriving.
In order to clear up the status, this is a general problem between Loggregator and logdrain systems (and Papertrail is one of those) and a fix has been testing and will be applied in the next versions of CloudFoundry.
Related
Question:
Is there an option within spring or its embedded servlet container to open ports when spring is ready to handle traffic?
Situation:
In the current setup i use a spring boot application running in google cloud run.
Circumstances:
Cloud run does not support liveness/readyness probes, it considers an open port as "application ready".
Cloud run sends request to the container although spring is not ready to handle requests.
Spring start its servlet container, open its ports while still spinning up its beans.
Problem:
Traffic to an unready application will result in a lot of http 429 status codes.
This affects:
new deployments
scaling capabilities of cloud run
My desire:
Configure spring/servlet container to delay opening ports when application is actually ready
Delaying opening ports to the time the application is ready would ease much pain without interfering too much with the existing code base.
Any alternatives not causing too much pain?
Things i found and considered not viable
Using native-image is not an option as it is considered experimental and consumes more RAM at compile time than our deployment pipeline agents allow to allocate (max 8GB vs needed 13GB)
another answer i found: readiness check for google cloud run - how?
which i don't see how it could satisfy my needs, since spring-boot startup time is still slow. That's why my initial idea was to delay opening ports
I did not have time to test the following, but one thing i stumbled upon is
a blogpost about using multiple processes within a container. Though it is against the recommendation of containers principles, it seems viable for the time until cloud run supports probes of any type.
As you are well aware of the fact that “Cloud Run currently does not have a readiness/liveness check to avoid sending requests to unready applications” I would say there is not much that can be done on Cloud Run’s side except :
Try and optimise the Spring boot app as per the docs.
Make a heavier entrypoint in Cloud Run service that takes care of
more setup tasks. This stackoverflow thread mentions how “A
’heavier’ entrypoint will help post-deploy responsiveness, at the
cost of slower cold-starts” ( this is the most relevant solution
from a Cloud Run perspective and outlines the issue correctly)
Run multiple processes in a container in Cloud Run as you
mentioned.
This question seems more directed at Spring Boot specifically and I found an article with a similar requirement.
However, if you absolutely need the app ready to serve when requests come in, we have another alternative to Cloud Run, Google Kubernetes Engine (GKE) which makes use of readiness/liveness probes.
I have a problem with a server that called server A:
Server A: Red Hat Enterprise Linux Server release 7.2 (Maipo)
Server B: Red Hat Enterprise Linux Server release 7.7 (Maipo)
jdk-8u231 installed on all of servers.
I have an Spring Boot application running on 2 servers.
Whenever i use Jmeter to send 100 concurrency request to application running on each servers, the application running on Server B have no problem.
But in Server A, the application will be not responding, that mean the Process (PID) still running but I can't visit actuator endpoint, cannot visit Swagger page, cannot send new request ... log file show nothing since that time.
Thread dump and heap dump have no significant difference.
Could anyone show me how to analysis that problem?
I still have no idea why the problem occur.
Well, I can only speculate here, but here some ideas that can help:
There are two possible sources of issue here Java Application and Linux (+its network policies, firewalls and so forth).
Since You don't know for sure, what happens, try working by "elimination".
Create a script that will run 100 concurrent requests. Place the script at the Server A (the problematic one) and run The script will run against "localhost" (obviously). If you see that it works, then the issue is not in Java at all. Probably some network policies or linux setup, who knows.
Place a log message in the controller of the java application and examine the log. The log should print the request number among other things, so that you'll be able to understand whether you get stuck after a well defined number of requests or its always a different number.
Check the configurations of Spring Boot application. Maybe there is a difference in the number of threads allocated to serve the request by the embedded web server that runs inside the spring boot application (assuming you're not using a reactive stack) and this number differs. In this case you won't be able to call rest endpoints, actuator, etc.
If JMX connection is available to the setup, connect via the JMX and check the MBean of Tomcat (again, assuming there is a tomcat under the hood) to check pretty much the same information as in 4.
You've mentioned thread dumps. Try to take more than one thread dump but one before you're running JMeter test, one during the running (when everything still works), one when everything is stuck.
In the thread dumps check the actual stacktraces, maybe all the threads are stuck working with Database or something and can't serve requests like I've explained in "4"
Examine GC logs, maybe GC works so hard that you can't really interact with the application.
We are migrating some apps from WAS full profile to WAS Liberty profile.
Some apps have MDBs and need JMS Activation Specs definitions connected to MQ.
In order to enforce strict FIFO ordering of messages in a cluster, we set the "WAS_EndpointInitialState" property to "INACTIVE" on those Activation Specs to tell WAS full profile to not start the Activation Spec on startup. When the cluster starts, we start (ie "resume") the activation on one server only.
Q: How to achieve this with Liberty (v16.0.x) ?
I don't see an equivalent parameter within the "properties.wmqJms" stanza.
Thanks
Liberty doesn't have an equivalent parameter/capability for activation specs.
You can open a request for enhancement here:
https://www.ibm.com/developerworks/rfe/?PROD_ID=544
In case it helps during the meantime, a crude way of simulating the capability is to start the server with the jmsActivationSpec elements commented out, and make configuration updates to uncomment as you want them activated.
Unfortunately as-is (with v16.0.0.3 and the current beta version), it is not possible to deploy an application with MDBs in production due to a serious lack on functionalities in Liberty profile (JMS Activations).
When using the jmsActivationSpec+ properties.wmqJms stanzas, it is impossible:
to configure the activation to stop after x failed tentatives to consume the message. Liberty tries to consume the message forever without any notification!!
to start the activation in an inactive state on startup., so it is impossible to enforce the FIFO paradigm on a Q when deployed in a cluster (or collective or other form of cluster)
Those are already captured in the following RFE:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95885
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95794
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=88543
For us it's a clear no-go to move to WebSphere Liberty profile for those reasons
This is way too late for the OP, but in case someone comes here looking for a current answer.
Liberty / Open Liberty now offer (as of 18.0.0.1) such a function, which you can enable via the autoStart attribute, e.g.:
<jmsActivationSpec autoStart="false" id="myJMSActSpec"/>
See here for a quick example of how you would use the EndpointControl MBean and/or the server resume CLI command to start message delivery into the server.
I'm looking for an option to stop my Liberty server if a specific application fails to start.
I can't find any option for this in the docs, the closest thing I've found to achieve this is health policies but they don't look to be a good fit.
You could write something (even as simple as a bash script) to check the logs for:
"CWWKZ0001I: Application {appName} started"
and if you don't see it in x time, then execute "/wlp/bin/server stop {serverName}"
Could do it all through mbean invocations via java or REST calls by checking the state of the app (WebSphere:service=com.ibm.websphere.application.ApplicationMBean,name=*) and if it's not 'Started' by x time, then invoke an api to stop it (for collective environment you could use WebSphere:feature=collectiveController,type=ServerCommands,name=ServerCommands, otherwise you could use the osgi framework api).
I have a console app that uses Quartz and sends out emails on a schedule. I want some basic logging so I can see if the app has started and configured (NHibernate) correctly, and also any stack traces from uncaught exceptions.
I could set up Log4net or similar to write log entries to a database table, or email them to me. But is there a simple way built into AppHarbor?
Use the LogEntries Add on. Then install the le_log4net nuget package and enter your api key in the app.config. There is a Free subscription that should tell you what you need.