spring-boot client unable to start with consul - spring

I have setup and run consul using docker on my system using following command:
sudo docker run -p 8500:8500 consul:0.9.2
Consul is running fine as I can check from the consul UI (Image attached below):
Now, I am trying to run my spring-boot service to use this consul instance for service discovery and registration. But, whenever i start it gives me following exception:
2017-09-02 18:58:17.091 ERROR 5578 --- [ restartedMain] o.s.c.c.c.ConsulPropertySourceLocator : Fail fast is set and there was an error reading configuration from consul.
2017-09-02 18:58:18.183 ERROR 5578 --- [ restartedMain] o.s.c.c.c.ConsulPropertySourceLocator : Fail fast is set and there was an error reading configuration from consul.
2017-09-02 18:58:19.375 ERROR 5578 --- [ restartedMain] o.s.c.c.c.ConsulPropertySourceLocator : Fail fast is set and there was an error reading configuration from consul.
2017-09-02 18:58:20.691 ERROR 5578 --- [ restartedMain] o.s.c.c.c.ConsulPropertySourceLocator : Fail fast is set and there was an error reading configuration from consul.
2017-09-02 18:58:22.114 ERROR 5578 --- [ restartedMain] o.s.c.c.c.ConsulPropertySourceLocator : Fail fast is set and there was an error reading configuration from consul.
2017-09-02 18:58:23.671 ERROR 5578 --- [ restartedMain] o.s.c.c.c.ConsulPropertySourceLocator : Fail fast is set and there was an error reading configuration from consul.
2017-09-02 18:58:23.691 ERROR 5578 --- [ restartedMain] o.s.boot.SpringApplication : Application startup failed
com.ecwid.consul.v1.OperationException: OperationException(statusCode=500, statusMessage='Internal Server Error', statusContent='No cluster leader')
at com.ecwid.consul.v1.kv.KeyValueConsulClient.getKVValues(KeyValueConsulClient.java:159)
at com.ecwid.consul.v1.ConsulClient.getKVValues(ConsulClient.java:487)
at org.springframework.cloud.consul.config.ConsulPropertySource.init(ConsulPropertySource.java:66)
at org.springframework.cloud.consul.config.ConsulPropertySourceLocator.create(ConsulPropertySourceLocator.java:157)
at org.springframework.cloud.consul.config.ConsulPropertySourceLocator.locate(ConsulPropertySourceLocator.java:131)
at org.springframework.cloud.consul.config.ConsulPropertySourceLocator$$FastClassBySpringCGLIB$$b35ebf8.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.retry.interceptor.RetryOperationsInterceptor$1.doWithRetry(RetryOperationsInterceptor.java:91)
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:163)
at org.springframework.retry.interceptor.RetryOperationsInterceptor.invoke(RetryOperationsInterceptor.java:118)
at org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:152)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
at org.springframework.cloud.consul.config.ConsulPropertySourceLocator$$EnhancerBySpringCGLIB$$66375879.locate(<generated>)
at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:93)
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:567)
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:338)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:301)
at com.pyg.auth.AuthServiceApp.main(AuthServiceApp.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
SpringBoot Main Class is annotated fine (I guess), because my spring-boot service was working fine with another consul instance earlier.

I was able to solve my issue by moving consul server and port configuration from application.yml to bootstrap.yml.
I don't know much about how it was solved and why it was unable to read from application.yml. If anyone has some details about it, pl. let me know.

Use sudo docker run -p 8500:8500 consul:0.9.2 -bootstrap to resolve the problem. I have
added -bootstrap one argument to bootstrap the server/servers and get elected as a leader. Manual bootstrapping with -bootstrap is not recommended in newer versions of Consul (0.5 and newer) as it is more error-prone.
Explanation:-
As said in docs Before a Consul cluster can begin to service requests, a server node must be elected leader. And this is reason of your exception on start of spring-boot service the leader has not been elected yet!!
Why the leader has not been elected? The list of servers involved in the cluster should be bootstrapped. And the servers can be bootstrapped using the
-bootstrap-expect configuration option. Recommended
Note:- Just for testing/learning purpose you can go ahead and create a single server because A single server deployment is highly discouraged as data loss is inevitable in a failure scenario.

Check your pom.xml.
Don't use spring-cloud-starter-consul-all. Use spring-cloud-starter-consul-discovery instead.
That worked for me.

Please check pom.xml to avoid using spring-cloud-starter-consul-all
If you use spring-cloud-starter-consul-all, please write the configuration to bootstrap.yml, the bootstrap.yml configuration file is the first to load.

Related

How to debug Spring Cloud Kubernetes Configuration Watcher when it is not working as expectted

There are two components in my k8s namespace.
The first component is a simple Spring Boot Web Service with actuator/refresh endpoint exposed. I have mannualy make post request to the endpoint and it will trigger a configuration context refresh succesfully.
The second component is Spring Cloud Kubernetes Configuration Watcher which I pull the image per official document guidance and make it run in my k8s env. Per official spring document, it should detect the changes on configmap with label spring.cloud.kubernetes.config="true" and make post request to the actuator/refresh endpoint of the application whose spring.application.name is equal to the configmap name.
The second component is not working as expectted, and I don't know how to trouble shoot the root cause.
My Springboot application is called spring-boot-demo, the configmap is also named as spring-boot-demo. But I never find any mentions of "spring-boot-demo" in Spring Cloud Kubernetes Configuration Watcher's log and nor can I assure if it has sent post request to related endpoint.
I can only see the logs showed up repeatedly below:
2021-11-22 02:58:53.332 INFO 1 --- [192.168.0.1/...] .w.HttpBasedConfigMapWatchChangeDetector : Added new Kubernetes watch: config-maps-watch-event
2021-11-22 02:58:53.332 INFO 1 --- [192.168.0.1/...] .w.HttpBasedConfigMapWatchChangeDetector : Kubernetes event-based configMap change detector activated
2021-11-22 03:34:06.555 WARN 1 --- [192.168.0.1/...] .f.c.r.EventBasedConfigMapChangeDetector : ConfigMaps watch closed
io.fabric8.kubernetes.client.WatcherException: too old resource version: 5491278743 (5554041906)
at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.onStatus(AbstractWatchManager.java:263) [kubernetes-client-5.5.0.jar:na]
at io.fabric8.kubernetes.client.dsl.internal.AbstractWatchManager.onMessage(AbstractWatchManager.java:247) [kubernetes-client-5.5.0.jar:na]
at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onMessage(WatcherWebSocketListener.java:93) [kubernetes-client-5.5.0.jar:na]
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:322) [okhttp-3.14.9.jar:na]
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219) [okhttp-3.14.9.jar:na]
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105) [okhttp-3.14.9.jar:na]
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:273) [okhttp-3.14.9.jar:na]
at okhttp3.internal.ws.RealWebSocket$1.onResponse(RealWebSocket.java:209) [okhttp-3.14.9.jar:na]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:174) [okhttp-3.14.9.jar:na]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [okhttp-3.14.9.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_312]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_312]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 5491278743 (5554041906)
... 13 common frames omitted
2021-11-22 03:34:06.605 INFO 1 --- [192.168.0.1/...] .f.c.r.EventBasedConfigMapChangeDetector : Added new Kubernetes watch: config-maps-watch-event
2021-11-22 03:34:06.605 INFO 1 --- [192.168.0.1/...] .f.c.r.EventBasedConfigMapChangeDetector : Kubernetes event-based configMap change detector activated
2021-11-22 03:34:06.607 INFO 1 --- [192.168.0.1/...] s.c.k.f.c.Fabric8ConfigMapPropertySource : Loading ConfigMap with name 'spring-cloud-kubernetes-configuration-watcher' in namespace 'my-namespace'
2021-11-22 03:34:06.621 WARN 1 --- [192.168.0.1/...] o.s.c.k.f.config.Fabric8ConfigUtils : config-map with name : 'spring-cloud-kubernetes-configuration-watcher' not present in namespace : 'my-namespace'
2021-11-22 03:34:06.625 WARN 1 --- [192.168.0.1/...] o.s.c.k.f.config.Fabric8ConfigUtils : config-map with name : 'spring-cloud-kubernetes-configuration-watcher-kubernetes' not present in namespace : 'my-namespace'
The docker image I use is springcloud/spring-cloud-kubernetes-configuration-watcher:2.1.0-RC1
Any hint to debug this issue is appreciated.

systemctl spring boot application The web application [ROOT] appears to have started a thread named

I developed the spring boot micro service on my windows 10 and it is running with no problems. after i deploying to my aws ec2 instance, i can run with
java -jar app.jar
however if i use systemctl to run it, it gave me
The web application [ROOT] appears to have started a thread named [RxIoScheduler-1 (Evictor)] but has failed to stop
i installed two java 11,
one is using yum install java,
the other is sdk install java
the reason why i installed two java is because systemctl cannot run the app with the java installed by sdkman.
I suspect it is because systemctl is always run under root.
i also have tried put it into .sh file
my app.service file:
[Unit]
Description=zuul service
[Service]
# The configuration file application.properties should be here:
#change this to your workspace
User=root
WorkingDirectory=/home/ec2-user
#path to executable.
#executable is a bash script which calls jar file
ExecStart=/usr/bin/java -jar /home/ec2-user/zuul/target/zuul.jar
SuccessExitStatus=143
TimeoutStopSec=10
RestartSec=5
[Install]
WantedBy=multi-user.target
this is the error message:
Stopping service [Tomcat]
2019-10-31 03:22:14.811 WARN 6973 --- [ main] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [spring.cloud.inetutils] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base#11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
java.base#11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
java.base#11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
java.base#11.0.5/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
java.base#11.0.5/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base#11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base#11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base#11.0.5/java.lang.Thread.run(Thread.java:834)
2019-10-31 03:22:14.811 WARN 6973 --- [ main] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [RxIoScheduler-1 (Evictor)] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base#11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
java.base#11.0.5/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
java.base#11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
java.base#11.0.5/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
java.base#11.0.5/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
java.base#11.0.5/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
java.base#11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
java.base#11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base#11.0.5/java.lang.Thread.run(Thread.java:834)
2019-10-31 03:22:14.818 INFO 6973 --- [ main] ConditionEvaluationReportLoggingListener :

The Tomcat connector configured to listen on port 28081 failed to start in the linux

I'm doing a spring boot project , and this project can run successfully in the eclipse and war use cmd. However it failed to start in the linux environment.
I have used netstat -ano | grep 28081 , the port is not in use.
2019-04-02 05:49:50.628 WARN 25548 --- [localhost-startStop-1] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [cluster-ClusterId{value='5ca33035b36bfd63cc6ff647', description='null'}-lswsstpfdb3u.nam.nsroot.net:28120] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:226)
com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:207)
com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:154)
java.lang.Thread.run(Thread.java:745)
2019-04-02 05:49:50.640 INFO 25548 --- [defaultTaskExecutor-4] utoConfigurationReportLoggingInitializer :
Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2019-04-02 05:49:50.646 ERROR 25548 --- [main] o.s.b.d.LoggingFailureAnalysisReporter :
***************************
APPLICATION FAILED TO START
***************************
Description:
The Tomcat connector configured to listen on port 28081 failed to start. The port may already be in use or the connector may be misconfigured.
Action:
Verify the connector's configuration, identify and stop any process that's listening on port 28081, or configure this application to listen on another port.
Local war instruction is :
C:\tmp\jdk1.8.0_31_x64\jre\bin\java -DaEnv=UAT -Denv=UAT -Da.domainName=pfDomain -Drealtime.nodeName=Node1 -jar C:\aServices.war -Djava.protocol.handler.pkgs=javax.net.ssl -Djavax.net.ssl.keyStore=C:\SSL-keystore.UAT.jks -Djavax.net.ssl.trustStore=C:\SSLTrust.UAT.jks
The application can start successfully.
In the application.properties file , the ssl location is wrong , it should like this :
server.ssl.key-store=/opt/a/etc/SSL-keystore.jks
server.ssl.trust-sotre=/opt/a/etc/certs/SSLTrust.jks

JMS ActiveMQ SpringBoot .FailoverTransport

iam trying to connected to remote broker url in activeMQ (activemq installed in unix vm)
iam able to connect from browser from my laptop.
while running springboot iam getting this error
--- [ActiveMQ Task-1] o.a.a.t.failover.FailoverTransport : Failed to connect to [tcp://http://199.247.18.11:61616] after: 8 attempt(s) continuing to retry.
what could be the issue?
Please remove https:// from your connection string. Port 61616 is expecting JMS connections.
Your connection string should be tcp://199.247.18.11:61616 or something similar. There is a rest API that (I think) goes through the built in HTTP server but it's not going to listen on 61616 and it's going to have a much longer URL. Something like
http://admin:admin#localhost:8161/api/message?destination=queue://myqueue
still issue
yml file
activemq:
broker-url: failover:(tcp://http://199.247.18.11:61616)?initialReconnectDelay=1000&maxReconnectDelay=60000&warnAfterReconnectAttempts=2
error:
2018-05-01 07:41:51.312 WARN 6560 --- [ActiveMQ Task-1] o.a.a.t.failover.FailoverTransport : Failed to connect to [tcp://http://199.247.18.11:61616] after: 2 attempt(s) continuing to retry.

Error starting jHipster Microservices with consul

I'm trying to build sample microservice app using this tutorial.
Jhipster version is 4.0.6
So i've created gateway, service and started consul using this command:
docker-compose -f src/main/docker/consul.yml up
from my gateway directory.
But the error occur on Spring Boot startup, here is the log:
2017-02-22 11:52:25.679 ERROR 3168 --- [ restartedMain] o.s.c.c.c.ConsulPropertySourceLocator : Fail fast is set and there was an error reading configuration from consul.
2017-02-22 11:52:32.491 WARN 3168 --- [ restartedMain] o.s.boot.SpringApplication : Error handling failed (ApplicationEventMulticaster not initialized - call 'refresh' before multicasting events via the context: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext#2a5b2096: startup date [Thu Jan 01 03:00:00 AST 1970]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext#5108df79)
com.ecwid.consul.transport.TransportException: java.net.ConnectException: Connection refused: connect
Could you please assist with this issue?
UPDATE:
I've found that app tries to make GET request to URL on staptup:
http://localhost:8500/v1/kv/config/armory,dev/?recurse&token=
But the only data stored in Consul K/V storage is:
KEY: config/application/data
VALUE:
configserver:
name: Docker Consul Service
status: Connected to Consul Server running in Docker
jhipster:
security:
authentication:
jwt:
secret: my-secret-token-to-change-in-production
You must copy your application yaml configurations into your consul instance as explained in the doc into central-server-config directory if consul is running in dev profile or in its git repo if consul is running in prod profile.
So assuming your app is named "armory" you should copy your src/main/resources/config/application.yml to armory.yml and for each profiles (e.g. application-dev.yml to armory-dev.yml)

Resources