I am trying to implement a spring actuator for readiness and liveness,
so if I deploy it with enabled probe, a graceful shutdown occurs.
However, for other services with the same settings and dependencies it works well, as well as if I run service with actuator on localhost it works well. What should I check?
my configs
probes:
enabled: true
ready:
path: /actuator/health/readiness
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 3
failureThreshold: 15
live:
path: /actuator/health/liveness
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 3
failureThreshold: 15
I have tried different settings for actuator
18:19:33,559 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
18:19:33,560 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
18:19:33,561 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/app.jar!/BOOT-INF/lib/sbol2-lib-commons2-1.0.0-SNAPSHOT.jar!/logback.xml]
18:19:33,578 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList#30c7da1e - URL [jar:file:/app.jar!/BOOT-INF/lib/sbol2-lib-commons2-1.0.0-SNAPSHOT.jar!/logback.xml] is not of type file
18:19:33,889 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
18:19:33,894 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [consoleAppender]
18:19:34,072 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.LoggingEventJsonProviders] for [providers] property
18:19:34,173 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.LoggingEventFormattedTimestampJsonProvider] for [timestamp] property
18:19:34,175 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.LogLevelJsonProvider] for [logLevel] property
18:19:34,176 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.ThreadNameJsonProvider] for [threadName] property
18:19:34,178 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.LoggerNameJsonProvider] for [loggerName] property
18:19:34,180 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.MdcJsonProvider] for [mdc] property
18:19:34,183 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.StackTraceJsonProvider] for [stackTrace] property
18:19:34,186 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.StackHashJsonProvider] for [stackHash] property
18:19:34,189 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.LoggingEventPatternJsonProvider] for [pattern] property
18:19:34,770 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to ALL
18:19:34,770 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [consoleAppender] to Logger[ROOT]
18:19:34,771 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
18:19:34,772 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator#3ecd23d9 - Registering current configuration as safe fallback point
{"#timestamp":"2022-10-20T18:19:35.182+06:00","level":"TRACE","thread_name":"main","logger_name":"o.s.c.i.s.SpringFactoriesLoader","message":"Loaded [org.springframework.boot.logging.LoggingSystemFactory] names: [org.springframework.boot.logging.logback.LogbackLoggingSystem.Factory, org.springframework.boot.logging.log4j2.Log4J2LoggingSystem.Factory, org.springframework.boot.logging.java.JavaLoggingSystem.Factory]"}
18:19:36,597 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList#bd4dc25 - URL [jar:file:/app.jar!/BOOT-INF/lib/sbol2-lib-commons2-1.0.0-SNAPSHOT.jar!/logback.xml] is not of type file
18:19:36,658 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
18:19:36,659 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [consoleAppender]
18:19:36,665 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.LoggingEventJsonProviders] for [providers] property
18:19:36,665 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.LoggingEventFormattedTimestampJsonProvider] for [timestamp] property
18:19:36,665 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.LogLevelJsonProvider] for [logLevel] property
18:19:36,666 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.ThreadNameJsonProvider] for [threadName] property
18:19:36,666 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.LoggerNameJsonProvider] for [loggerName] property
18:19:36,666 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.MdcJsonProvider] for [mdc] property
18:19:36,667 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.StackTraceJsonProvider] for [stackTrace] property
18:19:36,667 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.StackHashJsonProvider] for [stackHash] property
18:19:36,668 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [net.logstash.logback.composite.loggingevent.LoggingEventPatternJsonProvider] for [pattern] property
18:19:36,670 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to ALL
18:19:36,670 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator#f736069 - Propagating ALL level on Logger[ROOT] onto the JUL framework
18:19:36,671 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [consoleAppender] to Logger[ROOT]
18:19:36,671 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
18:19:36,671 |-INFO in org.springframework.boot.logging.logback.SpringBootJoranConfigurator#4229bb3f - Registering current configuration as safe fallback point
{"#timestamp":"2022-10-20T18:19:36.677+06:00","level":"TRACE","thread_name":"main","logger_name":"o.s.beans.BeanUtils","message":"No property editor [org.springframework.boot.logging.LogLevelEditor] found for type org.springframework.boot.logging.LogLevel according to 'Editor' suffix convention"}
18:19:36,680 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator#f736069 - Propagating INFO level on Logger[ROOT] onto the JUL framework
{"#timestamp":"2022-10-20T18:19:37.157+06:00","level":"INFO","thread_name":"main","logger_name":"k.s.s.a.l.LimitsApplication","message":"Starting LimitsApplication v1.0.13 using Java 15.0.2 on limits-5c859d4dbb-w642r with PID 1 (/app.jar started by 1000630000 in /)"}
{"#timestamp":"2022-10-20T18:19:37.159+06:00","level":"INFO","thread_name":"main","logger_name":"k.s.s.a.l.LimitsApplication","message":"The following profiles are active: ift"}
{"#timestamp":"2022-10-20T18:19:40.777+06:00","level":"INFO","thread_name":"main","logger_name":"o.s.d.r.c.RepositoryConfigurationDelegate","message":"Bootstrapping Spring Data JPA repositories in DEFAULT mode."}
{"#timestamp":"2022-10-20T18:19:41.484+06:00","level":"INFO","thread_name":"main","logger_name":"o.s.d.r.c.RepositoryConfigurationDelegate","message":"Finished Spring Data repository scanning in 694 ms. Found 9 JPA repository interfaces."}
{"#timestamp":"2022-10-20T18:19:44.194+06:00","level":"INFO","thread_name":"main","logger_name":"c.u.j.c.EnableEncryptablePropertiesBeanFactoryPostProcessor","message":"Post-processing PropertySource instances"}
{"#timestamp":"2022-10-20T18:19:44.296+06:00","level":"INFO","thread_name":"main","logger_name":"c.u.j.EncryptablePropertySourceConverter","message":"Converting PropertySource configurationProperties [org.springframework.boot.context.properties.source.ConfigurationPropertySourcesPropertySource] to AOP Proxy"}
{"#timestamp":"2022-10-20T18:19:44.298+06:00","level":"INFO","thread_name":"main","logger_name":"c.u.j.EncryptablePropertySourceConverter","message":"Converting PropertySource systemProperties [org.springframework.core.env.PropertiesPropertySource] to EncryptableMapPropertySourceWrapper"}
{"#timestamp":"2022-10-20T18:19:44.298+06:00","level":"INFO","thread_name":"main","logger_name":"c.u.j.EncryptablePropertySourceConverter","message":"Converting PropertySource systemEnvironment [org.springframework.boot.env.SystemEnvironmentPropertySourceEnvironmentPostProcessor$OriginAwareSystemEnvironmentPropertySource] to EncryptableSystemEnvironmentPropertySourceWrapper"}
{"#timestamp":"2022-10-20T18:19:44.299+06:00","level":"INFO","thread_name":"main","logger_name":"c.u.j.EncryptablePropertySourceConverter","message":"Converting PropertySource random [org.springframework.boot.env.RandomValuePropertySource] to EncryptablePropertySourceWrapper"}
{"#timestamp":"2022-10-20T18:19:44.299+06:00","level":"INFO","thread_name":"main","logger_name":"c.u.j.EncryptablePropertySourceConverter","message":"Converting PropertySource Config resource 'class path resource [application-ift.yaml]' via location 'optional:classpath:/' [org.springframework.boot.env.OriginTrackedMapPropertySource] to EncryptableMapPropertySourceWrapper"}
{"#timestamp":"2022-10-20T18:19:44.299+06:00","level":"INFO","thread_name":"main","logger_name":"c.u.j.EncryptablePropertySourceConverter","message":"Converting PropertySource Config resource 'class path resource [application.yaml]' via location 'optional:classpath:/' [org.springframework.boot.env.OriginTrackedMapPropertySource] to EncryptableMapPropertySourceWrapper"}
{"#timestamp":"2022-10-20T18:19:44.483+06:00","level":"INFO","thread_name":"main","logger_name":"c.u.j.f.DefaultLazyPropertyFilter","message":"Property Filter custom Bean not found with name 'encryptablePropertyFilter'. Initializing Default Property Filter"}
{"#timestamp":"2022-10-20T18:19:45.901+06:00","level":"INFO","thread_name":"main","logger_name":"c.u.j.r.DefaultLazyPropertyResolver","message":"Property Resolver custom Bean not found with name 'encryptablePropertyResolver'. Initializing Default Property Resolver"}
{"#timestamp":"2022-10-20T18:19:45.974+06:00","level":"INFO","thread_name":"main","logger_name":"c.u.j.d.DefaultLazyPropertyDetector","message":"Property Detector custom Bean not found with name 'encryptablePropertyDetector'. Initializing Default Property Detector"}
{"#timestamp":"2022-10-20T18:19:46.597+06:00","level":"INFO","thread_name":"main","logger_name":"o.s.b.w.e.t.TomcatWebServer","message":"Tomcat initialized with port(s): 9090 (http)"}
{"#timestamp":"2022-10-20T18:19:46.662+06:00","level":"INFO","thread_name":"main","logger_name":"o.a.c.http11.Http11NioProtocol","message":"Initializing ProtocolHandler [\"http-nio-9090\"]"}
{"#timestamp":"2022-10-20T18:19:46.662+06:00","level":"INFO","thread_name":"main","logger_name":"o.a.c.core.StandardService","message":"Starting service [Tomcat]"}
{"#timestamp":"2022-10-20T18:19:46.663+06:00","level":"INFO","thread_name":"main","logger_name":"o.a.c.core.StandardEngine","message":"Starting Servlet engine: [Apache Tomcat/9.0.48]"}
{"#timestamp":"2022-10-20T18:19:47.993+06:00","level":"INFO","thread_name":"main","logger_name":"o.h.j.internal.util.LogHelper","message":"HHH000204: Processing PersistenceUnitInfo [name: default]"}
{"#timestamp":"2022-10-20T18:19:48.185+06:00","level":"INFO","thread_name":"main","logger_name":"org.hibernate.Version","message":"HHH000412: Hibernate ORM core version 5.4.32.Final"}
{"#timestamp":"2022-10-20T18:19:48.506+06:00","level":"INFO","thread_name":"main","logger_name":"o.h.annotations.common.Version","message":"HCANN000001: Hibernate Commons Annotations {5.1.2.Final}"}
{"#timestamp":"2022-10-20T18:19:48.786+06:00","level":"WARN","thread_name":"main","logger_name":"com.zaxxer.hikari.HikariConfig","message":"HikariPool-1 - leakDetectionThreshold is less than 2000ms or more than maxLifetime, disabling it."}
{"#timestamp":"2022-10-20T18:19:48.787+06:00","level":"WARN","thread_name":"main","logger_name":"com.zaxxer.hikari.HikariConfig","message":"HikariPool-1 - idleTimeout has been set but has no effect because the pool is operating as a fixed size pool."}
{"#timestamp":"2022-10-20T18:19:48.787+06:00","level":"INFO","thread_name":"main","logger_name":"c.z.hikari.HikariDataSource","message":"HikariPool-1 - Starting..."}
{"#timestamp":"2022-10-20T18:19:49.595+06:00","level":"INFO","thread_name":"main","logger_name":"c.z.hikari.HikariDataSource","message":"HikariPool-1 - Start completed."}
{"#timestamp":"2022-10-20T18:19:49.806+06:00","level":"INFO","thread_name":"main","logger_name":"org.hibernate.dialect.Dialect","message":"HHH000400: Using dialect: org.hibernate.dialect.Oracle12cDialect"}
{"#timestamp":"2022-10-20T18:19:50.771+06:00","level":"INFO","thread_name":"main","logger_name":"o.h.e.b.i.EnversServiceImpl","message":"Envers integration enabled? : true"}
{"#timestamp":"2022-10-20T18:19:52.074+06:00","level":"INFO","thread_name":"main","logger_name":"o.h.v.internal.util.Version","message":"HV000001: Hibernate Validator 6.2.0.Final"}
{"#timestamp":"2022-10-20T18:19:55.159+06:00","level":"INFO","thread_name":"main","logger_name":"o.h.e.t.j.p.i.JtaPlatformInitiator","message":"HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]"}
{"#timestamp":"2022-10-20T18:19:55.173+06:00","level":"INFO","thread_name":"main","logger_name":"o.s.o.j.LocalContainerEntityManagerFactoryBean","message":"Initialized JPA EntityManagerFactory for persistence unit 'default'"}
{"#timestamp":"2022-10-20T18:20:08.260+06:00","level":"INFO","thread_name":"main","logger_name":"o.s.b.w.e.t.TomcatWebServer","message":"Tomcat started on port(s): 9090 (http) with context path ''"}
{"#timestamp":"2022-10-20T18:20:08.470+06:00","level":"INFO","thread_name":"main","logger_name":"o.s.b.w.e.t.TomcatWebServer","message":"Tomcat initialized with port(s): 8080 (http)"}
{"#timestamp":"2022-10-20T18:20:08.502+06:00","level":"INFO","thread_name":"main","logger_name":"o.s.b.a.e.w.EndpointLinksResolver","message":"Exposing 2 endpoint(s) beneath base path '/actuator'"}
{"#timestamp":"2022-10-20T18:20:08.703+06:00","level":"INFO","thread_name":"main","logger_name":"o.s.b.w.e.t.TomcatWebServer","message":"Tomcat started on port(s): 8080 (http) with context path ''"}
{"#timestamp":"2022-10-20T18:20:10.382+06:00","level":"INFO","thread_name":"main","logger_name":"k.s.s.a.l.LimitsApplication","message":"Started LimitsApplication in 35.283 seconds (JVM running for 37.72)"}
{"#timestamp":"2022-10-20T18:20:10.472+06:00","level":"INFO","thread_name":"main","logger_name":"k.sber.sbol2.app.limits.Runner","message":"Limits is running..."}
{"#timestamp":"2022-10-20T18:20:10.475+06:00","level":"INFO","thread_name":"SpringApplicationShutdownHook","logger_name":"o.s.b.w.e.t.GracefulShutdown","message":"Commencing graceful shutdown. Waiting for active requests to complete"}
{"#timestamp":"2022-10-20T18:20:10.487+06:00","level":"INFO","thread_name":"tomcat-shutdown","logger_name":"o.s.b.w.e.t.GracefulShutdown","message":"Graceful shutdown complete"}
{"#timestamp":"2022-10-20T18:20:10.563+06:00","level":"INFO","thread_name":"SpringApplicationShutdownHook","logger_name":"o.s.b.w.e.t.GracefulShutdown","message":"Commencing graceful shutdown. Waiting for active requests to complete"}
{"#timestamp":"2022-10-20T18:20:10.570+06:00","level":"INFO","thread_name":"tomcat-shutdown","logger_name":"o.s.b.w.e.t.GracefulShutdown","message":"Graceful shutdown complete"}
{"#timestamp":"2022-10-20T18:20:12.778+06:00","level":"INFO","thread_name":"SpringApplicationShutdownHook","logger_name":"o.s.o.j.LocalContainerEntityManagerFactoryBean","message":"Closing JPA EntityManagerFactory for persistence unit 'default'"}
{"#timestamp":"2022-10-20T18:20:12.781+06:00","level":"INFO","thread_name":"SpringApplicationShutdownHook","logger_name":"c.z.hikari.HikariDataSource","message":"HikariPool-1 - Shutdown initiated..."}
{"#timestamp":"2022-10-20T18:20:13.013+06:00","level":"INFO","thread_name":"SpringApplicationShutdownHook","logger_name":"c.z.hikari.HikariDataSource","message":"HikariPool-1 - Shutdown completed."}
Normal Scheduled 8m42s default-scheduler Successfully assigned x/limits-5c859d4dbb-p5jgt to ocp-t-w002
Normal AddedInterface 8m41s multus Add eth0 [10.111.6.127/24] from openshift-sdn
Normal Pulled 8m38s kubelet Successfully pulled image "nexus.x.x/x/limits:1.0.13" in 2.961535874s
Normal Killing 8m13s kubelet Container limits failed liveness probe, will be restarted
Warning Unhealthy 8m13s (x3 over 8m33s) kubelet Liveness probe failed: Get "http://10.111.6.127:8080/actuator/health/liveness": dial tcp 10.111.6.127:8080: connect: connection refused
Normal Pulling 7m43s (x2 over 8m41s) kubelet Pulling image "nexus.x.x:x/x/limits:1.0.13"
Normal Pulled 7m43s kubelet Successfully pulled image "nexus.x:x/x/limits:1.0.13" in 23.406574ms
Normal Created 7m42s (x2 over 8m38s) kubelet Created container limits
Normal Started 7m42s (x2 over 8m37s) kubelet Started container limits
Warning Unhealthy 3m41s (x51 over 8m36s) kubelet Readiness probe failed: Get "http://10.111.6.127:8080/actuator/health/readiness": dial tcp 10.111.6.127:8080:
I wrote a spring boot app that I am dockerizing. My app communicates with a mongodb (running locally on my box) and with a rabbitMq server (running in a docker container locally on my box). Running under my dev profile (having replaced localhost or 127.0.0.1 with *host.docker.internal) everything works.
Here is my Dockerfile
# Filename: Dockerfile
# It is assumed that you are in the root directory for this project
FROM openjdk:8
ADD /target/monitor-msg-consumer-0.0.1-SNAPSHOT.jar monitor-msg-consumer-0.0.1-SNAPSHOT.jar
EXPOSE 11371
## Profile: dev
ENTRYPOINT ["java","-jar", "-Dlog_home=/tmp","monitor-msg-consumer-0.0.1-SNAPSHOT.jar"]
I create an image, then run that image using the following command:
docker run --network=host -p 11371:11371 -e "SPRING_PROFILES_ACTIVE=dev" --name monitor-msg-consumer monitor-msg-consumer-img:latest
And as stated the app runs and I can interact with it under the dev profile.
Now, if I delete that container and run it with my test profile using the command:
docker run --network=host -p 11371:11371 -e "SPRING_PROFILES_ACTIVE=test" --name monitor-msg-consumer monitor-msg-consumer-img:latest
the app can't find an external rabbitMq server running in our test environment.
What am I missing?
I am new to docker. So, go easy on me!
Here are the logs from the docker container:
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.7.RELEASE)
2019-09-30T12:46:43.344207600Z
12:46:44,049 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList#62fdb4a6 - URL [jar:file:/monitor-msg-consumer-0.0.1-SNAPSHOT.jar!/BOOT-INF/classes!/logback.xml] is not of type file
12:46:44,053 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Will scan for changes in [jar:file:/monitor-msg-consumer-0.0.1-SNAPSHOT.jar!/BOOT-INF/classes!/logback.xml]
12:46:44,053 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Setting ReconfigureOnChangeTask scanning period to 30 seconds
12:46:44,054 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:46:44,054 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
12:46:44,055 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
12:46:44,056 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
12:46:44,057 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILE]
12:46:44,059 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy#500179317 - No compression will be used
12:46:44,061 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy#500179317 - Will use the pattern /tmp/archived/monitor_msg_consumer.%d{yyyy-MM-dd}.%i.log for the active file
12:46:44,062 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP#31206beb - The date pattern is 'yyyy-MM-dd' from file name pattern '/tmp/archived/monitor_msg_consumer.%d{yyyy-MM-dd}.%i.log'.
12:46:44,067 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP#31206beb - Roll-over at midnight.
12:46:44,067 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP#31206beb - Setting initial period to Mon Sep 30 12:46:42 UTC 2019
12:46:44,068 |-WARN in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP#31206beb - SizeAndTimeBasedFNATP is deprecated. Use SizeAndTimeBasedRollingPolicy instead
12:46:44,068 |-WARN in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP#31206beb - For more information see http://logback.qos.ch/manual/appenders.html#SizeAndTimeBasedRollingPolicy
12:46:44,068 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: /tmp/monitor_msg_consumer.log
12:46:44,068 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to [/tmp/monitor_msg_consumer.log]
12:46:44,069 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [gov.utah.dts.monitor] to DEBUG
12:46:44,069 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator#3e77a1ed - Propagating DEBUG level on Logger[gov.utah.dts.monitor] onto the JUL framework
12:46:44,069 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to DEBUG
12:46:44,069 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
12:46:44,069 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILE] to Logger[ROOT]
12:46:44,070 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
12:46:44,070 |-INFO in org.springframework.boot.logging.logback.SpringBootJoranConfigurator#3ffcd140 - Registering current configuration as safe fallback point
2019-09-30 12:46:44 | ERROR | [main] o.s.c.l.LoggingRebinder:83 - Cannot set level: WARN #(Options: DEBUG, ERROR, FATAL, INFO, WARN) for 'org.springframework'
12:46:44,074 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator#3e77a1ed - Propagating WARN level on Logger[ROOT] onto the JUL framework
2019-09-30 12:46:44 | ERROR | [main] o.s.c.l.LoggingRebinder:83 - Cannot set level: DEBUG #(Options: DEBUG, ERROR, FATAL, INFO, WARN) for 'gov.utah.dts.monitor'
2019-09-30 12:46:44 | ERROR | [main] o.s.c.l.LoggingRebinder:83 - Cannot set level: DEBUG #(Options: ALL, DEBUG, ERROR, FATAL, INFO, WARN) for 'gov.utah.dts'
2019-09-30 12:46:44 | INFO | [main] g.u.d.m.MainApp:651 - The following profiles are active: test
2019-09-30 12:46:54 | INFO | [main] g.u.d.m.MainApp:59 - Started MainApp in 13.437 seconds (JVM running for 14.251)
12:47:14,053 |-INFO in ReconfigureOnChangeTask(born:1569847604053) - Empty watch file list. Disabling
12:47:44,053 |-INFO in ReconfigureOnChangeTask(born:1569847604053) - Empty watch file list. Disabling
12:48:14,053 |-INFO in ReconfigureOnChangeTask(born:1569847604053) - Empty watch file list. Disabling
12:48:44,054 |-INFO in ReconfigureOnChangeTask(born:1569847604053) - Empty watch file list. Disabling
12:49:14,053 |-INFO in ReconfigureOnChangeTask(born:1569847604053) - Empty watch file list. Disabling
12:49:44,052 |-INFO in ReconfigureOnChangeTask(born:1569847604053) - Empty watch file list. Disabling
This code would make one think that things are running smoothly, but what I do not see on the rabbitMq server, itself, is a consumer that should be this application. That is why I believe it is not working.
When I run this application non-dockerized, then I see the consumer for the specified queue.
Here are application-test.properties:
#Port Range 11370 - 11379
server:
port: 11371
spring:
## Rabbit
rabbitmq:
host: 172.23.2.12
port: 5672
username: guest
password: guest
## MongoDB
data:
mongodb:
host: 172.23.1.242
port: 20000
database: **********
username: **********
password: **********
# Only required if you're hitting a local database w/o username and pswd
uri: mongodb://**********:**********#***.***.***.242:20000/**********
rabbitmq:
queue: erepMonitorQueue
durable: true
exchange: erepMonitorExchg
route: erepMonitorRoute
logging:
level:
root: WARN
# Only print warning messages and above from external libraries
gov.utah.dts: DEBUG #(Options: ALL, DEBUG, ERROR, FATAL, INFO, WARN)
gov.utah.dts.monitor: DEBUG #(Options: DEBUG, ERROR, FATAL, INFO, WARN)
org.springframework: INFO #(Options: DEBUG, ERROR, FATAL, INFO, WARN)
I have a storm project for which I have added some custom logback configurations using logback.xml (added configs to write logs to an index). When I build the jar and deploy the same on the cluster, I get the multiple bindings errors for this class StaticLoggerBinder.class due to it being present both in my jar and storm's own dependency list. Below is the error
SLF4J: Found binding in [jar:file:/usr/lib/storm/apache-storm-1.2.2/lib/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/app/test/test-storm-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
This causes none of my custom configs to be picked up.
When I remove the conflicting jar from Storm's lib folder, my logback.xml configs are reflected and my logs are written to the index with no issues. I also see these logs in the beginning when I deploy the topology showing the configs I have added
22:05:13,818 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/app/test/test-storm-0.0.55-SNAPSHOT-jar-with-dependencies.jar!/logback.xml]
22:05:13,839 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList#19bb089b - URL [jar:file:/app/test/test-storm-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/logback.xml] is not of type file
22:05:14,094 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
22:05:14,097 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [CONSOLE]
22:05:14,102 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
22:05:14,125 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [com.splunk.logging.HttpEventCollectorLogbackAppender]
22:05:14,126 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [SPLUNK]
22:05:14,132 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.test] to INFO
22:05:14,132 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [com.test] to false
22:05:14,132 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [SPLUNK] to Logger[com.test]
22:05:14,133 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [CONSOLE] to Logger[com.test]
22:05:14,133 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
22:05:14,133 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [CONSOLE] to Logger[ROOT]
22:05:14,133 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
However, I do not want to take this route since it cannot be maintained so I decided use the maven-shade-plugin to relocate the org.slf4j.impl to org.shaded.slf4j.impl. Now when I deploy the topology there is no multiple bindings anymore, however my logback.xml is not honored and none of my configs are reflected nor do I see the above messages. It probably falls back to using a different logger.
How can I make my configs to be reflected when I deploy the topology? Please help. (Storm Version 1.2.2)
Storm switched from Logback to Log4j2 a few years ago. You can configure Log4j2 via the storm/log4j2/worker.xml and storm/log4j2/cluster.xml files. The former configures logging for your topology, as well as the worker JVM.
You should exclude logback from your fat jar.
Please also see this answer https://stackoverflow.com/a/50229035/8845188.
Symptom: spring-boot basic app (no web, no jpa, no addons) defines a receiver in logback-spring.xml, but remote appender events never reach the local appenders.
logback-spring.xml:
<configuration debug="true">
<receiver name="logsink" class="ch.qos.logback.classic.net.server.ServerSocketReceiver">
<port>6004</port>
</receiver>
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d{HH:mm:ss} %-5level %logger{25} - %X{client} %msg %n</pattern>
</encoder>
</appender>
<appender name="crap" class="org.oclc.hadoop.logging.logbacksinksb.service.CrapAppender"/>
<root level="DEBUG">
<appender-ref ref="stdout"/>
<appender-ref ref="crap"/>
</root>
</configuration>
The crapAppender is stripped to bare-bones for this problem:
public class CrapAppender extends AppenderBase<ILoggingEvent> {
#Override
protected void append(ILoggingEvent eventObject) {
System.out.println("got event!");
}
}
On startup, the log says my components are being instantiated:
16:39:12,103 |-INFO in ch.qos.logback.classic.joran.action.ReceiverAction - About to instantiate receiver of type [ch.qos.logback.classic.net.server.ServerSocketReceiver]
16:39:12,119 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
16:39:12,119 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [stdout]
16:39:12,119 |-INFO in ch.qos.logback.classic.net.server.RemoteAppenderServerRunner#228dc826 - listening on 0.0.0.0:6004
16:39:12,120 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
16:39:12,135 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [org.oclc.hadoop.logging.logbacksinksb.service.CrapAppender]
16:39:12,135 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [crap]
16:39:12,136 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to DEBUG
16:39:12,136 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [stdout] to Logger[ROOT]
16:39:12,136 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [crap] to Logger[ROOT]
16:39:12,136 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
16:39:12,136 |-INFO in org.springframework.boot.logging.logback.SpringBootJoranConfigurator#1dd02175 - Registering current configuration as safe fallback point
16:39:12,138 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator#23bb8443 - Propagating INFO level on Logger[org.springframework] onto the JUL framework
I can even see that the remote appender has connected to the receiver port (6004):
java 5935 jamiesoh 32u IPv6 0xaf52811569ae62a3 0t0 TCP localhost:54160->localhost:6004 (ESTABLISHED)
java 6642 jamiesoh 53u IPv6 0xaf52811569ae5d63 0t0 TCP *:6004 (LISTEN)
java 6642 jamiesoh 54u IPv6 0xaf5281155d876d63 0t0 TCP localhost:6004->localhost:54160 (ESTABLISHED)
The local appenders only emit local events; the remote appender never reaches the local crap appender.
Any one have any ideas on what is preventing the remote events from passing thru the local appenders?
Just in case this bites someone else, I found the cause. One of the other artifacts had a dependency on logback-classic:1.2.3; the spring-boot version is 1.1.11. Excluding this transitive dependency corrected the problem.
Doctor, heal thyself...
I've got a spring boot app that is outputing around 1500 lines of JNDI debug logging to stdout at startup before logback is initialized. Does anyone know what's causing this and if there is a way to get logback initialized before this? I've got -Dlogback.debug=true but it doesn't help much.
10:44:56.453 [main] DEBUG org.springframework.jndi.JndiTemplate - Looking up JNDI object with name [java:comp/env/logging.exception-conversion-word]
10:44:56.457 [main] DEBUG jndi - InitialContextFactory.getInitialContext()
10:44:56.457 [main] DEBUG jndi - Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot#564fabc8
10:44:56.457 [main] DEBUG org.eclipse.jetty.jndi.java.javaURLContextFactory - >>> new root context requested
10:44:56.457 [main] DEBUG jndi - Looking up name="comp/env/logging.exception-conversion-word"
10:44:56.457 [main] DEBUG jndi - Trying thread context classloader
10:44:56.457 [main] DEBUG jndi - Looking up name="env/logging.exception-conversion-word"
10:44:56.458 [main] DEBUG org.springframework.jndi.JndiLocatorDelegate - Converted JNDI name [java:comp/env/logging.exception-conversion-word] not found - trying original name [logging.exception-conversion-word]. javax.naming.NameNotFoundException; remaining name 'env/logging.exception-conversion-word'
10:44:56.458 [main] DEBUG org.springframework.jndi.JndiTemplate - Looking up JNDI object with name [logging.exception-conversion-word]
...
10:44:56.487 [main] DEBUG jndi - Looking up name="env/LOGGING_PATTERNLEVEL"
10:44:56.487 [main] DEBUG org.springframework.jndi.JndiLocatorDelegate - Converted JNDI name [java:comp/env/LOGGING_PATTERNLEVEL] not found - trying original name [LOGGING_PATTERNLEVEL]. javax.naming.NameNotFoundException; remaining name 'env/LOGGING_PATTERNLEVEL'
10:44:56.487 [main] DEBUG org.springframework.jndi.JndiTemplate - Looking up JNDI object with name [LOGGING_PATTERNLEVEL]
10:44:56.487 [main] DEBUG jndi - InitialContextFactory.getInitialContext()
10:44:56.487 [main] DEBUG jndi - Created initial context delegate for local namespace:org.eclipse.jetty.jndi.local.localContextRoot#bcec361
10:44:56.487 [main] DEBUG jndi - Looking up name="LOGGING_PATTERNLEVEL"
10:44:56.487 [main] DEBUG org.springframework.jndi.JndiPropertySource - JNDI lookup for name [LOGGING_PATTERNLEVEL] threw NamingException with message: null. Returning null.
10:44:56,536 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
10:44:56,539 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
10:44:56,543 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
10:44:56,561 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
10:44:56,564 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILE]
10:44:56,565 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
10:44:56,571 |-INFO in my.log.TimestampRollingPolicy#24fcf36f - Will use the pattern /tmp/logs/qs.%d{yyyyMMdd-HHmmss}.log to archive files
10:44:56,574 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: /tmp/logs/qs.log
10:44:56,574 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to [/tmp/logs/qs.log]
10:44:56,576 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to WARN
10:44:56,576 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator#4a7f959b - Propagating WARN level on Logger[ROOT] onto the JUL framework
10:44:56,576 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILE] to Logger[ROOT]
10:44:56,576 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
10:44:56,576 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
10:44:56,577 |-INFO in org.springframework.boot.logging.logback.SpringBootJoranConfigurator#32b260fa - Registering current configuration as safe fallback point
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.4.4.RELEASE)
10:44:56,750 |-INFO in c.q.l.co.rolling.helper.RenameUtil - Renaming file [/tmp/logs/qs.log] to [/tmp/logs/qs.20170222-173130.log]
10:44:56 [main] INFO my.Application - Starting ...
I also saw this with Spring Boot 1.5.1 when using a custom logback-spring.xml. The debug statements don't appear if logback-spring.xml doesn't exist.
I was able to work around it by disabling the JNDIPropertySource. Create a file, spring.properties, in your classpath root:
# Disable the JNDIPropertySource. This avoids dozens of spurious DEBUG messages
# when starting up with a custom logback-spring.xml. We're not alone:
# http://stackoverflow.com/questions/42446013/spring-jndi-logging-before-logback-is-initialized
spring.jndi.ignore=true
spring.jndi.ignore was added in https://jira.spring.io/browse/SPR-14026.
This approach feels squicky, but as I'm not running in a container where JNDI would be useful, it doesn't break anything for me.
I also had the same problem, but I couldn't turn off jndi, because my application is using it on startup. So after some research, I ended up with the solution:
1) rename logback-spring.xml to some other name, for example logback-config.xml
2) put logging.config property into application.properties file.
As the result you configuration will be applied only on spring startup, and this debug logging will be skipped. Also in my case the first solution wasn't working because I needed jndi, in this case you also leave jndi enabled.
Update: this helped only for local environment, to make the same work on remote tomcat and remove this debug log from catalina.log file, I ended up with renaming this logback file to logback.xml, thus it's found on the startup, before jndi lookup.
you can avoid this by add this line to your logback xml file
<logger name="org.springframework.jndi" level="INFO" />