Unable to connect to RDS from Elastic Beanstalk Docker - spring

Question

My spring boot app deployed in Elastic Beanstalks docker is unable to connect to external RDS. It always stuck at "com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting..." during app startup.
Dockerfile
(I added the db connection details in ENTRYPOINT for troubleshooting purpose)
FROM openjdk:8
WORKDIR "/mrbs"
ARG JAR_FILE=target/*jar
COPY ${JAR_FILE} ./app.jar
EXPOSE 8080
#CMD ["java","-jar","./app.jar"]
ENTRYPOINT ["java","-jar","./app.jar","--spring.datasource.url=jdbc:mysql://[security].[security].ap-southeast-1.rds.amazonaws.com:3306/mrbs?serverTimezone=GMT%2B8","--spring.datasource.password=[security]","--spring.datasource.username=[security]","--logging.file.path=/mrbs/logs/"]
Dockerrun.aws.json
{
"AWSEBDockerrunVersion": 2,
"containerDefinitions": [
{
"name": "api",
"image": "[security]/mrbs-backend",
"hostname": "api",
"essential": true,
"memory": 128
}
]
}
.travis.yml
language: generic
sudo: required
services:
- docker
before_install:
install: skip
before_script:
script:
- mvn -f mrbs-backend clean package
- docker build -t [security]/mrbs-backend ./mrbs-backend
- echo "$DOCKER_PASSWORD" |docker login -u "$DOKCER_ID" --password-stdin
- docker push [security]/mrbs-backend
deploy:
skip_cleanup: true
provider: elasticbeanstalk
region: ap-southeast-1
#app: mrbs-docker
app: mrbs
env: Mrbs-env-3
bucket_name: elasticbeanstalk-ap-southeast-1-[security]351
bucket_patch: mrbs
on:
branch: master
access_key_id: $AWS_ACCESS_KEY
secret_access_key: $AWS_SECRET_KEY
Added security group mrbs-intra-docker to both RDS and Elastic Beanstalk
RDS Security Group
(image),
Elastic Beanstalk Security Group
(image),
EC2 Security Group
(image),
When I check the log from elastic beanstalk, it shows that spring boot app stuck at "com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting..." during startup.
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.3.RELEASE)
2020-11-01 09:14:16,590 [main] INFO c.j.m.MeetingRoomBookingSystemApplication - Starting MeetingRoomBookingSystemApplication v1.0 on api with PID 1 (/mrbs/app.jar started by root in /mrbs)
2020-11-01 09:14:16,599 [main] DEBUG c.j.m.MeetingRoomBookingSystemApplication - Running with Spring Boot v2.3.3.RELEASE, Spring v5.2.8.RELEASE
2020-11-01 09:14:16,600 [main] INFO c.j.m.MeetingRoomBookingSystemApplication - The following profiles are active: dev
2020-11-01 09:14:19,408 [main] INFO o.s.d.r.c.RepositoryConfigurationDelegate - Bootstrapping Spring Data JPA repositories in DEFERRED mode.
2020-11-01 09:14:19,729 [main] INFO o.s.d.r.c.RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 294ms. Found 7 JPA repository interfaces.
2020-11-01 09:14:21,469 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler#c03cf28' of type [org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-11-01 09:14:21,513 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'methodSecurityMetadataSource' of type [org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-11-01 09:14:22,602 [main] INFO o.s.b.w.e.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8080 (http)
2020-11-01 09:14:22,639 [main] INFO o.a.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8080"]
2020-11-01 09:14:22,640 [main] INFO o.a.catalina.core.StandardService - Starting service [Tomcat]
2020-11-01 09:14:22,640 [main] INFO o.a.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.37]
2020-11-01 09:14:22,826 [main] INFO o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2020-11-01 09:14:22,827 [main] INFO o.s.b.w.s.c.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 6052 ms
2020-11-01 09:14:24,415 [main] INFO o.s.s.c.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-11-01 09:14:24,516 [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
However, If I directly run the docker from EC2 command line, the spring boot can be started successfully, which means ec2 instance is able to connect to rds using the same image...
[ec2-user#ip-172-31-24-202 ~]$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
[secruity]/mrbs-backend latest bd84014e90df 9 minutes ago 570MB
amazon/amazon-ecs-agent latest ebac5fda27cb 8 weeks ago 67MB
[ec2-user#ip-172-31-24-202 ~]$ docker run [secruity]/mrbs-backend
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.3.3.RELEASE)
2020-11-01 09:36:39,075 [main] INFO c.j.m.MeetingRoomBookingSystemApplication - Starting MeetingRoomBookingSystemApplication v1.0 on 1d93b8a14d12 with PID 1 (/mrbs/app.jar started by root in /mrbs)
2020-11-01 09:36:39,087 [main] DEBUG c.j.m.MeetingRoomBookingSystemApplication - Running with Spring Boot v2.3.3.RELEASE, Spring v5.2.8.RELEASE
2020-11-01 09:36:39,088 [main] INFO c.j.m.MeetingRoomBookingSystemApplication - The following profiles are active: dev
2020-11-01 09:36:44,035 [main] INFO o.s.d.r.c.RepositoryConfigurationDelegate - Bootstrapping Spring Data JPA repositories in DEFERRED mode.
2020-11-01 09:36:44,558 [main] INFO o.s.d.r.c.RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 474ms. Found 7 JPA repository interfaces.
2020-11-01 09:36:47,066 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler#c03cf28' of type [org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-11-01 09:36:47,107 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'methodSecurityMetadataSource' of type [org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-11-01 09:36:49,047 [main] INFO o.s.b.w.e.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8080 (http)
2020-11-01 09:36:49,099 [main] INFO o.a.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8080"]
2020-11-01 09:36:49,106 [main] INFO o.a.catalina.core.StandardService - Starting service [Tomcat]
2020-11-01 09:36:49,107 [main] INFO o.a.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.37]
2020-11-01 09:36:49,546 [main] INFO o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2020-11-01 09:36:49,549 [main] INFO o.s.b.w.s.c.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 10161 ms
2020-11-01 09:36:51,992 [main] INFO o.s.s.c.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-11-01 09:36:52,141 [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Starting...
2020-11-01 09:36:54,622 [main] INFO com.zaxxer.hikari.HikariDataSource - HikariPool-1 - Start completed.
filePath=/mrbs/logs/
2020-11-01 09:36:55,296 [main] DEBUG com.jiangwensi.mrbs.AppContext - setApplicationContext is called
2020-11-01 09:36:55,528 [task-1] INFO o.h.jpa.internal.util.LogHelper - HHH000204: Processing PersistenceUnitInfo [name: default]
2020-11-01 09:36:56,161 [main] WARN o.s.b.a.o.j.JpaBaseConfiguration$JpaWebConfiguration - spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2020-11-01 09:36:58,427 [task-1] INFO org.hibernate.Version - HHH000412: Hibernate ORM core version 5.4.20.Final
2020-11-01 09:36:58,874 [main] DEBUG o.s.s.w.a.e.ExpressionBasedFilterInvocationSecurityMetadataSource - Adding web access control expression 'permitAll', for Ant [pattern='/auth/signUp', POST]
2020-11-01 09:36:58,887 [main] DEBUG o.s.s.w.a.e.ExpressionBasedFilterInvocationSecurityMetadataSource - Adding web access control expression 'permitAll', for Ant [pattern='/auth/verifyEmail', GET]
2020-11-01 09:36:58,888 [main] DEBUG o.s.s.w.a.e.ExpressionBasedFilterInvocationSecurityMetadataSource - Adding web access control expression 'permitAll', for Ant [pattern='/auth/requestResetForgottenPassword', POST]
2020-11-01 09:36:58,888 [main] DEBUG o.s.s.w.a.e.ExpressionBasedFilterInvocationSecurityMetadataSource - Adding web access control expression 'permitAll', for Ant [pattern='/auth/resetForgottenPassword', POST]
2020-11-01 09:36:58,889 [main] DEBUG o.s.s.w.a.e.ExpressionBasedFilterInvocationSecurityMetadataSource - Adding web access control expression 'permitAll', for Ant [pattern='/auth/resetPassword', POST]
2020-11-01 09:36:58,892 [main] DEBUG o.s.s.w.a.e.ExpressionBasedFilterInvocationSecurityMetadataSource - Adding web access control expression 'authenticated', for any request
2020-11-01 09:36:59,011 [main] DEBUG o.s.s.w.a.i.FilterSecurityInterceptor - Validated configuration attributes
2020-11-01 09:36:59,020 [main] DEBUG o.s.s.w.a.i.FilterSecurityInterceptor - Validated configuration attributes
2020-11-01 09:36:59,036 [main] INFO o.s.s.w.DefaultSecurityFilterChain - Creating filter chain: any request, [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter#31bcf236, org.springframework.security.web.context.SecurityContextPersistenceFilter#4c51cf28, org.springframework.security.web.header.HeaderWriterFilter#289710d9, org.springframework.web.filter.CorsFilter#4b3ed2f0, org.springframework.security.web.authentication.logout.LogoutFilter#3549bca9, com.jiangwensi.mrbs.security.LoginAuthenticationFilter#4fad9bb2, com.jiangwensi.mrbs.security.JwtAuthenticationFilter#517d4a0d, org.springframework.security.web.savedrequest.RequestCacheAwareFilter#5143c662, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter#71c27ee8, org.springframework.security.web.authentication.AnonymousAuthenticationFilter#7862f56, org.springframework.security.web.session.SessionManagementFilter#3da30852, org.springframework.security.web.access.ExceptionTranslationFilter#4eb386df, org.springframework.security.web.access.intercept.FilterSecurityInterceptor#134d26af]
2020-11-01 09:36:59,324 [main] DEBUG o.s.s.a.i.a.MethodSecurityInterceptor - Validated configuration attributes
2020-11-01 09:36:59,643 [main] INFO o.h.validator.internal.util.Version - HV000001: Hibernate Validator 6.1.5.Final
2020-11-01 09:37:01,174 [task-1] INFO o.h.annotations.common.Version - HCANN000001: Hibernate Commons Annotations {5.1.0.Final}
2020-11-01 09:37:03,566 [task-1] INFO org.hibernate.dialect.Dialect - HHH000400: Using dialect: org.hibernate.dialect.MySQL8Dialect
2020-11-01 09:37:05,670 [main] DEBUG o.s.s.c.a.a.c.AuthenticationConfiguration$EnableGlobalAuthenticationAutowiredConfigurer - Eagerly initializing {webSecurityConfigurator=com.jiangwensi.mrbs.security.WebSecurityConfigurator$$EnhancerBySpringCGLIB$$aa3555ad#1a20270e}
2020-11-01 09:37:05,692 [main] INFO o.a.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8080"]
2020-11-01 09:37:06,111 [main] INFO o.s.b.w.e.tomcat.TomcatWebServer - Tomcat started on port(s): 8080 (http) with context path ''
2020-11-01 09:37:06,136 [main] INFO o.s.d.r.c.DeferredRepositoryInitializationListener - Triggering deferred initialization of Spring Data repositories…
2020-11-01 09:37:09,726 [task-1] INFO o.h.e.t.j.p.i.JtaPlatformInitiator - HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2020-11-01 09:37:09,802 [task-1] INFO o.s.o.j.LocalContainerEntityManagerFactoryBean - Initialized JPA EntityManagerFactory for persistence unit 'default'
2020-11-01 09:37:12,036 [main] INFO o.s.d.r.c.DeferredRepositoryInitializationListener - Spring Data repositories initialized!
2020-11-01 09:37:12,065 [main] INFO c.j.m.MeetingRoomBookingSystemApplication - Started MeetingRoomBookingSystemApplication in 36.293 seconds (JVM running for 42.707)
2020-11-01 09:37:12,073 [main] DEBUG c.j.mrbs.InitializeApplicationData - onApplicationEvent is called

Answer 1

Based on the comments.
The issue was caused by not sufficient memory allocated to the container.
The solution was to increase the memory.