Connecting SpringBoot to Cloud SQL in Cloud Run error, connection refused - spring-boot

Question

I'm deploying my Maven Springboot project to Google Cloud Run and trying to connect to Cloud SQL from Cloud Run. I'm using https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-samples/spring-cloud-gcp-data-jpa-sample as a helper to connect to Cloud SQL. Working locally, I can connect to the instance using this library and gcloud auth application-default login to create a proxy. However, when deployed to Cloud Run, I got this exception:
I'm also running different profiles on Cloud Run (profile: qa) and local (default profile)
# For QA - application-qa.yml
spring:
jpa:
hibernate:
ddl-auto: create-drop
properties:
hibernate:
temp:
use_jdbc_metadata_defaults: false
datasource:
username: ${DATABASE_USERNAME}
password: ${DATABASE_PASSWORD}
cloud:
gcp:
sql:
database-name: ${DATABASE_NAME}
instance-connection-name: ${DATABASE_CONNECTION_NAME}
enabled: true
logging:
enabled: true
Clarification:
My local environment uses MySql from Docker image meaning that I connect to local database, that's why I have datasource.url inside application.yml for default profile.
# Default - application.yml
spring:
datasource:
url: jdbc:mysql://${DATABASE_HOSTNAME}:${DATABASE_PORT}/${DATABASE_NAME}?createDatabaseIfNotExist=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=true&useSSL=false&${DATABASE_ARGS:}
username: ${DATABASE_USERNAME}
password: ${DATABASE_PASSWORD}
driver-class-name: com.mysql.cj.jdbc.Driver
Moreover, I already tried to delete datasource.url from applcation.yml (default profile) to test if this makes spring confused about connection target or not, but turns out the error is still there.
Edit 1
In case needed, I already set cloud-sql-instance attached to my Cloud Run service. My SQL Instance allows both public and private IPs and allows non-secure (no-SSL) connections.
IAM policy for accounts:
${PROJECT_ID}-compute#developer.gserviceaccount.com (default SA) : Project Editor / Cloud SQL Admin
<Cloud Run Service Agent> : Cloud Run Service Agent
because official doc said that Cloud Run will authenticate to Cloud SQL using specific service account which, in this case, is default one so I didn't grant Cloud SQL Admin/Client to <Cloud Run Service Agent>
Edit 2
Here is my pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.8.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.backend</groupId>
<artifactId>backend</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>backend</name>
<description>Spring Project</description>
<properties>
<java.version>1.8</java.version>
<kotlin.version>1.2.71</kotlin.version>
<testcontainers.version>1.12.2</testcontainers.version>
</properties>
<repositories>
<repository>
<id>plugin-release</id>
<name>spring-plugin</name>
<url>https://repo.spring.io/plugins-release</url>
</repository>
<repository>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>jcenter-releases</id>
<name>jcenter</name>
<url>http://jcenter.bintray.com</url>
</repository>
</repositories>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-gcp-dependencies</artifactId>
<version>${project.version}</version>
<type>pom</type>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>net.sargue</groupId>
<artifactId>mailgun</artifactId>
<version>1.9.0</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-kotlin</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk8</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-test-autoconfigure</artifactId>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>2.9.2</version>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>2.9.2</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>1.11.560</version>
</dependency>
<dependency>
<groupId>org.modelmapper</groupId>
<artifactId>modelmapper</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.modelmapper.extensions</groupId>
<artifactId>modelmapper-jackson</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.modelmapper.extensions</groupId>
<artifactId>modelmapper-spring</artifactId>
<version>2.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-text</artifactId>
<version>1.6</version>
</dependency>
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-core</artifactId>
<version>1.21</version>
</dependency>
<dependency>
<groupId>io.github.swagger2markup</groupId>
<artifactId>swagger2markup</artifactId>
<version>1.3.3</version>
</dependency>
<dependency>
<groupId>com.corundumstudio.socketio</groupId>
<artifactId>netty-socketio</artifactId>
<version>1.7.17</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-jsr310</artifactId>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.inject</groupId>
<artifactId>jersey-hk2</artifactId>
<version>2.28</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-search-orm</artifactId>
<version>5.11.1.Final</version>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>1.4.197</version>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>testcontainers</artifactId>
<version>${testcontainers.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.testcontainers</groupId>
<artifactId>mysql</artifactId>
<version>1.12.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
</dependency>
<dependency>
<groupId>org.hibernate.validator</groupId>
<artifactId>hibernate-validator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-gcp-starter-sql-mysql</artifactId>
<version>1.1.3.RELEASE</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>${project.basedir}/src/main/kotlin</sourceDirectory>
<testSourceDirectory>${project.basedir}/src/test/kotlin</testSourceDirectory>
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.6.1</version>
</extension>
</extensions>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<configuration>
<compilerPlugins>
<plugin>jpa</plugin>
<plugin>spring</plugin>
</compilerPlugins>
<args>
<arg>-Xjsr305=strict</arg>
</args>
<languageVersion>1.2</languageVersion>
</configuration>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-noarg</artifactId>
<version>${kotlin.version}</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-allopen</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.6.1</version>
<configuration>
<protocArtifact>
com.google.protobuf:protoc:3.3.0:exe:${os.detected.classifier}
</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>
io.grpc:protoc-gen-grpc-java:1.4.0:exe:${os.detected.classifier}
</pluginArtifact>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>compile-custom</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Edit 3
I can now connect to Cloud SQL via locally deployment. Seems like I cannot connect to SQL Instance when deployed via Cloud Build triggers.
I'm using connection string as follow: <project_id>:asia-southeast1:farose-test-db
Cloud Run is using default service account, which is -compute#developer.gserviceaccount.com` with assigned role as follows:
Cloud SQL Client
Cloud Run Invoker
Editor
Edit 4
Seems like the issue came from missing several files in image during build step which explain why building and deploying locally works just fine. I disabled cache from kaniko executor right now.