Using same postgres container for a spring project database and a keycloak database - spring

I am trying to run three dockerized services:
Spring-boot app
Keycloak for authentication
Postgres as database
I would like to have both the Spring-boot app and the Keycloak app to use the same Postgres container as their database, but I couldn't find a way to make it work. My docker-compose.yml is as follows:
version: '3.7'
services:
db:
image: 'postgres:13.1-alpine'
container_name: db
ports:
- "5432:5432"
volumes:
- ./app_data:/var/lib/postgresql/data_app
- ./keycloak_data:/var/lib/postgresql/data_keycloak
- ../docker-postgresql-multiple-databases:/docker-entrypoint-initdb.d
environment:
POSTGRES_MULTIPLE_DATABASES: keycloak, app_user
POSTGRES_PASSWORD: password
healthcheck:
test: [ "CMD-SHELL", "pg_isready" ]
interval: 10s
timeout: 5s
retries: 5
keycloak:
image: jboss/keycloak:14.0.0
container_name: keycloak
environment:
- KEYCLOAK_USER=admin
- KEYCLOAK_PASSWORD=admin
- DB_VENDOR=postgres
- DB_ADDR=postgres
- DB_USER=keycloak
- DB_PASSWORD=password
- JDBC_PARAMS=useSSL=false
ports:
- "8080:8080"
depends_on:
- db
healthcheck:
test: "curl -f http://localhost:8080/auth || exit 1"
start_period: 20s
app:
image: 'app.postgre:latest'
build:
context: .
container_name: app
depends_on:
- db
- keycloak
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://db:5432/app
- SPRING_DATASOURCE_USERNAME=app_user
- SPRING_DATASOURCE_PASSWORD=password
- SPRING_JPA_HIBERNATE_DDL_AUTO=update
volumes:
app_data:
postgres_data:
(Note: I tried using the following code: https://github.com/mrts/docker-postgresql-multiple-databases to set-up the needed databases by hand, but even so it still fails. I also tried doing without this script, but that also failed.)
I have tried them and managed to make a docker-compose file which runs the spring app and the database together, and another docker-compose file which runs the keycloak app and the database together, but when I try to bring all three together it fails.

I have a very similar setup with Postgres, Keycloak, pgAdmin and a Golang API service. The skeleton of my docker-compose.yml is like this, give it a try (I omitted some parts for simplicity), it is working for me. I think the important parts here are networks and links, and also setting up multiple databases (as you already do). I use db as the hostname of Postgres server, when I connect to it via pgAdmin for example.
services:
db:
build:
context: .
dockerfile: ./Dockerfile.db
volumes:
networks:
- mynetwork
restart: unless-stopped
ports:
- ${POSTGRES_PORT}:5432
environment:
- POSTGRES_MULTIPLE_DATABASES=${POSTGRES_MULTIPLE_DATABASES}
- POSTGRES_USER=${POSTGRES_USER}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
healthcheck:
pgadmin:
image: dpage/pgadmin4
restart: unless-stopped
environment:
volumes:
ports:
networks:
- mynetwork
restart: unless-stopped
depends_on:
- db
api:
build:
context: .
dockerfile: ./Dockerfile.api
ports:
environment:
- POSTGRES_HOST=${POSTGRES_HOST}
- POSTGRES_PORT=${POSTGRES_PORT}
volumes:
networks:
- mynetwork
depends_on:
- db
links:
- db
restart: unless-stopped
keycloak:
image: quay.io/keycloak/keycloak:latest
environment:
- DB_VENDOR=${KEYCLOAK_DB_VENDOR}
- DB_ADDR=${KEYCLOAK_DB_ADDR}
- DB_DATABASE=${KEYCLOAK_DB_DATABASE}
- DB_USER=${KEYCLOAK_DB_USER}
- DB_SCHEMA=${KEYCLOAK_DB_SCHEMA}
- DB_PASSWORD=${KEYCLOAK_DB_PASSWORD}
- KEYCLOAK_USER=${KEYCLOAK_USER}
- KEYCLOAK_PASSWORD=${KEYCLOAK_PASSWORD}
ports:
- ${KEYCLOAK_PORT}:8080
depends_on:
- db
networks:
- mynetwork
links:
- db
restart: unless-stopped
volumes:
volumes:
networks:
mynetwork:
And some important a values from my .env:
POSTGRES_MULTIPLE_DATABASES=mydb,keycloak
POSTGRES_USER=
POSTGRES_PASSWORD=
POSTGRES_HOST=db
POSTGRES_PORT=5432
KEYCLOAK_PORT=8084
KEYCLOAK_DB_VENDOR=POSTGRES
KEYCLOAK_DB_ADDR=db
KEYCLOAK_DB_DATABASE=keycloak
KEYCLOAK_DB_USER=
KEYCLOAK_DB_SCHEMA=public
KEYCLOAK_DB_PASSWORD=
KEYCLOAK_USER=
KEYCLOAK_PASSWORD=
My Dockerfile.db is like this, you don't need the localedef part (I need it for Hungarian localization):
FROM postgres:latest
RUN localedef -i hu_HU -c -f UTF-8 -A /usr/share/locale/locale.alias hu_HU.UTF-8
COPY docker-postgresql-multiple-databases.sh /docker-entrypoint-initdb.d/
And docker-postgresql-multiple-databases.sh contains:
#!/bin/bash
set -e
set -u
function create_user_and_database() {
local database=$1
echo " Creating user and database '$database'"
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
CREATE USER $database;
CREATE DATABASE $database;
GRANT ALL PRIVILEGES ON DATABASE $database TO $database;
EOSQL
}
if [ -n "$POSTGRES_MULTIPLE_DATABASES" ]; then
echo "Multiple database creation requested: $POSTGRES_MULTIPLE_DATABASES"
for db in $(echo $POSTGRES_MULTIPLE_DATABASES | tr ',' ' '); do
create_user_and_database $db
done
echo "Multiple databases created"
fi

Related

Why is springboot connecting to the default postgres database instead of my specified one?

So I've got a springboot backend that I'm trying to connect to a postgres database. I've got a docker-compose file for it, yet the backend seems to connect to the wrong database. It seems to be connecting to the default postgres database instead of my specified vve database.
My docker-compose
version: '3'
services:
vve-postgres:
image: postgres:14.2
environment:
- POSTGRES_DATABASE=vve
- POSTGRES_USER=vve
- POSTGRES_PASSWORD=admin
volumes:
- vve-postgres-volume:/var/lib/postgresql/data
ports:
- "5432:5432"
vve-api:
build: .
restart: on-failure
env_file: .env
environment:
- DATABASE_HOST
- DATABASE_PORT
- DATABASE_NAME
- DATABASE_USER
- DATABASE_PASSWORD
depends_on:
- vve-postgres
volumes:
- ./data:/data
ports:
- "8080:8080"
volumes:
vve-postgres-volume:
My Environment file
DATABASE_HOST=vve-postgres
DATABASE_PORT=5432
DATABASE_NAME=vve
DATABASE_USER=vve
DATABASE_PASSWORD=admin
My application.properties
spring.config.import=optional:file:.env[.properties]
spring.jpa.generate-ddl=true
spring.jpa.hibernate.ddl-auto=create
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.PostgreSQLDialect
spring.sql.init.mode=always
spring.jpa.show-sql=true
spring.jpa.properties.hibernate.format_sql=true
spring.datasource.url=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT}/${DATABASE_NAME}
spring.datasource.username=${DATABASE_USER}
spring.datasource.password=${DATABASE_PASSWORD}
server.error.include-stacktrace=never
server.error.include-message=always
Picture showing the two databases, I'm wrongly connected to the postgres#localhost
How do I make sure my back end connects to the right database?
Environment in docker compose should be key-value pair:
vve-api:
build: .
restart: on-failure
env_file: .env
environment:
- DATABASE_HOST: ${DATABASE_HOST}
- DATABASE_PORT: ${DATABASE_PORT}
...
Etc for each one

Spring boot microservices and Docker, unable to connect to database

I'm writing a test app in microservices to learn Spring Boot. I'm done with the app and looking to deploy it on AWS using docker/docker-compose to host each microservices.
I'm unable to connect any of the Spring Boot instances to their Mysql database. I've been stuck on it for a few days and I can't see what's wrong.
Here is my docker-compose.yml
version: "3.3"
services:
card_mysql:
image: "mysql"
restart: always
environment:
MYSQL_ROOT_PASSWORD: "root"
MYSQL_DATABASE: "cards"
ports:
- "33061:3306"
auth_mysql:
image: "mysql"
restart: always
environment:
MYSQL_ROOT_PASSWORD: "root"
MYSQL_DATABASE: "auth"
ports:
- "33062:3306"
market_mysql:
image: "mysql"
restart: always
environment:
MYSQL_ROOT_PASSWORD: "root"
MYSQL_DATABASE: "market"
ports:
- "33063:3306"
user_mysql:
image: "mysql"
restart: always
environment:
MYSQL_ROOT_PASSWORD: "root"
MYSQL_DATABASE: "user"
ports:
- "33064:3306"
adminer:
image: adminer
restart: always
ports:
- 8000:8080
user:
image: "openjdk:11"
restart: always
entrypoint: java -jar /app/services/user/target/cardmarket-user-0.0.1-SNAPSHOT.jar
volumes:
- ./:/app
depends_on:
- "user_mysql"
card:
image: "openjdk:11"
restart: always
entrypoint: java -jar /app/services/card/target/cardmarket-card-0.0.1-SNAPSHOT.jar
volumes:
- ./:/app
depends_on:
- "card_mysql"
market:
image: "openjdk:11"
restart: always
entrypoint: java -jar /app/services/market/target/cardmarket-market-0.0.1-SNAPSHOT.jar
volumes:
- ./:/app
depends_on:
- "market_mysql"
proxy:
image: "openjdk:11"
restart: always
entrypoint: java -jar /app/zuul-proxy/target/zuul-proxy-0.0.1-SNAPSHOT.jar
volumes:
- ./:/app
auth:
image: "openjdk:11"
restart: always
entrypoint: java -jar /app/zuul-proxy/zuul-proxy-0.0.1-SNAPSHOT.jar
volumes:
- ./:/app
depends_on:
- "auth_mysql"
Here are each of my spring datasource settings:
user application.properties
spring.jpa.hibernate.ddl-auto=create
spring.datasource.url=jdbc:mysql://user_mysql:3306/user
spring.datasource.initialization-mode=always
spring.datasource.username=root
spring.datasource.password=root
market application.properties
spring.jpa.hibernate.ddl-auto=create-drop
spring.datasource.url=jdbc:mysql://market_mysql:3306/market
spring.datasource.initialization-mode=always
spring.datasource.username=root
spring.datasource.password=root
Auth application.properties
spring.jpa.hibernate.ddl-auto=create-drop
spring.datasource.url=jdbc:mysql://auth_mysql:3306/auth
spring.datasource.initialization-mode=always
spring.datasource.username=root
spring.datasource.password=root
Card application.properties
spring.jpa.hibernate.ddl-auto=create-drop
spring.datasource.url=jdbc:mysql://card_mysql:3306/cards
spring.datasource.initialization-mode=always
spring.datasource.username=root
spring.datasource.password=root
I'm sure it is something small I'm missing or misunderstood, but I can't figure out what.
Edit 1
I ran a couple more tests but still wasn't able to figure out what's wrong.
I commented out everything but the card and card_mysql containers. The spring boot instance throw an exception for every configuration possible (using docker dns name, localhost, shared and network ip), none of them seems to work.
The two containers do communicate, I can ping between them using the dns name.
I'm using docker-desktop with wsl and deploy my containers on a wsl Ubuntu 20.04. I haven't tried on an actual Linux machine.
I've also tried to make the spring boot instance wait for the mysql one using wait-for-it.sh, it does wait correctly but the exception still occurs.
Here is parts of the exception spring boot triggers:
2021-05-16 12:23:48.948 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2021-05-16 12:23:50.277 ERROR 1 --- [ main] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Exception during pool initialization.
com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
...
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
...
Caused by: java.net.ConnectException: Connection refused (Connection refused)
Edit 2
I tried having just one instance of spring boot and mysql and linking them using a network, unfortunately the exception still triggers. The two containers still pings. Here is the docker-compose I used for this test:
version: "3.9"
services:
card_mysql:
image: "mysql"
restart: always
environment:
MYSQL_ROOT_PASSWORD: "root"
MYSQL_DATABASE: "cards"
networks:
- "card_network"
card:
image: "openjdk:11"
restart: always
entrypoint: ["/app/wait-for-it.sh", "card_mysql:3306", "--", "java", "-jar", "/app/services/card/target/cardmarket-card-0.0.1-SNAPSHOT.jar"]
# entrypoint: ["/app/wait-for-it.sh", "card_mysql:3306", "--", "ping", "card_mysql"]
volumes:
- ./:/app
depends_on:
- "card_mysql"
networks:
- "card_network"
networks:
card_network: {}
Hi Spinarial please add network and gather it all in one network
version: "3.4"
services:
springMvcRestApi:
image: springbootproject:0.0.1
container_name: app
networks:
- postgres
ports:
- 8085:8085
depends_on:
- postgres
- pgadmin
links:
- postgres
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://postgres:5432/reactiveDbcd?createDatabaseIfNotExist=true
SPRING_DATASOURCE_USERNAME: postgres
SPRING_DATASOURCE_PASSWORD: 12345
postgres:
container_name: postgres
image: postgres
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: 12345
PGDATA: /data/postgres
volumes:
- postgres:/data/postgres
expose:
- 5432
ports:
- 5432:5432
networks:
- postgres
restart: unless-stopped
pgadmin:
container_name: pgadmin_container
image: dpage/pgadmin4
environment:
PGADMIN_DEFAULT_EMAIL: ${PGADMIN_DEFAULT_EMAIL:-pgadmin4#pgadmin.org}
PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD:-admin}
PGADMIN_CONFIG_SERVER_MODE: 'False'
volumes:
- pgadmin:/root/.pgadmin
ports:
- "${PGADMIN_PORT:-80}:80"
networks:
- postgres
restart: unless-stopped
networks:
postgres:
driver: bridge
volumes:
postgres:
pgadmin:

Connect service container to db container

I'm new to docker and started to play with it on my small project.
I have dockerized the service itself with the following Docker file:
ROM adoptopenjdk:11-jdk-hotspot AS DEPENDENCIES_BUILD_IMAGE
ENV APP_HOME=/usr/app/
WORKDIR $APP_HOME
COPY build.gradle settings.gradle gradlew $APP_HOME
COPY gradle $APP_HOME/gradle
RUN ./gradlew build || return 0
COPY . .
RUN ./gradlew build
FROM adoptopenjdk/openjdk11:jdk-11.0.7_10-alpine AS FINAL
ENV JAR_TEMPLATE=myapp-0.0.1-SNAPSHOT.jar
ENV ARTIFACT_NAME=myapp.jar
ENV APP_HOME=/usr/app
WORKDIR $APP_HOME
COPY --from=DEPENDENCIES_BUILD_IMAGE $APP_HOME/build/libs/$JAR_TEMPLATE .
RUN mv $JAR_TEMPLATE $ARTIFACT_NAME
EXPOSE 8080
CMD ["java", "-jar", "budget-calculator.jar"]
Side note - I know that there's a problem that I'm always copying 0.0.1-SNAPSHOT - but I'm not sure how to solve it at the moment.
After that I wanted to connect my service to a Postgres DB with docker-compose using this confirmation:
version: '3'
services:
backend:
build: .
container_name: myapp
ports:
- "8080:8080"
links:
- "db"
depends_on:
- db
networks:
- backend
db:
restart: unless-stopped
image: postgres:10
container_name: myapp-db
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=secret
- POSTGRES_DB=myapp
ports:
- 5436:5436
networks:
- backend
networks:
backend:
After that I've updated my application.properties file to indicate that the DB link is on the other container as follow:
spring.flyway.url=jdbc:postgresql://db:5436/myapp
spring.flyway.user=postgres
spring.flyway.password=secret
spring.flyway.baseline-on-migrate=true
spring.datasource.url=jdbc:postgresql://db:5436/myapp
spring.datasource.username=postgres
spring.datasource.password=secret
spring.datasource.driverClassName=org.postgresql.Driver
Now I had 2 problems:
While I assumed that build: . will rebuild my image every time
that I'm running docker-compose up if something changed in
practice I saw that it's not the case.
When the backend service starts flyway (a migration DB library) try
to connect to the database and cannot resolve the connection.
I've seen online that the usage of - links is deprecated and I should use the networks but both do not seem to work - what am I missing?
There are 2 problems with my configurations, the first one - the internal port of Postgres was configured as 5436 while the default port of the image is 5432 (I've updated both of them to 5432)
the second one, in order to pass the IP of the DB to the service I've added the following environment variables to the service image:
environment: # Pass environment variables to the service
SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/budget
SPRING_DATASOURCE_USERNAME: postgres
SPRING_DATASOURCE_PASSWORD: secret
SPRING_FLYWAY_URL: jdbc:postgresql://db:5432/budget
SPRING_FLYWAY_USER: postgres
SPRING_FLYWAY_PASSWORD: secret
So my current working configuration is this:
version: '3.8'
services:
backend:
build: .
container_name: app-service
ports:
- "8080:8080"
depends_on:
- db
environment: # Pass environment variables to the service
SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/myapp
SPRING_DATASOURCE_USERNAME: postgres
SPRING_DATASOURCE_PASSWORD: secret
SPRING_FLYWAY_URL: jdbc:postgresql://db:5432/myapp
SPRING_FLYWAY_USER: postgres
SPRING_FLYWAY_PASSWORD: secret
db:
restart: unless-stopped
image: postgres:10
environment:
- POSTGRES_DB=myapp
- POSTGRES_PASSWORD=postgres
- POSTGRES_USER=secret
volumes:
- myapp_data:/var/lib/postgresql/data
ports:
- 5432:5432
volumes:
myapp_data:

Docker postgres container loses data that should be stored in volume

I am running a postgres database generated by the below docker-compose file on Windows. Before running docker-compose up --build, I created a docker volume with docker volume --name postgresdata --driver local. The latter is done to avoid mounting a Windows folder into Postgres.
However, when I run docker-compose down followed by docker-compose up --build, the database is empty which I would not have expected. Any ideas or suggestions?
This is the docker-compose.yml file I am using:
version: '3.0'
services:
db:
image: postgres:latest
restart: always
ports:
- 5432:5432
env_file:
- env_file
volumes:
- postgresdata
networks:
- db1
market_data:
build: .
environment:
PYTHONUNBUFFERED: 'true'
stdin_open: true
tty: true
links:
- db:db
container_name: market_data_container
volumes:
- '.:/market_data'
depends_on:
- db
networks:
- db1
adminer:
image: adminer
restart: always
ports:
- 8080:8080
networks:
- db1
depends_on:
- db
volumes:
market_data:
postgresdata:
external: true
networks:
db1:
driver: bridge
Postgres uses already a volume to persist data, but docker-compose down deletes this volume. You are using named volumes in your compose file, but don't mount it correctly.
version: '3.0'
services:
db:
image: postgres:latest
restart: always
ports:
- 5432:5432
env_file:
- env_file
volumes:
- postgresdata:/var/lib/postgresql/data
networks:
- db1
Add the default path for postgres data to your volume postgresdata:/var/lib/postgresql/data. This should fix it.

Connecting Spring Cloud Applications in Docker Container

I am attempting to host a Spring Cloud application in Docker containers.The underlying exception is as follows:
search_1 | Caused by: java.lang.IllegalStateException: Invalid URL: config:8888
I understand the reason is because of the URL specified in my config server.
spring.application.name=inventory-client
#spring.cloud.config.uri=http://localhost:8888
spring.cloud.config.uri=config:8888
On my development machine, I am able to use localhost. However, based on a past question (relating to connecting to my database), I learned that localhost is not appropriate in containers. For my database, I was able to use the following:
spring.jpa.hibernate.ddl-auto=none
spring.jpa.show-sql=false
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect
spring.datasource.url=jdbc:postgresql://db:5432/leisurely_diversion
#spring.datasource.url=jdbc:postgresql://localhost:5000/leisurely_diversion
spring.datasource.driver-class-name=org.postgresql.Driver
but this obviously did not work as expected for the configuration server.
My docker-compose file:
# Use postgres/example user/password credentials
version: '3.2'
services:
db:
image: postgres
ports:
- 5000:5432
environment:
POSTGRES_PASSWORD: example
volumes:
- type: volume
source: psql_data
target: /var/lib/postgresql/data
networks:
- app
restart: always
config:
image: kellymarchewa/config_server
networks:
- app
volumes:
- /root/.ssh:/root/.ssh
restart: always
search:
image: kellymarchewa/search_api
networks:
- app
restart: always
ports:
- 8082:8082
depends_on:
- db
- config
- inventory
inventory:
image: kellymarchewa/inventory_api
depends_on:
- db
- config
ports:
- 8081:8081
networks:
- app
restart: always
volumes:
psql_data:
networks:
app:
Both services are running under the same user defined network; how I allow the services to find the configuration service?
Thanks.

Resources