java.net.UnknownHostException: host.docker.internal: Name or service not known on AWS EC2 - spring-boot

I ran into this "java.net.UnknownHostException: host.docker.internal: Name or service not known" problem when deploying a dockerized spring boot application on an AWS EC2 T2.micro instance. The spring boot application failed to start because of this error.
But the weird part is, I did not use the variable "host.docker.internal" anywhere in my application: not in the code, not in the yaml file, not in the .env file:
$ sudo grep -Rl "host.docker.internal" ~
/home/ec2-user/.bash_history
And when I run the following command it shows nothing but previous command to search for it:
$ cat /home/ec2-user/.bash_history | grep "host.docker.internal"
Locally I am using Windows 10 for development, and I can successfully bring up the stack with docker-compose.
Here is the EC2 instance OS version info:
$ cat /etc/*release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)
And here is the docker-compose file that I used on the EC2 instance:
version: '2'
services:
backend:
container_name: backend
image: 'dockerhubuser/backend:0.0.4'
ports:
- '8080:8080'
volumes:
- /var/log/backend/logs:/var/log/backend/logs
- ./backend-ssl:/etc/ssh/backend
env_file:
- .env
depends_on:
- mysql
- redis
redis:
container_name: redis
image: 'redis:alpine'
ports:
- '6379:6379'
volumes:
- $PWD/redis/redis-data:/var/lib/redis
- $PWD/redis/redis.conf:/usr/local/etc/redis/redis.conf
mysql:
container_name: mysql
image: 'mysql:8.0.21'
ports:
- '3306:3306'
environment:
MYSQL_DATABASE: dbname
MYSQL_USER: dbuser
MYSQL_PASSWORD: dbpass
MYSQL_ROOT_PASSWORD: dbrootpass
volumes:
- ./my_volume/mysql:/var/lib/mysql
volumes:
my_volume:
And here is my .env file on the EC2 instance:
SERVER_PORT=8080
KEY_STORE=/etc/ssh/backend/keystore.p12
KEY_STORE_PASSWORD=keystorepass
REDIS_HOST=redis
REDIS_PORT=6379
DB_HOST=mysql
DB_PORT=3306
DB_USERNAME=dbuser
DB_PASSWORD=dbpass
I am pretty sure that this .env file is being used when bringing up the stack with "docker-compose up" because I can see the SERVER_PORT in the log matches this file when I change it.
2021-01-02 20:55:44.870 [main] INFO o.s.b.w.e.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8080 (https)
But I keep getting the error complaining about "host.docker.internal".
Here are things that I have tried but not working:
Hard-code the db host in property spring.datasource.url in application.yml
Add the following entry to /etc/hosts file (see https://stackoverflow.com/a/48547074/1852496)
172.17.0.1 host.docker.internal
Add the following entry to /etc/hosts file, where "ip-172-31-33-56.us-east-2.compute.internal" is what I got when running command "echo $HOSTNAME"
ip-172-31-33-56.us-east-2.compute.internal host.docker.internal
Terminate the instance and created another T2.micro instance, but got same result.
Edit inbound rules to allow TCP:3306 from anywhere.
Can someone take a look? Any help appreciated.

It works on Ubuntu 20.04 after adding "172.17.0.1 host.docker.internal" to /etc/hosts file.
Make sure the docker engine version is 20.10-beta1 or newer.

Related

Docker / Oracle Database / Change Port 1521

I have set an Oracle docker image (https://github.com/oracle/docker-images/tree/main/OracleDatabase/SingleInstance/dockerfiles) which by default is running on port 1521.
I would like to change the port in the Image to 1531.
I know that in the docker-compose I can set "1531:1521" BUT the other container still searching for port 1521 in the created network.
I tried to modify the port referenced in the Dockerfile of the version I want to use (19.3.0) and also in the createDB.sh but when I try to connect with the SID it fails, the listener is not working as expected.
Anybody already succeeded?
Update 1:
Here is the error message when I try to connect to the database after I changed the port.
SQL> CONNECT sys/HyperSecuredPassword#ORCLCDB AS sysdba; ERROR: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
Update 2:
I have the following docker-compose.yaml to set up the other containers for my project.
version: "3.8"
services:
hadea-database:
image: hadea_oracle_1521:19.3.0
container_name: hadea_oracle_1930
ports:
- "1521:1521"
environment:
- ORACLE_SID=ORCLCDB
- ORACLE_PDB=ORCLPDB
- ORACLE_PWD=Oracle4System
- ORACLE_MEM=2000
volumes:
- ./database/OracleDB/oradata:/opt/oracle/oradata
- ./database/OracleDB/setup:/opt/oracle/scripts/setup
- ./database/OracleDB/startup:/opt/oracle/scripts/startup
networks:
- hadea-network
hadea-maildev:
image: maildev/maildev
container_name: hadea_maildev
command: bin/maildev --web 80 --smtp 25 --hide-extensions STARTTLS
ports:
- "8081:80"
networks:
- hadea-network
hadea-server:
build:
context: ./server
dockerfile: Dockerfile
container_name: hadea_back
environment:
- HTTP_PORT=3000
- HTTP_HOST=0.0.0.0
- DATABASE_HOST=hadea-database
- DATABASE_PORT=1521 # CONTAINER port NOT the HOST port
- DATABASE_SID=ORCLCDB
- MAIL_HOST=hadea-maildev
- MAIL_PORT=25 # CONTAINER port NOT the HOST port
ports:
- "3000:3000"
working_dir: /usr/src/app
volumes:
- ./server:/usr/src/app
networks:
- hadea-network
depends_on:
- hadea-database
- hadea-maildev
hadea-front:
build:
context: ./front
dockerfile: Dockerfile
container_name: hadea_front
ports:
- "4200:4200"
- "3001:3001"
volumes:
- ./front:/usr/src/app
networks:
- hadea-network
depends_on:
- hadea-database
- hadea-maildev
- hadea-server
networks:
hadea-network:
If you want to change the port used WITHIN the container (I think this is the question), you could try building a new image after modifying the conf file, e.g. (for the 18c image). The other images hard code the 1521 port in various files in that repo depending on the oracle version you are using, so those would have to be changed prior to building the image.
I have been using this image: container-registry.oracle.com/database/express:latest. This is version 18c and it has a conf file within the image located at /etc/sysconfig/oracle-xe-18c.conf, I would just build a new Dockerfile and overwrite that file with a new one that has the port you require. Or, you could extract the entire contents of that directory, dump it to a host directory, modify the file as needed, and map a volume to etc/sysconfig (make sure the permissions are correct). This way you could tweak the file from the host. It might be possible to set the variable in that conf file from an environment variable within a docker-compose.yaml file or on the docker command line. This variable is named LISTENER_PORT. Some of the variables in these scripts are defined locally and do not pull their values from environment variable though.

Problems deploying a project with docker

I recently started trying to use the docker in a more advanced way. I had already used it to create mysql server and postgres server containers. At the moment I have a Spring test project, which runs normally when starting up. But I tried to instantiate the same with docker, it successfully creates a mysql server, and instantiates a tomcat, but it does not go up the project itself.
Dockerfile content:
FROM tomcat:9-jre11
ENV CATALINA_OPTS="$CATALINA_OPTS -Duser.timezone=America/Sao_Paulo -Xms1024m -Xmx2560m -XX:+UseParallelGC -XX:-OmitStackTraceInFastThrow"
ADD ./smartCircuit/target/*.war $CATALINA_HOME/webapps/
EXPOSE 8080
CMD catalina.sh run;
docker-compose.yaml content:
version: '3.5'
services:
db:
image: mysql:5.7
restart: always
container_name: mysql-container
environment:
MYSQL_DATABASE: 'smart_circuit'
MYSQL_USER: 'user'
MYSQL_PASSWORD: 'password'
MYSQL_ROOT_PASSWORD: 'password'
ports:
# <Port exposed> : < MySQL Port running inside container>
- '3307:3306'
expose:
# Opens port 3306 on the container
- '3306'
# Where our data will be persisted
volumes:
- my-db:/var/lib/mysql
web:
build: .
container_name: tomcat
ports:
- 8080:8080
environment:
DB_URL: jdbc:mysql://mysql-container:3306/smart_circuit?useSSL=false
DB_USER: user
DB_PASSWORD: password
links:
- db:mysql-container
# Names our volume
volumes:
my-db:
I'm expecting my application to start and be accessible when I use the command "sudo docker-compose up --build". For example I have one RestController mapped "project" with a method mapped "find" which receives an id and either retrieves it when found or throws a NotFoundException.
When I run it through InteliJ without docker it works, but when I start it through docker I get a Tomcat 404 Not Found return.
Basically I can access the mysql instance with no problem and the tomcat is kind of working, but the project itself is not being deployed.

Can't reach server inside docker container from host

I am hosting a mysql server and a go http server in docker. I am unable to hit the http server from my host machine. My host machine is a mac.
I have tried using localhost:8080 and ipofserver:8080. I get the ip from the docker inspect. I am able to connect to my mysql server from my host, but i can't hit the server from the host.
Here is my docker ps output.
0.0.0.0:8080->8080/tcp
0.0.0.0:3306->3306/tcp, 33060/tcp
Below are my details:
Docker Desktop version 2.0.0.3.
docker-compose
version: '3.1'
services:
mysql:
image: mysql:latest
restart: always
environment:
MYSQL_ROOT_PASSWORD: password
MYSQL_DATABASE: mydb
volumes:
- mysql:/var/lib/mysql
ports:
- "3306:3306"
networks:
- mynetwork
server:
image: server:latest
networks:
- mynetwork
ports:
- "8080:8080"
volumes:
mysql: ~
networks:
mynetwork:
driver: "bridge"
mysql dockerfile
FROM mysql:8.0.16
COPY ./scripts/mysql/dbgen-v1.sql /docker-entrypoint-initdb.d/
EXPOSE 3306
server dockerfile
FROM golang:1.12.5
WORKDIR a/go/path
COPY . .
ENV GOBIN=/usr/local/bin
RUN go get github.com/go-sql-driver/mysql
RUN go get github.com/iancoleman/strcase
RUN go get github.com/jmoiron/sqlx
RUN go get github.com/spf13/cobra
RUN go get github.com/gorilla/websocket
RUN go get github.com/spf13/viper
RUN go install -v cmd/project/main.go
EXPOSE 8080
CMD ["main"]
(This answer is based on the chat we had in the comments)
In order to expose the web server from inside the container to the host it needs to bind to 0.0.0.0 and not to 127.0.0.1. Using 0.0.0.0 ensures that the web server binds to the bridge interface that can be accessed from the host side.
Relevant Docker docs: https://docs.docker.com/v17.09/engine/userguide/networking/default_network/binding/

Running Sonarqube with docker-compose using bind mount volumes

I’m trying to run Sonarqube in a Docker container on a Centos 7 server using docker-compose. Everything works as expected using named volumes as configured in this docker-compose.yml file:
version: "3"
services:
sonarqube:
image: sonarqube
ports:
- "9000:9000"
networks:
- sonarnet
environment:
- sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
volumes:
- sonarqube_conf:/opt/sonarqube/conf
- sonarqube_data:/opt/sonarqube/data
- sonarqube_extensions:/opt/sonarqube/extensions
- sonarqube_bundled_plugins:/opt/sonarqube/lib/bundled-plugins
db:
image: postgres
networks:
- sonarnet
environment:
- POSTGRES_USER=sonar
- POSTGRES_PASSWORD=sonar
volumes:
- postgresql:/var/lib/postgresql
- postgresql_data:/var/lib/postgresql/data
networks:
sonarnet:
driver: bridge
volumes:
sonarqube_conf:
sonarqube_data:
sonarqube_extensions:
sonarqube_bundled_plugins:
postgresql:
postgresql_data:
However, my /var/lib/docker/volumes directory is not large enough to house the named volumes. So, I changed the docker-compose.yml file to use bind mount volumes as shown below.
version: "3"
services:
sonarqube:
image: sonarqube
ports:
- "9000:9000"
networks:
- sonarnet
environment:
- sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
volumes:
- /data/sonarqube/conf:/opt/sonarqube/conf
- /data/sonarqube/data:/opt/sonarqube/data
- /data/sonarqube/extensions:/opt/sonarqube/extensions
- /data/sonarqube/bundled_plugins:/opt/sonarqube/lib/bundled-plugins
db:
image: postgres
networks:
- sonarnet
environment:
- POSTGRES_USER=sonar
- POSTGRES_PASSWORD=sonar
volumes:
- /data/postgresql:/var/lib/postgresql
- /data/postgresql_data:/var/lib/postgresql/data
networks:
sonarnet:
driver: bridge
However, after running docker-compose up -d, the app starts up but none of the bind mount volumes are written to. As a result, the Sonarqube plugins are not loaded and the sonar postgreSQL database is not initialized. I thought it may be a selinux issue, but I temporarily disabled it with no success. I’m unsure what to look at next.
I think my answer from "How to persist configuration & analytics across container invocations in Sonarqube docker image" would help you as well.
For good measure I have also pasted it in here:
.....
Notice this line SONARQUBE_HOME in the Dockerfile for the docker-sonarqube image. We can control this environment variable.
When using docker run. Simply do:
txt
docker run -d \
...
...
-e SONARQUBE_HOME=/sonarqube-data
-v /PERSISTENT_DISK/sonarqubeVolume:/sonarqube-data
This will make Sonarqube create the conf, data and so forth folders and store data therein. As needed.
Or with Kubernetes. In your deployment YAML file. Do:
txt
...
...
env:
- name: SONARQUBE_HOME
value: /sonarqube-data
...
...
volumeMounts:
- name: app-volume
mountPath: /sonarqube-data
And the name in the volumeMounts property points to a volume in the volumes section of the Kubernetes deployment YAML file.
This again will make Sonarqube use the /sonarqube-data mountPath for creating extenions, conf and so forth folders, then save data therein.
And voila your Sonarqube data is thereby persisted.
I hope this will help others.
N.B. Notice that the YAML and Docker run examples are not exhaustive. They focus on the issue of persisting Sonarqube data.
Try it out BobC and let me know.
Have a great day.
The below code will help you in a single command I hope so.
Create a new docker-compose file named as docker-compose.yaml,
version: "3"
services:
sonarqube:
image: sonarqube:8.2-community
depends_on:
- db
ports:
- "9000:9000"
networks:
- sonarqubenet
environment:
SONAR_JDBC_URL: jdbc:postgresql://db:5432/sonarqube
SONAR_JDBC_USERNAME: sonar
SONAR_JDBC_PASSWORD: sonar
volumes:
- sonarqube_data:/opt/sonarqube/data
- sonarqube_extensions:/opt/sonarqube/extensions
- sonarqube_logs:/opt/sonarqube/logs
- sonarqube_temp:/opt/sonarqube/temp
restart: on-failure
container_name: sonarqube
db:
image: postgres
networks:
- sonarqubenet
environment:
POSTGRES_USER: sonar
POSTGRES_PASSWORD: sonar
volumes:
- postgresql:/var/lib/postgresql
- postgresql_data:/var/lib/postgresql/data
restart: on-failure
container_name: postgresql
networks:
sonarqubenet:
driver: bridge
volumes:
sonarqube_data:
sonarqube_extensions:
sonarqube_logs:
sonarqube_temp:
postgresql:
postgresql_data:
Then, execute the command,
$ docker-compose up -d
$ docker container ps
Sounds like the container is running and, as you mentioned, Sonarqube starts-up. When it starts, is it showing that it's using the H2 in memory db? After running docker-compose up -d, use docker logs -f <container_name> to see what's happening on Sonarqube startup.
To simplify viewing your logs with a known name, I suggest you also add a container name to your Sonarqube service. For example, container_name: sonarqube.
Also, while I know the plan is to deprecate the use of environment variables for the username, password and jdbc connection, I've had better luck in docker-compose using environment variables rather than the corresponding property value. For the connection string, try: SONARQUBE_JDBC_URL: jdbc:postgresql://db/sonar without specifying the default port for postgres.

Docker Compose + Spring Boot + Postgres connection

I have a Java Spring Boot app which works with a Postgres database. I want to use Docker for both of them. I initially put just the Postgres in Docker, and I had a docker-compose.yml file defined like this:
version: '2'
services:
db:
container_name: sample_db
image: postgres:9.5
volumes:
- sample_db:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=sample
- POSTGRES_USER=sample
- POSTGRES_DB=sample
- PGDATA=/var/lib/postgresql/data/pgdata
ports:
- 5432:5432
volumes:
sample_db: {}
Then, when I issued the commands sudo dockerd and sudo docker-compose -f docker-compose.yml up, it was starting the database. I could connect using pgAdmin for example, by using localhost as server and port 5432. Then, in my Spring Boot app, inside the application.properties file I defined the following properties.
spring.datasource.url=jdbc:postgresql://localhost:5432/sample
spring.datasource.username=sample
spring.datasource.password=sample
spring.jpa.generate-ddl=true
At this point I could run my Spring Boot app locally through Spring Suite, and it all was working fine. Then, I wanted to also add my Spring Boot app as Docker image. I first of all created a Dockerfile in my project directory, which looks like this:
FROM java:8
EXPOSE 8080
ADD /target/manager.jar manager.jar
ENTRYPOINT ["java","-jar","manager.jar"]
Then, I entered to the directory of the project issued mvn clean followed by mvn install. Next, issued docker build -f Dockerfile -t manager . followed by docker tag 9c6b1e3f1d5e myuser/manager:latest (the id is correct). Finally, I edited my existing docker-compose.yml file to look like this:
version: '2'
services:
web:
image: myuser/manager:latest
ports:
- 8080:8080
depends_on:
- db
db:
container_name: sample_db
image: postgres:9.5
volumes:
- sample_db:/var/lib/postgresql/data
environment:
- POSTGRES_PASSWORD=sample
- POSTGRES_USER=sample
- POSTGRES_DB=sample
- PGDATA=/var/lib/postgresql/data/pgdata
ports:
- 5432:5432
volumes:
sample_db: {}
But, now if I issue sudo docker-compose -f docker-compose.yml up command, the database again starts correctly, but I get errors and exit code 1 for the web app part. The problem is the connection string. I believe I have to change it to something else, but I don't know what it should be. I get the following error messages:
web_1 | 2017-06-27 22:11:54.418 ERROR 1 --- [ main] o.a.tomcat.jdbc.pool.ConnectionPool : Unable to create initial connections of pool.
web_1 |
web_1 | org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections
Any ideas?
Each container has its own network interface with its own localhost. So change how Java points to Postgres:
spring.datasource.url=jdbc:postgresql://localhost:5432/sample
To:
spring.datasource.url=jdbc:postgresql://db:5432/sample
db will resolve to the proper Postgres IP.
Bonus. With docker-compose you don't need to build your image by hand. So change:
web:
image: myuser/manager:latest
To:
web:
build: .
I had the same problem and I lost some time to understand and solve this problem:
org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
I show all the properties so that everyone understands.
application.properties:
spring.datasource.url=jdbc:postgresql://localhost:5432/testdb
spring.datasource.driver-class-name=org.postgresql.Driver
spring.datasource.username=postgres
spring.datasource.password=postgres
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQL82Dialect
spring.jpa.hibernate.ddl-auto=update
docker-compose.yml:
version: "3"
services:
springapp:
build: .
container_name: springapp
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/testdb
ports:
- 8000:8080
restart: always
depends_on:
- db
db:
image: postgres
container_name: db
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=testdb
- PGDATA=/var/lib/postgresql/data/pgdata
ports:
- 5000:5432
volumes:
- pgdata:/var/lib/postgresql/data
restart: always
volumes:
pgdata:
For start spring application with local database we use url localhost.
For connect to container with database we need change 'localhost' on your database service, in my case 'localhost' to 'db'.
Solution: add SPRING_DATASOURCE_URL environment in docker-compose.yml wich rewrite spring.datasource.url value for connect:
environment:
SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/testdb
I hope this helps someone save his time.
You can use this.
version: "2"
services:
sample_db-postgresql:
image: postgres:9.5
ports:
- 5432:5432
environment:
- POSTGRES_PASSWORD=sample
- POSTGRES_USER=sample
- POSTGRES_DB=sample
volumes:
- sample_db:/var/lib/postgresql/data
volumes:
sample_db:
You can use ENV variable to change the db address in your docker-compose.
Dockerfile:
FROM java:8
EXPOSE 8080
ENV POSTGRES localhost
ADD /target/manager.jar manager.jar
ENTRYPOINT exec java $JAVA_OPTS -jar manager.jar --spring.datasource.url=jdbc:postgresql://$POSTGRES:5432/sample
docker-compose:
`
container_name: springapp
environment:
- POSTGRES=db`

Resources