I am running Apache Airflow using docker-compose (below docker-compose.yml that I am using) and can't get debug information in case of any errors.
If any error occurs i.e. some requirements can't be installed (see docker-compose.yml below) I don't see any DAGS in a webserver.
I am using the command docker-compose up -d and I tried docker-compose --verbose up -d but it doesn't produce any meaningful outcome.
What is a proper way to debug those issues during day-to-day development?
docker-compose.yml file
version: "3.7"
services:
postgres:
...
init:
...
webserver:
...
scheduler:
...
command: ["-c", "pip install -r /opt/airflow/env/requirements.txt && airflow scheduler"]
volumes:
logs:
Related
What am I wanting to do?
I want to convert to run my containerized of Spring-Boot's app which is can run on amd64 systems, on Raspberry Pi 4's Docker. How can I fix the my problem?
What's the running OS on Raspberry Pi?
I've installed Ubuntu Server 20.04.2 LTS for arm64 architectures using via Raspberry Pi Imager v1.6.1 on Raspberry Pi 4.
What are the steps I have done to fix the problem?(problem is still continue)
In default version does not work on my Raspberry Pi. Thus, I've tried some couple of things at below.
I changed row of "FROM" from openjdk:15-jdk-slim to arm64v8/openjdk:17 at my dockerfile as you can see:
FROM arm64v8/openjdk:17
COPY . /projects/red-dir
WORKDIR /projects/red-dir
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} /projects/red-dir/my_red_app.jar
ENTRYPOINT ["java","-jar","/projects/red-dir/my_red_app.jar"]
Afterwards, I've built and pushed my app using with Maven and Docker:
$ ./mvnw clean package -Dmaven.test.skip=true && java -jar target/my_red_app.jar
$ docker build -t user/my_red_app:v1.0.0 .
Then, My app's depend on a MongoDB, so I changed this image from mongo to arm64v8/mongo on docker-compose.yml as you can see at below:
version: "3"
services:
mongodb:
container_name: mongodb
image: arm64v8/mongo
restart: always
ports:
- 27017:27017
environment:
MONGO_INITDB_ROOT_USERNAME: admin
MONGO_INITDB_ROOT_PASSWORD: 12345
networks:
- shared-net
colour_app:
container_name: my_red_app
image: user/my_red_app
restart: always
ports:
- 7070:7070
depends_on:
- mongodb
networks:
- shared-net
networks:
shared-net:
driver: bridge
Things that work and things that don't work
MongoDB is running. I can reach using via MongoDBCompass, but my Spring-Boot app doesn't work. In docker-compose up command, I've encountered an output like this:
...
...
colour_app | standard_init_linux.go:219: exec user process caused: exec format error
colour_app | standard_init_linux.go:219: exec user process caused: exec format error
colour_app | standard_init_linux.go:219: exec user process caused: exec format error
colour_app | standard_init_linux.go:219: exec user process caused: exec format error
Let me know what things I didn't catch. Thank you for reading.
I've solved my problem and tried to create simple guide for ones who are encountered this problem.
BUILD FOR OTHER OS (multi-architecture / buildx command)
Turn On "experimental" feature's on Docker Settings.
For Linux OS:
sudo nano /etc/docker/daemon.json
Add below content to it:
{
"experimental": true
}
For Win10 OS / MAC OS:
Open Docker Desktop application.
Go to Settings.
Select Docker Engine Tab on the left-side.
Find "experimental": false.
Change it to "experimental": true.
Restart Docker.
Create multi-architecture os. (Important: The image you are using should contain the OS you want to create it)
Creating a just SINGLE OS :
You can use load or push commands.
--load means that docker saves it to local disk:
docker buildx build --load --platform linux/arm64 -t <dockerhub_username>/<repository_name>:<tag_name> .
--push means that docker doesn't save it to local disk but save it to cloud (docker hub):
docker buildx build --push --platform linux/arm64 -t <dockerhub_username>/<repository_name>:<tag_name> .
--load and --push can not be set together.
Creating a MANY OS :
You can just use push command with many OS.
--push means that docker doesn't save it to local disk but save it to cloud (docker hub):
docker buildx build --push --platform linux/amd64,linux/arm64,linux/ppc64le -t <dockerhub_username>/<repository_name>:<tag_name> .
I'm facing a strange problem (or better: two different, weird problems) trying to pass build-args to my Dockerfile through docker-compose up.
My files - initial setup
Dockerfile:
ARG NODE_VERSION
FROM node:${NODE_VERSION}
ARG NPM_REGISTRY_TOKEN
RUN echo "=====> token ${NPM_REGISTRY_TOKEN}"
... ... ...
docker-compose.yml:
version: '3'
services:
myservice:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
With this initial setup in place, I have the following behaviour (on Linux Mint 20, docker-compose version 1.26.2, build eefe0d31):
running docker build --build-arg NPM_REGISTRY_TOKEN=xyz123 produces in output =====> token xyz123: the NPM_REGISTRY_TOKEN arg flows to the Dockerfile
running docker-compose build --build-arg NPM_REGISTRY_TOKEN=xyz123 myservice produces in output =====> token xyz123: the NPM_REGISTRY_TOKEN arg flows to the Dockerfile
running NPM_REGISTRY_TOKEN=xyz123 docker-compose up myservice produces in output =====> token : the NPM_REGISTRY_TOKEN env arg should flow to the Dockerfile due to - NPM_REGISTRY_TOKEN (according to https://docs.docker.com/compose/compose-file/#args: You can omit the value when specifying a build argument, in which case its value at build time is the value in the environment where Compose is running) but it seems to not be available during build
My files - reloaded
Simply changing my docker-compose.yml file to
version: '3'
services:
myservice:
build:
context: ./myservice
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
dockerfile: ../Dockerfile
seems to solve the problem: switching args and dockerfile entries in yml file unlocks the capability to pass environment variables to Dockerfile as build-args through docker-compose up, too. Problem solved. Or not?
Changing OS, getting new problem
So, developers in my team use a bunch of different operating systems: Linux, Mac Os, and Windows, too.
Running the same commands on the same version (1.26.2) of docker-compose on Windows 10 Professional 1909 we're getting the same problem we faced initially, both using the initial version of the docker-compose.yml file and using the version that works on Linux.
We tried passing env var from command line, setting them in the command prompt, setting them as system variables through GUI... we tried launching docker-compose up for git-bash, too, but we're not able to get the variable value in Dockerfile.
I googled a bit aaround but I've not found any reference to known bugs or limitation of the Windows version of docker-compose.
Anyone have any idea what the problem might be? Thank you very much in advance!
So, finally, after some try-and-fail on different OSs and with different configurations, I ended up with an explanation of my problem - and therefore with a viable workaround, which allowed me to reach a satisfactory configuration for my docker-compose-yml file.
Short answer: it wasn't a matter of OSs nor env var passing nor order of context / dockerfile sections - it was a matter of clash between different services in my compose file.
More in detail: my docker-compose.yml file contained an additional service, too, whose job was to initialize the database the application was pointing to:
version: '3'
services:
myservice:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
depends_on:
- persistence
- db_initializer
command: sh -c './wait-for localhost:5432 -- ./wait-for localhost:15672 -- npm run start:dev'
persistence:
# Setting up the DBMS here
db_initializer:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
depends_on:
- persistence
command: sh -c './wait-for localhost:5432 -- ./wait-for localhost:15672 -- npm run db:migrate'
So, the problem was that I was configuring two services based on the same, self-build image, launching it with different commands (npm run db:migrate for the db_initializer service, npm run start:dev for the application service). Apparently compose took the configuration provided for the first initialized service (db_initializer, because myservice was dependant on it) and used that configuration for both services, ignoring the (different) args section I was providing for the second container: so I was able to solve (this time really!) the problem simply merging services declaration, including all args I needed:
version: '3'
services:
myservice:
build:
context: ./myservice
dockerfile: ../Dockerfile
args:
- NODE_VERSION=10.15.1-alpine
- NPM_REGISTRY_TOKEN
depends_on:
- persistence
- db_initializer
command: sh -c './wait-for localhost:5432 -- ./wait-for localhost:15672 -- run db:migrate && npm run start:dev'
persistence:
# Setting up the DBMS here
So, after a bunch of months without collecting answers, I think it's time to share my experience, hoping it can help someone encountering this weird behaviour.
I have an up and running containers and I wish to execute a database backup. Apparently, a simple command from the docker such as: sudo mkdir new_folder result in: bash: sudo: command not found
What have I tried (on an intuitive level) I accessed one of the running container with docker exec -i -t 434a38fedd69/bin/bash and RUN
apt-get update
apt-get install sudo
when exit back to docker and tried to perform sudo mkdir new_folder but I got the same message bash: sudo: command not found
Baresp#adhg MINGW64 /c/Program Files/Docker Toolbox/postgre
$ mkdir new_folder
mkdir: cannot create directory ‘new_folder’: Permission denied
Baresp#adhg MINGW64 /c/Program Files/Docker Toolbox/postgre
$ sudo mkdir new_folder
bash: sudo: command not found
BTW, I'm not sure if this is relevant but the docker-compose file I was using is:
version: '2'
services:
postgres:
image: postgres
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: changeme
PGDATA: /data/postgres
volumes:
- /data/postgres:/data/postgres
ports:
- "5432:5432"
networks:
- postgres
restart: unless-stopped
container_name: xx_postgres
pgadmin:
links:
- postgres:postgres
image: fenglc/pgadmin4
volumes:
- /data/pgadmin:/root/.pgadmin
ports:
- "5050:5050"
networks:
- postgres
restart: unless-stopped
container_name: xx_pgadmin
networks:
postgres:
driver: bridge
First, nothing you do in a docker exec is persistent outside of that particular running container (copy of the image), so if you want future containers run from that image to include sudo, those apt-get commands need to go into the Dockerfile that builds the image. Which, since you're using docker-compose, would require you to first make a Dockerfile and specify its location in the YAML.
Second, what do you mean "exit back to docker"? Nothing you do inside a container is going to have any effect on the system that Docker itself is running on, but it looks like you're running software install commands inside a Docker container and then expecting that to result in the newly-installed software being available outside the container on the Windows system that is running Docker.
To do a backup of the postgres database in the container, you first have to enter the container (similar to how you do it):
docker exec -it postgres bash
(substitude postgres with the real container name you get from docker-compose ps)
Now you are in the container as root. That means, you don't need sudo for anything. Next create your backup folder:
mkdir /tmp/backup
Now run the backup command, from a quick Google I found the following (you might know better):
pg_dumpall > /tmp/backup/filename
Then exit the shell within the container by typing exit. From your host system run the following to copy the backup file out of the container:
docker cp postgres:/tmp/backup/filename .
(postgres is your container name again)
I set a env variable in docker-compose, like so:
cloud:
build:
context: foobar/.
ports:
- "5000:5000"
depends_on:
- redis
- rabbitmq
- postgresql
links:
- redis
- rabbitmq
- postgresql
environment:
- RABBITMQ_HOST=rabbitmq
And I can see that it's listed in the running container and I can echo it just fine.
WORKON_HOME=/opt/virtualenvs
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHLVL=1
HOME=/root
no_proxy=*.local, 169.254/16
LESSOPEN=| /usr/bin/lesspipe %s
RABBITMQ_HOST=rabbitmq
LESSCLOSE=/usr/bin/lesspipe %s %s
_=/usr/bin/env
But when I try to use it in a configuration file in the running container, like so:
amqp://guest:guest#${RABBITMQ_HOST}
Then I run docker-compose up, and the app just exits because it can't find the RABBITMQ_HOST variable? What gives?
I'm using envdir like this to start in a bash script the gunicorn server:
envdir /apps/foobar/.envdir gunicorn -w 2 -b 0.0.0.0:5000 dispatch:app --reload
update:
I'm using Docker Enterprise Edition (EE) on a mac (el capitan).
I'm using Flask (python 3) with the Gunicorn application server (dockerfile).
I want to test ELK.
It works fine
BUt when I want to do a
docker-compose up
behind a proxy
docker-compose up --no-recreate
Building kibana
Step 1 : FROM kibana:latest
---> 544887fbfa30
Step 2 : RUN apt-get update && apt-get install -y netcat
---> Running in 794342b9d807
It failed
W: Some index files failed to download. They have been ignored, or old ones used instead.
Is' OK with
docker build --build-arg http_proxy=http://proxy:3128 --build-arg https_proxy=http://proxy:3128 kibana
But when I redo a docker-compose up, il tries to re-build, and failed to pass through proxy
Any help ?
You will need docker-compose 1.6.0-rc1 in order to pass the proxy to your build through docker-compose.
See commit 47e53b4 from PR 2653 for issue 2163.
Move all build related configuration into a build: section in the service.
Example:
web:
build:
context: .
dockerfile: Dockerfile.name
args:
key: value
As mkjeldsen points out in the comments
If key should assume the value of an environment variable of the same name, value can be omitted (docker-compose ARGS):
Especially useful for https_proxy: if the envvar is unset or empty, the builder will not apply proxy, otherwise it will.
I ran into the same problem. What helped me was using the explicit version 2.2 and then build - args and - network as described in the documentation.
VonC is right, it works for me by adding args section under the build lines in docker-compose file:
original:
ssh:
build: ssh/.
container_name: ssh
ports:
- "3000:22"
networks:
vault_net:
ipv4_address: 172.16.238.20
Modified:
ssh:
build:
context: "ssh/."
args:
HTTP_PROXY: http://X.X.X.X:XXXX
HTTPS_PROXY: http://X.X.X.X:XXXX
NO_PROXY: .domain.ltd,127.0.0.1
container_name: ssh
ports:
- "3000:22"
networks:
vault_net:
ipv4_address: 172.16.238.20
Note that I have to add quotes for context since it needs to be formatted as string.
Thanks a lot.
did you try it on clean machine?
docker-machine stop default
docker-machine create -d virtualbox test
docker-machine start test
eval $(docker-machine env test)
docker-compose up