Could somebody explain to me why Google Cloud Build creates intermediate containers to run commands?
Step #0 - "Build": Step 2/25 : ARG NODE_ENV
Step #0 - "Build": ---> Running in 17281bea0e29
Step #0 - "Build": Removing intermediate container 17281bea0e29
Step #0 - "Build": ---> a43229632036
Step #0 - "Build": Step 3/25 : ARG DB_NAME
Step #0 - "Build": ---> Running in 04a199971761
Step #0 - "Build": Removing intermediate container 04a199971761
Step #0 - "Build": ---> 3f15b2ad5662
Step #0 - "Build": Step 4/25 : ARG DB_USER
Step #0 - "Build": ---> Running in 4fb95096aab1
Step #0 - "Build": Removing intermediate container 4fb95096aab1
Step #0 - "Build": ---> 97c619b21472
Step #0 - "Build": Step 5/25 : ARG DB_PASSWORD
Step #0 - "Build": ---> Running in 2280e3095a20
Step #0 - "Build": Removing intermediate container 2280e3095a20
Step #0 - "Build": ---> a79acad9a411
Step #0 - "Build": Step 6/25 : ARG CLOUD_SQL_CONNECTION_NAME
Step #0 - "Build": ---> Running in 1f63bde588f0
Step #0 - "Build": Removing intermediate container 1f63bde588f0
Step #0 - "Build": ---> 8e22be5c6191
Step #0 - "Build": Step 7/25 : ENV NODE_ENV ${NODE_ENV}
Step #0 - "Build": ---> Running in 6f5495791d72
Step #0 - "Build": Removing intermediate container 6f5495791d72
Step #0 - "Build": ---> 2413826fa5f6
Step #0 - "Build": Step 8/25 : ENV DB_NAME ${DB_NAME}
Step #0 - "Build": ---> Running in e3099226a900
Step #0 - "Build": Removing intermediate container e3099226a900
Step #0 - "Build": ---> 68f11d7cee19
Step #0 - "Build": Step 9/25 : ENV DB_USER ${DB_USER}
Step #0 - "Build": ---> Running in c1c3dc6ca115
Step #0 - "Build": Removing intermediate container c1c3dc6ca115
Step #0 - "Build": ---> 5fd5610b2de5
Step #0 - "Build": Step 10/25 : ENV DB_PASSWORD ${DB_PASSWORD}
Step #0 - "Build": ---> Running in 67c6ed7a56e7
Step #0 - "Build": Removing intermediate container 67c6ed7a56e7
Step #0 - "Build": ---> 7c8ecb080c59
Step #0 - "Build": Step 11/25 : ENV CLOUD_SQL_CONNECTION_NAME ${CLOUD_SQL_CONNECTION_NAME}
Step #0 - "Build": ---> Running in 8dddcddd80c0
Step #0 - "Build": Removing intermediate container 8dddcddd80c0
Step #0 - "Build": ---> 789cc6e25b55
Step #0 - "Build": Step 12/25 : WORKDIR /usr/src/app
Step #0 - "Build": ---> Running in 2c5e9083a8a5
Step #0 - "Build": Removing intermediate container 2c5e9083a8a5
I am interested in Removing intermediate container ... after each step. Does it hurt the performance of the build?
cloudbuild.yml
steps:
- name: gcr.io/cloud-builders/docker
args:
- build
- '--build-arg'
- NODE_ENV=$_NODE_ENV
- '--build-arg'
- DB_NAME=$_DB_NAME
- '--build-arg'
- DB_USER=$_DB_USER
- '--build-arg'
- DB_PASSWORD=$$DB_PASSWORD
- '--build-arg'
- CLOUD_SQL_CONNECTION_NAME=$_CLOUD_SQL_CONNECTION_NAME
- '--no-cache'
- '-t'
- '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
- .
- '-f'
- Dockerfile
id: Build
secretEnv:
- DB_PASSWORD
- name: gcr.io/cloud-builders/docker
...
Cloud Build uses Docker to execute builds. To understand why Cloud Build creates intermediate containers, first you must understand the Docker build process.
For each build step, Cloud Build executes a Docker container as an instance of docker run. Each step is processed in an intermediate container.
"Those intermediate containers can succeed or fail. If they succeed, the intermediate container is merged with the image from the last successful build step, and then the intermediate container is deleted."
On the performance perspective, removing immediate containers are part of the build process and it helps reduce the size of your container image.
There are already some existing articles that further explains the Docker build process. Here are some interesting links:
How are intermediate containers formed?
https://blog.hipolabs.com/understanding-docker-without-losing-your-shit-cf2b30307c63
https://medium.com/ihme-tech/troubleshooting-the-docker-build-process-454583c80665
Related
I try to create docker image and run a container but maven build fails due failing the tests with testcontainers. Also should say that Im a windows user, but there is a Ubuntu-22.04 over Windows 10. Docker successfully finds WSL2 in settings
P.S. tests are passed if I run mvn clean package/clean install or manually start them
When I run docker build -t *someName* OR docker-compose up --build app I faced with this (most useful as I think) stacktrace after successful dependencies downloading:
#0 78.13 12:55:52.042 [main] INFO org.testcontainers.utility.ImageNameSubstitutor - Image name
substitution will be performed by: DefaultImageNameSubstitutor
(composite of 'ConfigurationFileImageNameSubstitutor' and
'PrefixingImag eNameSubstitutor')
#0 78.18 12:55:52.093 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy -
$XDG_RUNTIME_DIR is not set.
#0 78.18 12:55:52.094 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy -
'/root/.docker/run' does not exist.
#0 78.24 12:55:52.151 [main] DEBUG org.testcontainers.dockerclient.RootlessDockerClientProviderStrategy -
'/run/user/0' does not exist.
#0 78.24 12:55:52.151 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy - Trying
out strategy: UnixSocketClientProviderStrategy
#0 78.24 12:55:52.154 [main] DEBUG org.testcontainers.dockerclient.DockerClientProviderStrategy -
UnixSocketClientProviderStrategy: failed with exception
InvalidConfigurationException (Could not find unix domain socket).
Root caus e NoSuchFileException (/var/run/docker.sock)
#0 78.25 12:55:52.157 [main] INFO org.testcontainers.dockerclient.DockerMachineClientProviderStrategy -
docker-machine executable was not found on PATH
([/opt/java/openjdk/bin, /usr/local/sbin, /usr/local/bin, /usr/sbin,
/usr/bin, /sbin, /bin])
#0 78.25 12:55:52.158 [main] ERROR org.testcontainers.dockerclient.DockerClientProviderStrategy - Could
not find a valid Docker environment. Please check configuration.
Attempted configurations were:
#0 78.25 UnixSocketClientProviderStrategy: failed with exception InvalidConfigurationException (Could not find unix domain
socket). Root cause NoSuchFileException (/var/run/docker.sock)As no
valid configuration was found, e xecution cannot continue.
#0 78.25 See https://www.testcontainers.org/on_failure.html for more details.
===Dockerfile===
FROM maven:3.8.5 AS maven
WORKDIR /usr/src/app
COPY . /usr/src/app
RUN mvn clean package
FROM openjdk:17-jdk-slim
ARG JAR_FILE=practise.jar
WORKDIR /opt/app
COPY --from=maven /usr/src/app/target/${JAR_FILE} /opt/app/
ENTRYPOINT \["java","-jar","practise.jar"\]
EXPOSE 8080
===docker-compose.yml===
version: '3.1'
services:
app:
build: .
image: 'practise'
ports:
"8080:8080"
links:
postgresdb
environment:
SPRING_DATASOURCE_URL=jdbc:postgresql://postgresdb:5432/practise
SPRING_DATASOURCE_USERNAME=root
SPRING_DATASOURCE_PASSWORD=root
SPRING_JPA_HIBERNATE_DDL_AUTO=update
postgresdb:
image: 'postgres:13.1-alpine'
ports:
"5432:5432"
expose:
5432
environment:
POSTGRES_PASSWORD=root
POSTGRES_USER=root
POSTGRES_DB=practise
There is a declaration of test class:
#AutoConfigureMockMvc
#SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT)
#Testcontainers
#TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class CompanyControllerTest {
#Container
public static final PostgreSQLContainer<?> container =
new PostgreSQLContainer<>("postgres")
.withUsername("root")
.withPassword("root");
#DynamicPropertySource
static void properties(DynamicPropertyRegistry registry) {
registry.add("hibernate.connection.url", container::getJdbcUrl);
registry.add("hibernate.connection.username", container::getUsername);
registry.add("hibernate.connection.password", container::getPassword);
}
// tests
}
I use GitHub Action for CI/CD, I write some cypress tests and YAML files from it. But when I push the repository I got an error.
name: Testing EDGE Portal
runs-on: self-hosted
needs: [deploy_edge_service]
steps:
- name: Install node
uses: actions/setup-node#v3
with:
node:version: '16'
- name: setup cypress env
run: |
echo "setup cypress env"
- name: run all test
uses: cypress-io/github-action#v5
with:
start: npm start
working-directory: ./integration_test/NetProbe_E2E/cypress
command: npx cypress run --env grepTags="#trap #probe"
The failing error
enter image description here
How I fix that problem.
i have springboot application, which is created via initializr and i created Dockerfile like following
FROM adoptopenjdk/openjdk11:alpine-jre
ARG JAR_FILE=target/tender-api.jar
WORKDIR /opt/app
COPY ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","app.jar"]
i want to deploy in my ubuntu server. Because it i created in another path docker-compose.yml
version: '3.8'
services:
appDB:
image: postgres:14.1-alpine
restart: always
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
ports:
- '5432:5432'
volumes:
- db:/var/lib/postgresql/data
tender:
image: tender-api:1
build:
context: .
dockerfile: /root/dev/tender-api/Dockerfile
volumes:
- /data/tender-api
ports:
- "8080:8080"
depends_on:
- appDB
volumes:
db:
driver: local
And verbose logs are like following
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 32B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/adoptopenjdk/openjdk11:alpine-jre
#3 DONE 0.9s
#4 [internal] load build context
#4 transferring context: 2B done
#4 DONE 0.0s
#5 [3/4] RUN echo target/tender-api.jar
#5 CACHED
#6 [2/4] WORKDIR /opt/app
#6 CACHED
#7 [4/4] COPY target/tender-api.jar app.jar
#7 ERROR: failed to walk /var/lib/docker/tmp/buildkit-mount4259571673/target: lstat /var/lib/docker/tmp/buildkit-mount4259571673/target: no such file or directory
#8 [1/4] FROM docker.io/adoptopenjdk/openjdk11:alpine-jre#sha256:c4e70e7696899eae647575724b77cc71efa8bea35c17b8d58fbb9bb6485af353
#8 resolve docker.io/adoptopenjdk/openjdk11:alpine-jre#sha256:c4e70e7696899eae647575724b77cc71efa8bea35c17b8d58fbb9bb6485af353 done
#8 sha256:c4e70e7696899eae647575724b77cc71efa8bea35c17b8d58fbb9bb6485af353 433B / 433B done
#8 sha256:b565a288907ff71371d4b22e2d1756ecdab0f5df188a8da6e5af388c330b4615 951B / 951B done
#8 sha256:00e2ce5eeb8afe4d1487c5976172e0fcb4258cbf7628836e6476be083e9e3f3e 6.13kB / 6.13kB done
#8 CANCELED
------
> [4/4] COPY target/tender-api.jar app.jar:
------
failed to solve: failed to compute cache key: failed to walk /var/lib/docker/tmp/buildkit-mount4259571673/target: lstat /var/lib/docker/tmp/buildkit-mount4259571673/target: no such file or directory
now, i can't understand what it is "ERROR: failed to walk ...", i just executed
docker compose up and afterwards I thought that docker can't create file, because of that i have tried with sudo , Nevertheless it's not working
i resolved it and wanted to share with you my solution
changed Dockerfile like following
#
# Build stage
#
FROM maven:3.6.0-jdk-11-slim AS build
COPY src /app/src
COPY pom.xml /app
RUN mvn -q -f /app/pom.xml clean package -DskipTests
# without -q you can take limit size problem
#
# Package stage
#
FROM adoptopenjdk/openjdk11:alpine-jre
ARG JAR_FILE=/app/target/tender-api.jar
COPY --from=build ${JAR_FILE} app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
as second, changed context and dockerfile in docker-compose.yml
context: /root/dev/tender-api/
dockerfile: Dockerfile
I'm using the GitHub action "Build and push Docker images" as it's from Docker and a top rated verified action.
The relevant snippet of my YAML file is as follows
- name: Set up QEMU
uses: docker/setup-qemu-action#v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action#v1
- name: Login to DockerHub
uses: docker/login-action#v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
- name: Build and push
id: docker_build
uses: docker/build-push-action#v2
with:
push: true
tags: user/app:latest
- name: Image digest
run: echo ${{ steps.docker_build.outputs.digest }}
Just as it was shown in the example. When the workflow runs, I consistently see the error
10 [stage-1 2/2] COPY --from=build /workspace/target/*.jar app.jar
#10 DONE 0.9s
#12 exporting to image
#12 exporting layers
#12 exporting layers 4.3s done
#12 exporting manifest sha256:dafb0869387b325491aed0cdc10c2d0206aca28006b300554f48e4c389fc3bf1 done
#12 exporting config sha256:f64316c3b529b43a6cfcc933656c77e556fea8e5600b6d0cce8dc09f775cf107 done
#12 pushing layers
#12 pushing layers 0.8s done
#12 ERROR: server message: insufficient_scope: authorization failed
------
> exporting to image:
------
failed to solve: rpc error: code = Unknown desc = server message: insufficient_scope: authorization failed
Error: The process '/usr/bin/docker' failed with exit code 1
The contents of my Dockerfile for a standard spring-boot application is as shown below
FROM maven:3.6.3-jdk-11-slim AS build
RUN mkdir -p /workspace
WORKDIR /workspace
COPY pom.xml /workspace
COPY src /workspace/src
RUN mvn -B -f pom.xml clean package -DskipTests
FROM openjdk:11-jdk-slim
COPY --from=build /workspace/target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","app.jar"]
Any clue how this can be fixed?
I'm able to publish to docker-hub when using a different GitHub action as shown below
- name: Build and push docker image
uses: elgohr/Publish-Docker-Github-Action#master
with:
name: bloque/sales-lead-management
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_ACCESS_TOKEN }}
You need to set a path context while using the Docker's build-push-action. It should look something like this:
- name: Build and push
id: docker_build
uses: docker/build-push-action#v2
with:
context: .
file: Dockerfile
push: true
tags: user/app:latest
The file option is entirely optional, but if left out it will find the Dockerfile inside the root directory.
It's also recommended to use the metadata action that provides more relevant metadata and tags for your Docker image.
Here is an example of how I did it for Spring Boot apps in few of my projects: https://github.com/moja-global/FLINT.Reporting/blob/d7504909f8f101054e503a2993f4f70ca92c2577/.github/workflows/docker.yml#L153
I am using the below-mentioned code as my 'bitbucket-pipeline.yml' file
image: node:6.9.4
pipelines:
branches:
dev:
- step:
script:
- npm install
At the time of committing the code and pushing to my dev branch, it's not getting triggered to my Pipeline dashboard.