I'm working on a Go project that has the following structure (schematic):
./lib1
./prog1
./...
./prog2
./...
Where:
lib1 is a library
prog1 and prog2 are executables (both depend on lib1)
In a nutshell: How do I create Dockerfiles for prog1 and prog2?
I tried several approaches, all to no avail:
Creating a Dockerfile in prog1/
Failed because docker cannot ADD ../lib1 to the container since it is out of the context (see http://docs.docker.com/reference/builder/#add).
Creating a Dockerfile in the root directory
Ignoring the fact that I actually need two Dockerfiles (for prog1, and for prog2), I tried placing the Dockerfile in the root directory of the project.
However, the docker image I need to use (golang:1.4.1-onbuild) fails to find any Go files (since they are in ./prog1, and not in the root):
+ exec go install -v
can't load package: package app: no buildable Go source files in /go/src/app
What is the recommended project structure for the golang:1.4.1-onbuild image?
Edit: Better asked here: https://stackoverflow.com/questions/29209202/dockerizing-gos-hello-world
Most pull the files in via their VCS or just go get instead of using ADD. You add the the source tree in the same way regardless of what tool your building, since you need the structure of $GOPATH to be the same for each.
Note, you can also name your docker file via docker build -f, so if you want to use ADD, you can have multiple DOCKERFILEs in your root directory.
Related
I am making a Go project containing a few packages. Those are data structures (and there will be algorithms as well). My project root looks like this:
C:.
├───array
├───binary_tree
├───heap
└───list
The thing is, I want to add like a CI. So I would have a job checking if all packages build and if all tests pass.
Unfortunately, I can't run go build on project root. I have to pass whole path to it. I mean I could write a script which calls go build X/Y/foo and then go build X/Y/bar, but the CI on GitLab (docker image) won't have those paths, it will just git clone my repo and that's it (cuz I cannot run it on a relative path from project root, but a relative path from GOPATH, so like github.com/dabljues/project_name/array). And what about the test?
So the question there would be: Can I somehow run go build and go test for all the packages in the Go project? (located in separate folders)
Use a Makefile. Define the targets and instruction to compile them.
For example define the following in a Makefile
build:
go build -o bin/main main.go
run:
go run main.go
To run just invoke : make build
My docker build is failing due to the following error:
COPY failed: CreateFile \?\C:\ProgramData\Docker\tmp\docker-builder117584470\Aeros.Services.Kubernetes\Aeros.Services.Kubernetes.csproj: The system cannot find the path specified.
I am fairly new to docker and have went with the basic project template that is set up when you create a Kubernetes container project template, so I'd figure it would work out of the box, but I'm mistaken.
I'm having problems trying to figure out what it's attempting to due in the temp directory structure and the reason it is failing. Can anyone offer some assistance? I've done some searching and others have said the default docker template was incorrect in Visual Studio, but I'm not seeing any of the files being copied over to the temp directory to begin with, so figuring out what is going on is being rather problematic at the time.
Here is the docker file, the only thing I've added is a publishingProfile arg so I can tell it which profile to use in the Build and Publish steps :
ARG publishingProfile
FROM microsoft/dotnet:2.1-aspnetcore-runtime AS base
WORKDIR /app
EXPOSE 80
FROM microsoft/dotnet:2.1-sdk AS build
WORKDIR /src
COPY ["Aeros.Services.Kubernetes/Aeros.Services.Kubernetes.csproj", "Aeros.Services.Kubernetes/"]
RUN dotnet restore "Aeros.Services.Kubernetes/Aeros.Services.Kubernetes.csproj"
COPY . ./
WORKDIR "/src/Aeros.Services.Kubernetes"
RUN dotnet build "Aeros.Services.Kubernetes.csproj" -c $publishingProfile -o /app
FROM build AS publish
RUN dotnet publish "Aeros.Services.Kubernetes.csproj" -c $publishingProfile -o /app
FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "Aeros.Services.Kubernetes.dll"]
I haven't touched the yaml file, but if you need that I can provide it as well. Again, all I've done with this is add a few NuGet packages to the project reference. Build in VisualStudio runs fine, but the docker command:
docker build . --build-arg publishingProfile=Release
is failing with the error mentioned above.
Can someone be so kind as to offer some enlightenment? Thanks!
Edit 1:
I am executing this from the project's folder via a PowerShell command line.
Leandro's comments helped come across the solution.
So first a rundown of that COPY command, it takes two parameters, source and destination.
Within the template for the Dockerfile for Visual Studio, it includes the folder location of the .csproj file it is attempting to copy. In my case, the command read as follows:
COPY ["Aeros.Services.Kubernetes/Aeros.Services.Kubernetes.csproj", "Aeros.Services.Kubernetes/"]
So it is looking for my Aeros.Services.Kubernetes.csproj file in the Aeros.Services.Kubernetes project folder and copying it to the Aeros.Services.Kubernetes folder in the src folder of Docker.
The problem with this is that if you use the default setup, your dockerfile is included inside the project folder. If you are executing the docker build from within the project folder, the syntax for the COPY command is actually looking in the wrong file location. For instance, if your project is TestApp.csproj located in the TestApp project folder, and you are executing the Docker build command for the dockerfile within the same folder, the syntax for that COPY command:
COPY ["TestApp/TestApp.csproj", "TestApp/"]
is actually looking for: TestApp/TestApp/TestApp.csproj.
The correct syntax for the COPY command in this situation should be:
COPY ["TestApp.csproj", "TestApp/"]
since you are already within the TestApp project folder.
Another problem with the default template that may trouble some is that it doesn't copy the web files for the project either, so once you get past the COPY and dotnet restore steps, you will fail during the BUILD with a:
CSC : error CS5001: Program does not contain a static 'Main' method
suitable for an entry point
This is resolved by adding:
COPY . ./
following your RUN dotnet restore command to copy your files.
Once these pieces have been addressed in the default template provided, everything should be functioning as expected.
Thanks for the help!
In which line the problem happens? I do not remember if docker build shows it.
Where are you executing this build? The problem is that it is not finding the file you are trying to copy. It should be local to where the command is executed.
I saw now, the problem is on the first COPY.
I have a google cloud container build with the following steps
gcr.io/cloud-builders/mvn to run a mvn clean package cmd
gcr.io/cloud-builders/docker to create a docker image
My docker image includes and will run tomcat.
Both these steps work fine independently.
How can I copy the artifacts built by step 1 into the correct folder of my docker container? I need to move either the built wars or specific lib files from step 1 to the tomcat dir in my docker container.
Echoing out the /workspace and /root dir in my Dockerfile doesn't show the artifacts. I think I'm misunderstanding this relationship.
Thanks!
Edit:
I ended up changing the Dockerfile to set the WORKDIR to /workspace
and
COPY /{files built by maven} {target}
The working directory is a persistent volume mounted in the builder containers, by default under /workdir. You can find more details in the documentation here https://cloud.google.com/container-builder/docs/build-config#dir
I am not sure what is happening in your case. But there is an example with a Maven step and a Docker build step in the documentation of the gcr.io/cloud-builders/mvn builder. https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/mvn/examples/spring_boot. I suggest you compare with your files.
If it does not help, could you share your Dockerfile and cloudbuild.yaml. Please make sure you remove any sensitive information.
Also, you can inspect the working directory by running the build locally https://cloud.google.com/container-builder/docs/build-debug-locally#preserve_intermediary_artifacts
Background
Related question: Google Container Builder: How to install govendor dependencies during build step?
I am trying to use Google Cloud Container Builder to automate the building of my Docker containers using Build Triggers.
My code is in Go, and I have a vendor folder (checked in to Git) in my project root which contains all of my Go dependencies.
My project has four binaries that need to be Dockerized, structured as follows:
vendor/
...
program1/
program1.go
main/
main.go
Dockerfile
program2/
program2.go
main/
main.go
Dockerfile
...
Each program's Dockerfile is simple:
FROM alpine
ADD main main
ENTRYPOINT ["/main"]
I have a Build Trigger set up to track my master branch. The trigger runs the following build request (cloudbuild.yaml), which makes use of the open sourced Docker build step:
steps:
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/program1:0.1.15-$SHORT_SHA', '.']
dir: 'program1/main'
... (repeated for each program)
(images, tags omitted)
To summarize, my current build process is as follows:
Edit code.
Build each Go executable using go build. The executable is called main, and is saved in the programX/main/ directory, alongside main.go.
Commit and push code (since the main executables are tracked by Git) to my master branch.
Build Trigger makes four Docker images using the main file built in step 1.
Goal
I would like to eliminate Step 1 from my build process, so that I no longer need to compile my executables locally, and do not need to track my main executables in Git.
In sum, here is my ideal process:
Edit code, commit, push to remote.
Build Trigger compiles all four programs, builds all four images.
Relax :)
Attempted solution
I used the open source Go build step, as follows:
cloudbuild.yaml: (updated)
steps:
- id: 'build-program1'
name: 'gcr.io/cloud-builders/go'
args: ['build', '-a', '-installsuffix', 'cgo', '-ldflags', '''-w''', '-o', 'main', './main.go']
env: ['PROJECT_ROOT=/workspace', 'CGO_ENABLED=0', 'GOOS=linux']
dir: 'program1/main'
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/program1:0.1.15-$SHORT_SHA', '.']
dir: 'program1/main'
waitFor: ['build-program1']
... (repeated for each program)
(images, tags omitted)
After trying various combinations of setting PROJECT_ROOT and GOPATH in the env field of the build-programX, I kept getting the same error for every single package used in my project (file path varies):
cannot find package "github.com/acoshift/go-firebase-admin" in any of
Step #0 - "build-program1": /usr/local/go/src/github.com/acoshift/go-firebase-admin (from $GOROOT)
Step #0 - "build-program1": /workspace/auth/main/gopath/src/github.com/acoshift/go-firebase-admin (from $GOPATH)
It's not even looking for a vendor directory?
What next?
My guess is that one of the following is true:
I am not specifying the GOPATH/PROJECT_ROOT correctly in the build request file. But if so, what is the correct setting?
My project is not structured correctly.
What I am trying to do is impossible :(*
I need to make a custom build step, somehow.
The version of Go used is old - but how can I check this?
I can find no examples online of what I am trying to achieve, and I find GCP's documentation on this subject quite lacking.
Any help would be greatly appreciated!
The issue is with #1: PROJECT_ROOT refers to the desired import path of your binaries. For example, if in program1/main/main.go you import "github.com/foo/bar/program1" to get the package defined in program1/program1.go, you'd set PROJECT_ROOT=github.com/foo/bar.
Fixed the problem (but not exactly sure how...), thanks to these changes:
Set PROJECT_ROOT to my_root, such that the code I want to compile sits at my_root/program1/main/main.go (thanks to John Asmuth for his answer: https://stackoverflow.com/a/46526875/6905609)
Remove the dirs field for the Go build step
Set the -o flag to ./program1/main/main, and the final build arg to ./program1/main/main.go
Previously I was cding into the program1/main directory during the build step, and for some reason go build was looking for packages within my_root/program1/main instead of my_root. Weird!
I have a following Dockerfile in my pet project:
FROM java:8
ADD target/sources-registry-0.0.1-SNAPSHOT.jar sources-registry.jar
RUN bash -c 'touch /sources-registry.jar'
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/sources-registry.jar"]
EXPOSE 8761
And everything perfectly works - image is created and can be run. Now a bit of description about the project: it is a maven based project and before actually executing the Dockerfile I have to manually run mvn package.
However, if I change the Dokerfile to (because I do not want manually to run mvn package and want to automate it)
FROM java:8
RUN ls target
RUN ./mvnw package
ADD target/sources-registry-0.0.1-SNAPSHOT.jar sources-registry.jar
RUN bash -c 'touch /sources-registry.jar'
#ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/sources-registry.jar"]
EXPOSE 8761
then while execution I get /bin/sh: 1: ./mvnw: not found. However, mvnw is in my project files near the pom.xml.
Even more if I just do following Dockerfile
FROM java:8
RUN ls target/
then I get ls: cannot access target/: No such file or directory.
Can someone please explain this behaviour?
I mean why I can actually do something with target folder (first
Dockerfile) even if does not exist (third Dockerfile)?
How project files (and what files) get copied into a created
container?
The main question: Why second Dockerfile is not working? And how I can make it work?
The ADD command copies over the sources-registry.jar file into the Docker image, so that the first example is able to execute it. If you want to use any other files inside the container, you need to include them in the image as well (using ADD or COPY). See the reference docs for more information