I'm trying to get to grips with working with workspaces in go 1.18, and how to make it work well in a monorepo.
As a minimum example, I have the following project structure:
.
├── docker-compose.yml
├── go.work
├── makefile
└── project
├── go.mod
├── module1
│ ├── Dockerfile
│ ├── go.mod
│ └── main.go
├── module2
│ ├── Dockerfile
│ ├── go.mod
│ └── main.go
└── shared-module
├── go.mod
└── shared.go
module1 and module2 can both be built into binaries with their respective main.go files. However, module1 uses shared-module, but module2 does not.
When building the binaries without Docker, I can cd into module1 or module2 and run go build and everything works fine. However problems occur when I want to use Docker for the builds.
Here is the Dockerfile for module1:
# FIRST STAGE
FROM golang:1.18 AS builder
WORKDIR /app/
# copy workfiles
COPY go.* ./
WORKDIR /app/project/
# dependency management files
COPY project/go.* ./
COPY project/module1/go.* ./module1/
COPY project/shared-module/go.* ./shared-module/
WORKDIR /app/project/module1/
RUN go mod download
WORKDIR /app/project/
# copy shared module
COPY project/shared-module/ ./shared-module/
# copy module to compile
COPY project/module1/ ./module1/
WORKDIR /app/project/module1/
RUN CGO_ENABLED=0 GOOS=linux go build -o bin/module1
# SECOND STAGE
FROM alpine:3.14.2
WORKDIR /app
COPY --from=builder /app/project/module1/bin/module1 /app
ENTRYPOINT [ "./module1" ]
With this build i'm trying to maximise caching by specifying files which change infrequently first. I'm also excluding any files which I don't need for the module (like module2).
Running docker-compose build module1 to build the image using that Dockerfile, the build fails with error:
go: open /app/project/module2/go.mod: no such file or directory
This initially surprised me, as module2 is not a dependency of either module1, or shared-module, but after a bit of consideration I realised it was because of the go.work file which specifies ./project/module2. Removing the line in go.work that specifies that module allows the image to be built successfully.
My question is therefore, if I want to have streamlined image builds, do I have to create multiple go.work files for each of the modules I want to build in Docker? For example, I would need another go.work file for module2 which omits module1 and shared-module.
Related
Suppose I have a monorepo. Apps (app1, app2) use a Gradle as a build system and share some build-logic with includeBuild("../shared-build-logic") which is outside of the root of each app.
├── shared-build-logic
│ └── src/...
└── app1
├── Earthfile
├── build.gradle
├── src/...
└── app2
├── Earthfile
├── build.gradle
├── src/...
Is it possible for Earthfile to access the files from outside of its root folder or Earthly has the same restrictions as Dockerfile?
I get the following error on attempt to COPY ../shared-build-logic ./:
============================ ❌ FAILURE [2. Build 🔧] ============================
Repeating the output of the command that caused the failure
+compile *failed* | --> COPY ../shared-build-logic ./
+compile *failed* | [no output]
+compile *failed* | ERROR Earthfile line 22:4
+compile *failed* | The command
+compile *failed* | COPY ../shared-build-logic ./
+compile *failed* | failed: "/shared-build-logic": not found
I would also like to perform integration testing with the docker-compose.yaml file located one level above the Eartfile root, but facing the similar problem:
integration-tests:
FROM earthly/dind:alpine
COPY ../docker-compose.yaml ./ # <------- does not work
WITH DOCKER --compose docker-compose.yaml --load build-setup=+compile --allow-privileged
RUN docker run -e SPRING_DATA_MONGODB_URI=mongodb://mongodb:27017/test-db --network=default_dev-local build-setup ./gradlew test
END
Is my the only solution to the problem to move Earthfile itself one level upper?
While you can't directly access targets outside of your Earthfile directory you can reference targets.
This allows you to write a target in an Earthfile under shared-build-logic that saves an artifact containing those files
You can expose the files you need by using a target.
shared-build-logic/Earthfile
files:
WORKDIR files
# Copy all of files you want to share
SAVE ARTIFACT ./*
app/Earthfile
use-files:
COPY ../shared-build-logic+files/* .
# do stuff
You should be able to do something similar with you integration-test target.
Earthfile
files:
WORKDIR files
SAVE ARTIFACT docker-compose.yaml
folder-with-integration-tests/Earthfile
integration-tests:
FROM earthly/dind:alpine
COPY ../+files/docker-compose.yaml ./
WITH DOCKER --compose docker-compose.yaml --load build-setup=+compile --allow-privileged
RUN docker run -e SPRING_DATA_MONGODB_URI=mongodb://mongodb:27017/test-db --network=default_dev-local build-setup ./gradlew test
END
I encountered a problem with gradle project structure. I have a task that needs to be realized and some tests are meant to be executed to check whether my project structure is correct and the tasks in gradle execute correctly. However I think I misunderstood instruction a bit and I'm wondering whether I can do something with my current folders structure or If I will have to rewrite the whole project. My current project structure looks like this:
main-repo-folder/
├── docker-related-file
├── rootProject
│ ├── sub-project-1
│ ├── build(output from tasks is created here)
│ ├── build.gradle
│ ├── sub-project-2
│ ├── gradle
│ ├── gradlew
│ ├── gradlew.bat
│ ├── settings.gradle
│ └── src
As you can see, the root project is a directory inside a repo. In order for my tests to execute I think the repo itself must be a root folder (or act as one) because the tests seem to be trying executing there. And here is my question, is it possible to add f.e settings.gradle file in main-repo-folder (at the same level as rootProject folder) to "point" gradle to build from rootProject and treat that folder as the root?(I mean f.e if I call gradle clean build task_name in main-repo-folder I want to make gradle execute it as I would be in rootProject folder)
I've tried to find some information but I'm at the path of learning gradle and I don't know if it is even possible :/ .
Rename main-repo-folder/rootProject to main-repo-folder.
In my maven application i have multiple projects:
Core
Application 1
Application 2
Application 1 and Application 2 are two projects that uses the core (for example, those application are built for two different customers)
In order to Dockerize all of them, the simplest way would be to create a multi-module project, but the downside is that i have all inside a single project (core + Application 1 + Application 2).
I would like to have the core separated from them.
The main problem with this configuration is that the core project need to built before the other two, and App 1 and App 2 use this as maven dependency:
App 1
<dependency>
<groupId>it.myorg</groupId>
<artifactId>core-project</artifactId>
<version>1.12.0-SNAPSHOT</version>
</dependency>
If i try to dockerize the App 1 its fail when i package it, because inside the docker container core-project 1.12.0-SNAPSHOT do not exists.
I was thinking to setup a "local maven repo", pushing the core there and App 1 will "pull" the jar from the repo and not from .m2 folder, but i dont like this soulution.
I can provide more information, sorry if i dont provide examples, but my paper is white right now :(
Folder structure
+- Core
--- pom.xml
--- src
+- Application1
--- pom.xml
--- src
The solution i'm trying to do now is create a Dockerfile for core project (FROM maven:latest), building the image with a tag and in Dockerfile of App1 use this image (so, creating a multi stage build but in two separate moments).
The best would be
FROM maven:latest as core-builder
## build the core
FROM maven:latest
## Copy jar from builder
Because the project are in separate folder, i cant build the core in this way. I need to build del core BEFORE (running docker build -t) and later copy from them.
UPDATE
After the correct answer from #mihai i'm asking if its possible a structure like this:
-- myapp-docker
- Dockerfile
- docker-compose.yml
-- core-app
-- application_1
Having Dockerfile at the same level of core-app and application_1 its totally fine and 100% working. The only "problem" is that i should put all the projects in the same repo.
This is the proposed solution with multi-stage builds.
To replicate your setup I created this structure:
.
├── Dockerfile-app1
├── application1
│ ├── pom.xml
│ └── src
│ └── main
│ ├── resources
│ └── webapp
│ ├── WEB-INF
│ │ └── web.xml
│ └── index.jsp
├── core
│ ├── pom.xml
│ └── src
│ ├── main
│ │ └── java
│ │ └── com
│ │ └── test
│ │ └── App.java
│ └── test
│ └── java
│ └── com
│ └── test
│ └── AppTest.java
In the pom.xml file from Application 1 I added the dependency to core:
<dependency>
<groupId>com.test</groupId>
<artifactId>core</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
I named the Dockerfile Dockerfile-app1, this way you can have more than 1 of them.
This is the Dockerfile-app1:
FROM maven:3.6.0-jdk-8 as build
WORKDIR /apps
COPY ./core .
RUN mvn clean install
FROM maven:3.6.0-jdk-8
# If you comment this out then the build fails because it cannot find the dependency to 'core'
COPY --from=build /root/.m2 /root/.m2
COPY ./application1 ./
RUN mvn clean install
You should probably add an entrypoint at the end to run your project or even better add another 3rd stage that only copies the generated artefacts and runs your project (this way the final image will not have your sourced in).
The first stage only builds the core submodule.
The second stage used the results of the first stage, copies only the source for application1 and builds it.
You can easily replicate this for application2 by creating a similar file Dockerfile-app2.
Since you're using maven, try dockerfile-maven to build the image. You don't want any of your build information inside of your image (like what the dependencies are), you should just add the jar at the end. I usually use it together with spring-boot-maven-plugin and repackage, to get a fully self-contained jar.
I'm new to go modules, and am taking them for a spin in a new project which I'm trying to model after the structure described here
Here is an example of my directory structure:
.
├── cmd
│ └── app_name
│ └── main.go
├── go.mod
├── go.sum
├── internal
│ └── bot
│ └── bot.go
└── pkg
├── website_name
│ ├── client.go
│ ├── client.options.go
│ ├── server.go
│ └── server.options.go
└── lib
└── lib.go
Is this idiomatically correct? I know there's not a whole lot of consensus out there, but I'd like to follow best practices.
When I run go build I get 'unexpected module path "github.com/ragurney/app_name/cmd/app_name"', but when I run go build ./... it works. Why?
When I move main.go to the top level everything works as expected. Should I just not use the /cmd pattern with modules?
To answer your first question, its completely opinionated and whatever you like best that is also easy to understand for others you should go with (I think it's fine).
To answer your second question the reason go build ./... works as opposed to go build from the root directory is because ./... starts in the current directory (the root) and searches for all program entry-points and builds them. When you move main.go to the root directory, with this new information, go build working then makes sense, as its only looking in the current directory.
You can explicitly say go build ./cmd/app_name which would also work.
Your application structure works perfectly fine with modules, as I use something very similar to it (https://www.ardanlabs.com/blog/2017/02/package-oriented-design.html) and modules work very well for me.
from what i can tell there is nothing wrong with your project structure. What has worked for me is to run the go build/run command from the project root
eg.
go run github.com/username/project/cmd/somecommand
go build -o somebinary github.com/username/project/cmd/somecommand
I prefer to add the specific file to build, there are some projects with more than one executable
go build -o app ./cmd/server/main.go
my working tree is like this:
/opt/go/src/tb-to-composer/
├── apis
│ └── rtb.go
├── config.yaml
├── jsondef
│ └── structures.go
├── LICENSE.md
├── README.md
├── tb-to-composer
└── thingsToComposer.go
when I do go build inside /opt/go/src/tb-to-composer/ the build doesn't recompile rtb.go and structures.go even though there was changes in them. In order to achieve build I need to run go build -a every time I do a change to rtb.go or structures.go, is that the expected behavior from go build? How to I recompile only custom libs inside my package folder without recompile the whole /opt/go/src tree?
You can try the -i flag, or (this does not work, sorry) specify the files in the directories explicitly as arguments to go build, i.e. go build thingsToComposer.go apis/rtb.go jsondef/structures.go