I am using multiple packages that I import into different projects, these range from custom adapters for my business logic that are shared by lambda and google cloud functions and other public packages. The way I do this right now is that I vendor them and include them for cloud functions. For applications that can be compiled and deployed on a VM, I compile them separately. This works fine for me, however, its a pain developing these modules.
If I update the method signature and names in the package, I have to push my changes to github / gitlab (my package path is something like gitlab.com/groupName/projectName/pkg/packageName) and then do a go get -u <pacakgeName> to update the package.
This also, does not really update it, sometimes am stuck with an older version with no idea on how to update it. Is there an easier way of working with this I wonder.
For sake of clarity:
Exported package 1
Path: gitlab.com/some/name/group/pkg/clients/psql
psql-client
|
|_ pkg
|
|_psql.go
Application 1 uses psql-client
Path: gitlab.com/some/name/app1
Application 2 uses psql-client
Path: gitlab.com/some/name/app2
My understanding is that (a) you are using the new Go modules system, and that (b) part of the problem is that you don't want to keep pushing changes to github or gitlab across different repositories when you are doing local development.
In other words, if you have your changes locally, it sounds like you don't want to round-trip those changes through github/gitlab in order for those changes to be visible across your related repositories that you are working on locally.
Most important advice
It is greatly complicating your workflow to have > 1 module in a single repository.
As is illustrated by your example, in general it is almost always more work on an on-going basis to have > 1 module in a single repository. It is also very hard to get right. For most people, the cost is almost always not worth it. Also, often the benefit is not what people expect, or in some cases, there is no practical benefit to having > 1 module in a repo.
I would definitely recommend you follow the commonly followed rule of "1 repo == 1 module", at least for now. This answer has more details about why.
Working with multiple repos
Given you are using Go modules, one approach is you can add a replace directive to a module's go.mod file that informs that Go module about the on-disk location of other Go modules.
Example structure
For example, if you had three repos repo1, repo2, repo3, you could clone them so that they all sit next to each other on your local disk:
myproject/
├── repo1
├── repo2
└── repo3
Then, if repo1 depends on repo2 and repo3, you could set the go.mod file for repo1 to know the relative on-disk location of the other two modules:
repo1 go.mod:
replace github.com/me/repo2 => ../repo2
replace github.com/me/repo3 => ../repo3
When you are inside the repo1 directory or any of its child directories, a go command like go build or go test ./.... will use the on-disk versions of repo2 and repo3.
repo2 go.mod:
If repo2 depends on repo3, you could also set:
replace github.com/me/repo3 => ../repo3
repo3 go.mod:
If for example repo3 does not depend on either of repo1 or repo2, then you would not need to add a replace to its go.mod.
Additional details
The replace directive is covered in more detail in the replace FAQ on the Modules wiki.
Finally, it depends on your exact use case, but a common solution at this point is to use gohack, which automates some of this process. In particular, it creates a mutable copy of a dependency (by default in $HOME/gohack, but the location is controlled by $GOHACK variable). gohackalso sets your current go.mod file to have a replace directive to point to that mutable copy.
go get is transitive, so you can just add it to your build process. A typical Go project build is basically:
go get -u ./... && go test ./... && go build ./cmd/myapp
Which gets & updates dependencies, runs all project tests, then builds the binary.
Related
I am getting this error from go mod tidy when I am trying to update my dependancies. I am primarily developing a webhook service to use with cert-manager and I cannot figure out how to resolve this dependency isssue since the packages I am dependent on are created by other developers and I cannot control those "sub dependency".
This is my output:
go.opentelemetry.io/otel/semconv: module go.opentelemetry.io/otel#latest found (v1.9.0), but does not contain package go.opentelemetry.io/otel/semconv
I looked the package up here: https://pkg.go.dev/go.opentelemetry.io/otel/semconv
and the problem for me seems like that the package have been restructured like this:
go.opentelemetry.io/otel/semconv/v1.9.0
as a subdirectory instead of package version.
Is there a way I can manipulate the underlying dependancy of the packages that my service is depending on?
Please leave a comment if you need addictional information.
Take a look at the accepted solution's comments
You may want to use a local copy of the module where you can fix the issue and use it. Steps for that
Clone the module repository git clone https://github.com/open-telemetry/opentelemetry-go.git
If needed, checkout to a branch/tag git checkout branch_name
In the go.mod file of your module, add the following lines
replace (
go.opentelemetry.io => /path/where/cloned/opentelemetry-go
)
Now you should be able to modify code in the cloned opentelemetry-go repo and use it in your module
Currently I am working on an API which uses Serverless Framework with Go.
I'm using the Serverless-offline plugin for local testing.
This API depends on a few other repositories (which I also maintain), which I import using the go.mod file.
However I am having a hard time refining my developer workflow.
Currently, if I want to make changes in a repository which this API depends upon, I have to alter the projects go.mod to include replace directives for the purpose of testing, but then I'm having to manually change it back for deployment to production.
Basically I'm looking for a way to include replace directives, which only get applied during local development. How has everyone else dealt with this problem?
Bonus question: Is there any way to run Serverless offline in docker? I'm finding that serverless-offline running on the bare metal is causing inconsistencies between different developers environments.
You can run go commands with an alternate go.mod file with the -modfile option:
From Build commands:
The -modfile=file.mod flag instructs the go command to read (and
possibly write) an alternate file instead of go.mod in the module root
directory. The file’s name must end with .mod. A file named go.mod
must still be present in order to determine the module root directory,
but it is not accessed. When -modfile is specified, an alternate
go.sum file is also used: its path is derived from the -modfile flag
by trimming the .mod extension and appending .sum.
Create a local.go.mod file with the necessary replace directive for development and build, for example, with:
go build -modfile=local.go.mod ./...
We are converting our internal codebase from the dep dependency manager to go modules (vgo or built in with go1.11.2). Imagine we have code like this:
$GOPATH/src/mycompany/myprogram/main.go:
package main
import (
"fmt"
lib "mycompany/mylib" )
func main() {
fmt.Println("2+3=", lib.add(2, 3))
}
$GOPATH/src/mycompany/myprogram/go.mod:
module mycompany/myprogram
(it doesn't have any dependencies; our real-world code does).
$GOPATH/src/mycompany/mylib/lib.go:
package mylib
func Add(x int, y int) int {
return x + y
}
I didn't module-ize this code; it doesn't seem to matter whether I do or don't.
These are trivial examples but our internal code follows a similar structure as this worked historically.
Since these directories are on the Gopath, export GO111MODULE=auto still builds as before and this works fine (modules not used because we are on the gopath). However, when I set export GO111MODULE=on I immediately get the error:
build mycompany/myprogram: cannot find module for path mycompany/mylib
So I did some research and I would like to validate my understanding. First let me say our old approach worked, but I am more interested in changing to use go modules as it appears to be where the go project itself is headed. So.
It seems the intention of the golang authors was that "dotless" paths belong to the standard repository only; that is there should be a binding between domain name and project. We don't use go get on our internal project, unsurprisingly. Here is the source specifically:
Dotless paths in general are reserved for the standard library; go get has (to my knowledge) never worked with them, but go get is also the main entry point for working with versioned modules.
Can anyone with more knowledge of golang than me confirm this?
My key assumption is that once go decides to use modules, all dependencies must be modules and the gopath becomes somewhat irrelevant, except as a cache (for downloaded modules). Is this correct?
If this is true, we need to use a private gitlab (in our case) repository on the path. There's an open issue on handling this that I'm aware of so we can implement this if necessary. I'm more interested in the consequences, specifically for iterating in the private repositories. Previously we could develop these libraries locally before committing any changes; now it seems we have a choice:
Accept this remote dependency and iterate. I was hoping to avoid needing to push and pull remotely like this. There are workarounds to needing an internet connection if strictly necessary.
Merge everything into one big git repository.
If it matters, I'm using go version go1.11.2 linux/amd64 and my colleagues are using darwin/amd64. If it helps, my golang is exactly as installed by Fedora's repositories.
So, tl;dr, my question is: are go modules all-or-nothing, in that any dependency must be resolved using the module system (go get, it seems) and the gopath has become redundant? Or is there something about my setup that might trigger this to fail? Is there some way to indicate a dependency should be resolved explicitly from the gopath?
Updates since asking the question:
I can move myprogram out of the gopath. The same issue occurs (mylib has been left in the gopath).
I can run, or not run, go mod init mycompany/mylib in the mylib directory; it makes no difference at all.
I came across Russ Cox's blog post on vgo. My concerns about offline development that I tried not to dive into too far are resolved by $GOPROXY.
I use a workaround with GITHUB_TOKEN to solve this.
Generate GITHUB_TOKEN here https://github.com/settings/tokens
export GITHUB_TOKEN=xxx
git config --global url."https://${GITHUB_TOKEN}:x-oauth-basic#github.com/mycompany".insteadOf "https://github.com/mycompany"
I wrote up a solution for this on Medium: Go Modules with Private Git Repositories.
The way we handle it is basically the same as the answer above from Alex Pliutau, and the blog goes into some more detail with examples for how to set up your git config with tokens from GitHub/GitLab/BitBucket.
The relevant bit for GitLab:
git config --global \
url."https://oauth2:${personal_access_token}#privategitlab.com".insteadOf \
"https://privategitlab.com"
#or
git config --global \
url."https://${user}:${personal_access_token}#privategitlab.com".insteadOf \
"https://privategitlab.com"
I hope it's helpful.
I use ls-remote git command to help resolve private repo tags and go get after it.
$ go env GO111MODULE=on
$ go env GOPRIVATE=yourprivaterepo.com
$ git ls-remote -q https://yourprivaterepo.com/yourproject.git
$ go get
My Problem
Elastic Beats is an open source project for log shippers written in Go. It features several log outputs, including console, Elasticsearch and Redis. I would like to add an output of my own - to AWS Kinesis.
I have cloned the repo to ~/github/beats, and tried building it:
$ cd filebeat; go build main.go
However, it failed due to a missing library which is a part of the project:
main.go:6:2: cannot find package "github.com/elastic/beats/filebeat/cmd" in any of:
/usr/local/go/src/github.com/elastic/beats/filebeat/cmd (from $GOROOT)
/Users/adam/go/src/github.com/elastic/beats/filebeat/cmd (from $GOPATH)
A directory of the project is dependent on a package from the same repo, but instead of looking one directory up the hierarchy it looks in the GOPATH.
So, go get github.com/elastic/beats/filebeat/cmd fetched the code, and now go build main.go works. Changing the code in my GOPATH is reflected in these builds.
This leaves me with an structural inconvenience. Some of my code is at a working directory, and some of it is at my GOPATH and included by the code in my working directory.
I would like to have all my code in a single directory for various reasons, not the least being keeping everything under version control.
What Have I Tried
Mostly searching for the problem. I am quite new to Go, so I might have missed the correct terminology.
My Question
What is the right way to edit the code of an imported library in Go?
One of the recommended ways to work with other's packages is:
Get the sources of the original package:
go get github.com/elastic/beats
As a result you will clone project's git repository to the folder
$GOPATH/src/github.com/elastic/beats
Make some fixes, compile code, fix, compile... When you make go install package will be compiled and installed to your system. When you need merge updates from original repository you can git pull them.
Everything is OK. What's next? How to share your work with others?
Fork project on github, suppose it will be github.com/username/beats
Add this fork as another remote mycopy (or any other name you like) to your local repository
git remote add mycopy git://github.com/username/beats.git
When all is done you can push updated sources to your repo on github
git push mycopy
and then open a pull-request to original sources. This way you can share your work with others. And keep your changes in sync with mainstream.
Previous answers to this question are obsolete when developing projects that using Go Modules.
For projects that using Go Modules, one may use the following command to replace an imported library(eg. example.com/imported/module) with a local module(eg. ./local/module):
go mod edit -replace=example.com/imported/module=./local/module
Or by adding the following line into the go.mod file:
replace example.com/imported/module => ./local/module
Reference Docs: https://golang.org/doc/modules/managing-dependencies#unpublished
A project working copy should be checked out into $GOPATH/src/package/import/path - for example, this project should be checked out into /Users/adam/go/src/github.com/elastic/beats. With the project in the correct location, the go tooling will be able to operate on it normally; otherwise, it will not be able to resolve imports correctly. See go help gopath for more info.
Trying to move to the officially supported Golang vendoring solution from legacy Godeps workflow.
Scenario:
Repo A===
\
========> Repo C (shared library code)
/
Repo B===
What is the best workflow I can choose for a mid-size (roughly 5-10 member) team of engineers to vendor Repo C for both Repo A and Repo B? Engineers of varying abilities, most of which probably shouldn't need to know the details of this at all?
I'm currently using govendor for this. I'd prefer not to switch but would if there is a tool that provides a better workflow.
This needs to integrate with a CI server running the builds. I can think of 3 scenarios:
Vendor Repo C into A & B:
Pros:
Reproducible Builds
Easy integration with CI
Cons:
Manual and Error Prone - Can easily vendor incorrect code
Engineers need decent knowledge of vendor tool and methodology
Symlink Repo C trunk branch into vendor folders of A & B:
Pros:
Engineers need no knowledge of vendor tool
Low Developer Maintenance
Cons:
Builds not (easily) reproducible
Possibility of including code in build that shouldn't be released
Less Flexible (Repo A and Repo B can't have differing versions of C)
Include Repo C as a git submodule or subtree in Repo A and Repo B (either utilizing vendor or not):
Pros:
Engineers need no knowledge of vendor tool
Easy Setup
Less Maintenance
Reproducible Builds
Cons:
Having to use git submodule or subtree
Finding surprisingly little about this question on the internet. Is there some idiomatic way of doing this? I'm sure there are other ways of doing this; what am I missing?
i suggest to use a manifest based approach with version constraint.
Project A == Manifest
|- Repo A#~1.0.1
|- Repo B#~1.0.1
Repo A == Manifest
|- Repo C#~1.0.1
Repo B == Manifest
|- Repo C#~1.0.5
Repo C == Manifest empty
Which will resolve into
Project A == Resolved Manifest
|- Repo A#1.0.1
|- Repo B#1.0.1
|- Repo C#1.0.5
where ~1.0.1 means >=1.0.1 <1.1.0.
As you see B and A dependency to C are independent, yet within project they are resolved correctly.
In the event A and B would define incompatible dependency to C, an error should occur as the project should not be build-able.
You may prefer to use caret ^ rather than tilde ~, ^1.0.1 -> >=1.0.1 < 2.0.0.
Note that you are not forced to use those 'helpers' such tilde and caret, you may define explicit version range.
You shall decide which constraint to apply given the level of confidence you give to the remote author to correctly upgrade its version number.
Finally,you can use glide to solve that for you.
Starting with Repo C, assuming you already tagged the repos, run glide init, git commit -am 'glide init', git push
Repo A, glide init, glide get git#repo.com/repoc, git commit -am 'glide init', git push
Repo B, glide init, glide get git#repo.com/repoc, git commit -am 'glide init', git push
Finally, Project A, glide init, glide get git#repo.com/repoa, glide get git#repo.com/repob, git commit -am 'glide init', git push
To re install the project, glide install, go build.
Nothing prevent you to tarball ProjectA with its vendor folder, in order to skip the glide install command when you execute the remote installation.
But you normally don t want to commit the vendor folder for a development environment. You d usually add vendor/ to your .gitignore file and run glide install or glide update.
Expect some difficulties in the begin, passed that step, things will work.
Once you jumped to that workflow, note that you ll have to bump every changes on your repos.
That is bloatware when you work both project A and repo B to reach a viable change, so in that case, rather than vendoring repoB into project A (you can leave the manifest definition, but get ride of the repoB folder into vendor/), install repoB as a go module with the go get command.
Doing so the changes are taken in effect immediately on re-build. Once the change set is completed, browse into each repos and bump them appropriately.
Finally you may want to use a version bumper to help you to get it done quick and fast, it happens i did one for my personal usage.
hope this helps.