I have a golang application structure like this:
.
├── calc
│ ├── go.mod
│ ├── main.go
│ └── Makefile
├── go.mod
├── LICENSE
├── num
│ ├── go.mod
│ └── num.go
└── README.md
Where calc is an "application" where I'm importing the num package to add 2 numbers.
calc/go.mod
go 1.15
require github.com/github_username/goapp/num v0.2.1
num/go.mod
module github.com/github_username/goapp/num/v0.2.1
go 1.15
go.mod
module github.com/github_username/goapp/v0.2.1
go 1.15
When in /calc, and I run go run main.go, I get the following:
go: github.com/github_username/goapp/num#v0.2.1: reading github.com/github_username/goapp/num/num/go.mod at revision num/v0.2.1: unknown revision num/v0.2.1
What am I doing wrong? The github repo has the annotated tags.
For further context, I'm mimicking a production setup where we have six different mini golang services in folders such as calc, calc2, etc. where each "calc" service has a go.mod file.
module github.com/github_username/goapp/num/v0.2.1
Is nonsense. The semver version tag "v0.2.1" does not belong into the module name.
(Note that for major versions > 1, e.g. 4.3.1, the major version becomes part of the name like in module github.com/user/proj/folder/v4).
And one more: There are no source code belonging to the root go.mod so this module makes no sense whatsoever.
You really should not make that many modules.
Are you working with private repositories?
if yes, then you need to configure OAuth authentication:
export GITHUB_TOKEN=MY_GITHUB_TOKEN
git config --global url."https://${GITHUB_TOKEN}:x-oauth-basic#github.com/".insteadOf "https://github.com/"
Now, if you don't have the necessity to use private repositories!
Turn your repositories to the public that will solve the problem too.
Related
I've got a git repo on a local server. I can clone from it with git clone user#10.xxx.yyy.zzz:/srv/git/liqid.git where the user has ssh access and read/write to the git/ directory.
When I try to use it with go get -v user#10.xxx.yyy.zzz:/srv/git/liqid.git it gives
go: cannot use path#version syntax in GOPATH mode
I've tried various other combinations like leaving out the :, but they all fail.
Can go get work with a repo on a local server?
Here's what I've learned about using go get for packages and modules when the git repo is on a private server. I hope it helps someone else to have all the steps documented in one place.
Using Packages And Modules With git Repositories on Private Servers
Here are all the steps needed to use Go packages and modules with git repositories on private servers. These are servers with IP addresses such as 10.xxx.yyy.zzz or 192.168.xxx.yyy. No github or gitlab is assumed to exist on these servers so there is no web server running on port 80, which is the assumption that go get is based on. Your private server only needs to have sshd running.
Install the latest version of Go and be sure the GOPATH is set. The first element in your GOPATH is where the code from these examples will be downloaded to.
You'll need another computer on your network where you will create git repositories. This will be your private git server.
If you want to skip all of the setup steps and example code you can check the two Key Differences sections that list the differences when using packages or modules with a private server instead of a public git repository.
The source code
Put dateutil.go and stringutil.go in the directory structures shown below for both packages and modules.
dateutil.go:
package datepackage
import "time"
func GetTime() time.Time {
return time.Now().Local()
}
stringutil.go:
package stringpackage
import "strings"
func ToUpperCase(s string) string {
return strings.ToUpper(s)
}
main.go (put main.go somewhere outside of the directory structure shown below):
package main
import (
"fmt"
"github.com/your-github-username/go-package-test-dateutil/datepackage"
"github.com/your-github-username/go-package-test-stringutil/stringpackage"
)
func main() {
fmt.Println("github: " + stringpackage.ToUpperCase("test"))
fmt.Println("github: " + datepackage.GetTime().String())
}
These files can be found on github.com
git#github.com:dwschulze/go-package-test-dateutil.git
git#github.com:dwschulze/go-package-test-stringutil.git
git#github.com:dwschulze/go-module-package-test-drivers.git
Create Packages Using the GOPATH Convention
Create a directory structure like this outside of your GOPATH and add the files above. This follows the GOPATH convention, but you don't need these files in you GOPATH.
package
├── github
│ ├── dateutil
│ │ └── src
│ │ └── datepackage
│ │ └── dateutil.go
│ └── stringutil
│ └── src
│ └── stringpackage
│ └── stringutil.go
└── your-local-git-repo-hostname
├── dateutil
│ └── src
│ └── datepackage
│ └── dateutil.go
└── stringutil
└── src
└── stringpackage
└── stringutil.go
your-local-git-repo-hostname is the hostname or IP address of your private git server where you'll create git repos (not the machine where you currently have this code). There is an undocumented requirement that the hostname for the go get command have a . in it. If the hostname of your private git server doesn't have a . in it then use its IP address. Add ssh key access to your private git server with ssh-copy-id.
Using a Private repo on github
We'll start with the simplest case, using packages from github.com. You'll need a github account with ssh key access setup.
Create git repos in the two src/ directories under github/ that you created above using git init
package
├── github
│ ├── dateutil
│ │ └── src
│ │ └── ...
│ └── stringutil
│ └── src
| └── ...
Add and commit the datepackage/ and stringpackage/ dirs respectively to your git repos.
Create two private github repos named go-package-test-dateutil and go-package-test-stringutil in your account on github.com. Follow the instructions to set the remotes in your local git repos in your src/ directories to their respective github repositories. Push the code.
Since your repos are private you'll have to use ssh public key access to download the code. Since go get uses https by default you'll need to add this to your ~/.gitconfig file:
git config --global url."git#github.com:".insteadOf "https://github.com/"
Run these commands to put the code you just pushed to github.com into your GOPATH:
go get github.com/your-github-username/go-package-test-stringutil/stringpackage/
go get github.com/your-github-username/go-package-test-dateutil/datepackage/
The packages get downloaded into the pkg/ and src/ directories of the first element in your GOPATH.
In the directory with the main.go file you created above type go run main.go and the results will be printed to the console.
Create Git Repositories on Your Private Server
Now you'll create git repos on your private git server. To keep this simple you'll just use git init --bare. There's no need to install github or gitlab. On your private git server you'll need to have sshd running and have ssh key access from the machine where your code is. The new repos will be in /home/myusername/gitrepo.
git init /home/myusername/gitrepo/go-package-test-dateutil --bare
git init /home/myusername/gitrepo/go-package-test-stringutil --bare
The go get command has an undocumented requirement that the hostname have a . in it. If the hostname of your private git server doesn't have a . in it then use its IP address, which is what I'll do throughout the rest of this example. Assume the machine's IP address is 192.168.0.12
Copy the dateutil.go and stringutil.go files that you created earlier into the directories under your-local-git-repo-hostname:
package
├── github
| ...
└── your-local-git-repo-hostname
├── dateutil
│ └── src
│ └── datepackage
│ └── dateutil.go
└── stringutil
└── src
└── stringpackage
└── stringutil.go
In the two src/ directories create local git repos as you did before and add and commit the code. Set the remotes to the git repo that you created on your private git server
git remote add origin myusername#your-local-git-repo-hostname:gitrepo/go-package-test-dateutil
git remote add origin myusername#your-local-git-repo-hostname:gitrepo/go-package-test-stringutil
You'll need another entry in ~/.gitconfig for your private git server:
git config --global url."myusername#your-local-git-repo-hostname:".insteadOf "https://192.168.0.12/"
Now push the code. Note that this code is not yet in your GOPATH.
Use go get to retrieve the code from your private git server. It's necessary to add a ".git" suffix to the directory name where you created the git repos with git init --bare on your private git server. That tells go get that this is a git repository rather than some other version control system.
go get 192.168.0.12/gitrepo/go-package-test-stringutil.git/stringpackage
go get 192.168.0.12/gitrepo/go-package-test-dateutil.git/datepackage
The packages get downloaded into the pkg/ and src/ directories of the first element in your GOPATH.
└── src
├── 192.168.0.12
│ └── gitrepo
│ ├── go-module-test-dateutil.git
│ │ ├── dateutil.go
│ │ └── go.mod
│ ├── go-package-test-dateutil.git
│ │ └── datepackage
│ │ └── dateutil.go
│ └── go-package-test-stringutil.git
│ └── stringpackage
│ └── stringutil.go
In your main.go code add two import statements to the packages on your private git server
package main
import (
dpkg "192.168.0.12/gitrepo/go-package-test-dateutil.git/datepackage"
strpkg "192.168.0.12/gitrepo/go-package-test-stringutil.git/stringpackage"
"fmt"
"github.com/your-github-username/go-package-test-dateutil/datepackage"
"github.com/your-github-username/go-package-test-stringutil/stringpackage"
)
func main() {
fmt.Println("github: " + stringpackage.ToUpperCase("test"))
fmt.Println("github: " + datepackage.GetTime().String())
fmt.Println("local: " + strpkg.ToUpperCase("test"))
fmt.Println("local: " + dpkg.GetTime().String())
}
Note the use of the aliases dpkg and strpkg in the two new import statements because the import statements have duplicate package names (the last element in the import statement).
Now you can run the code with go run main.go.
Key differences between using a private repo on guthub.com and a repo on a private git server:
Modifying ~/.gitconfig
Adding the .git suffix in go get statements
The hostname in go get and import statements must have a dot in it
Creating Modules Using Private Repositories
Create a directory structure like this parallel to the packages/ directory you created above. Copy the dateutil.go and stringutil.go files from above. Change the package statements to datemod and stringmod respectively. You'll create the go.mod files later.
module
├── github
│ ├── datemodule
│ │ ├── dateutil.go
│ │ └── go.mod
│ └── stringmodule
│ ├── go.mod
│ └── stringutil.go
└── 192.168.0.12
├── datemodule
│ ├── dateutil.go
│ └── go.mod
└── stringmodule
├── go.mod
└── stringutil.go
Note that the directory names are different from the package names. Packages don't have to follow directory names.
Generate the go.mod files by executing the following commands in these directories:
In github/datemodule/
go mod init github.com/your-github-username/go-module-test-dateutilmod
In github/stringmodule/
go mod init github.com/your-github-username/go-module-test-stringutilmod
In 192.168.0.12/datemodule/
go mod init 192.168.0.12/gitrepo/go-module-test-dateutil.git
In 192.168.0.12/stringmodule/
go mod init 192.168.0.12/gitrepo/go-module-test-stringutil.git
In the four datemodule/ and stringmodule/ directories above create a git repo with git init and add and commit the files.
Create two private repos on github.com named go-module-test-dateutilmod and go-module-test-stringutilmod. Follow the instructions to set the remotes in your respective local git repos under the github/ directory. Push the code.
On your private git server create two git repos with
git init /home/myusername/gitrepo/go-module-test-dateutil --bare
git init /home/myusername/gitrepo/go-module-test-stringutil --bare
Set the remotes in the respective git repos under 192.168.0.12/ directory with
git remote add origin myusername#192.168.0.12:gitrepo/go-package-test-dateutil
git remote add origin myusername#192.168.0.12:gitrepo/go-package-test-stringutil
Push the code.
Now you have four different modules, two each in repositories in your github.com account and your private git server.
In another directory create a main.go program to use these modules:
package main
import (
datemodlocal "192.168.0.12/gitrepo/go-module-test-dateutil.git"
stringmodlocal "192.168.0.12/gitrepo/go-module-test-stringutil.git"
"fmt"
"github.com/your-github-username/go-module-test-dateutilmod"
"github.com/your-github-username/go-module-test-stringutilmod"
)
func main() {
fmt.Println("github: " + stringmod.ToUpperCase("test"))
fmt.Println("github: " + datemod.GetTime().String())
fmt.Println("local: " + stringmodlocal.ToUpperCase("test"))
fmt.Println("local: " + datemodlocal.GetTime().String())
fmt.Println("local toString: " + datemodlocal.GetTimeStr())
}
To use modules with private repos we have to set GOPRIVATE
go env -w GOPRIVATE=192.168.0.12/gitrepo/*,github.com/your-github-username/*
When GOPRIVATE is set modules will be pulled directly from the specified git repos instead of the Go public proxy.
Now run
$ go mod init module-driver
go: creating new go.mod: module module-driver
$ cat go.mod
module module-driver
go 1.15
Now execute main.go. It will download the modules from github.com and your private git server before running the code:
$ go run main.go
go: finding module for package github.com/dwschulze/go-module-test-dateutilmod
go: finding module for package github.com/dwschulze/go-module-test-stringutilmod
go: finding module for package 192.168.0.12/gitrepo/go-module-test-stringutil.git
go: finding module for package 192.168.0.12/gitrepo/go-module-test-dateutil.git
go: downloading 192.168.0.12/gitrepo/go-module-test-stringutil.git v0.0.1
go: downloading 192.168.0.12/gitrepo/go-module-test-dateutil.git v0.0.3
go: downloading github.com/dwschulze/go-module-test-dateutilmod v0.0.1
go: downloading github.com/dwschulze/go-module-test-stringutilmod v0.0.1
go: found 192.168.0.12/gitrepo/go-module-test-dateutil.git in 192.168.0.12/gitrepo/go-module-test-dateutil.git v0.0.3
go: found 192.168.0.12/gitrepo/go-module-test-stringutil.git in 192.168.0.12/gitrepo/go-module-test-stringutil.git v0.0.1
go: found github.com/dwschulze/go-module-test-dateutilmod in github.com/dwschulze/go-module-test-dateutilmod v0.0.1
go: found github.com/dwschulze/go-module-test-stringutilmod in github.com/dwschulze/go-module-test-stringutilmod v0.0.1
github: TEST
github: 2020-12-08 07:57:02.969147007 -0700 MST
local: TEST
local: 2020-12-08 07:57:02.969220121 -0700 MST
local toString: 2020-12-08 07:57:02.969222359 -0700 MST (dev2 branch)
local GetTimeStr3: 2020-12-08 07:57:02.96925053 -0700 MST (dev2 branch)
You didn't have to run go get to download these modules before running the code. The go.mod was also modified
$ cat go.mod
module module-driver
go 1.15
require (
192.168.0.12/gitrepo/go-module-test-dateutil.git v0.0.3 // indirect
192.168.0.12/gitrepo/go-module-test-stringutil.git v0.0.1 // indirect
github.com/dwschulze/go-module-test-dateutilmod v0.0.1 // indirect
github.com/dwschulze/go-module-test-stringutilmod v0.0.1 // indirect
)
You can download your modules with go get to test that your environment is set up correctly:
go get 192.168.0.12/gitrepo/go-module-test-dateutil.git
go get 192.168.0.12/gitrepo/go-module-test-stringutil.git
go get github.com/your-github-username/go-module-test-dateutilmod
go get github.com/your-github-username/go-module-test-stringutilmod
Again note the use of the .git suffix on the repository path for your private git server. Without this go get will use https instead of git (which would use ssh).
Running go get modifies the go.mod file. While some people say you can manually modify the go.mod file others say that you should not edit go.mod manually and instead use go get to make any modifications.
Downloading the module code either with go get or go run main.go downloads the code into $GOPATH/pkg/mod. Since no version was specified it will pull the latest semantic version tag for that module.
Semantic versioning is done with tags which are just a commit. Tags are independent of branches so if the latest semver was done on a branch other than master it will pull that version.
To specify a specific version use
go get 192.168.0.12/gitrepo/go-module-test-dateutil.git#v0.0.1
This will update the entry in go.mod if one already exists
You should be able to delete $GOPATH/pkg/mod/ directory and do go run main.go again. You'll see Go downloading the needed modules before running the code.
Key Differences When Using Modules From a Private Repository
Modifying ~/.gitconfig
Adding the .git suffix for the repos on your private server
The hostname of your private server must have a dot in it, or use its IP address
Set GOPRIVATE with go env -w GOPRIVATE=...
Using code on a different branch
Semantic version tags are independent of branches, but there is one case where go get can use a branch. If you want to go get the latest commit from a branch you can append #branchname like this:
go get 192.168.0.12/gitrepo/go-module-test-dateutil.git#branchname
If the latest commit on the branch does not have a semver tag go get will create a pseudo-version using the next semver tag number and a timestammp and hash.
FAQs
What does a 410 Gone error mean when downloading a module?
go get github.com/your-github-username/go-module-test-dateutilmod
go: downloading github.com/your-github-username/go-module-test-dateutilmod v0.0.1
go get github.com/your-github-username/go-module-test-dateutilmod: github.com/your-github-username/go-module-test-dateutilmod#v0.0.1: verifying module: github.com/your-github-username/go-module-test-dateutilmod#v0.0.1: reading https://sum.golang.org/lookup/github.com/your-github-username/go-module-test-dateutilmod#v0.0.1: 410 Gone
server response: not found: github.com/your-github-username/go-module-test-dateutilmod#v0.0.1: invalid version: unknown revision v0.0.1
This can happen if you don't have GOPRIVATE set. Go tries to retrieve your (private) modules from the Go public proxy, but it can't find them there.
Resources
Using private repos from public git hosting services. A demonstration of how GOPRIVATE and GOPROXY work.
I am having issues trying to deploy a Google cloud function in Go 1.11 using Go modules. I have have the following code structure in my GOPATH:
└── example
├── models
│ ├── go.mod
│ └── models.go
└── load
├── fn.go
├── go.mod
├── go.sum
└── vendor
└── ....
the load/go.mod looks like the following:
module github.com/example/load
require (
github.com/example/models v0.0.0
)
replace github.com/example/models => ../models
When I try to deploy the function using the command
gcloud functions deploy load-data --entry-point GCSNewFileTrigger --runtime go111 --trigger-resource new_data --trigger-event google.storage.object.finalize
I get the following error:
Deploying function (may take a while - up to 2 minutes)...failed.
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Build failed: go: parsing /models/go.mod: open /models/go.mod: no such file or directory
go: error loading module requirements
The commands go mod vendor and go mod verify run successfully locally and I can see my local package models in the vendor folder of load
Modules are preferred by the builder over vendor. If there is a go.mod, modules will be used. When you upload your function, it only includes the directory with your function at the root, not any directories one level up. So, when there is a go.mod and you have a replace directive pointing one level up, it will not work.
The solution is to vendor and not upload the go.mod/go.sum files. When using gcloud, you can create a .gcloudignore file to do this for you. See https://cloud.google.com/functions/docs/concepts/go-runtime#specifying_dependencies for more detail.
Disclaimer: I work at Google and on this product.
I have a repository structure as follows :-
xyz/src
1. abc
- p
- q
- r
2. def
- t
- u
- v
3. etc
- o
- m
- n
I have created a .mod file in src and run go build ./...
Except for local packages everything is fine. So if abc/p is being used in def then it throws the following exception :- cannot find module providing package abc/p. The idea behind keeping the .mod file in src package was to make sure the path is being found from where the mod file is located. Can anyone suggest where should the mod file ideally should be? also i tried placing it one directory above in xyz but still same issue as well as i created one for each sub directory. I am bit confused on this. Will I have to create separate repository for abc and etc. But considering gopath which earlier used to work for the same I think module should also be able to do the same. Any suggestions?
The most common and easiest approach is a single go.mod file in your repository, where that single go.mod file is placed in the root of your repository.
Russ Cox commented in #26664:
For all but power users, you probably want to adopt the usual convention that one repo = one module. It's important for long-term evolution of code storage options that a repo can contain multiple modules, but it's almost certainly not something you want to do by default.
The Modules wiki says:
For example, if you are creating a module for a repository
github.com/my/repo that will contain two packages with import paths
github.com/my/repo/foo and github.com/my/repo/bar, then the first
line in your go.mod file typically would declare your module path as
module github.com/my/repo, and the corresponding on-disk structure
could be:
repo/
├── go.mod <<<<< Note go.mod is located in repo root
├── bar
│ └── bar.go
└── foo
└── foo.go
In Go source code, packages are imported using the full path including
the module path. For example, if a module declared its identity in its
go.mod as module github.com/my/repo, a consumer could do:
import "example.com/my/repo/bar"
That imports package bar from the module github.com/my/repo.
I have a single go.mod in the root of my go application. I am using the following structure inspired by Kat Zien - How Do You Structure Your Go Apps
At the minute one of my applications looks like this
.
├── bin
├── cmd
│ ├── cli
│ └── server
│ └── main.go
├── pkg
│ ├── http
│ │ └── rest
| │ # app-specific directories excluded
│ └── storage
│ └── sqlite
All packages are imported via their full path, i.e. import "github.com/myusername/myapp/pkg/http/rest" otherwise it causes problems all over the place and this was the one change I had to make going from $GOPATH to go mod.
go mod then handles all the dependencies it discovers properly as far as I've discovered so far.
I'm new to go modules, and am taking them for a spin in a new project which I'm trying to model after the structure described here
Here is an example of my directory structure:
.
├── cmd
│ └── app_name
│ └── main.go
├── go.mod
├── go.sum
├── internal
│ └── bot
│ └── bot.go
└── pkg
├── website_name
│ ├── client.go
│ ├── client.options.go
│ ├── server.go
│ └── server.options.go
└── lib
└── lib.go
Is this idiomatically correct? I know there's not a whole lot of consensus out there, but I'd like to follow best practices.
When I run go build I get 'unexpected module path "github.com/ragurney/app_name/cmd/app_name"', but when I run go build ./... it works. Why?
When I move main.go to the top level everything works as expected. Should I just not use the /cmd pattern with modules?
To answer your first question, its completely opinionated and whatever you like best that is also easy to understand for others you should go with (I think it's fine).
To answer your second question the reason go build ./... works as opposed to go build from the root directory is because ./... starts in the current directory (the root) and searches for all program entry-points and builds them. When you move main.go to the root directory, with this new information, go build working then makes sense, as its only looking in the current directory.
You can explicitly say go build ./cmd/app_name which would also work.
Your application structure works perfectly fine with modules, as I use something very similar to it (https://www.ardanlabs.com/blog/2017/02/package-oriented-design.html) and modules work very well for me.
from what i can tell there is nothing wrong with your project structure. What has worked for me is to run the go build/run command from the project root
eg.
go run github.com/username/project/cmd/somecommand
go build -o somebinary github.com/username/project/cmd/somecommand
I prefer to add the specific file to build, there are some projects with more than one executable
go build -o app ./cmd/server/main.go
my working tree is like this:
/opt/go/src/tb-to-composer/
├── apis
│ └── rtb.go
├── config.yaml
├── jsondef
│ └── structures.go
├── LICENSE.md
├── README.md
├── tb-to-composer
└── thingsToComposer.go
when I do go build inside /opt/go/src/tb-to-composer/ the build doesn't recompile rtb.go and structures.go even though there was changes in them. In order to achieve build I need to run go build -a every time I do a change to rtb.go or structures.go, is that the expected behavior from go build? How to I recompile only custom libs inside my package folder without recompile the whole /opt/go/src tree?
You can try the -i flag, or (this does not work, sorry) specify the files in the directories explicitly as arguments to go build, i.e. go build thingsToComposer.go apis/rtb.go jsondef/structures.go