I have two git repositories:
common
this repository contains protobuf files (.proto), these files should be fetched via git submodule init from other vendor repository. There are also go files with directive: //go:generate protoc --proto_path=a/xxx --go_out=. --go_opt=paths=source_relative a/xxx/b.proto
main
this repository contains main project, which imports the package from common. Then I call git submodule update --init and the go generate. After generate i can see the log: go: finding github.xxx.com/xxx latest
but the main go file shows some problems:
build github.xxx.com/main: cannot load github.xxx.com/xxx/proto: module github.xxx.com/common#latest found, but does not contains package github.xxx.com/xxx/proto
It seems that the go generate command doesn't generate proto files. I think if I import something then these files are somewhere in cache. I'm not sure if my approach is possible.
Could you tell me if this solution is feasible or what I should change.
ok, my solution is project layout like this: https://github.com/golang-standards/project-layout
there I have more main.go files inside cmd folder. In this way I have only one module and inside the module I can have different applicaions. I think it's very cool solution.
Related
Not so familiar with Golang, it's probably a stupid mistake I made... But still, I can't for the life of me figure it out.
So, I got a proto3 file (let's call it file.proto), whose header is as follows:
syntax = "proto3";
package [package_name];
option go_package = "github.com/[user]/[repository]";
And I use protoc:
protoc --go_out=$GOPATH/src --go-grpc_out=$GOPATH/src file.proto
So far so good, I end up with two generated files (file.pb.go and file_grpc.pb.go) inside /go/src/github.com/[user]/[repository]/, and they are defined inside the package [package_name].
Then, the code I'm trying to build has the following import:
import (
"github.com/[user]/[repository]/[package_name]"
)
And I naively thought it would work. However, it produces the following error when running go mod tidy:
go: downloading github.com/[user]/[repository] v0.0.0-20211105185458-d7aab96b7629
go: finding module for package github.com/[user]/[repository]/[package_name]
example/xxx imports
github.com/[user]/[repository]/[package_name]: module github.com/[user]/[repository]#latest found (v0.0.0-20211105185458-d7aab96b7629), but does not contain package github.com/[user]/[repository]/[package_name]
Any idea what I'm doing wrong here? Go version is go1.19 linux/amd64 within Docker (golang:1.19-alpine).
Note: I also tried to only import github.com/[user]/[repository], same issue obviously.
UPDATE:
OK so what I do is that I get the proto file from the git repository that only contains the proto file:
wget https://raw.githubusercontent.com/[user]/[repository]/file.proto
Then I generate go files from that file with protoc:
protoc --go_out=. --go-grpc_out=. file.proto
Right now, in current directory, it looks like:
- directory
| - process.go
| - file.proto
| - github.com
| - [user]
| - [repository]
| - file.pb.go
| - file_grpc.pb.go
In that same directory, I run:
go mod init xxx
go mod tidy
CGO_ENABLED=0 go build process.go
The import directive in process.go is as follows:
import (
"xxx/github.com/[user]/[repository]"
)
Now it looks like it finds it, but still getting a gRPC error, which is weird because nothing changed. I still have to figure out if it comes from the issue above or not. Thanks!
Your question is really a number of questions in one; I'll try to provide some info that will help. The initial issue you had was because
At least one file with the .go extension must be present in a directory for it to be considered a package.
This makes sense because importing github.com/[user]/[repository] would be fairly pointless if that repository does not contain any .go files (i.e. the go compiler could not really do anything with the files).
Your options are:
Copy the output from protoc directly into your project folder and change the package declarations to match your package. If you do this there is no need for any imports.
Copy (or set go_out argument to protoc) the output from protoc into a subfolder of your project. The import path will then be the value of the module declaration in your go.mod plus the path from the folder that the go.mod is in (this is what you have done).
Store the files in a repo (on github or somewhere else). This does not need to be the same repo as your .proto files if you "want it to be agnostic" (note that 2 & 3 can be combined if the generated files will only be used within one code base or the repo is accessible to all users).
Option 1 is simple but its often beneficial to keep the generated code separate (makes it clear what you should not edit and improves editor autocomplete etc).
Option 2 is OK (especially if protoc writes the files directly and you set go_package appropriately). However issues may arise when the generated files will be used in multiple modules (e.g. as part of your customers code) and your repo is private. They will need to change go_package before running protoc (or search/replace the package declarations) and importing other .proto files may not work well.
Option 3 is probably the best approach in most situations because this works with the go tooling. You can create github.com/[user]/goproto (or similar) and put all of your generated code in there. To use this your customers just need to import github.com/[user]/goproto (no need to run protoc etc).
Go Modules/package intro
The go spec does not detail the format of import paths, leaving it up to the implementation:
The interpretation of the ImportPath is implementation-dependent but it is typically a substring of the full file name of the compiled package and may be relative to a repository of installed packages.
As you are using go modules (pretty much the default now) the implementations rules for resolving package paths (synonym of import path) can be summarised as:
Each package within a module is a collection of source files in the same directory that are compiled together. A package path is the module path joined with the subdirectory containing the package (relative to the module root). For example, the module "golang.org/x/net" contains a package in the directory "html". That package’s path is "golang.org/x/net/html".
So if your "module path" (generally the top line in a go.mod) is set to xxx (go mod init xxx) then you would import the package in subfolder github.com/[user]/[repository] with import xxx/github.com/[user]/[repository] (as you have found). If you got rid of the intervening folders and put the files into the [repository] subfolder (directly off your main folder) then it would be import xxx/[repository]
You will note in the examples above that the module names I used are paths to repo (as opposed to the xxx you used in go mod init xxx). This is intentional because it allows the go tooling to find the package when you import it from a different module. For example if you had used go mod init github.com/[user]/[repository] and option go_package = "github.com/[user]/[repository]/myproto";" then the generated files should go into the myproto folder in your project and you import them with import github.com/[user]/[repository]/myproto.
While you do not have to follow this approach I'd highly recommend it (it will save you from a lot of pain!). It can take a while to understand the go way of doing this, but once you do, it works well and makes it very clear where a package is hosted.
I wrote a program and want to encapsulate some logic.
So I did new module and pull it in my git. Link for git looks like
gitlab.xxx.ru/group/subgroup/proj
but when I tried to get it with go get, I got error
fatal: «https://xxx.ru:#gitlab.xxx.ru/group/subgroup.git/» unreachable: URL using bad/illegal format or missing URL
Go tried to load subgroup instead project.
I made folder gitlab.xxx.ru/group/subgroup/ in $GOPATH/src/ and clone my project.
And now it wrote
could not import gitlab.xxx.ru/group/subgroup/proj (no required module provides package "gitlab.xxx.ru/group/subgroup/proj")
So, if I understand correctly, in Golang 1.16 I can't just put my project in the correct directory and I can't use local packages.
How to fix loading from my GitLab and load it with ssh?
Thank you.
UDP go.mod in my proj.
module gitlab.xxx.ru/group/subgroup/proj
go 1.16
require (
golang.org/x/sys v0.0.0-20210608053332-aa57babbf139
golang.org/x/text v0.3.6
)
You may be hitting the intended behaviour from Gitlab which will prevent go from fetching the list of subgroups while trying to compute project dependencies. Since go get requests are unauthenticated, existing projects not under the root group are invisible and cannot be found.
To overcome this limitation, which Gitlab has yet to provide a solution, you can use one of the following two approaches:
Have the project located at the root and not in a subgroup (not always possible)
Leverage the .git extension as well as the replace directive in the project which imports your module (see below)
Go is able to fetch the project living under a subgroup if it knows the version control qualifier (.git). To indicate this, make sure you import the project using the following format gitlab.xxx.ru/group/subgroup/proj.git
While this alone works, it would force you to have all those .git in your imports. To overcome this, you also need a replace directive in your go.mod so you can use the original import path.
In the end, the project which imports your module should have a go.mod that look like this:
require(
gitlab.xxx.ru/group/subgroup/proj v1.7.0
)
replace(
gitlab.xxx.ru/group/subgroup/proj => gitlab.xxx.ru/group/subgroup/proj.git v1.7.0
)
I have my projects that have many packages which import each other and import outside packages. When I make a change to one of my low lever packages, and then push it to git it is fine and works in that section. When I go get it for use in another project that was working perfectly I now get this go get this error:
module declares its path as: github.com/xdg-go/scram
but was required as: github.com/xdg/scram
None of my code uses either of those directly. It looks like it automatically updated some lower external packages and broke things the used to then old import.
How do I either find out the package that is importing the wrong name or stop all auto-updates?
The go.mod file at github.com/xdg/scram declares itself as github.com/xdg-go/scram:
module github.com/xdg-go/scram
go 1.11
require (
github.com/xdg-go/stringprep v1.0.2
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2
)
The go.mod file should be updated to reflect the correct import path.
Unfortunately if this module is for you an indirect dependency, the best fix possible is to update whatever project you import that is directly importing it.
When that is not an option, a solution to this error is to clone the problematic repository locally and use the replace directive in your go.mod file:
module mymodule
replace github.com/xdg/stringprep => ../strprep
go 1.16.2
require (
github.com/divjotarora/mgo v0.0.0-20190308170442-1d451d2a3149
)
where ../strprep is where the code of the required module exists in your local machine, relative to the go.mod file of your project.
The downside of this of course is that you have to replicate this palliative fix wherever you plan to go get your modules.
Note also:
divjotarora/mgo is just a random example of a project that imports one of those packages using their old import path.
I'm using xdg/stringprep as an example because I can't find modules that import xdg/scram instead, but apparently it suffers from the same issue
Beside, you can use:
go mod why <package> to find out why a certain package is listed as a dependency of your project
go mod graph to show the full dependency graph. The output is in <package> <requirement> format
I'm trying to understand how to organize my golang project using go1.11 modules. I tried several options, but none of them worked.
I have some code in the main package under the application folder and a local package that the main package uses.
$GOPATH
+ src
+ application/
+ main/
+ main.go
+ otherFileUnderMainPackage.go
+ aLocalPackage/
+ someCode.go
+ someCode_test.go
+ someMoreCode.go
+ someMoreCode_test.go
Files in the main package, imports ../aLocalPackage. When I compile by go build main/*.go it's working.
Then, I ran go mod init application: V.0.9.9 and got the go.mod file, but the build always fails. I always get error about not finding the local package: build application:V0.9.9/main: cannot find module for path _/.../src/application/aLocalPackage. I also tried to place the local package right under src/, place it under main/ etc. but none of these methods worked for me.
What is the way to use modules and local packages?
Thanks.
Relative import paths are not supported in module mode. You will need to update your import statements to use a full (absolute) import path.
You should also choose a module name other than application. Your module path should generally begin with a URL prefix that you control — either your own domain name, or a unique path such as github.com/$USER/$REPO.
I had some problems working with local packages myself.
There are two tricks to make it work:
you run "go build" in the package directory
This compiles the package and places it in the build cache.
This link about code organisation in go explains more.
You can identify where the cache is using:
>go env GOCACHE
/home/<user>/.cache/go-build
Import using a path relative to the project
I puzzled loads over what the correct import path was and finally discovered that go doc or go list will tell you.
>go doc
package docs // import "tools/src/hello/docs"
>go list
tools/src/hello/docs
For example. I have a hello world API project and was using swaggo to generate documentation which it does in a docs sub-directory.
To use it I add an import:
_ "tools/src/hello/docs"
For my case the _ is important as docs is not used directly but we its init() function to be invoked.
Now in hello/main.go I can add "tools/src/hello/docs" and it will import the correct package.
The path is relative to the location of go.mod if you have one.
I have tools/ here as I have a go.mod declaring "modules tools".
Modules are a different kettle of fish - see https://github.com/golang/go/wiki/Modules.
Recent versions of go (1.11 and later) can create a go.mod file which you may use to fix the version of a module that is used and avoid go's crazy default behaviour of just downloading the latest version of any package you import.
I have written a blogpost on how to start your first Go project using modules.
https://marcofranssen.nl/start-on-your-first-golang-project/
In general it boils down to just create a new folder somewhere on your system (doesn't have to be in GOPATH).
mkdir my-project
cd my-project
go mod init github.com/you-user/my-project
This will create the go.mod file. Now you can simply create your project layout and start building whatever you like.
Maybe one of my other blogs can inspire you a bit more on how to do things.
https://marcofranssen.nl/categories/software-development/golang/
How to setup project structure to produce library and cli with same name in
single repository?
Suppose my project name is project. I want to make it importable with name
project and have executable binary with name project when installed with
go get. My setup currently is like this:
host.com/project/
project/
main.go
core/
project.go
Then, when installed with:
go get host.com/project/project
it installs project as executable which pulls core as dependency. In
core/project.go file, the package has this:
package project
The problem is it is imported with:
import (
"host.com/project/core"
)
And it exports project as name space not core which violates go's convention.
How can I do this?
This isn't a convention, and you may do however you like, but it's a common practice to have a cmd folder in the project root, and all the executables go under that, each in is own folder.
So a good solution in your case could be simply:
host.com/project/
cmd/
project/
project.go
filea.go
fileb.go
In filea.go and fileb.go the package declaration should be:
package project
In cmd/project/project.go package and import declaration should be:
package main
import "host.com/project"
Of course if your project is more complex, you may create further packages under the project root, and it may have multiple commands (multiple main packages), e.g.:
host.com/project/
cmd/
project/
project.go
prjtool/
prjtool.go
packagex/
x.go
packagey/
y.go
filea.go
fileb.go
An example following this layout is the very popular Go Delve debugger (4.5k stars currently), available at https://github.com/derekparker/delve.
A very complex repository example is the golang.org/x/tools package which is a collection of multiple, essential Go tools, available at https://github.com/golang/tools/. Its cmd folder contains more than 20 subfolders (commands), many of the tools (main packages) even consist of multiple .go files (but each using package main declaration, forming a single package of the executable).
Also check out the following related question: What is a sensible way to layout a Go project