I am trying to migrate an existing Go project to use modules but having difficulties working out the correct directory structure. The repo contains two regular packages (p1, p2) along with example programs to demonstrate how to use the packages. I am using Go 1.13.
The current layout is
<reponame>
|
+--- p1
| |
| +-- p1a.go
| +-- p1b.go
|
+--- p2
| |
| +-- p2a.go
| +-- p2b.go
|
+-- examples
|
+-- e1.go (which refers to packages p1 and p2)
The github repository is already public with a version tag > 1. I know I need a new major number, v5, so it looks like I need to move to having v5 somewhere in the path to the packages.
But should it be <repo>/v5/p1 or <repo>/p1/v5? Where do go.mod files go? I suspect I need one to cover the two packages, and one for the example programs.
All the examples I can find for version > 1 seem to only have a single package in the module.
And importantly, how do I get the examples to compile and run? I've tried all kinds of permutations of a go.mod file for the examples but they all keep failing with an error like "no matching versions for query 'v5'" even with the "replace" directive to try to point at the local directories.
It looks like it is expecting a v5 version of code to be already published to github before I've done local testing.
Modules are about versioning. If p1, p2 and example are versioned together then put the go.mod at the repo level. This is probably the what you want.
Make sure the module declares itself properly, i.e. module <repo>/v5 in the go.mod file.
You do not need the v5 in the file system path. There are basically three ways: A v5 folder containing the v5 stuff, a v5 branch containing the v5 stuff or none of the two in which case your repo will be v5 only.
Make sure the import paths are correct. E.g. in package examples you must import p1 via import "<repo>/v5/p1". (Same for p2 or even p1 importing p2, etc.)
Thats all.
cd into the example folder and go build: Go will look up the filesystem tree for a go.mod and will find it at the repo root. It thus knows that this package belongs to module <repo>/v5 and thus where to find all packages from the module <repo>/v5 and can import <repo>/v5/p1 without the need for any replace directive. Like this you can work locally on the v5 without need to push to a remote repo.
(A common mistake is to not declare the module as v5: In the go.mod file you must have a line module github.com/<user>/<repo>/v5. If you falsely just write module github.com/<user>/<repo> then when compiling e.g. examples the compiler thinks "well, the examples package belongs to module github.com/<user>/<repo> and I should import github.com/<user>/<repo>/v5/p1 so let's peek at github.com// to see what we find there..." and wont find anything and complain.)
Related
Not so familiar with Golang, it's probably a stupid mistake I made... But still, I can't for the life of me figure it out.
So, I got a proto3 file (let's call it file.proto), whose header is as follows:
syntax = "proto3";
package [package_name];
option go_package = "github.com/[user]/[repository]";
And I use protoc:
protoc --go_out=$GOPATH/src --go-grpc_out=$GOPATH/src file.proto
So far so good, I end up with two generated files (file.pb.go and file_grpc.pb.go) inside /go/src/github.com/[user]/[repository]/, and they are defined inside the package [package_name].
Then, the code I'm trying to build has the following import:
import (
"github.com/[user]/[repository]/[package_name]"
)
And I naively thought it would work. However, it produces the following error when running go mod tidy:
go: downloading github.com/[user]/[repository] v0.0.0-20211105185458-d7aab96b7629
go: finding module for package github.com/[user]/[repository]/[package_name]
example/xxx imports
github.com/[user]/[repository]/[package_name]: module github.com/[user]/[repository]#latest found (v0.0.0-20211105185458-d7aab96b7629), but does not contain package github.com/[user]/[repository]/[package_name]
Any idea what I'm doing wrong here? Go version is go1.19 linux/amd64 within Docker (golang:1.19-alpine).
Note: I also tried to only import github.com/[user]/[repository], same issue obviously.
UPDATE:
OK so what I do is that I get the proto file from the git repository that only contains the proto file:
wget https://raw.githubusercontent.com/[user]/[repository]/file.proto
Then I generate go files from that file with protoc:
protoc --go_out=. --go-grpc_out=. file.proto
Right now, in current directory, it looks like:
- directory
| - process.go
| - file.proto
| - github.com
| - [user]
| - [repository]
| - file.pb.go
| - file_grpc.pb.go
In that same directory, I run:
go mod init xxx
go mod tidy
CGO_ENABLED=0 go build process.go
The import directive in process.go is as follows:
import (
"xxx/github.com/[user]/[repository]"
)
Now it looks like it finds it, but still getting a gRPC error, which is weird because nothing changed. I still have to figure out if it comes from the issue above or not. Thanks!
Your question is really a number of questions in one; I'll try to provide some info that will help. The initial issue you had was because
At least one file with the .go extension must be present in a directory for it to be considered a package.
This makes sense because importing github.com/[user]/[repository] would be fairly pointless if that repository does not contain any .go files (i.e. the go compiler could not really do anything with the files).
Your options are:
Copy the output from protoc directly into your project folder and change the package declarations to match your package. If you do this there is no need for any imports.
Copy (or set go_out argument to protoc) the output from protoc into a subfolder of your project. The import path will then be the value of the module declaration in your go.mod plus the path from the folder that the go.mod is in (this is what you have done).
Store the files in a repo (on github or somewhere else). This does not need to be the same repo as your .proto files if you "want it to be agnostic" (note that 2 & 3 can be combined if the generated files will only be used within one code base or the repo is accessible to all users).
Option 1 is simple but its often beneficial to keep the generated code separate (makes it clear what you should not edit and improves editor autocomplete etc).
Option 2 is OK (especially if protoc writes the files directly and you set go_package appropriately). However issues may arise when the generated files will be used in multiple modules (e.g. as part of your customers code) and your repo is private. They will need to change go_package before running protoc (or search/replace the package declarations) and importing other .proto files may not work well.
Option 3 is probably the best approach in most situations because this works with the go tooling. You can create github.com/[user]/goproto (or similar) and put all of your generated code in there. To use this your customers just need to import github.com/[user]/goproto (no need to run protoc etc).
Go Modules/package intro
The go spec does not detail the format of import paths, leaving it up to the implementation:
The interpretation of the ImportPath is implementation-dependent but it is typically a substring of the full file name of the compiled package and may be relative to a repository of installed packages.
As you are using go modules (pretty much the default now) the implementations rules for resolving package paths (synonym of import path) can be summarised as:
Each package within a module is a collection of source files in the same directory that are compiled together. A package path is the module path joined with the subdirectory containing the package (relative to the module root). For example, the module "golang.org/x/net" contains a package in the directory "html". That package’s path is "golang.org/x/net/html".
So if your "module path" (generally the top line in a go.mod) is set to xxx (go mod init xxx) then you would import the package in subfolder github.com/[user]/[repository] with import xxx/github.com/[user]/[repository] (as you have found). If you got rid of the intervening folders and put the files into the [repository] subfolder (directly off your main folder) then it would be import xxx/[repository]
You will note in the examples above that the module names I used are paths to repo (as opposed to the xxx you used in go mod init xxx). This is intentional because it allows the go tooling to find the package when you import it from a different module. For example if you had used go mod init github.com/[user]/[repository] and option go_package = "github.com/[user]/[repository]/myproto";" then the generated files should go into the myproto folder in your project and you import them with import github.com/[user]/[repository]/myproto.
While you do not have to follow this approach I'd highly recommend it (it will save you from a lot of pain!). It can take a while to understand the go way of doing this, but once you do, it works well and makes it very clear where a package is hosted.
My folder structure looks something like this... (say my git repo name is demorepo)
demorepo
|--directory1
|-- (no go.mod at this level)
|--module1
|--package1 --------->--------------->--------------------->----|
|--go.mod (github.com/demorepo/directory1/module1) |
|--go.sum |
|--module2 |
|--module3 |
|--directory2 |
|-- (no go.mod at this level) |
|--newmodule ------<------------<------------------<----------------|
Now, I want to use a function defined in "package1" in my "newmodule"
When I hit
go get <repo_address>/directort1/module1/package1 at "new_module"
it says ....
github.com/<repo>#upgrade found (v0.0.0-20211215055943-92e412ad4a12), but does not contain package github.com/<repo>/directory1/module1/package1
There is a proposal for a Go Workspace File for Go 1.18 which should simplify this task.
Meanwhile, you can use the replace directive in your go.mod file to refer to a module located on a local filesystem.
demorepo/directory1/module1/go.mod:
module github.com/<repo>/directory1/module1
demorepo/directory2/newmodule/go.mod:
module github.com/<repo>/directory2/newmodule
replace github.com/<repo>/directory1/module1 => ../../directory1/module1
Now you can import github.com/<repo>/directory1/module1/package1 in newmodule normally and it'll refer to the local module1.
You might not want the replace directive in the go.mod file itself, and instead make a copy of it e.g. go.mod.local and use it when building your project: go build -modfile go.mod.local . (can also add *.local to .gitignore).
Approach :
If you want to use module1 inside newModule then you should make one new repo for module1 put your logic there push it in github. please make sure you should use appropriate version for the library.
import it as a library and it will work.
Also refer the official docs of module dependency and also check root level dependency.
Official docs : https://go.dev/blog/using-go-modules
I am developing multiple microservices, which require different modules (which shall be available like modules on github, but private)
My first tests with Go were all located within one package, which gets quite messy after some time
I'm coming from the Java side of programming - with loads of packages - which keep stuff clear and clean.
(This also works with public modules like on github.com/xyz/module1 github.com/xyz/module2 github.com/xyz/module3)
I just need this for private modules - how can I do so?
This is what I tried yet:
Directory of my go sources:
C:\my\path\to\go\src\
In this dir I have multiple subdirs containing modules (actually more than listed here)
my-module1
my-module2
my-module3
For each folder I called go mod init but I get the message that
package my-module1/util is not in GOROOT (c:\go\src\my-module1\util)
which is obviously right, as my private libraries reside in C:\my\path\to\go\src\
Importing packages from github with go get ... is working without troubles (those packages will be loaded but copied to c:\go\src)
Working with all files in one folder works but is not the desired solution (I need to create multiple microservices therefore I want to be able to create different projects with custom executeables and or tests)
What am I doing wrong?
If more information is needed I will provide it - just let me know what
NOTE: packages without go file in package main cannot be installed via go install. This system looks pretty confusing to me - as modules cannot be found...
finally i created a multimodule
means i have a root folder (lets call it root) which contains a go.mod file
this defines the root module
it requires all my submodules
and it replaces the absolute path into a relative.
root
* aa
* ab
* bc
main gets defined as
module root
go 1.15
require (
root/aa v0.0.0
root/ab v0.0.0
root/bc v0.0.0
)
replace (
root/aa => ./aa
root/ab => ./ab
root/bc => ./bc
)
all the submodules have a very simple definition (this is equal for all, as long as you do not have any dependencies)
module root/aa
When running glide install on my project, I get the following error:
[ERROR] Error scanning github.com/golang/protobuf/proto/testdata: cannot find package "." in:
/Users/bevernie/.glide/cache/src/https-github.com-golang-protobuf/proto/testdata
[ERROR] Failed to retrieve a list of dependencies: Error resolving imports
When checking protobuf's source code, I can in fact see that there is no such package. I however don't directly use protobuf, so the error must come from one of the dependencies I use.
When running glide tree on my project, there is only one instance of github.com/golang/protobuf/proto/testdata:
|-- github.com/golang/protobuf/proto (/Users/bevernie/Programmation/work/src/github.com/golang/protobuf/proto)
| |-- github.com/golang/protobuf/proto/test_proto (/Users/bevernie/Programmation/work/src/github.com/golang/protobuf/proto/test_proto)
| | |-- (Recursion) github.com/golang/protobuf/proto (/Users/bevernie/Programmation/work/src/github.com/golang/protobuf/proto)
| |-- github.com/golang/protobuf/ptypes/any (/Users/bevernie/Programmation/work/src/github.com/golang/protobuf/ptypes/any)
| | |-- (Recursion) github.com/golang/protobuf/proto (/Users/bevernie/Programmation/work/src/github.com/golang/protobuf/proto)
github.com/golang/protobuf/proto/testdata (glide get github.com/golang/protobuf/proto/testdata)
|-- github.com/golang/protobuf/ptypes/any (/Users/bevernie/Programmation/work/src/github.com/golang/protobuf/ptypes/any)
| |-- github.com/golang/protobuf/proto (/Users/bevernie/Programmation/work/src/github.com/golang/protobuf/proto)
| | |-- github.com/golang/protobuf/proto/test_proto (/Users/bevernie/Programmation/work/src/github.com/golang/protobuf/proto/test_proto)
| | | |-- (Recursion) github.com/golang/protobuf/proto (/Users/bevernie/Programmation/work/src/github.com/golang/protobuf/proto)
| | |-- (Recursion) github.com/golang/protobuf/ptypes/any (/Users/bevernie/Programmation/work/src/github.com/golang/protobuf/ptypes/any)
which doesn't really help me pinpoint the source of the problem.
Do you have any suggestions as of how to fix the issues?
My project was compiling just fine until a week or two ago (I use Docker to deploy in production, so the glide install was run every time and never failed before that, and I haven't added any new dependency recently).
Before your own PR (995), there was glide issue 968
It looks like it's caused by a repository's structure changing, i.e. a sub-package being moved, or removed entirely.
Workarounds proposed by Elliot Wright (seeruk):
If the package that has been updated is under your own control, then I've since found it easier to use some of the newer Go features like type aliases to ease the pain from refactoring.
So, instead of just moving a package, move it and then make aliases to the new location in the old one so that your older code still works.
Then, gradually move things over. Basically just mark things as deprecated but make sure they're still usable for a little while until you've ported new code over.
If the package is not in your control, then you can always clone the version you want manually to your vendor folder and make your updates in your code.
Once you're done, Glide should let you update again.
If it's much more complex, sometimes it's even easier to revert to using go get until you're done updating packages, and rely on your $GOPATH contents.
It's far from ideal, but there are ways you can work around it at least.
In the mean time, I've also made an issue about this on dep.
I think they're looking into a way of disabling this kind of check if you just want the tool to trust you as the developer.
So you can check if you have the same issue using godep, or even the bleeding-edge vgo.
I don't know if I totally got the concept wrong, but I want to create several projects with dependencies to other projects which are not part of the directory structure of a parent project. I know that the normal way of doing this would be to use an external dependency which fetches from some external repository. But in this case, where let's say in project called 'F' a framework is developed, which is used in project 'P'., then P uses F, but F should IMO not necessarily be a sub-project of P as P is only used to test-drive the development of F (but it's not only a unit test). Later in the process, when F is stable, F is separated and can be consumed by other projects via a repository. But during development of F with P as it's test case, it would be nice if that round-trip through the repository could be omitted.
To make matters worse, for the initial development there is more than one test-driving consumer project, which all need to have a dependency to F, but not via an external repository.
My idea is to develop F in some place on the disk with it's own git reposity. The other P like projects reside somewhere else on the disk and have a local file system based dependency to F. Would such a construct be possible in Gradle? If so, where do I start? I scanned the Java examples but couldn't find an appropriate example.
Any ideas?
The Gradle project hierarchy is fully virtual. It just has the default that the physical location corresponds to the virtual hierarchy. But you have complete control over this. See: http://gradle.org/0.9-rc-1/docs/userguide/build_lifecycle.html#sec:settings_file
Regarding your other ideas have a look at the following Jira: http://jira.codehaus.org/browse/GRADLE-1014
You could consider a folder hierarchy like this one:
Main folder
|- F folder
| |- .git
| |- sources
| |- build.gradle (with parts specific to F)
|- P folder
| |- sources
| |- build.gradle (with part specific to P)
|- build.gradle (with common parts)
|- settings.gradle
So you can always decide to run gradle on either the F project, the P project or the two alltoegether. It will also allows you to promote you F project alone without the P or any other side projects.
For more up-to-date information, check the Multi Project Builds chapter of the Gradle documentation.