How to get the path to a Go module dependency? - go

I have two Go modules, let's name them example.com/a and example.com/b.
Let this be example.com/a's go.mod:
module example.com/a
go 1.12
require (
example.com/b v0.4.2
)
In example.com/b's root directory, there is a file named data.yaml. example.com/a needs to autogenerate some code as part of its build process. This autogeneration needs to read data.yaml.
How can I in the directory of example.com/a query for the path of example.com/b to read that file? I know that after downloading, the module will be somewhere in (go env GOPATH)/pkg/mod but I don't know how the path will be constructed from there as it contains some ! characters that are not part of the import path. I hoped that there is some subcommand of go mod or go list that will output the path, but I haven't found it in the documentation.
I have thought about including data.yaml in Go code via go-bindata (yes I'm aware of //go:embed but I don't want to require Go 1.16 for now) but then I would only have access at run-time when I need it at compile-time.

You can use go list with the -m flag and the -f flag like so:
go list -m -f '{{.Dir}}' example.com/b
The -m flag:
causes go list to list modules instead of packages. In this mode, the
arguments to go list may be modules, module patterns (containing the
... wildcard), version queries, or the special pattern all, which
matches all modules in the build list. If no arguments are specified,
the main module is listed.
(reference)
The -f flag:
specifies an alternate format for the output, using the syntax of
package template. The struct being passed to the template, when using
the -m flag, is:
type Module struct {
Path string // module path
Version string // module version
Versions []string // available module versions (with -versions)
Replace *Module // replaced by this module
Time *time.Time // time version was created
Update *Module // available update, if any (with -u)
Main bool // is this the main module?
Indirect bool // is this module only an indirect dependency of main module?
Dir string // directory holding files for this module, if any
GoMod string // path to go.mod file for this module, if any
GoVersion string // go version used in module
Error *ModuleError // error loading module }
type ModuleError struct {
Err string // the error itself
}
[the above quote was altered for context]
(reference)

You can figure out the module path like this:
package main
import (
"fmt"
"os"
"path"
"golang.org/x/mod/module"
)
func GetModulePath(name, version string) (string, error) {
// first we need GOMODCACHE
cache, ok := os.LookupEnv("GOMODCACHE")
if !ok {
cache = path.Join(os.Getenv("GOPATH"), "pkg", "mod")
}
// then we need to escape path
escapedPath, err := module.EscapePath(name)
if err != nil {
return "", err
}
// version also
escapedVersion, err := module.EscapeVersion(version)
if err != nil {
return "", err
}
return path.Join(cache, escapedPath+"#"+escapedVersion), nil
}
func main() {
var path, err = GetModulePath("github.com/jakubDoka/mlok", "v0.4.7")
if err != nil {
panic(err)
}
if _, err := os.Stat(path); os.IsNotExist(err) {
fmt.Println("you don't have this module/version installed")
}
fmt.Println("module found in", path)
}

Related

How to check if a public go function/struct is not used outside of the package?

Is there a way to check if a public function/struct is used outside of the package in which it's declared? I'm not writing a public go module that's consumed anywhere else, and simply want to scan whether func Foo() it's used anywhere in my codebase outside of the package in which it's declared.
I'm using GoLand but any programmatic solution would do.
Simplest solution: manually rename Foo() to Foo2(). Build/compile your project: if there are no compilation errors, it's not referenced in your code. Same check also works with any identifiers and with any IDEs (this doesn't use any of the IDE's features).
Obviously if you already have a Foo2 identifier, this will fail. But the idea is to rename it to a non-existing identifier...
You can scan a particular package to see all the available function in it.
In this main.go, app the root package name and there is another package in database directory under the package name database.
By running the code you will found all the function name available inside database package
package main
import (
"fmt"
"app/database"
"go/ast"
"go/parser"
"go/token"
"os"
)
// Name of the package you want to scan
const subPackage = "database"
func main() {
set := token.NewFileSet()
packs, err := parser.ParseDir(set, subPackage, nil, 0)
if err != nil {
fmt.Println("Failed to parse package:", err)
os.Exit(1)
}
funcs := []*ast.FuncDecl{}
for _, pack := range packs {
for _, f := range pack.Files {
for _, d := range f.Decls {
if fn, isFn := d.(*ast.FuncDecl); isFn {
funcs = append(funcs, fn)
}
}
}
}
fmt.Println("All the functions in the package:",subPackage)
for _, fn := range funcs {
fmt.Println(fn.Name.Name)
}
// database Package is called/used
database.Connection()
}
This will get all function declarations in the stated subpackage as an ast.FuncDecl. This isn't an invokable function; it's just a representation of its source code of it.
If you wanted to do anything like call these functions, you'd have to do something more sophisticated. After gathering these functions, you could gather them and output a separate file that calls each of them, then run the resulting file.

Get Name of Current Module in Go

I am attempting to create named loggers automatically for HTTP handlers that I'm writing, where I am passed a function (pointer).
I'm using the code mentioned in this question to get the name of a function:
package utils
import (
"reflect"
"runtime"
)
func GetFunctionName(fn interface{}) string {
value := reflect.ValueOf(fn)
ptr := value.Pointer()
ffp := runtime.FuncForPC(ptr)
return ffp.Name()
}
I'm using this in my main function to try it out like so:
package main
import (
"github.com/naftulikay/golang-webapp/experiments/functionname/long"
"github.com/naftulikay/golang-webapp/experiments/functionname/long/nested/path"
"github.com/naftulikay/golang-webapp/experiments/functionname/utils"
"log"
)
type Empty struct{}
func main() {
a := long.HandlerA
b := path.HandlerB
c := path.HandlerC
log.Printf("long.HandlerA: %s", utils.GetFunctionName(a))
log.Printf("long.nested.path.HandlerB: %s", utils.GetFunctionName(b))
log.Printf("long.nested.path.HandlerC: %s", utils.GetFunctionName(c))
}
I see output like this:
github.com/naftulikay/golang-webapp/experiments/functionname/long.HandlerA
This is okay but I'd like an output such as long.HandlerA, long.nested.path.HandlerB, etc.
If I could get the Go module name (github.com/naftulikay/golang-webapp/experiments/functionname), I can then use strings.Replace to remove the module name to arrive at long/nested/path.HandlerB, then strings.Replace to replace / with . to finally get to my desired value, which is long.nested.path.HandlerB.
The first question is: can I do better than runtime.FuncForPC(reflect.ValueOf(fn).Pointer()) for getting the qualified path to a function?
If the answer is no, is there a way to get the current Go module name using runtime or reflect so that I can transform the output of runtime.FuncForPC into what I need?
Once again, I'm getting values like:
github.com/naftulikay/golang-webapp/experiments/functionname/long.HandlerA
github.com/naftulikay/golang-webapp/experiments/functionname/long/nested/path.HandlerB
github.com/naftulikay/golang-webapp/experiments/functionname/long/nested/path.HandlerC
And I'd like to get values like:
long.HandlerA
long.nested.path.HandlerB
long.nested.path.HandlerC
EDIT: It appears that Go does not have a runtime representation of modules, and that's okay, if I can do it at compile time that would be fine too. I've seen the codegen documentation and I'm having a hard time figuring out how to write my own custom codegen that can be used from go generate.
The module info is included in the executable binary, and can be acquired using the debug.ReadBuildInfo() function (the only requirement is that the executable must be built using module support, but this is the default in the current version, and likely the only in future versions).
BuildInfo.Path is the current module's path.
Let's say you have the following go.mod file:
module example.com/foo
Example reading the build info:
bi, ok := debug.ReadBuildInfo()
if !ok {
log.Printf("Failed to read build info")
return
}
fmt.Println(bi.Main.Path)
// or
fmt.Println(bi.Path)
This will output (try it on the Go Playground):
example.com/foo
example.com/foo
See related: Golang - How to display modules version from inside of code
If your goal is to just have the name of the module available in your program, and if you are okay with setting this value at link time, then you may use the -ldflags build option.
You can get the name of the module with go list -m from within the module directory.
You can place everything in a Makefile or in a shell script:
MOD_NAME=$(go list -m)
go build -ldflags="-X 'main.MODNAME=$MOD_NAME'" -o main ./...
With main.go looking like:
package main
import "fmt"
var MODNAME string
func main() {
fmt.Println(MODNAME) // example.com
}
With the mentioned "golang.org/x/mod/modfile" package, an example might look like:
package main
import (
"fmt"
"golang.org/x/mod/modfile"
_ "embed"
)
//go:embed go.mod
var gomod []byte
func main() {
f, err := modfile.Parse("go.mod", gomod, nil)
if err != nil {
panic(err)
}
fmt.Println(f.Module.Mod.Path) // example.com
}
However embedding the entire go.mod file in your use case seems overkill. Of course you could also open the file at runtime, but that means you have to deploy go.mod along with your executable. Setting the module name with -ldflags is more straightforward IMO.

How to list all non-standard/custom packages in Go?

As mentioned here one can get all the standard Go packages using https://godoc.org/golang.org/x/tools/go/packages 's Load() function in which one can give "pattern" as input.
pkgs, err := packages.Load(nil, pattern)
For example, if pattern = "std" then it returns all the standard packages.
But, if I want to get a list of custom/user-defined packages having custom patterns such as only the vendor folders of the form github.com/X/Y/vendor/... then how exactly I can specify the pattern?
I have tried using /vendor/, github.com/X/Y/vendor/ and some other combinations as pattern in the Load() function. None of them have worked.
You can use the ... syntax in pattern field of the Load() function.
Example
My Go module requires github.com/hashicorp/go-multierror package :
module mymodule
require github.com/hashicorp/go-multierror v1.0.0
So, the following code :
package main
import (
"fmt"
"golang.org/x/tools/go/packages"
)
func main() {
pkgs, err := packages.Load(nil, "github.com/hashicorp...")
if err == nil {
for _, pkg := range pkgs {
fmt.Println(pkg.ID)
}
}
}
returns all the required packages starting with github.com/hashicorp (even transitive ones) :
github.com/hashicorp/errwrap
github.com/hashicorp/go-multierror
Note that you can also use ... anywhere in your pattern (...hashicorp..., ...ha...corp..., github.com/...).

API to get the module name

Is there an API to get the module name of a project which uses go 1.11 module system?
so I need to get abc.com/a/m from the module definition module abc.com/a/m in go.mod file.
As of this writing, I am not aware of any exposed APIs for that. However, looking at go mod sources, there is a function that can be quite useful in Go mod source file
// ModulePath returns the module path from the gomod file text.
// If it cannot find a module path, it returns an empty string.
// It is tolerant of unrelated problems in the go.mod file.
func ModulePath(mod []byte) string {
//...
}
func main() {
src := `
module github.com/you/hello
require rsc.io/quote v1.5.2
`
mod := ModulePath([]byte(src))
fmt.Println(mod)
}
Which outputs github.com/you/hello
Try this?
package main
import (
"fmt"
"io/ioutil"
"os"
modfile "golang.org/x/mod/modfile"
)
const (
RED = "\033[91m"
RESET = "\033[0m"
)
func main() {
modName := GetModuleName()
fmt.Fprintf(os.Stdout, "modName=%+v\n", modName)
}
func exitf(beforeExitFunc func(), code int, format string, args ...interface{}) {
beforeExitFunc()
fmt.Fprintf(os.Stderr, RED+format+RESET, args...)
os.Exit(code)
}
func GetModuleName() string {
goModBytes, err := ioutil.ReadFile("go.mod")
if err != nil {
exitf(func() {}, 1, "%+v\n", err)
}
modName := modfile.ModulePath(goModBytes)
fmt.Fprintf(os.Stdout, "modName=%+v\n", modName)
return modName
}
If your starting point is a go.mod file and you are asking how to parse it, I would suggest starting with go mod edit -json, which outputs a particular go.mod file in JSON format. Here is the documentation:
https://golang.org/cmd/go/#hdr-Edit_go_mod_from_tools_or_scripts
Alternatively, you could use rogpeppe/go-internal/modfile, which is a go package that can parse a go.mod file, and which is used by rogpeppe/gohack and some other tools from the broader community.
Issue #28101 I think tracks adding a new API to the Go standard library to parse go.mod files.
Here is a snippet of the documentation for go mod edit -json:
The -json flag prints the final go.mod file in JSON format instead of
writing it back to go.mod. The JSON output corresponds to these Go
types:
type Module struct {
Path string
Version string
}
type GoMod struct {
Module Module
Go string
Require []Require
Exclude []Module
Replace []Replace
}
type Require struct {
Path string
Version string
Indirect bool
}
Here is an example snippet of JSON output from go mod edit -json that shows the actual module path (aka module name), which was your original question:
{
"Module": {
"Path": "rsc.io/quote"
},
In this case, the module name is rsc.io/quote.
As of Go 1.12 (for those finding this via a search who are using modules but not necessarily the older version the OP mentioned), the runtime/debug package includes functionality for getting information about the build, including the module name. For example:
import (
"fmt"
"runtime/debug"
)
func main() {
info, _ := debug.ReadBuildInfo()
fmt.Printf("info: %+v", info.Main.Path)
}
You can run this example on the playground: https://go.dev/play/p/5oGbCRxSnjM
For more information, see the documentation for "runtime/debug".BuildInfo

Why does proto marshal then unmarshal fail when it is run outside of project containing vendor directory?

I have a main.go file that uses the proto files in pkg/models to Marshal and Unmarshal a proto struct like this:
// Convert to string
protoStr := proto.MarshalTextString(proto)
// Unmarshal string back to proto struct
var proto2 models.Stuff
err := proto.UnmarshalText(protoStr, &proto2)
The setup is here: https://github.com/chuyval/qqs/tree/master/q2
The project contains a vendor directory that only has the github.com/golang/protobuf repo checked out. (run glide install to create vendor if it doesnt exist)
The main.go program works fine when running go run main.go from inside the project.
When I move the main.go file one level up to the parent directory, and run the same command go run main.go at the parent level, it reports the following error:
line 2: unknown field name "value_list" in models.Stuff
When I delete the vendor directory in the project directory, and run go run main.go at the parent level, I get no error.
Why would having a vendor directory in the project repository make it error out?
Another thing to note, if I run the same main.go application inside of the dependent repo, it works every time (with or without the vendor repository).
Sample code:
package main
import (
"github.com/chuyval/qqs/q2/pkg/models"
"github.com/golang/protobuf/proto"
"log"
)
func main () {
stuff := createProtoStuff()
log.Printf("Stuff: %+v", stuff)
// Convert to string
stuffStr := proto.MarshalTextString(stuff)
// Unmarshal string back to proto struct
var stuff2 models.Stuff
err := proto.UnmarshalText(stuffStr, &stuff2)
if err != nil {
log.Printf("It didnt work. Error: %s", err.Error())
} else {
log.Printf("It worked. Proto: %+v", stuff2)
}
}
func createProtoStuff() *models.Stuff {
someValueList := []*models.SomeValue{&models.SomeValue{Id: "Some Value ID"}}
valueList := &models.SomeValueList{SomeValue: someValueList}
stuffValueList := &models.Stuff_ValueList{
ValueList: valueList,
}
stuff := &models.Stuff{
Id: "Stuff List Id",
Stuff: stuffValueList,
}
return stuff
}
Software Versions
glide version 0.13.1
go version go1.10.3 darwin/amd64
protoc version libprotoc 3.6.0

Resources