Say you are using Go 1.13 and set up a project initialized for Go Modules.
$ mkdir my-project
$ cd my-project
$ git mod init github.com/bmuschko/my-project
Instead of using the default Google proxy to download dependencies, you set GoCenter or a different proxy.
$ export GOPROXY=https://gocenter.io
Once you download dependencies, it doesn't seem that Go keeps track of the originating proxy. In the end, there's no way to know or verify that a dependency came from the Google proxy, a custom proxy or directly from the source code repository. In theory, the checksums could be different depending on the originating proxy if you switch between them even if you pull the same version.
$ go get github.com/spf13/cobra
Does Go store this information somewhere in the cache? I couldn't find this information. Any advice would be appreciated.
The originating proxy should not matter and is not recorded: if you have downloaded the module from anywhere, then the bytes in your module cache should match the checksum found in either your go.sum file or the global checksum database.
(The go command fetches checksums for any new module dependencies from the database before downloading the module or adding the checksum to your go.sum file.)
Related
Currently I am working on an API which uses Serverless Framework with Go.
I'm using the Serverless-offline plugin for local testing.
This API depends on a few other repositories (which I also maintain), which I import using the go.mod file.
However I am having a hard time refining my developer workflow.
Currently, if I want to make changes in a repository which this API depends upon, I have to alter the projects go.mod to include replace directives for the purpose of testing, but then I'm having to manually change it back for deployment to production.
Basically I'm looking for a way to include replace directives, which only get applied during local development. How has everyone else dealt with this problem?
Bonus question: Is there any way to run Serverless offline in docker? I'm finding that serverless-offline running on the bare metal is causing inconsistencies between different developers environments.
You can run go commands with an alternate go.mod file with the -modfile option:
From Build commands:
The -modfile=file.mod flag instructs the go command to read (and
possibly write) an alternate file instead of go.mod in the module root
directory. The file’s name must end with .mod. A file named go.mod
must still be present in order to determine the module root directory,
but it is not accessed. When -modfile is specified, an alternate
go.sum file is also used: its path is derived from the -modfile flag
by trimming the .mod extension and appending .sum.
Create a local.go.mod file with the necessary replace directive for development and build, for example, with:
go build -modfile=local.go.mod ./...
I work in a closed environment in which machines do not have access to internet. I'm wondering if there is a way for me to load the dependency modules for development with Go. One way I am thinking is to get the package files, if there is any. However, I couldn't find any such file for the dependencies I am using.
There are several things you could do:
You can use replace directives in your go.mod file to redirect packages to local directories where you have copies of the dependency packages.
You can use go mod vendor (see docs) to "vendor" dependency packages locally.
You can use a local proxy and set the GOPROXY env var.
I'm using Artifactory 7.10.6.
go version 1.15.6 (also tested with older versions)
I am not using the jfrog cli, and would prefer not to.
I'm trying to sort out what I'm doing wrong here. I've used Artifactory to pull down content from remote connections to be stored on an local repository for other package types, but this doesn't seem to be working for me fully with GO. Disclaimer, I'm not super versed in GO...
Here is what I have setup.
a local go repo called "go-ext-release"
a remote of gocenter called "go-gocenter"
a virtual called "go-virtual" that contains only "go-ext-release"
a virtual called "go-virtual-dev" that contains "go-virtual" followed by "go-gocenter"
The idea here of course. Run a build with my GOPROXY set to "go-virtual-dev", copy the downloaded files from go-gocenter-cache to "go-ext-release". That should get me all the files I need to reset my environment, point to GOPROXY to "go-virtual" and run a build.
My build pointing to "go-virtual-dev" works fine. Build works, content is pulled down (mostly .mod and .info).
I move that content to the local (go-ext-release) and build in a clean environment using "go-virtual" and the build fails. it says it can't access .zip files. i.e. a 404 on /github.com/gorilla/mux/#v/v1.7.4.zip
Of course when I look for that zip, it doesn't exist.
If I take the url its trying to access and change the url from the "go-virtual" path to "go-virutal-dev" and punch it into a web browser the correct zip file gets downloaded to the "go-gocenter-cache" repo (as expected).
I did this process for the 4 or 5 zip files the build needed (its a small test build), and then moved the zips from the cached location over to the "go-ext-release" repo. After that, the build works using the "go-virtual" repository (i.e. the repo that just sees into our local repo).
So what am I doing wrong here? My expectation was that the initial build would have pulled all the files , zips included, to the cache as well. I know the build pulled them down because I can see them in my GOCACHE folder. Its as though it isn't using my GOPROXY to pull the zips down
Any help would be appreciated.
is there any commanline switch to force go to show me the exact URL it is using for pulls? I've experimented with using go get -v, but it doesn't give the full url.
Can you try running the build against go-virtual-dev using an empty GOPATH. I believe the Go client will not trigger the module zip download if you already have it locally which will not allow Artifactory to cache it from the remote repo.
BTW, Running go get -x should show you all the URLs being fetched.
I'm trying to deploy a Google Cloud Function written in Go.
By doing some research I found out that vendor files are prefered over go.mod so I'm vendoring everything I use (which includes some local dependencies) and ignoring the go.mod/sum files in the .gcloudignore file.
The problem is that after trying to deploy, I get the following error:
go: nimbus#v0.0.0-00010101000000-000000000000: parsing /nimbus/go.mod: open /nimbus/go.mod: no such file or directory; Error ID: 03a1e2f7
nimbus is my local dependency and it has the following structure:
My Function repository has the following structrure:
and my go.mod file is:
module my_function
go 1.13
require nimbus v0.0.0-00010101000000-000000000000
replace nimbus => ../../../nimbus
I've tried this solution https://stackoverflow.com/questions/5441096 already. But it did not fix my issue.
I've tried everything to solve this issue, but nothing seems to work.
If you have a go.mod file and a vendor directory, the vendor directory will be ignored when you deploy your function.
https://cloud.google.com/functions/docs/writing/specifying-dependencies-go
I have used modules when deploying GCP functions in Go. Haven't had any problems. But I can't speak to the preference of using vendor/ instead. It should work, just without the go.mod file.
Turns out the problem was very complicated and I hope Google finds a solution for it asp.
By deploying my function using Cloud Build, It would read from my repository on Google Source, but, by reading from there it would bypass the .gcloudignore file and deploy both the go.mod/sum files and the vendor directory with my local code.
As said in https://stackoverflow.com/a/62050872/10316247:
If you have a go.mod file and a vendor directory, the vendor directory will be ignored when you deploy your function.
So the error would occur because of my go.mod not being able to find local repository.
My solution was to rename my go.mod/sum files so it would not be considered:
When you use golang 1.16 and specify golang 1.16 in the go.mod folder it will instead default to using the vendor files with the --mod=vendor flag set, which will solve this issue.
You'll just need to ensure your module name is formatted correctly (something like example.com/module).
My Problem
Elastic Beats is an open source project for log shippers written in Go. It features several log outputs, including console, Elasticsearch and Redis. I would like to add an output of my own - to AWS Kinesis.
I have cloned the repo to ~/github/beats, and tried building it:
$ cd filebeat; go build main.go
However, it failed due to a missing library which is a part of the project:
main.go:6:2: cannot find package "github.com/elastic/beats/filebeat/cmd" in any of:
/usr/local/go/src/github.com/elastic/beats/filebeat/cmd (from $GOROOT)
/Users/adam/go/src/github.com/elastic/beats/filebeat/cmd (from $GOPATH)
A directory of the project is dependent on a package from the same repo, but instead of looking one directory up the hierarchy it looks in the GOPATH.
So, go get github.com/elastic/beats/filebeat/cmd fetched the code, and now go build main.go works. Changing the code in my GOPATH is reflected in these builds.
This leaves me with an structural inconvenience. Some of my code is at a working directory, and some of it is at my GOPATH and included by the code in my working directory.
I would like to have all my code in a single directory for various reasons, not the least being keeping everything under version control.
What Have I Tried
Mostly searching for the problem. I am quite new to Go, so I might have missed the correct terminology.
My Question
What is the right way to edit the code of an imported library in Go?
One of the recommended ways to work with other's packages is:
Get the sources of the original package:
go get github.com/elastic/beats
As a result you will clone project's git repository to the folder
$GOPATH/src/github.com/elastic/beats
Make some fixes, compile code, fix, compile... When you make go install package will be compiled and installed to your system. When you need merge updates from original repository you can git pull them.
Everything is OK. What's next? How to share your work with others?
Fork project on github, suppose it will be github.com/username/beats
Add this fork as another remote mycopy (or any other name you like) to your local repository
git remote add mycopy git://github.com/username/beats.git
When all is done you can push updated sources to your repo on github
git push mycopy
and then open a pull-request to original sources. This way you can share your work with others. And keep your changes in sync with mainstream.
Previous answers to this question are obsolete when developing projects that using Go Modules.
For projects that using Go Modules, one may use the following command to replace an imported library(eg. example.com/imported/module) with a local module(eg. ./local/module):
go mod edit -replace=example.com/imported/module=./local/module
Or by adding the following line into the go.mod file:
replace example.com/imported/module => ./local/module
Reference Docs: https://golang.org/doc/modules/managing-dependencies#unpublished
A project working copy should be checked out into $GOPATH/src/package/import/path - for example, this project should be checked out into /Users/adam/go/src/github.com/elastic/beats. With the project in the correct location, the go tooling will be able to operate on it normally; otherwise, it will not be able to resolve imports correctly. See go help gopath for more info.