I have a project to do at my job and we're using Bitbucket. So we have all our repos like this :
bitbucket.org/company/project Nothing new here.
I have created a repository called go-tools, his module name is bitbucket.org/company/go-tools and his path his bitbucket.org/company/go-tools
Following this medium post I could achieve a go mod tidy
package whatever
import (
"bitbucket.org/company/go-tools"
"bitbucket.org/company/go-tools/env"
// and so on ...
)
The problem occurs when I try to replace "bitbucket.org/company" by "company.com" because we would like to have our company name instead bitbucket.
So my module name become company.com/go-tools instead of bitbucket.org/company/go-tools
And my imports become :
package whatever
import (
"company.com/go-tools"
"company.com/go-tools/env"
// and so on ...
)
I have set my GOPRIVATE to use bitbucket and configured git to use bitbucket instead of company.com
git config --global url."https://{username}:{app password}#bitbucket.com/company".insteadOf "https://company.com"
go env -w GOPRIVATE=bitbucket.org/company
And from there I only get a 404 error telling me that my package can't be found.
Did anyone have an idea why ? Am I misunderstanding something ?
NOTE : I also read this
Thanks #adrian for your reply this answer my question for at least a part.
I was more looking for a way of just go get 'company.com/whatever' but this is ok.
So if I understand correctly I need to go get bitbucket.org/company/whatever first and then go mod edit -replace bitbucket.org/company/whatever=company.com/whatever
Thanks
Related
The steps I took:
1- Configure .gitconfig
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
2- export GOPRIVATE=bitbucket.org/myproject/helpers-go
3- go get
my go.mod file ->
module bitbucket.org/myproject/x/test-service
go 1.16
require (
bitbucket.org/myproject/helpers-go v0.0.2
)
replace bitbucket.org/myproject/helpers-go => bitbucket.org/myproject/helpers-go.git v0.0.2
My ssh key works. I can push/pull each services.
I also have the v0.0.2 tag on my last helpers-go commit.
I am trying to import private repo(helpers-go) into (test-service) and the error I get is :
go: bitbucket.org/myproject/helpers-go#v0.0.2: invalid version control suffix in bitbucket.org/ path
Even if I change the v0.0.2 to something random like v0.0.9 which I don`t have a tag like this, I still get the same error.
Appreciate the help...
it is solved:
1- I was using go 1.16 and there was an issue with bitbucket private repos not being supported for this version. I don`t think it has the same issue with gitlab. Upgrading it to go 1.19
2- Small change here .org/ -> .org:
[url "git#bitbucket.org:"]
insteadOf = https://bitbucket.org/
3- Also making sure the private repo I am importing has a file that ends with .go. My .go files were all in other folders leaving main folder without a .go file.
My module is gitlab.com/getsote/utilities/slogger
My repository is gitlab.com/getsote/utilities/slogger.git
When I run go get gitlab.com/getsote/utilities/slogger, I get the message below.
Scotts-Mac-mini:seeding syacko$ go get gitlab.com/getsote/utilities/slogger
go get gitlab.com/getsote/utilities/slogger: module gitlab.com/getsote/utilities/slogger: git ls-remote -q origin in /Users/syacko/workspace/sotesoft/golang/pkg/mod/cache/vcs/80b3644beae1b986f1c659355360479e2463820660aa328d2edb1e571aba259b: exit status 128:
remote: The project you were looking for could not be found.
fatal: repository 'https://gitlab.com/getsote/utilities.git/' not found
Scotts-Mac-mini:seeding syacko$
The gitlab.com/getsote/utilities.git is a sub-directory and not a repository. I don't understand why go get is going to the utilities as a repository?
==========================
PREVIOUS Updates
Directory Structure:
GOPATH/src/slogger
|----go.mod
|----slogger.go
|----slogger_test.go
go.mod file
module slogger or gitlab.com/getsote/utilities/slogger -> still gets the error below
go 1.14
gitlab.com/getsote/utilities contains repository slogger.git
I have run a test to see if the issue is the number of nodes in the path. So, I create a new repository with no sub-directory and pushed the slogger code. Then ran go get gitlab.com/getsote/slogger which generate a different error message.
GOPATH/gitlab.com/getsote/test-go-mod -> create new directory and added slogger files listed above
gitblab.com/getsote/test-go-mod -> new repository with one less level
Scotts-Mac-mini:test-go-mod syacko$ go get gitlab.com/getsote/test-go-mod
go: downloading gitlab.com/getsote/test-go-mod v0.0.0-20200409023538-794310bf7cf9
go get gitlab.com/getsote/test-go-mod: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: verifying module: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: reading https://sum.golang.org/lookup/gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: 410 Gone
server response:
not found: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/7753c92c9bd1419156d8120684b7f3707fd207e01a2947ba89e2acfd2ecfb4d0: exit status 128:
fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled
Scotts-Mac-mini:test-go-mod syacko$
This is still getting the status error of 128 for the missing version. Additionally, it is looking in the right location for the code. If this is true, then I just need help with the version missing. Moving to a shorted directory structure is doable.
========================
Newest Update
#praveent > The solution at https://medium.com/cloud-native-the-gathering/go-modules-with-private-git-repositories-dfe795068db4 didn't work for me. So I started from scratch to see how to resolve the issue.
The reason is because for a git repository it assumes that utilities is the repo and not utilities/slogger
There is a way to override this behavior by implementing go get API. But, gitlab is yet to implement the same due to security concerns. You can read more here. Gitlab issue
Update: Add reference to gitlab issue tracking this problem.
So, here is how I got this to work using gitlab.com. I'm not saying other ways will not work, they just didn't for me and my setup. First, since I don't care if the code is available to the public, I created a new group at gitlab.com. This new group is public from the start, so no need to adjust permissions. Then I create a repository called packages and cloned the repository to my local machine with the same directory structure that is in gitlab.com, gitlab.com/soteapps/packages with ~/workspace/soteapps/packages on my machine. Both of these are out side the GOPATH. I'm not sure this matters, but it is working this way, so I'm putting it here.
Under packages, I copied the slogger directory and code.
cp -R slogger ~/workspace/soteapps/packages/.
Edited the go.mod file to match the repository structure, which is in the packages directory. There is no go.mod file in the slogger directory.
module gitlab.com/soteapps/packages
go 1.14
Edited the hello.go import to match the package.
package main
import (
"fmt"
"rsc.io/quote"
"gitlab.com/soteapps/packages/slogger"
)
func main() {
fmt.Println(quote.Hello())
slogger.Info("Test message")
}
Built the program using go build -o hello and then ran it hello with the following results:
Scotts-Mac-mini:hello syacko$ hello
Hello, world.
INFO:2020/04/10 21:11:33 Test message
Scotts-Mac-mini:hello syacko$
Worked! Thank you all that helped. This wouldn't of gotten solved without your help.
Note: This only works for public repositories.
It's a go module question.
My PC is running with go 1.13 and go module mode is on.
This my go env:
GO111MODULE="on"
GOPRIVATE=""
GOPROXY="https://goproxy.io,direct"
At first I wrote a public package for testing and uploaded it to github. And then I "go get" this package for independence. After that I made it private and even deleted it from github (I tried "git clone" and couldn't download it again). But I could "go get" it after I removed it from github. Soon I realized it that maybe this package was cached in goproxy.io or other proxy databases. So I set GOPROXY="direct" and "go get" this package again and failed. This action proved my guess.
Now it's my question:
How to remove this whole package from goproxy.io or other database.
Is there a safe way to use go module, I don't want to upload my private code to other databases by mistake.
I tried to STFW and found nothing. Thank U to all people for reading and answering this question.
If you accidentally published a package / module you intend to be private, then go.dev: About page:
Removing a package
If you would like a package removed, please send an email to go-discovery-feedback#google.com, with the import path or module path that you want to remove.
But as Adrian mentioned in the comments, there is no guarantee that no one downloaded your published packages.
If you want some packages / modules to remain private, you may enumerate them (using glob patterns) in the GOPRIVATE, GONOPROXY, GONOSUMDB environment variables, which are respected by the go tool.
Command go: Environment variables:
GOPRIVATE, GONOPROXY, GONOSUMDB
Comma-separated list of glob patterns (in the syntax of Go's path.Match)
of module path prefixes that should always be fetched directly
or that should not be compared against the checksum database.
See 'go help module-private'.
From command go help module-private:
The GOPRIVATE environment variable controls which modules the go command
considers to be private (not available publicly) and should therefore not use the
proxy or checksum database. The variable is a comma-separated list of
glob patterns (in the syntax of Go's path.Match) of module path prefixes.
These vars can also be set using the go env -w command. You may get more help about it using go help env.
For Question2, I have tried several times today.
If u want to build a private package safely, the most important thing you should do is setting the GOPRIVATE BEFORE u build your package. And GONOPROXY,GONOSUMDB will be automatically set to same as GOPRIVATE.
Now u can write the code and git push to a private remote repo and try to go get it, and u will fail and see some error message like it:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.
The link show us that we should use https link with your username/password or just replace your https link with ssh link if u upload a key to remote ropo. After doing it, u will succeed to go get it from your source address (your package name) but NOT PROXY.
Futhurmore, let's delete all go module cache by using go clean -modcache and set GOPRIVATE="". I will test whether our private package was uploaded to proxy.
case 1:
Set GOPROXY="https://goproxy.io" or other proxy and thenen go get the private package.
u will face a 404 Not Found which show u that u find NOTHING from proxy (it meant nothing was cached/stored in proxy) .
case 2:
Set GOPROXY="direct"(use the direct address of your pacakage name) and then go get the private package.
u will face error message like it:
verifying {PACKAGENAME}: {PACKAGENAME}: initializing sumweb.Conn: checking tree#{ID}: Get https://sum.golang.org/tile/8/2/000.p/11: dial tcp {IP}:{PORT}: i/o timeout.
The go get function will try to check the checksum from golang.org database with your package (I can't link to google server directly so it shows timeout error / I guess u will get 404 if u can link to google server). It shows that if we try to go get a private package with not setting GOPRIVATE, u will fail because u can not pass the check.
Suggestion:
If you want to build your private go package, u should set your GOPRIVATE firstly and make sure it is wide enough to INCLUDE your package name.
Thank U to all people for reading and answering this question.
I am using golang 1.13 .
I have a project that depends on a private gitlab project.
I have the ssh keys for the same.
When I try to retrieve the dependencies for a newly created module, I am getting the following error:
$ go version
go version go1.13 linux/amd64
$ go mod why
go: downloading gitlab.com/mycompany/myproject v0.0.145
verifying gitlab.com/mycompany/myproject#v0.0.145: gitlab.com/mycompany/myproject#v0.0.145: reading https://sum.golang.org/lookup/gitlab.com/mycompany/myproject#v0.0.145: 410 Gone
I have no idea why it is trying to ping sum.golang.org/lookup since it is a private gitlab project.
My ~/.gitconfig contains the following (based on my looking up in google search for similar errors)
# Enforce SSH
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
[url "git#gitlab.com:"]
insteadOf = https://gitlab.com/
The error still persists.
I would expect the package to be downloaded from my private gitlab project repository to the current project.
Is there anything I need to do in my private gitlab project repository to make it ready for 'go get' ?
The private gitlab project repository already contains the go.sum and go.mod for the project as well.
Anything that I am missing ?
edit: 1) The private repo name and the company name contains no asterisks or any other special characters. only alphabets and not even numeric characters.
Answering my own question after looking up,
Setting the GOPRIVATE variable seems to help.
GOPRIVATE=gitlab.com/mycompany/* go mod why
"
The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database.
" from https://golang.org/doc/go1.13
Aliter:
Setting the env variable GONOSUMDB also seems to work.
Specifically, invoking the following command seems to help.
GONOSUMDB=gitlab.com/mycompany/* go mod why
The above env variable prevents the ping to sum.golang.org/lookup for a checksum match. It also prevents leaking the names of private repos to a public checksum db. [ Source - https://docs.gomods.io/configuration/sumdb/ ]
Also - here at
* GONOSUMDB=prefix1,prefix2,prefix3 sets a list of module path prefixes, again possibly containing globs, that should not be looked up using the database.
source: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md
Related Issues:
https://github.com/golang/go/issues/32291
https://github.com/golang/go/issues/33985
["Go 1.13 has been released, and this issue was filed well after the freeze window. The proposed changes will not happen in 1.13, but don't assume they will necessarily happen in 1.14 either." from issue 33985 above. ]
Basically it failed to verify private repository. However I don't like turning off checksum, but you can easily set GOSUMDB to off before trying to get module. something like this:
GOSUMDB=off go get github.com/mycompany/myproject
ref: https://github.com/golang/go/issues/35164#issuecomment-546503518
A second and better solution is to set GOPRIVATE environment variable that controls which modules the go command considers to be private (not available publicly) and should therefore NOT use the proxy or checksum database. The variable is a comma-separated list of glob patterns (same syntax of Go's path.Match) of module path prefixes. For example,
export GOPRIVATE=*.corp.example.com,rsc.io/private
Or
go env -w GOPRIVATE=github.com/mycompany/*
Last solution you can try is to turn off such checks for all private repositories that you don't want to go public or being verified through sum.golang.org/lookup/github.com/mycompany/...
GONOSUMDB=gitlab.com/mycompany/* go mod why
Note that:
If you have issues fetching modules or repos over https, you may want to add the following to your ~/.gitconfig to make go get/fetch repositories using ssh instead of https
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
Change following go variable's setting and then upgrade your package,
$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOSUMDB=off
$ go get -u <your dependency package>
I have this scenario too and this works for me.
edit your .git/config and add two lines in it.( I have this in a global .gitconfig in home dir)
[url "ssh://youprivate.com"]
insteadOf = https://yourprivate.com
export GOSUMDB=off
Then everything will OK.
Suppose you have a repository at github.com/someone/repo and you fork it to github.com/you/repo. You want to use your fork instead of the main repo, so you do a
go get github.com/you/repo
Now all the import paths in this repo will be "broken", meaning, if there are multiple packages in the repository that reference each other via absolute URLs, they will reference the source, not the fork.
Is there a better way as cloning it manually into the right path?
git clone git#github.com:you/repo.git $GOPATH/src/github.com/someone/repo
If you are using go modules. You could use replace directive
The replace directive allows you to supply another import path that might
be another module located in VCS (GitHub or elsewhere), or on your
local filesystem with a relative or absolute file path. The new import
path from the replace directive is used without needing to update the
import paths in the actual source code.
So you could do below in your go.mod file
module some-project
go 1.12
require (
github.com/someone/repo v1.20.0
)
replace github.com/someone/repo => github.com/you/repo v3.2.1
where v3.2.1 is tag on your repo. Also can be done through CLI
go mod edit -replace="github.com/someone/repo#v0.0.0=github.com/you/repo#v1.1.1"
To handle pull requests
fork a repository github.com/someone/repo to github.com/you/repo
download original code: go get github.com/someone/repo
be there: cd "$(go env GOPATH)/src"/github.com/someone/repo
enable uploading to your fork: git remote add myfork https://github.com/you/repo.git
upload your changes to your repo: git push myfork
http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html
To use a package in your project
https://github.com/golang/go/wiki/PackageManagementTools
One way to solve it is that suggested by Ivan Rave and http://blog.campoy.cat/2014/03/github-and-go-forking-pull-requests-and.html -- the way of forking.
Another one is to workaround the golang behavior. When you go get, golang lays out your directories under same name as in the repository URI, and this is where the trouble begins.
If, instead, you issue your own git clone, you can clone your repository onto your filesystem on a path named after the original repository.
Assuming original repository is in github.com/awsome-org/tool and you fork it onto github.com/awesome-you/tool, you can:
cd $GOPATH
mkdir -p {src,bin,pkg}
mkdir -p src/github.com/awesome-org/
cd src/github.com/awesome-org/
git clone git#github.com:awesome-you/tool.git # OR: git clone https://github.com/awesome-you/tool.git
cd tool/
go get ./...
golang is perfectly happy to continue with this repository and doesn't actually care some upper directory has the name awesome-org while the git remote is awesome-you. All import for awesome-org are resovled via the directory you have just created, which is your local working set.
In more length, please see my blog post: Forking Golang repositories on GitHub and managing the import path
edit: fixed directory path
If your fork is only temporary (ie you intend that it be merged) then just do your development in situ, eg in $GOPATH/src/launchpad.net/goamz.
You then use the features of the version control system (eg git remote) to make the upstream repository your repository rather than the original one.
It makes it harder for other people to use your repository with go get but much easier for it to be integrated upstream.
In fact I have a repository for goamz at lp:~nick-craig-wood/goamz/goamz which I develop for in exactly that way. Maybe the author will merge it one day!
Here's a way to that works for everyone:
Use github to fork to "my/repo" (just an example):
go get github.com/my/repo
cd ~/go/src/github.com/my/repo
git branch enhancement
rm -rf .
go get github.com/golang/tools/cmd/gomvpkg/…
gomvpkg <<oldrepo>> ~/go/src/github.com/my/repo
git commit
Repeat each time when you make the code better:
git commit
git checkout enhancement
git cherry-pick <<commit_id>>
git checkout master
Why? This lets you have your repo that any go get works with. It also lets you maintain & enhance a branch that's good for a pull request. It doesn't bloat git with "vendor", it preserves history, and build tools can make sense of it.
Instead of cloning to a specific location, you can clone wherever you want.
Then, you can run a command like this, to have Go refer to the local version:
go mod edit -replace github.com/owner/repo=../repo
https://golang.org/cmd/go#hdr-Module_maintenance
The answer to this is that if you fork a repo with multiple packages you will need to rename all the relevant import paths. This is largely a good thing since you've forked all of those packages and the import paths should reflect this.
Use vendoring and submodules together
Fork the lib on github (go-mssqldb in this case)
Add a submodule which clones your fork into your vendor folder but has the path of the upstream repo
Update your import statements in your source code to point to the vendor folder, (not including the vendor/ prefix). E.g. vendor/bob/lib => import "bob/lib"
E.g.
cd ~/go/src/github.com/myproj
mygithubuser=timabell
upstreamgithubuser=denisenkom
librepo=go-mssqldb
git submodule add "git#github.com:$mygithubuser/$librepo" "vendor/$upstreamgithubuser/$librepo"
Why
This solves all the problems I've heard about and come across while trying to figure this out myself.
Internal package refs in the lib now work because the path is unchanged from upstream
A fresh checkout of your project works because the submodule system gets it from your fork at the right commit but in the upstream folder path
You don't have to know to manually hack the paths or mess with the go tooling.
More info
https://git-scm.com/book/en/v2/Git-Tools-Submodules
How do I fix the error message "use of an internal package not allowed" when go getting a golang package?
https://github.com/denisenkom/go-mssqldb/issues/406
https://github.com/golang/go/wiki/PackageManagementTools#go15vendorexperiment
The modern answer (go 1.15 and higher, at least).
go mod init github.com/theirs/repo
Make an explicit init arg that is the ORIGINAL package names. If you don't include the repo name, it will assume the one in gopath. But when you use go modules, they no longer care where they are on disk, or where git actually pulls dependencies from.
To automate this process, I wrote a small script. You can find more details on my blog to add a command like "gofork" to your bash.
function gofork() {
if [ $# -ne 2 ] || [ -z "$1" ] || [ -z "$2" ]; then
echo 'Usage: gofork yourFork originalModule'
echo 'Example: gofork github.com/YourName/go-contrib github.com/heirko/go-contrib'
return
fi
echo "Go get fork $1 and replace $2 in GOPATH: $GOPATH"
go get $1
go get $2
currentDir=$PWD
cd $GOPATH/src/$1
remote1=$(git config --get remote.origin.url)
cd $GOPATH/src/$2
remote2=$(git config --get remote.origin.url)
cd $currentDir
rm -rf $GOPATH/src/$2
mv $GOPATH/src/$1 $GOPATH/src/$2
cd $GOPATH/src/$2
git remote add their $remote2
echo Now in $GOPATH/src/$2 origin remote is $remote1
echo And in $GOPATH/src/$2 their remote is $remote2
cd $currentDir
}
export -f gofork
You can use command go get -f to get you a forked repo
in your Gopkg.toml file add these block below
[[constraint]]
name = "github.com/globalsign/mgo"
branch = "master"
source = "github.com/myfork/project2"
So it will use the forked project2 in place of github.com/globalsign/mgo