I have a go module that is mirrored to several locations. One is in gitlab and the other bitbucket. When trying to base it off of the new gitlab location I get:
go: gitlab.com/company/core#v1.0.21: parsing go.mod:
module declares its path as: bitbucket.org/company/core
but was required as: gitlab.com/company/core
I know why this is happening, but how can I define my go.mod to be either or?
Short answer: You can't.
But all may not be lost...
If you have code in both the location referenced by the module path and in some other location that for some reason you prefer to download from, you can configure a git alias to redirect the reference from the stated module path to the desired download source:
git config --global --add url."https://<download path>".insteadOf "https://<module path>"
The alias doesn't have to be a complete replacement. You can use an alias to replace a prefix on any module path. So if you have a number of modules hosted on server X that you have identified (in their module path) on server Y, you only need to alias the server host prefix and path to cover all of these modules (as long as the repo names are the same).
For example, we use this to create 'friendly' module paths for modules hosted in our on-prem AzureDevOps Server git repos, for example:
git config --global --add url."https://tfs.myorg.com/myorgcollection/_git".insteadOf "https://tfs.myorg.com"
Our modules are then able to identify themselves as tfs.myorg.com/somemodule.git but when go get attempts to fetch this git will look for it at https://tfs.myorg.com/myorgcollection/_git/somemodule.git
(the .git suffix is a necessary Azure DevOps peculiarity)
To be honest, I haven't tried this with entirely different servers, only for "rewriting" paths on the same server, but in principle it should work. So in your case:
git config --global --add url."https://gitlab.com/company/".insteadOf "https://bitbucket.org/company/"
Because this works at the git level of things, as far as go is concerned it is still getting from bitbucket.org even though git is going to gitlab.com under the hood.
i.e. when you go get you need to reference the declared module path:
go get bitbucket.org/company/core
The only downside of this is that you need to configure the alias on any/all workstations (and build machines) that need to go get from any location other than that declared in the module path.
I'm searching for the way to get $ go get work with private repository, after many google try.
The first try:
$ go get -v gitlab.com/secmask/awserver-go
Fetching https://gitlab.com/secmask/awserver-go?go-get=1
https fetch failed.
Fetching http://gitlab.com/secmask/awserver-go?go-get=1
Parsing meta tags from http://gitlab.com/secmask/awserver-go?go-get=1 (status code 200)
import "gitlab.com/secmask/awserver-go": parse http://gitlab.com/secmask/awserver-go?go-get=1: no go-import meta tags
package gitlab.com/secmask/awserver-go: unrecognized import path "gitlab.com/secmask/awserver-go
Yep, it did not see the meta tags because I could not know how to provide login information.
The second try:
Follow https://gist.github.com/shurcooL/6927554. Add config to .gitconfig.
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
$ go get -v gitlab.com/secmask/awserver-go --> not work
$ go get -v gitlab.com/secmask/awserver-go.git --> work but I got src/gitlab.com/secmask/awserer-go.git
Yes it work but with .git extension with my project name, I can rename it to original but do it everytime $ go get is not so good, is there an otherway?
You have one thing to configure. The example is based on GitHub but this shouldn't change the process:
$ git config --global url.git#github.com:.insteadOf https://github.com/
$ cat ~/.gitconfig
[url "git#github.com:"]
insteadOf = https://github.com/
$ go get github.com/private/repo
For Go modules to work (with Go 1.11 or newer), you'll also need to set the GOPRIVATE variable, to avoid using the public servers to fetch the code:
export GOPRIVATE=github.com/private/repo
The proper way is to manually put the repository in the right place. Once the repository is there, you can use go get -u to update the package and go install to install it. A package named
github.com/secmask/awserver-go
goes into
$GOPATH/src/github.com/secmask/awserver-go
The commands you type are:
cd $GOPATH/src/github.com/secmask
git clone git#github.com:secmask/awserver-go.git
I had a problem with go get using private repository on gitlab from our company.
I lost a few minutes trying to find a solution. And I did find this one:
You need to get a private token at:
https://gitlab.mycompany.com/profile/account
Configure you git to add extra header with your private token:
$ git config --global http.extraheader "PRIVATE-TOKEN: YOUR_PRIVATE_TOKEN"
Configure your git to convert requests from http to ssh:
$ git config --global url."git#gitlab.mycompany.com:".insteadOf "https://gitlab.mycompany.com/"
Finally you can use your go get normally:
$ go get gitlab.com/company/private_repo
For people using private GitLabs, here's a snippet that may help: https://gist.github.com/MicahParks/1ba2b19c39d1e5fccc3e892837b10e21
Also pasted below:
Problem
The go command line tool needs to be able to fetch dependencies from your private GitLab, but authenticaiton is required.
This assumes your private GitLab is hosted at privategitlab.company.com.
Environment variables
The following environment variables are recommended:
export GO111MODULE=on
export GOPRIVATE=privategitlab.company.com # this is comma delimited if using multiple private repos
The above lines might fit best in your shell startup, like a ~/.bashrc.
Explanation
GO111MODULE=on tells Golang command line tools you are using modules. I have not tested this with projects not using
Golang modules on a private GitLab.
GOPRIVATE=privategitlab.company.com tells Golang command line tools to not use public internet resources for the hostnames
listed (like the public module proxy).
Get a personal access token from your private GitLab
To future proof these instructions, please follow this guide from the GitLab docs.
I know that the read_api scope is required for Golang command line tools to work, and I may suspect read_repository as
well, but have not confirmed this.
Set up the ~/.netrc
In order for the Golang command line tools to authenticate to GitLab, a ~/.netrc file is best to use.
To create the file if it does not exist, run the following commands:
touch ~/.netrc
chmod 600 ~/.netrc
Now edit the contents of the file to match the following:
machine privategitlab.company.com login USERNAME_HERE password TOKEN_HERE
Where USERNAME_HERE is replaced with your GitLab username and TOKEN_HERE is replaced with the access token aquired in the
previous section.
Common mistakes
Do not set up a global git configuration with something along the lines of this:
git config --global url."git#privategitlab.company.com:".insteadOf "https://privategitlab.company.com"
I beleive at the time of writing this, the SSH git is not fully supported by Golang command line tools and this may cause
conflicts with the ~/.netrc.
Bonus: SSH config file
For regular use of the git tool, not the Golang command line tools, it's convient to have a ~/.ssh/config file set up.
In order to do this, run the following commands:
mkdir ~/.ssh
chmod 700 ~/.ssh
touch ~/.ssh/config
chmod 600 ~/.ssh/config
Please note the permissions on the files and directory above are essentail for SSH to work in it's default configuration on
most Linux systems.
Then, edit the ~/.ssh/config file to match the following:
Host privategitlab.company.com
Hostname privategitlab.company.com
User USERNAME_HERE
IdentityFile ~/.ssh/id_rsa
Please note the spacing in the above file matters and will invalidate the file if it is incorrect.
Where USERNAME_HERE is your GitLab username and ~/.ssh/id_rsa is the path to your SSH private key in your file system.
You've already uploaded its public key to GitLab. Here are some instructions.
All of the above did not work for me. Cloning the repo was working correctly but I was still getting an unrecognized import error.
As it stands for Go v1.13, I found in the doc that we should use the GOPRIVATE env variable like so:
$ GOPRIVATE=github.com/ORGANISATION_OR_USER_NAME go get -u github.com/ORGANISATION_OR_USER_NAME/REPO_NAME
Generate a github oauth token here and export your github token as an environment variable:
export GITHUB_TOKEN=123
Set git config to use the basic auth url:
git config --global url."https://$GITHUB_TOKEN:x-oauth-basic#github.com/".insteadOf "https://github.com/"
Now you can go get your private repo.
If you've already got git using SSH, this answer by Ammar Bandukwala is a simple workaround:
$ go get uses git internally. The following one liners will make git and consequently $ go get clone your package via SSH.
Github:
$ git config --global url."git#github.com:".insteadOf "https://github.com/"
BitBucket:
$ git config --global url."git#bitbucket.org:".insteadOf "https://bitbucket.org/"
I came across .netrc and found it relevant to this.
Create a file ~/.netrc with the following content:
machine github.com
login <github username>
password <github password or Personal access tokens >
Done!
Additionally, for latest GO versions, you might need to add this to the environment variables GOPRIVATE=github.com
(I've added it to my .zshrc)
netrc also makes my development environment setup better as my personal github access for HTTPS is been configured now to be used across the machine (just like my SSH configuration).
Generate GitHub personal access tokens: https://github.com/settings/tokens
See this answer for its use with Git on Windows specifically
Ref: netrc man page
If you want to stick with the SSH authentication, then mask the request to use ssh forcefully
git config --global url."git#github.com:".insteadOf "https://github.com/"
More methods for setting up git access: https://gist.github.com/technoweenie/1072829#gistcomment-2979908
That looks like the GitLab issue 5769.
In GitLab, since the repositories always end in .git, I must specify .git at the end of the repository name to make it work, for example:
import "example.org/myuser/mygorepo.git"
And:
$ go get example.org/myuser/mygorepo.git
Looks like GitHub solves this by appending ".git".
It is supposed to be resolved in “Added support for Go's repository retrieval. #5958”, provided the right meta tags are in place.
Although there is still an issue for Go itself: “cmd/go: go get cannot discover meta tag in HTML5 documents”.
After trying multiple solutions my problem still persisted. The final solution after setting up the ~/.netrc and SSH config file, was to add the following line to my ~/.bash_profile
export GOPRIVATE="github.com/[organization]"
I have created a user specific ssh-config, so my user automatically logs in with the correct credentials and key.
First I needed to generate an key-pair
ssh-keygen -t rsa -b 4096 -C "my#email.here"
and saved it to e.g ~/.ssh/id_my_domain. Note that this is also the keypair (private and public) I've connected to my Github account, so mine is stored in~/.ssh/id_github_com.
I have then created (or altered) a file called ~/.ssh/config with an entry:
Host github.com
HostName github.com
User git
IdentityFile ~/.ssh/id_github_com
On another server, the "ssh-url" is admin#domain.com:username/private-repo.git and the entry for this server would have been:
Host domain.com
HostName domain.com
User admin
IdentityFile ~/.ssh/id_domain_com
Just to clarify that you need ensure that the User, Host and HostName is set correctly.
Now I can just browse into the go path and then go get <package>, e.g go get main where the file main/main.go includes the package (from last example above) domain.com:username/private-repo.git.
For me, the solutions offered by others still gave the following error during go get
git#gl.nimi24.com: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
What this solution required
As stated by others:
git config --global url."git#github.com:".insteadOf "https://github.com/"
Removing the passphrase from my ./ssh/id_rsa key which was used for authenticating the connection to the repository. This can be done by entering an empty password when prompted as a response to:
ssh-keygen -p
Why this works
This is not a pretty workaround as it is always better to have a passphrase on your private key, but it was causing issues somewhere inside OpenSSH.
go get uses internally git, which uses openssh to open the connection. OpenSSH takes the certs necessary for authentication from .ssh/id_rsa. When executing git commands from the command line an agent can take care of opening the id_rsa file for you so that you do not have to specify the passphrase every time, but when executed in the belly of go get, this did not work somewhy in my case. OpenSSH wants to prompt you then for a password but since it is not possible due to how it was called, it prints to its debug log:
read_passphrase: can't open /dev/tty: No such device or address
And just fails. If you remove the passphrase from the key file, OpenSSH will get to your key without that prompt and it works
This might be caused by Go fetching modules concurrently and opening multiple SSH connections to Github at the same time (as described in this article). This is somewhat supported by the fact that OpenSSH debug log showed the initial connection to the repository succeed, but later tried it again for some reason and this time opted to ask for a passphrase.
However the solution of using SSH connection multiplexing as put forward in the mentioned article did not work for me. For the record, the author suggested adding the collowing conf to the ssh config file for the affected host:
ControlMaster auto
ControlPersist 3600
ControlPath ~/.ssh/%r#%h:%p
But as stated, for me it did not work, maybe I did it wrong
After setting up GOPRIVATE and git config ...
People may still meeting problems like this when fetching private source:
https fetch: Get "https://private/user/repo?go-get=1": EOF
They can't use private repo without .git extension.
The reason is the go tool has no idea about the VCS protocol of this repo, git or svn or any other, unlike github.com or golang.org them are hardcoded into go's source.
Then the go tool will do a https query before fetching your private repo:
https://private/user/repo?go-get=1
If your private repo has no support for https request, please use replace to tell it directly :
require private/user/repo v1.0.0
...
replace private/user/repo => private.server/user/repo.git v1.0.0
https://golang.org/cmd/go/#hdr-Remote_import_paths
Make sure you remove your previous gitconfigs, I had the same issue.
Previously I executed gitconfig whose token was expired, when you execute the command next time with new token make sure to delete previous one.
first I tried
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
but it didn't worked for my local.
I tried
ssh -t git#github.com
and it shows my ssh is fine.
finally, I fix the problem to tell the go get to consider all as private and use ssh instead of HTTPS.
adding export GOPRIVATE=*
For standalone/final repos, an as a quick fix, why don't just to name the module within the go.mod as a package using your company's domain ... ?
module go.yourcompany.tld/the_repo
go.yourcompany.tld don't even have to exist as a valid (sub)domain...
Also, in the same go.mod you can use replacement block/lines to use private repos previously cloned the same way (within a respective folder cloned also in $GOPATH/src/go.yourcompany.tld) (why do we have to depend too much in GitHub?)
Edit
Needless to say that a private repo usually shall be a private repo, typically a standard git repo, right? With that, why not just to git clone and then go get within the cloned folder?
It's Hard Code In Go Get. Not The Git Reason. So Modify Go Source.
Reason:
repoRootForImportDynamic Will Request: https://....go-get
// RepoRootForImportPath analyzes importPath to determine the
// version control system, and code repository to use.
func RepoRootForImportPath(importPath string, mod ModuleMode, security web.SecurityMode) (*RepoRoot, error) {
rr, err := repoRootFromVCSPaths(importPath, security, vcsPaths)
if err == errUnknownSite {
rr, err = repoRootForImportDynamic(importPath, mod, security)
if err != nil {
err = importErrorf(importPath, "unrecognized import path %q: %v", importPath, err)
}
}
if err != nil {
rr1, err1 := repoRootFromVCSPaths(importPath, security, vcsPathsAfterDynamic)
if err1 == nil {
rr = rr1
err = nil
}
}
So add gitlab domain to vcsPaths will ok.
Download go source code:
vi ./src/cmd/go/internal/vcs/vcs.go
Look for code below:
var vcsPaths = []*vcsPath{
// GitHub
{
pathPrefix: "github.com",
regexp: lazyregexp.New(`^(?P<root>github\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`),
vcs: "git",
repo: "https://{root}",
check: noVCSSuffix,
},
add Code As Follow,XXXX Is Your Domain:
// GitLab
{
pathPrefix: "gitlab.xxxx.com",
regexp: lazyregexp.New(`^(?P<root>gitlab.xxxx\.com/[A-Za-z0-9_.\-]+/[A-Za-z0-9_.\-]+)(/[A-Za-z0-9_.\-]+)*$`),
vcs: "git",
repo: "https://{root}",
check: noVCSSuffix,
},
compile and replace go.
I'm trying out WSL 2 on windows 10 and it's gone well so far, but I've been struggling for 2 weeks to make this work, because for some reason go get doesn't use or is not able to make the Git Credentials Manager to prompt for my credentials.
I followed this blog to set up WSL2 with GCM https://www.edwardthomson.com/blog/git_credential_manager_with_windows_subsystem_for_linux.html
And it works very good for most of the daily tasks like cloning, read and write. But when using go get I get this error.
go get <remote github repo>#<latest commit id>
go: <remote github repo> 681dceefc81203e094872401c184d038090d6049 => v0.0.17-0.20200501212733-681dceefc812
go get: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812/go.mod: verifying module: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812/go.mod: reading https://sum.golang.org/lookup/<remote github repo>#v0.0.17-0.20200501212733-681dceefc812: 410 Gone
server response:
not found: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/232ff028cb2fdebd254e30bfc612843483f0fe3fbeb18d5fc8fb4b20f21c9021: exit status 128:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
Already tried ssh-keys and the solutions proposed here go get results in 'terminal prompts disabled' error for github private repo
But the error remains the same, when enabling env GIT_TERMINAL_PROMPT=1 nothing happens, I guess it's because WSL 2 doesn't have the permissions to do that. Anyway I also tried this tool https://github.com/microsoft/Git-Credential-Manager-for-Mac-and-Linux and by setting a variable for plain credentials store, it prompts in the terminal for credentials. But I'm using 2FA because it's required by the organization and the prompt only asks for username and password, so the authentication fails.
So I have to reach out to a mate who is using Mac. He is able to go get the dependency to affect go.mod, make a commit and push the change so I can pull it and continue from there. But of course this is not ideal, and he doesn't have any problem, he uses osxkeychain to manage his git credentials.
Anyone has faced this issue? or know how to solve it? Thank you so much in advance.
Go is not able to understand that certain modules are private and their checksum should not be validated against go's checksum library. The following error comes from that
verifying module: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812/go.mod: reading https://sum.golang.org/lookup/<remote github repo>#v0.0.17-0.20200501212733-681dceefc812: 410 Gone
If possible use at least go 1.13 or, higher. Go had introduced an env variables by name GOPRIVATE, GONOPROXY and GONOSUMDB for managing private modules better. Simplest way to signal to Go that you are importing a private repo is to use GOPRIVATE. Set the pattern of private repos to GOPRIVATE env variable to suppress checksum validation and usage of GOPROXY. Example below avoids checksum for all repos in that hierarchy:
GOPRIVATE=github.com/<your org>/*
Check out answers here and here. You can also do go help module-private for help.
I followed this and added the keys in SSH Permissions as well as the fingerprint in my circle config file.
I also added this to my ~/.gitconfig as part of my circle compile step.
[url "ssh://git#github.com/MYORGANIZATION/"]
insteadOf = https://github.com/MYORGANIZATION/
following the official recommendation
When I SSH into the circle image I can see the fingerprint being added using this command ssh-add -l -E md5. But there's no keys added in ~/.ssh/. I'd expect to have ~/.ssh/id_rsa_<fingerprint> in there.
However I still get access denied when I try to retrieve the package.
The easiest way to get this to work is to follow the instructions for adding a machine user: https://circleci.com/docs/2.0/gh-bb-integration/#enable-your-project-to-check-out-additional-private-repositories
For a more complicated solution, read on.
I recently attempted the same thing. The add_ssh_keys keys should (and did in my case) add the id_rsa_<fingerprint> file.
The problem I ran into was that the key is added with an ssh config that contains:
Host !github.com *
I believe the problem is that it uses the default CircleCI key to authenticate with github. That key is valid, so github accepts it, but it most likely does not have access to the private repo in your dependencies.
To get it to work what I had to do was:
# Disable ssh-agent which seemed to override `-i`
export SSH_AUTH_SOCK=none
# Tell get to ssh with the key I want to use
export GIT_SSH_COMMAND="ssh -i /root/.ssh/id_rsa_FINGERPRINT
# Run some command to pull dependencies
go test ./...
I have a launch configuration and auto-scaling group set up. The launch config uses an AMI that I've already created, based on Ubuntu 14.04, that installs Nginx, Git, and has my static files stored as a Git repo in Nginx's /usr/share/nginx/html/ directory.
The problem: the static files in my nginx/html directory are only as new as the files that were loaded in the AMI when I created it.
To remedy this, I have tried to add a "User Data" field into the launch config. The field is defined as:
#! /bin/bash
cd /usr/share/nginx/html/
git pull origin master
<my git repo's password>
But when I check to see if the instance has the latest version of the repo, I see that it doesn't. Something is wrong with my script, and I'm not sure what.
I have tested entering these commands one-by-one exactly as is into the EC2 instance via SSH, and it works exactly as expected.
Why doesn't this work in the user data field?
Note: I have verified that the 'bash' file is indeed present in /bin/bash.
You need to pass username and password of your repository with the repo url
Sample example :
#! /bin/bash
cd /usr/share/nginx/html/
git clone https://username:password#yourRepoURL.git
Problem is definitely in the bash script. Everything it contains is executed by bash, so it actually tries to execute your password as a command.
There are multiple ways to provide a password to Git in a script. See for example this question: How to provide username and password when run "git clone git#remote.git"?
It basically depends on how secure do you need it. Maybe it's enough to have a plain text password in Git's config (it doesn't have to be so bad if you set a restricted mode for that file, it would be similar to using a private key without passphrase).
It's been a while since I asked this- I've learned a lot since then.
You can pass your username and password as part of the URL, but that is bad form as if you share the code with anyone or give anyone access to your script then they will know your account credentials.
The best way to do this would be to set up your server to connect to your Git repo over SSH - I believe this is industry best practice as it is more secure and password-less.