I have a go module that is mirrored to several locations. One is in gitlab and the other bitbucket. When trying to base it off of the new gitlab location I get:
go: gitlab.com/company/core#v1.0.21: parsing go.mod:
module declares its path as: bitbucket.org/company/core
but was required as: gitlab.com/company/core
I know why this is happening, but how can I define my go.mod to be either or?
Short answer: You can't.
But all may not be lost...
If you have code in both the location referenced by the module path and in some other location that for some reason you prefer to download from, you can configure a git alias to redirect the reference from the stated module path to the desired download source:
git config --global --add url."https://<download path>".insteadOf "https://<module path>"
The alias doesn't have to be a complete replacement. You can use an alias to replace a prefix on any module path. So if you have a number of modules hosted on server X that you have identified (in their module path) on server Y, you only need to alias the server host prefix and path to cover all of these modules (as long as the repo names are the same).
For example, we use this to create 'friendly' module paths for modules hosted in our on-prem AzureDevOps Server git repos, for example:
git config --global --add url."https://tfs.myorg.com/myorgcollection/_git".insteadOf "https://tfs.myorg.com"
Our modules are then able to identify themselves as tfs.myorg.com/somemodule.git but when go get attempts to fetch this git will look for it at https://tfs.myorg.com/myorgcollection/_git/somemodule.git
(the .git suffix is a necessary Azure DevOps peculiarity)
To be honest, I haven't tried this with entirely different servers, only for "rewriting" paths on the same server, but in principle it should work. So in your case:
git config --global --add url."https://gitlab.com/company/".insteadOf "https://bitbucket.org/company/"
Because this works at the git level of things, as far as go is concerned it is still getting from bitbucket.org even though git is going to gitlab.com under the hood.
i.e. when you go get you need to reference the declared module path:
go get bitbucket.org/company/core
The only downside of this is that you need to configure the alias on any/all workstations (and build machines) that need to go get from any location other than that declared in the module path.
Related
I'm attempting to use private repositories as go libraries.
Whenever i try to run go get og god mod tidy, i get this kind of error
>go get bitbucket.org/myworkspace/myRepo
go get bitbucket.org/myworkspace/myRepo: reading https://api.bitbucket.org/2.0/repositories/myworkspace/myRepo?fields=scm: 404 Not Found
I've found multiple suggestions to fix this, with git config insteadOf url reqriting, but it doesn't work, and it all seems to assume that go will clone the library repo via git, and not the api.
My colleague who is running Linux, tried this and it worked, and at no point does it appear to use api.bitbucket.org instead of just bitbucket.org.
I've tried calling https://api.bitbucket.org/2.0/repositories/myworkspace/myRepo?fields=scm via Insomnia, with credentials, and i get the repo back just fine.
Why does go use the bitbucket api on windows, and how can i have it use credentials, so it can find the repo ?
This is due to a change made by Bitbucket (rolling out from June 1st 2022):
Rolling out these changes will break previous versions of Go due to the fact that the go command relies on a 403 response to fetch repositories hosted on Bitbucket Cloud. This means that users who use older versions of Go with private repositories, for example CI/CD builds with Go dependencies, will run into 404 errors.
Go has been updated to support these changes; version 1.18 includes the change but if you are running an earlier version you may need to upgrade to a later minor revision (change is in 1.17.7 and 1.16.14). The relevant Go issue is here (the aim of the change is something different but it resolves the issue).
Why does go use the bitbucket api on windows...
Go was using the API to determine if the Bitbucket repo was using Git or Mercurial (Bitbucket is dropping support for Mercurial).
As mentioned in the comments I've found that the new Git Credential Manager removes the need for the workarounds previously required to access private Bitbuicket repos. Using the credential manager and setting GOPRIVATE was all that was needed..
You can first export the private repository with the command export GOPRIVATE=<remote module name>. Then you can run the command env GIT_TERMINAL_PROMPT=1 go get <remote module name> so that if the credentials are not configured, you get a prompt.
I am using Go modules.
In order to use module version, I cannot use local module. For example:
replace locakpkg => ../localpkg v0.1.0
The above will fail because replacement local path cannot have version so far (go 1.15).
Thus, to make the module version work, I decided to use a private ssh repo.
I did search how to make private ssh repo work for two days.
By following many online articles, I did
git config --global url.user#private.com:.insteadOf https://private.com/
go env -w GOPRIVATE=private.com
I found out go get will always do https fetch to check ssl credential. So I configured a https server properly too.
But in the end, I still get an error message:
unrecognized import path "private.com/foo": reading https://private.com/foo?go-get=1: 404 Not Found
I did google this error and found out this spec https://golang.org/ref/mod#vcs-find which says I have to let the server reply with <meta name="go-import" content="root-path vcs repo-url"> for https fetch request.
If there is a way to use git tag versioning in local module packages, I am OK to use local replace in go.mod instead of configuring a private ssh repo.
If the above point is not possible, how to avoid https fetch when I configure a private ssh repo? I think ssh repo has nothing to do with https protocol.
(I am using go 1.15 at linux. The latest stable version while posting this answer)
I solved the problem and posting here, hopefully, this will help other people one day. I don't find any correct answer by my search online.
In short, the answer is to use .git suffix in all places. Without .git suffix, go mod tidy and go get will use https instead of ssh (git).
At Client:
The file ~/.gitconfig (at linux) if you use /repopath/foo.git path at server:
[url "ssh://user#private.com"]
insteadOf = https://private.com
The file ~/.gitconfig (at linux) if you use ~/repopath/foo.git path at server:
[url "user#private.com:"]
insteadOf = https://private.com/
Execute the following to update ~/.config/go/env at linux:
go env -w GOPRIVATE=private.com
In go.mod, it should use
require private.com/repopath/foo.git v0.1.0
In file.go, it should be
import private.com/repopath/foo.git
At SSH Server
in foo.git/go.mod at private server should have:
module private.com/repopath/foo.git
And make sure the git repo at server has tag version v0.1.0. Don't forget to use git push --tags at client to update the tag version to the server. Without --tags, tag version will not be pushed.
After adding .git suffix to all the required places, go mod tidy and go get will no longer send https request.
Good day! I try to migrate from glide to go modules (private gitlab repos)
and checkout code via ssh
I have a simple project with an import from private gitlab repo.
go.mod looks like:
module my.private.package/modtest
go 1.12
require my.private.package/statistics v1.0.0
when I try to build my app or run test I get:
go: my.private.package/statistics#v1.0.0: unrecognized import path "my.private.package/statistics" (parse https://my.private.package/statistics?go-get=1: no go-import meta tags ())
go: error loading module requirements
I Tried to add settings to git config:
[url "ssh://git#my.gitlab.server:9999"]
insteadOf = https://my.private.package
But still getting this error.
Is there any way to make it work?
Thank you.
I've dealt with Go modules and a private GitLab before. Our private GitLab has groups and subgroups. The piece you are likely missing is ~/.netrc and you may have an improper global git configuration.
I've made a GitHub gist for this. You can find it here: https://gist.github.com/MicahParks/1ba2b19c39d1e5fccc3e892837b10e21
You can find the gist pasted below:
Problem
The go command line tool needs to be able to fetch dependencies from your private GitLab, but authenticaiton is required.
This assumes your private GitLab is hosted at privategitlab.company.com.
Environment variables
The following environment variables are recommended:
export GO111MODULE=on
export GOPRIVATE=privategitlab.company.com
The above lines might fit best in your shell startup, like a ~/.bashrc.
Explanation
GO111MODULE=on tells Golang command line tools you are using modules. I have not tested this with projects not using
Golang modules on a private GitLab.
GOPRIVATE=privategitlab.company.com tells Golang command line tools to not use public internet resources for the hostnames
listed (like the public module proxy).
Get a personal access token from your private GitLab
To future proof these instructions, please follow this guide from the GitLab docs.
I know that the read_api scope is required for Golang command line tools to work, and I may suspect read_repository as
well, but have not confirmed this.
Set up the ~/.netrc
In order for the Golang command line tools to authenticate to GitLab, a ~/.netrc file is best to use.
To create the file if it does not exist, run the following commands:
touch ~/.netrc
chmod 600 ~/.netrc
Now edit the contents of the file to match the following:
machine privategitlab.company.com login USERNAME_HERE password TOKEN_HERE
Where USERNAME_HERE is replaced with your GitLab username and TOKEN_HERE is replaced with the access token aquired in the
previous section.
Common mistakes
Do not set up a global git configuration with something along the lines of this:
git config --global url."git#privategitlab.company.com:".insteadOf "https://privategitlab.company.com"
I beleive at the time of writing this, the SSH git is not fully supported by Golang command line tools and this may cause
conflicts with the ~/.netrc.
Bonus: SSH config file
For regular use of the git tool, not the Golang command line tools, it's convient to have a ~/.ssh/config file set up.
In order to do this, run the following commands:
mkdir ~/.ssh
chmod 700 ~/.ssh
touch ~/.ssh/config
chmod 600 ~/.ssh/config
Please note the permissions on the files and directory above are essentail for SSH to work in it's default configuration on
most Linux systems.
Then, edit the ~/.ssh/config file to match the following:
Host privategitlab.company.com
Hostname privategitlab.company.com
User USERNAME_HERE
IdentityFile ~/.ssh/id_rsa
Please note the spacing in the above file matters and will invalidate the file if it is incorrect.
Where USERNAME_HERE is your GitLab username and ~/.ssh/id_rsa is the path to your SSH private key in your file system.
You've already uploaded its public key to GitLab. Here are some instructions.
What version of Go are you using? If it's Go 1.13 or later, the default is to download modules through proxy.golang.org. You can change that for a particular set of packages using the GOPRIVATE environment variable.
Here is a quote from go help module-private, which I highly recommend reading in full. Once a module is fetched directly, it should use the same git/ssh logic as before modules.
The GOPRIVATE environment variable controls which modules the go command
considers to be private (not available publicly) and should therefore not use the
proxy or checksum database. The variable is a comma-separated list of
glob patterns (in the syntax of Go's path.Match) of module path prefixes.
For example,
GOPRIVATE=*.corp.example.com,rsc.io/private
causes the go command to treat as private any module with a path prefix
matching either pattern, including git.corp.example.com/xyzzy, rsc.io/private,
and rsc.io/private/quux.
I need some help.
I have a Windows Server 2003 computer that I'm setting up with Git and Review-Board.
My Git repositories are managed by Gitolite. The setup is successful and everything is working fine (tested with multiple workstations).
The Review-Board site has installed successfully and is hosted under Apache 2.2. This is also working fine in itself (the website is accessible and responds for workstations), except the part where I'm trying to link Git repositories to Review-Board.
Most tutorials (example: http://ericholscher.com/blog/2011/jan/24/using-reviewboard-git/) provide good details on how to setup a Git repository under Review-Board, except that they refer to Linux/Unix systems, and I am running Windows Server 2003.
My Gitolite repositories are stored under D:\Repositories\Git
Example: D:\Repositories\Git\sdu-test\.git
My understanding is that Review-Board needs a local clone of the repository(ies). Therefore, I created a directory named LocalClones and I cloned my repository.
Example: D:\Repositories\LocalClones\[I cloned sdu-test.git here]
Command used (from LocalClones directory): git clone ../Git/sdu-test.git(the clone was successful)
Now, in Review-Board, I'm trying to declare my sdu-test repository.
Name is 'sdu-test'
Hosting type is 'Custom'
Repository type is 'Git'
Path is 'D:\Repository\LocalClones\sdu-test' (I have also tried with 'sdu-test.git', and 'sdu-test/.git')
Mirror path is 'ssh://git#localhost/sdu-test' (I have also tried simply 'git#localhost:sdu-test', and no mirror path at all)
When I click Save, I get the following error: (11001, 'getaddrinfo failed'). I have no idea what to do about that.
If I try to change the path to a Unix-style path ('/D/Repository/LocalClones/sdu-test'), I get a different error message: Permission denied accessing the local Git repository '/D/Repository/LocalClones/sdu-test'. With a Unix-style path, I could even write a directory path that doesn't exist ('/I/Dont/Exist') and the same permission denied error is returned (the path in the error message does reflect the change though).
Can anyone help me out and tell me how to declare my local Git repository in Review-Board under Windows?
Thank you very much!
** UPDATE ** thank you Tamagochi and VonC for your answers, but unfortunately they're not working for me. Even after fixing the git.py file, I still get the Permission denied accessing the local Git repository '/D/Repository/LocalClones/sdu-test' error message.
There appears to be a bug in ReviewBoard 1.5.5 that causes this error. You can resolve this problem in either of two ways:
Move your repositories to drive C.
Edit \reviewboard\scmtools\git.py file, and replace the following line:
self.git_dir = url_parts[2]
with:
self.git_dir = url_parts[1] + url_parts[2]
Then use the following path: file://D:/Repository/LocalClones/sdu-test
I don't have any experience with ReviewBoard, but from what I can gather (from your link):
you do need to put the full path up to the .git directory of the local clone.
you should make your local clone through an url-based address. If the local file protocol is to be used, you should try with file///D/Repository/LocalClones/sdu-test.
git#localhost:sdu-test would only work if you have a git daemon running.
ssh://git#localhost/sdu-test would only work if you have a ssh daemon running.
I am currently setting up a git schema as described here.
The server in question is running Windows Server 2008 R2, using copSSH and msysGit.
The jist of it is a pair of git repos: one bare repo that is pushed to and then another non-bare repo that contains a live application.
The repo that is pushed to contains a post-update hook that looks as follows:
echo
echo "**** Pushing data to Live Server."
echo
cd /cygdrive/c/Repositories/Live || exit
unset GIT_DIR
git pull hub master
exec git-update-server-info
Of note: Live is the non-bare repo which is, as the name implies, live. Live.git is the bare repo which contains the above hook and is pushed to. I was unable to cd to the repo without /cygdrive/, which was uncovered by echoing $pwd in a diagnostic push.
Pull is successfully called in the Live repo, the problem comes in locating the Live.git repo in that update. I get "odd ball" errors that read like:
remote: Pushing data to Live Server.ommand not foundnd not foundle or directory
remote: : not foundpdate-server-infoand not foundand not founddommand not found
hub is defined as a remote repository via the local git shell as both /c/Repositories/Live.git and /cygdrive/c/Repositories/Live.git.
I've replaced "hub" with direct folder paths, even trying things like "../Live.git"
Everything I have tried has ended in a git update failure. All of the attempted paths for Live.git in the update command were attempted via a git bash with the working directory set to the Live repo and worked.
Thank you for any leads or help!
An update:
My schema requires that I pull from the hub repository, it would take an unnecessary amount of room to explain why, but such is the case.
I have done more testing and found more out about my issue:
When I push via SSH to my repository, the prime hub is successfully mounted and the pull command is executed. When said pull command tries to resolve the hub repository it fails (as I detailed above), but it turns out with a series of ls commands in the hook that no folders can be seen. I even ran an ls on /cygdrive/c/ and can only see a folder called usr. I have ensured that the copSSH user has full administrative privileges on the entire file system (only for testing and debugging purposes).
I do not know why my paths have to use the /cygdrive/ path, I do not have cygwin installed besides the cygwin packaged in copSSH. Is this indicative of a problem in my configuration? or is this normal for copSSH + msysGit setups? How is it possible that I can mount the prime repository from the hook, even if ls will not list the repositories folder?
You shouldn't need /cygdrive in paths with msysgit, unless you also have cygwin installed (in which case the combination of msysgit mingw and cygwin might be the source of your problem)
You shouldn't try to run git pull from a repo hook. While some people will say that it works fine, it does not and the behavior can be unpredictable due to various factors.
Instead you should use a method such as this one. http://toroid.org/ams/git-website-howto
That method uses a post-receive hook in a bare repo that calls git checkout to update a directory.