I'm attempting to use private repositories as go libraries.
Whenever i try to run go get og god mod tidy, i get this kind of error
>go get bitbucket.org/myworkspace/myRepo
go get bitbucket.org/myworkspace/myRepo: reading https://api.bitbucket.org/2.0/repositories/myworkspace/myRepo?fields=scm: 404 Not Found
I've found multiple suggestions to fix this, with git config insteadOf url reqriting, but it doesn't work, and it all seems to assume that go will clone the library repo via git, and not the api.
My colleague who is running Linux, tried this and it worked, and at no point does it appear to use api.bitbucket.org instead of just bitbucket.org.
I've tried calling https://api.bitbucket.org/2.0/repositories/myworkspace/myRepo?fields=scm via Insomnia, with credentials, and i get the repo back just fine.
Why does go use the bitbucket api on windows, and how can i have it use credentials, so it can find the repo ?
This is due to a change made by Bitbucket (rolling out from June 1st 2022):
Rolling out these changes will break previous versions of Go due to the fact that the go command relies on a 403 response to fetch repositories hosted on Bitbucket Cloud. This means that users who use older versions of Go with private repositories, for example CI/CD builds with Go dependencies, will run into 404 errors.
Go has been updated to support these changes; version 1.18 includes the change but if you are running an earlier version you may need to upgrade to a later minor revision (change is in 1.17.7 and 1.16.14). The relevant Go issue is here (the aim of the change is something different but it resolves the issue).
Why does go use the bitbucket api on windows...
Go was using the API to determine if the Bitbucket repo was using Git or Mercurial (Bitbucket is dropping support for Mercurial).
As mentioned in the comments I've found that the new Git Credential Manager removes the need for the workarounds previously required to access private Bitbuicket repos. Using the credential manager and setting GOPRIVATE was all that was needed..
You can first export the private repository with the command export GOPRIVATE=<remote module name>. Then you can run the command env GIT_TERMINAL_PROMPT=1 go get <remote module name> so that if the credentials are not configured, you get a prompt.
Related
Everything was working fine until I added a dependency of a private repository. I noticed that autocomplete stopped working and there was an error notification by vscode:
Error loading workspace: err: exit status 1: stderr: go:
bitbucket.org/my-group/my-private-repo#v0.0.0-20210512194559-2c29669c4ecc:
reading https://api.bitbucket.org/2.0/repositories/my-group/my-private-repo?fields=scm:
403 Forbidden server response: Access denied. You must have write or admin access.
go: bitbucket.org/my-group/my-private-repo#v0.0.0-20210512194559-2c29669c4ecc:
reading https://api.bitbucket.org/2.0/repositories/my-group/my-private-repo?fields=scm:
403 Forbidden server response: Access denied. You must have write or admin access.
: packages.Load error
It seems like the go extension is trying to access my private repo:
Why is this necessary for code competition to be able to work?
If it is necessary, how do I provide my SSH-key to the go extension?
Other workaround?
It attempts to get the sources for your dependency so it can generate the actual suggestions for your. It wants to load the source go module style and I am guessing you have your dependency located in the GOPATH. You would hope it would gracefully handle the connection errors and still provide completions for the code it can access, so maybe this is a bug, consider creating an issue on github(I don't know which project).
In this case it is, also by doing the following fix you will be able to use go mod for private dependencies as well instead of having to manually manage dependencies the "old fashion" way. Go modules uses Git by default, you can instruct git to always use SSH instead of HTTPS by adding the following to your global git config file:
[url "git#bitbucket.org:"]
insteadOf = https://bitbucket.org/
Your private key should be used automatically, assuming you don't require any special config for normal git over ssh operations.
Go modules supports inclusion of dependencies via the vendor directory. If you place your dependency in there the code completion should use it instead of trying to download it. But I can't confirm this, have never tried it
I am using Go modules.
In order to use module version, I cannot use local module. For example:
replace locakpkg => ../localpkg v0.1.0
The above will fail because replacement local path cannot have version so far (go 1.15).
Thus, to make the module version work, I decided to use a private ssh repo.
I did search how to make private ssh repo work for two days.
By following many online articles, I did
git config --global url.user#private.com:.insteadOf https://private.com/
go env -w GOPRIVATE=private.com
I found out go get will always do https fetch to check ssl credential. So I configured a https server properly too.
But in the end, I still get an error message:
unrecognized import path "private.com/foo": reading https://private.com/foo?go-get=1: 404 Not Found
I did google this error and found out this spec https://golang.org/ref/mod#vcs-find which says I have to let the server reply with <meta name="go-import" content="root-path vcs repo-url"> for https fetch request.
If there is a way to use git tag versioning in local module packages, I am OK to use local replace in go.mod instead of configuring a private ssh repo.
If the above point is not possible, how to avoid https fetch when I configure a private ssh repo? I think ssh repo has nothing to do with https protocol.
(I am using go 1.15 at linux. The latest stable version while posting this answer)
I solved the problem and posting here, hopefully, this will help other people one day. I don't find any correct answer by my search online.
In short, the answer is to use .git suffix in all places. Without .git suffix, go mod tidy and go get will use https instead of ssh (git).
At Client:
The file ~/.gitconfig (at linux) if you use /repopath/foo.git path at server:
[url "ssh://user#private.com"]
insteadOf = https://private.com
The file ~/.gitconfig (at linux) if you use ~/repopath/foo.git path at server:
[url "user#private.com:"]
insteadOf = https://private.com/
Execute the following to update ~/.config/go/env at linux:
go env -w GOPRIVATE=private.com
In go.mod, it should use
require private.com/repopath/foo.git v0.1.0
In file.go, it should be
import private.com/repopath/foo.git
At SSH Server
in foo.git/go.mod at private server should have:
module private.com/repopath/foo.git
And make sure the git repo at server has tag version v0.1.0. Don't forget to use git push --tags at client to update the tag version to the server. Without --tags, tag version will not be pushed.
After adding .git suffix to all the required places, go mod tidy and go get will no longer send https request.
I'm trying out WSL 2 on windows 10 and it's gone well so far, but I've been struggling for 2 weeks to make this work, because for some reason go get doesn't use or is not able to make the Git Credentials Manager to prompt for my credentials.
I followed this blog to set up WSL2 with GCM https://www.edwardthomson.com/blog/git_credential_manager_with_windows_subsystem_for_linux.html
And it works very good for most of the daily tasks like cloning, read and write. But when using go get I get this error.
go get <remote github repo>#<latest commit id>
go: <remote github repo> 681dceefc81203e094872401c184d038090d6049 => v0.0.17-0.20200501212733-681dceefc812
go get: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812/go.mod: verifying module: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812/go.mod: reading https://sum.golang.org/lookup/<remote github repo>#v0.0.17-0.20200501212733-681dceefc812: 410 Gone
server response:
not found: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/232ff028cb2fdebd254e30bfc612843483f0fe3fbeb18d5fc8fb4b20f21c9021: exit status 128:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
Already tried ssh-keys and the solutions proposed here go get results in 'terminal prompts disabled' error for github private repo
But the error remains the same, when enabling env GIT_TERMINAL_PROMPT=1 nothing happens, I guess it's because WSL 2 doesn't have the permissions to do that. Anyway I also tried this tool https://github.com/microsoft/Git-Credential-Manager-for-Mac-and-Linux and by setting a variable for plain credentials store, it prompts in the terminal for credentials. But I'm using 2FA because it's required by the organization and the prompt only asks for username and password, so the authentication fails.
So I have to reach out to a mate who is using Mac. He is able to go get the dependency to affect go.mod, make a commit and push the change so I can pull it and continue from there. But of course this is not ideal, and he doesn't have any problem, he uses osxkeychain to manage his git credentials.
Anyone has faced this issue? or know how to solve it? Thank you so much in advance.
Go is not able to understand that certain modules are private and their checksum should not be validated against go's checksum library. The following error comes from that
verifying module: <remote github repo>#v0.0.17-0.20200501212733-681dceefc812/go.mod: reading https://sum.golang.org/lookup/<remote github repo>#v0.0.17-0.20200501212733-681dceefc812: 410 Gone
If possible use at least go 1.13 or, higher. Go had introduced an env variables by name GOPRIVATE, GONOPROXY and GONOSUMDB for managing private modules better. Simplest way to signal to Go that you are importing a private repo is to use GOPRIVATE. Set the pattern of private repos to GOPRIVATE env variable to suppress checksum validation and usage of GOPROXY. Example below avoids checksum for all repos in that hierarchy:
GOPRIVATE=github.com/<your org>/*
Check out answers here and here. You can also do go help module-private for help.
So, I recently made two big changes ... moved my code from bitbucket to github, and set up a pipeline on heroku with a new staging app (original app is now production).
I got a new github token and placed it into the auth.json file as was done with the previous bitbucket repo (it's a private repo). However, when I push to heroku to build the code with composer there, I cannot connect with the laravel spark repo.
Error:
Installing laravel/spark (v3.0.5): Downloading (failed) Failed to download laravel/spark
from dist: The "https://api.github.com/repos/laravel/spark/zipball/512af184c15d793c33328ff03313553ea6feacba"
file could not be downloaded (HTTP/1.1 404 Not Found)
Now trying to download from source
Installing laravel/spark (v3.0.5): Cloning 512af184c1
[RuntimeException]
Failed to execute git clone --no-checkout 'https://***:***#github.com/laravel/spark.git' '/tmp/build_9916d292e7eb72e0fbe34f47e3d9854c/vendor/laravel/spark' && cd '/tmp/build_9916d292e7eb72e0fbe34f47e3d9854c/vendor/laravel/spark' && git remote add composer 'https://***:***#github.com/laravel/spark.git' && git fetch composer
remote: Repository not found.
fatal: repository 'https://***:***#github.com/laravel/spark.git/' not found
What I have tried ...
Setting the github api token on heroku with
heroku config:set GITHUB_API_TOKEN=<token>
Setting the composer github token
composer config -g github-oauth.github.com <token>
I am connected to the Laravel Spark repo on github and when I run composer on my local machine I am not prompted for a spark token. Every other dependency that I have runs fine - I can change the auth.json and that is not the case, so I don't think this is a problem with lack of access to my github.
Does anyone know how Laravel - Spark checks to grant access and how we can check to see where we are going wrong? There should be a checklist of things that can be looked at if access is denied.
Any help is appreciated. Been stuck for almost a week. I really need some way to figure out how to connect to the Spark repo.
(Edit) Spark is a composer satis repo. I can't really find any info on how to prompt this type of repo to tell me why I can't clone it or how best to communicate with it.
(Edit 2) Also tried changing the git config to ensure that it had the right token. This should be overwritten by the files, but I tried it anyway.
git config github.accesstoken <token>
The response from the software providers is to use an alternative method and place the code under my source control so that composer is not trying to load it. I do not wish to do this for a number of reasons. Again, I need a way to clone the satis repo in composer.
Edit 3: I have also tried going to the URL of the repo and attempting to access one of the versions. This displays the same error as when you go to the URL in the error directly (it's the same URL).
{
"message": "Not Found",
"documentation_url": "https://developer.github.com/v3/repos/contents/#get-archive-link"
}
This seems to back up the belief that this is not a composer issue, but something to do with a github setting or spark setting.
Edit 4: It occurred to me that my problems started after upgrading to V6 and I am getting denied access to the spark repo containing versions 1-5 and version 6 is separate. I upgraded my spark version to 6 and had access to that repo. I then tried uploaded the code base to heroku that had version 6 but was denied access to the repo there.
I also tried ...
heroku config:set github_oauth=<token>
Edit 5: I noticed that the output from pushing to heroku included the phrase
NOTICE: Using $COMPOSER_GITHUB_OAUTH_TOKEN for GitHub OAuth.
In response, I found an article asserting that the oauth token should be set in the config portion of composer.json as ...
"config": {
"github-oauth": {
"github.com": "<token>"
}
}
I tried it, but it didn't work
So, it turns out there were several issues. The final big one was that for some reason, I had to delete my api personal token used for github access and create a new one with full privileges for everything. Once that was set up, I had access and was able to reduce the privileges to repo only.
https://github.com/ladybirdweb/agorainvoicing Use open source Agora Invoicing software. It has all the tools you need to start software selling business. It is build on Laravel framework and is very similar to Laravel Spark
C:\Users\me
> go get -insecure github.com/denisenkom/go-mssqldb
# cd .; git clone https://github.com/denisenkom/go-mssqldb C:\Users\me\Projects\Go\src\github.com\denisenkom\go-mssqldb
Cloning into 'C:\Users\me\Projects\Go\src\github.com\denisenkom\go-mssqldb'...
fatal: unable to access 'https://github.com/denisenkom/go-mssqldb/': SSL certificate problem: unable to get local issuer certificate
package github.com/denisenkom/go-mssqldb: exit status 128
According to go help get this connection should drop down to http? Yes? Do I misunderstand? How do I get this to work w/o https?
PS: I'm not interested in trying to fix https (which on this Win10 image I have no control over anyway) - I already fought that battle with npm and lost...
Edit: I found a passable answer by fixing the global git config. I hate to do it, but needs must... atom.io/go-plus does not seem to pick up this config change, I will ping the author.
C:\Users\me
> git config --global http.sslVerify false
Update Q2 2021: with the newly released Go 1.17 (beta), you now have, for deprecations:
go get
The go get -insecure flag is deprecated and has been removed.
To permit the use of insecure schemes when fetching dependencies, please use the GOINSECURE environment variable.
The -insecure flag also bypassed module sum validation, use GOPRIVATE or GONOSUMDB if you need that functionality.
See go help environment for details.