I cannot get this to work, even after checking other topics on stackoverflow. My project on gitlab.com/my_company/backend needs a module, found at gitlab.com/my_company/pkg/auth.
Locally, I've setup GOPRIVATE / git's configuration to make it work (and it works), though in Gitlab's CI Pipelines on a merge request, this fails.
Pipeline log / go.mod
I've added some debugging logs just to make sure everything was setup like I thought. Here's a failing pipeline's log :
$ git config --global url."ssh://git#gitlab.com/my_company/".insteadOf "https://gitlab.com/my_company/"
$ git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
$ git config -l | grep instead
url.ssh://git#gitlab.com/my_company/.insteadof=https://gitlab.com/my_company/
url.git#gitlab.com:.insteadof=https://gitlab.com/
$ env | grep GOPRIVATE
GOPRIVATE=gitlab.com/my_company
$ go mod download
go: gitlab.com/my_company/pkg/auth#v1.1.0: reading gitlab.com/my_company/pkg/auth/auth/go.mod at revision auth/v1.1.0: unknown revision auth/v1.1.0
One weird part of this log I've found is :
reading gitlab.com/my_company/pkg/auth/auth/go.mod - why is it repeating auth/auth? It actually happened once before locally, but it was because I wrote "github" instead of "gitlab" :)
The relevant go.mod line just in case :
require (
gitlab.com/my_company/pkg/auth v1.1.0 // indirect
)
Repository tags
Here are the tags setup on the repository gitlab.com/my_company/pkg :
$ git tag -l
auth/v1.0.0
auth/v1.1.0
cache/v1.0.0
cache/v1.0.1
$ git ls-remote --tags
From git#gitlab.my_company/pkg.git
9efcb02d5489adaac9d525dcb496d868d65e856a refs/tags/auth/v1.0.0
13730d4f61df978c6d690fd2678e2ed924808e0c refs/tags/auth/v1.1.0
2b8dff0ec1b737d975290720933180a9b591a1db refs/tags/cache/v1.0.0
9a3e598bbf83bea57b29d8a908b514861ae37b12 refs/tags/cache/v1.0.1
I'm not that familiar with Gitlab CI so I'm out of things to try. Any ideas?
Thank you!
Update: I finally got gitlab-runner installed so I could try running the yml directly, no luck. It still works locally (not a big surprise).
In you project should be file .gitlab-ci.yml and you can add GOPRIVATE variable to your CI and runner will use it for you project.
More details how to add env vars:
https://docs.gitlab.com/ee/ci/variables/#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file
My module is gitlab.com/getsote/utilities/slogger
My repository is gitlab.com/getsote/utilities/slogger.git
When I run go get gitlab.com/getsote/utilities/slogger, I get the message below.
Scotts-Mac-mini:seeding syacko$ go get gitlab.com/getsote/utilities/slogger
go get gitlab.com/getsote/utilities/slogger: module gitlab.com/getsote/utilities/slogger: git ls-remote -q origin in /Users/syacko/workspace/sotesoft/golang/pkg/mod/cache/vcs/80b3644beae1b986f1c659355360479e2463820660aa328d2edb1e571aba259b: exit status 128:
remote: The project you were looking for could not be found.
fatal: repository 'https://gitlab.com/getsote/utilities.git/' not found
Scotts-Mac-mini:seeding syacko$
The gitlab.com/getsote/utilities.git is a sub-directory and not a repository. I don't understand why go get is going to the utilities as a repository?
==========================
PREVIOUS Updates
Directory Structure:
GOPATH/src/slogger
|----go.mod
|----slogger.go
|----slogger_test.go
go.mod file
module slogger or gitlab.com/getsote/utilities/slogger -> still gets the error below
go 1.14
gitlab.com/getsote/utilities contains repository slogger.git
I have run a test to see if the issue is the number of nodes in the path. So, I create a new repository with no sub-directory and pushed the slogger code. Then ran go get gitlab.com/getsote/slogger which generate a different error message.
GOPATH/gitlab.com/getsote/test-go-mod -> create new directory and added slogger files listed above
gitblab.com/getsote/test-go-mod -> new repository with one less level
Scotts-Mac-mini:test-go-mod syacko$ go get gitlab.com/getsote/test-go-mod
go: downloading gitlab.com/getsote/test-go-mod v0.0.0-20200409023538-794310bf7cf9
go get gitlab.com/getsote/test-go-mod: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: verifying module: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: reading https://sum.golang.org/lookup/gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: 410 Gone
server response:
not found: gitlab.com/getsote/test-go-mod#v0.0.0-20200409023538-794310bf7cf9: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/7753c92c9bd1419156d8120684b7f3707fd207e01a2947ba89e2acfd2ecfb4d0: exit status 128:
fatal: could not read Username for 'https://gitlab.com': terminal prompts disabled
Scotts-Mac-mini:test-go-mod syacko$
This is still getting the status error of 128 for the missing version. Additionally, it is looking in the right location for the code. If this is true, then I just need help with the version missing. Moving to a shorted directory structure is doable.
========================
Newest Update
#praveent > The solution at https://medium.com/cloud-native-the-gathering/go-modules-with-private-git-repositories-dfe795068db4 didn't work for me. So I started from scratch to see how to resolve the issue.
The reason is because for a git repository it assumes that utilities is the repo and not utilities/slogger
There is a way to override this behavior by implementing go get API. But, gitlab is yet to implement the same due to security concerns. You can read more here. Gitlab issue
Update: Add reference to gitlab issue tracking this problem.
So, here is how I got this to work using gitlab.com. I'm not saying other ways will not work, they just didn't for me and my setup. First, since I don't care if the code is available to the public, I created a new group at gitlab.com. This new group is public from the start, so no need to adjust permissions. Then I create a repository called packages and cloned the repository to my local machine with the same directory structure that is in gitlab.com, gitlab.com/soteapps/packages with ~/workspace/soteapps/packages on my machine. Both of these are out side the GOPATH. I'm not sure this matters, but it is working this way, so I'm putting it here.
Under packages, I copied the slogger directory and code.
cp -R slogger ~/workspace/soteapps/packages/.
Edited the go.mod file to match the repository structure, which is in the packages directory. There is no go.mod file in the slogger directory.
module gitlab.com/soteapps/packages
go 1.14
Edited the hello.go import to match the package.
package main
import (
"fmt"
"rsc.io/quote"
"gitlab.com/soteapps/packages/slogger"
)
func main() {
fmt.Println(quote.Hello())
slogger.Info("Test message")
}
Built the program using go build -o hello and then ran it hello with the following results:
Scotts-Mac-mini:hello syacko$ hello
Hello, world.
INFO:2020/04/10 21:11:33 Test message
Scotts-Mac-mini:hello syacko$
Worked! Thank you all that helped. This wouldn't of gotten solved without your help.
Note: This only works for public repositories.
I am using golang 1.13 .
I have a project that depends on a private gitlab project.
I have the ssh keys for the same.
When I try to retrieve the dependencies for a newly created module, I am getting the following error:
$ go version
go version go1.13 linux/amd64
$ go mod why
go: downloading gitlab.com/mycompany/myproject v0.0.145
verifying gitlab.com/mycompany/myproject#v0.0.145: gitlab.com/mycompany/myproject#v0.0.145: reading https://sum.golang.org/lookup/gitlab.com/mycompany/myproject#v0.0.145: 410 Gone
I have no idea why it is trying to ping sum.golang.org/lookup since it is a private gitlab project.
My ~/.gitconfig contains the following (based on my looking up in google search for similar errors)
# Enforce SSH
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
[url "git#gitlab.com:"]
insteadOf = https://gitlab.com/
The error still persists.
I would expect the package to be downloaded from my private gitlab project repository to the current project.
Is there anything I need to do in my private gitlab project repository to make it ready for 'go get' ?
The private gitlab project repository already contains the go.sum and go.mod for the project as well.
Anything that I am missing ?
edit: 1) The private repo name and the company name contains no asterisks or any other special characters. only alphabets and not even numeric characters.
Answering my own question after looking up,
Setting the GOPRIVATE variable seems to help.
GOPRIVATE=gitlab.com/mycompany/* go mod why
"
The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database.
" from https://golang.org/doc/go1.13
Aliter:
Setting the env variable GONOSUMDB also seems to work.
Specifically, invoking the following command seems to help.
GONOSUMDB=gitlab.com/mycompany/* go mod why
The above env variable prevents the ping to sum.golang.org/lookup for a checksum match. It also prevents leaking the names of private repos to a public checksum db. [ Source - https://docs.gomods.io/configuration/sumdb/ ]
Also - here at
* GONOSUMDB=prefix1,prefix2,prefix3 sets a list of module path prefixes, again possibly containing globs, that should not be looked up using the database.
source: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md
Related Issues:
https://github.com/golang/go/issues/32291
https://github.com/golang/go/issues/33985
["Go 1.13 has been released, and this issue was filed well after the freeze window. The proposed changes will not happen in 1.13, but don't assume they will necessarily happen in 1.14 either." from issue 33985 above. ]
Basically it failed to verify private repository. However I don't like turning off checksum, but you can easily set GOSUMDB to off before trying to get module. something like this:
GOSUMDB=off go get github.com/mycompany/myproject
ref: https://github.com/golang/go/issues/35164#issuecomment-546503518
A second and better solution is to set GOPRIVATE environment variable that controls which modules the go command considers to be private (not available publicly) and should therefore NOT use the proxy or checksum database. The variable is a comma-separated list of glob patterns (same syntax of Go's path.Match) of module path prefixes. For example,
export GOPRIVATE=*.corp.example.com,rsc.io/private
Or
go env -w GOPRIVATE=github.com/mycompany/*
Last solution you can try is to turn off such checks for all private repositories that you don't want to go public or being verified through sum.golang.org/lookup/github.com/mycompany/...
GONOSUMDB=gitlab.com/mycompany/* go mod why
Note that:
If you have issues fetching modules or repos over https, you may want to add the following to your ~/.gitconfig to make go get/fetch repositories using ssh instead of https
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
Change following go variable's setting and then upgrade your package,
$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOSUMDB=off
$ go get -u <your dependency package>
I have this scenario too and this works for me.
edit your .git/config and add two lines in it.( I have this in a global .gitconfig in home dir)
[url "ssh://youprivate.com"]
insteadOf = https://yourprivate.com
export GOSUMDB=off
Then everything will OK.
I think the below should work based on what I've read but I get an error. I'm surely not expected hard code my password in .netrc. I've ran heroku:set JSPM_GITHUB_AUTH_TOKEN obvs! I've not base64 encoded the token, maybe thats the problem
heroku run jspm config registries.github.auth $JSPM_GITHUB_AUTH_TOKEN
$ heroku run jspm init
Running jspm init on test... up, run.7011
ok Verified package.json at package.json
Verified config file at frontend/system.config.js
Looking up loader files...
warn Error on download for github:systemjs/systemjs
Unauthorized response for GitHub API.
Use jspm registry config github to reconfigure the credentials, or update them in your ~/.netrc file.
err TypeError: Cannot read property 'toString' of undefined
at Object.exports.log (/app/node_modules/jspm/lib/ui.js:99:27)
at /app/node_modules/jspm/lib/core.js:361:8
at process._tickCallback (node.js:377:9)
I am trying to install a github project using composer and get the following error
Composer [UnexpectedValueException]
Your Github oauth token for github.com contains invalid characters: ""
Can anyone explain what I need to do to correct this error?
I am using the following command
composer create-project --prefer-dist --stability=dev vova07/yii2-start yii2-start
Thank you
I started getting a similar error and the reason was that Github recently changed the format of their auth tokens:
https://github.blog/changelog/2021-03-31-authentication-token-format-updates-are-generally-available/
To resolve the error:
Find the composer/auth.json file (if you're running the project in a container, you'll have to bash into it and find the file in there)
Remove its github.com entry. Your file will probably look like the following after removing the entry: {"github-oauth": {}}
Run composer self-update. The issue got resolved in version 2.0.12. See the first item in the changelog for that version here: https://getcomposer.org/changelog/2.0.12
After that, you can restore your composer/auth.json file to its initial state as the newer version of composer will recognize the new key format.
You can try Basic Auth instead:
Change this (oauth):
"github-oauth": {
"github.com": "ghp_[YOUR-PERSONAL-TOKEN]"
}
To this (basic auth):
"http-basic": {
"github.com": {
"username": "[YOUR-GITHUB-USERNAME]",
"password": "ghp_[YOUR-PERSONAL-TOKEN]"
}
}
You can find instructions on how to create a Personal Access Token
Inspired from github docs. Apparently, you can use Basic Authentication with a Personal Access token instead of oauth in some cases (e.g. like mine: installing a private git repo with composer).
I fixed it.
Goto C:\Users\XXXXX\AppData\Roaming\Composer
Open the auth.json
delete the github.com entry under "github-oauth": {}
That's it.
Update answer for Masiorama and Ruchir Mehta:
If you looking for file auth.json but don't know how, use this command:
locate auth.json
And here's the result:
You can see that auth.json will look like this:
/home/{your user name}/.config/composer/auth.json
Then you could use this command to edit the file:
sudo gedit /home/dev/.config/composer/auth.json
And remove content inside github-oauth.
If you're on MacOS, the auth.json file is at ~/.composer/auth.json. Then from there, you can remove the value for github-oauth. I tried fully deleting the file but I got a parse error, Expected one of: 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['. Your auth.json file should look like this:
{
"github-oauth": {}
}
This is similar to other answers posted but I wasn't able to use the locate command on MacOS so this might be helpful to other Mac users
This error recently popped up from nowhere.
Simply deleting the whole auth file worked for me..! Not sure why / when it appeared in the first place.
~/.composer/auth.json
As far as I know (I'm a beginner with composer too), the problem is with your authentication, so you have to fix your credentials in auth.json inside path-to-composer/.composer/
Inside you will find a json which will probably looks like:
{
"github-oauth": {
"github.com": null
}
}
Fix that and you should be ok ;)
The solution is just to upgrade your Composer version
using command composer self-update.
Go to C:\Users\UserName\AppData\Roaming\Composer
Open the auth.json file.
Clear everything and paste the below code
{
"bitbucket-oauth": {},
"github-oauth": {},
"gitlab-oauth": {},
"gitlab-token": {},
"http-basic": {},
"bearer": {}
}
I hope it will be solved
I run in the same problem after upgrading githup api token to the new format.
The answer is you need to upgrade composer version 1.10.21 or higher that fixes this problem.
Same solution as the answer of Paulina Khew but with command lines on MacOS :
cd ~/.composer/
nano auth.json
Delete what is inside th bracket :
{
"github-oauth": {}
}
When you're ready to save the file, hold down the Ctrl key and press the letter O
Press the Enter key on your keyboard to save.
When finished, press Ctrl + X to close nano and return to your shell.
Edit the composer authentication configuration file ~/.composer/auth.json
Then replace the following.
"http-basic": {
"github.com": {
"username": "[YOUR-GITHUB-USERNAME]",
"password": "ghp_[YOUR-PERSONAL-TOKEN]"
}
}
Now run the command composer install
That's a bug.
If you have Debian or Ubuntu, try this patch. Otherwise read the last line.
Quick copy-paste patch
If you have Debian 10 buster or Ubuntu 20.LTS or similar distributions, try this copy-paste command:
wget https://gist.githubusercontent.com/valerio-bozzolan/84364c28a3bba13751c504214016adcf/raw/c1356d529c89c10de4c959058e2e86ffe58fa407/fix-composer.patch -O /tmp/fix-composer.patch
sudo patch /usr/share/php/Composer/IO/BaseIO.php /tmp/fix-composer.patch
If it does not work, write it in the comments.
Step-by-step explaination
Your Composer version has a bug: you are able to save a valid GitHub token, but then it's not able to read that token again because Composer thinks that your GitHub token cannot contain underscores or stuff like that. Moreover, it's strange that Composer checks its syntax only the second time. Why? that's another story.
The fix is simple. You can temporary disable that wrong validation in your Composer version. Also because GitHub is a proprietary service and their specifications can change over time (as you demonstrated today). So it makes sense not to validate the syntax of GitHub tokens. The only person who should hard-validate GitHub tokens is GitHub itself, not Composer.
If you installed Composer via apt install composer, probably you will not have any update available and surely you cannot use self-update because Composer is read-only for security reasons (and for a similar reason, you should not execute Composer from root). Instead, you can create a safe hot-patch to fix that specific issue.
To create a patch, create a file called /tmp/fix-composer.patch with this exact content:
103,105c103,105
< if (!preg_match('{^[.a-z0-9]+$}', $token)) {
< throw new \UnexpectedValueException('Your github oauth token for '.$domain.' contains invalid characters: "'.$token.'"');
< }
---
> // if (!preg_match('{^[.a-z0-9]+$}', $token)) {
> // throw new \UnexpectedValueException('Your github oauth token for '.$domain.' contains invalid characters: "'.$token.'"');
> //
That content can also be seen from here:
https://gist.github.com/valerio-bozzolan/84364c28a3bba13751c504214016adcf
Then run this command to apply that patch:
sudo patch /usr/share/php/Composer/IO/BaseIO.php /tmp/fix-composer.patch
If it does not work, probably you have not installed composer via apt.
In short, whatever operating system, and whatever installation method, locate the file BaseIO.php in your Composer and comment out the validation check.