I cannot get this to work, even after checking other topics on stackoverflow. My project on gitlab.com/my_company/backend needs a module, found at gitlab.com/my_company/pkg/auth.
Locally, I've setup GOPRIVATE / git's configuration to make it work (and it works), though in Gitlab's CI Pipelines on a merge request, this fails.
Pipeline log / go.mod
I've added some debugging logs just to make sure everything was setup like I thought. Here's a failing pipeline's log :
$ git config --global url."ssh://git#gitlab.com/my_company/".insteadOf "https://gitlab.com/my_company/"
$ git config --global url."git#gitlab.com:".insteadOf "https://gitlab.com/"
$ git config -l | grep instead
url.ssh://git#gitlab.com/my_company/.insteadof=https://gitlab.com/my_company/
url.git#gitlab.com:.insteadof=https://gitlab.com/
$ env | grep GOPRIVATE
GOPRIVATE=gitlab.com/my_company
$ go mod download
go: gitlab.com/my_company/pkg/auth#v1.1.0: reading gitlab.com/my_company/pkg/auth/auth/go.mod at revision auth/v1.1.0: unknown revision auth/v1.1.0
One weird part of this log I've found is :
reading gitlab.com/my_company/pkg/auth/auth/go.mod - why is it repeating auth/auth? It actually happened once before locally, but it was because I wrote "github" instead of "gitlab" :)
The relevant go.mod line just in case :
require (
gitlab.com/my_company/pkg/auth v1.1.0 // indirect
)
Repository tags
Here are the tags setup on the repository gitlab.com/my_company/pkg :
$ git tag -l
auth/v1.0.0
auth/v1.1.0
cache/v1.0.0
cache/v1.0.1
$ git ls-remote --tags
From git#gitlab.my_company/pkg.git
9efcb02d5489adaac9d525dcb496d868d65e856a refs/tags/auth/v1.0.0
13730d4f61df978c6d690fd2678e2ed924808e0c refs/tags/auth/v1.1.0
2b8dff0ec1b737d975290720933180a9b591a1db refs/tags/cache/v1.0.0
9a3e598bbf83bea57b29d8a908b514861ae37b12 refs/tags/cache/v1.0.1
I'm not that familiar with Gitlab CI so I'm out of things to try. Any ideas?
Thank you!
Update: I finally got gitlab-runner installed so I could try running the yml directly, no luck. It still works locally (not a big surprise).
In you project should be file .gitlab-ci.yml and you can add GOPRIVATE variable to your CI and runner will use it for you project.
More details how to add env vars:
https://docs.gitlab.com/ee/ci/variables/#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file
I'm trying to use a private theme/module with a personal access token. I can get this working by adding the following to my global git config.
git config --global url."https://{USER}:{TOKEN}#github.com".insteadOf "https://github.com"
Then running hugo mod get -u it will pull changes as expected.
I don't want this set in my global config and if I set it locally I get an error, because Go doesn't seem to be using the local config.
Set my configurations locally within the root of the site/repository:
git config --local url."https://{USER}:{TOKEN}#github.com".insteadOf "https://github.com"
Then running hugo mod get -u I get the following error:
go get: module github.com/USER/REPOSITORY: git ls-remote -q origin in /var/folders/26/gqnv01_55p964v8yz39d51fw0000gn/T/hugo_cache/modules/filecache/modules/pkg/mod/cache/vcs/b410fc7b91fbc1121b5f6ec2bb2711c27cd172b4084c213e1430a33cde552597: exit status 128:
remote: Repository not found.
fatal: repository 'https://github.com/USER/REPOSITORY/' not found
How can I get Go/Hugo to use my local git config rather than the global?
From the hugo mod source code, hugo will look for a go.mod in your project:
filepath.Walk(dirname, func(path string, info os.FileInfo, err error) error {
if info.IsDir() {
return nil
}
if info.Name() == "go.mod" {
// Found a module.
dir := filepath.Dir(path)
fmt.Println("Update module in", dir)
Check where your go.mod is, and do (in that go.mod parent folder):
git config -l --show-origin --show-scope
That will tell you if your expected local config is actually there or not.
Look for any .git folder which would indicate a nested git repository/submodule, which would ignore your initial git config --local command
An issue like 34513 seems to suggests though that go mod won't take into account the local repository:
The git configuration only affects operations on the underlying git repo.
The error that you're seeing is coming from before that, when the go command is attempting to resolve the repo for the requested package path.
The official documentation only references the global config .gitconfig.
I solved this by adding a directory replacement mapping to the site’s config, instead of modifying the git url. This points to my locally cloned theme and updates the served site whenever I modify the theme.
module:
imports:
path: 'github.com/[USER]/[REPO-NAME]'
replacements: 'github.com/[USER]/[REPO-NAME] -> ../../[REPO-NAME]/'
I am trying to build a go application that makes use of git repositories that are private. I have an ssh key for my github account and have the following in my .gitconfig file:
[url "https://username:token#github.com"]
insteadOf = https://github.com
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
when I execute go test or go build I am asked for the passphrase. I then get a response:
go: found github.com/x/businesses in github.com/x/businesses v0.0.0-yy
go: cmd/main/cmd imports
github.com/x/businesses: github.com/x/businesses#v0.0.0-yy/go.mod:
verifying module: github.com/x/businesses#v0.0.0-yy/go.mod:
reading https://sum.golang.org/lookup/github.com/xx/businesses#v0.0.0-yy: 410 Gone
server response:
not found: github.com/x/businesses#v0.0.0-yy:
invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*
in tmp/gopath/pkg/mod/cache/vcs/zz:
exit status 128:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
I tried removing the top insteadOf in the .gitconfig for no reason other then to try something.
Your git config should look something like
[url "git#github.com:"] insteadOf = https://github.com/
or similar to what you have.
Then you can go get your pacakge by telling Go that your using a private repo like this:
GOPRIVATE="github.com/your_username_or_org" go get github.com/name_or_org/repo_name
Also, I usually dont put a passprahse on my SSH keys since is kinda annoying to type it in each time but of course is more secure by adding it like someone pointed out in the comments.
I am using golang 1.13 .
I have a project that depends on a private gitlab project.
I have the ssh keys for the same.
When I try to retrieve the dependencies for a newly created module, I am getting the following error:
$ go version
go version go1.13 linux/amd64
$ go mod why
go: downloading gitlab.com/mycompany/myproject v0.0.145
verifying gitlab.com/mycompany/myproject#v0.0.145: gitlab.com/mycompany/myproject#v0.0.145: reading https://sum.golang.org/lookup/gitlab.com/mycompany/myproject#v0.0.145: 410 Gone
I have no idea why it is trying to ping sum.golang.org/lookup since it is a private gitlab project.
My ~/.gitconfig contains the following (based on my looking up in google search for similar errors)
# Enforce SSH
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
[url "ssh://git#gitlab.com/"]
insteadOf = https://gitlab.com/
[url "ssh://git#bitbucket.org/"]
insteadOf = https://bitbucket.org/
[url "git#gitlab.com:"]
insteadOf = https://gitlab.com/
The error still persists.
I would expect the package to be downloaded from my private gitlab project repository to the current project.
Is there anything I need to do in my private gitlab project repository to make it ready for 'go get' ?
The private gitlab project repository already contains the go.sum and go.mod for the project as well.
Anything that I am missing ?
edit: 1) The private repo name and the company name contains no asterisks or any other special characters. only alphabets and not even numeric characters.
Answering my own question after looking up,
Setting the GOPRIVATE variable seems to help.
GOPRIVATE=gitlab.com/mycompany/* go mod why
"
The new GOPRIVATE environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level GONOPROXY and GONOSUMDB variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database.
" from https://golang.org/doc/go1.13
Aliter:
Setting the env variable GONOSUMDB also seems to work.
Specifically, invoking the following command seems to help.
GONOSUMDB=gitlab.com/mycompany/* go mod why
The above env variable prevents the ping to sum.golang.org/lookup for a checksum match. It also prevents leaking the names of private repos to a public checksum db. [ Source - https://docs.gomods.io/configuration/sumdb/ ]
Also - here at
* GONOSUMDB=prefix1,prefix2,prefix3 sets a list of module path prefixes, again possibly containing globs, that should not be looked up using the database.
source: https://go.googlesource.com/proposal/+/master/design/25530-sumdb.md
Related Issues:
https://github.com/golang/go/issues/32291
https://github.com/golang/go/issues/33985
["Go 1.13 has been released, and this issue was filed well after the freeze window. The proposed changes will not happen in 1.13, but don't assume they will necessarily happen in 1.14 either." from issue 33985 above. ]
Basically it failed to verify private repository. However I don't like turning off checksum, but you can easily set GOSUMDB to off before trying to get module. something like this:
GOSUMDB=off go get github.com/mycompany/myproject
ref: https://github.com/golang/go/issues/35164#issuecomment-546503518
A second and better solution is to set GOPRIVATE environment variable that controls which modules the go command considers to be private (not available publicly) and should therefore NOT use the proxy or checksum database. The variable is a comma-separated list of glob patterns (same syntax of Go's path.Match) of module path prefixes. For example,
export GOPRIVATE=*.corp.example.com,rsc.io/private
Or
go env -w GOPRIVATE=github.com/mycompany/*
Last solution you can try is to turn off such checks for all private repositories that you don't want to go public or being verified through sum.golang.org/lookup/github.com/mycompany/...
GONOSUMDB=gitlab.com/mycompany/* go mod why
Note that:
If you have issues fetching modules or repos over https, you may want to add the following to your ~/.gitconfig to make go get/fetch repositories using ssh instead of https
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
Change following go variable's setting and then upgrade your package,
$ export GO111MODULE=on
$ export GOPROXY=direct
$ export GOSUMDB=off
$ go get -u <your dependency package>
I have this scenario too and this works for me.
edit your .git/config and add two lines in it.( I have this in a global .gitconfig in home dir)
[url "ssh://youprivate.com"]
insteadOf = https://yourprivate.com
export GOSUMDB=off
Then everything will OK.
I'm trying to use Bower for a web app, but find myself hitting some sort of proxy issues:
D:\>bower search jquery
bower retry Request to https://bower.herokuapp.com/packages/search/jquery failed with ECONNRESET, retrying in 1.2s
bower retry Request to https://bower.herokuapp.com/packages/search/jquery failed with ECONNRESET, retrying in 2.5s
bower retry Request to https://bower.herokuapp.com/packages/search/jquery failed with ECONNRESET, retrying in 6.8s
bower retry Request to https://bower.herokuapp.com/packages/search/jquery failed with ECONNRESET, retrying in 15.1s
bower retry Request to https://bower.herokuapp.com/packages/search/jquery failed with ECONNRESET, retrying in 20.3s
bower ECONNRESET Request to https://bower.herokuapp.com/packages/search/jquery failed: tunneling socket could not be established, cause=Parse Error
Relevant points:
I can browse to https://bower.herokuapp.com/packages/search/jquery and it returns a full json response.
I can use git to clone, both using the git:// protocol and http(s).
I can use NPM directly without these issues
I've tried using Fiddler to determine what's being blocked, but it doesn't detect any calls from the Bower command. I can see calls from NPM commands in Fiddler.
I've searched the Bower issues list, seen similar issues, but they either have no solution or it doesn't seem quite the same as mine.
Any ideas?
Thanks #user3259967
This did the job.
I would like to add that if you are behind a proxy that needs to be authenticated, you can add the username/password to your .bowerrc file.
{
"directory": "library",
"registry": "http://bower.herokuapp.com",
"proxy":"http://<USERNAME>:<PASSWORD>#<PROXY_IP>:<PROXY_PORT>/",
"https-proxy":"http://<USERNAME>:<PASSWORD>#<PROXY_IP>:<PROXY_PORT>/"
}
NOTICE the use of http:// in https-proxy
The solution for me is this config .bowerrc
{
"directory": "vendor",
"registry": "http://bower.herokuapp.com",
"proxy": "http://<user>:<pwd>#proxy.host.br:8080",
"https-proxy": "http://<user>:<pwd>#proxy.host.br:8080",
"strict-ssl": false
}
Using the http protocol in https-proxy plus registry entry with http protocol.
Remember to change 8080 port number to whatever is yours proxy server port.
Are you behind a proxy?
Have you set up environment variables HTTP_PROXY and HTTPS_PROXY?
SET HTTP_PROXY=http://yourproxyserver:yourproxyport
SET HTTPS_PROXY=http://yourproxyserver:yourproxyport
Try changing the registry value in your .bowerrc:
{
"registry": "http://bower.herokuapp.com"
}
I did not have a .bowerrc file to configure my bower settings. I found the settings living in a file called defaults.js. found under "C:\...\bower\node_modules\bower-config\lib\util\defaults.js"
I hope this helps others:
var defaults = {
'cwd': process.cwd(),
'directory': 'bower_components',
'registry': 'http://bower.herokuapp.com',
'shorthand-resolver': 'git://github.com/{{owner}}/{{package}}.git',
'tmp': paths.tmp,
'proxy': '<<http://user:pass#proxy:port>>', // change proxy here or at the top
'https-proxy': '<<http://user:pass#proxy:port>>', // change proxy here or at the top
'timeout': 30000,
'ca': { search: [] },
'strict-ssl': false,
'user-agent': userAgent,
'color': true,
'interactive': null,
'storage': {
packages: path.join(paths.cache, 'packages'),
links: path.join(paths.data, 'links'),
completion: path.join(paths.data, 'completion'),
registry: path.join(paths.cache, 'registry'),
empty: path.join(paths.data, 'empty') // Empty dir, used in GIT_TEMPLATE_DIR among others
}
};
module.exports = defaults;
you can try suggest #thebignet into same issue on GitHub
Set proxy, https-proxy and strict-ssl configuration into .bowerrc File :
{
"proxy" : "http://<host>:<port>",
"https-proxy" : "http://<host>:<port>",
"strict-ssl" : false
}
But you must run Command from terminal:
git config --global url."https://".insteadOf git://
"strict-ssl": false
in .bowerrc did for me
For Win 7.
What worked for me , are below steps as suggested at this link - read #nanowizard answer.
In .bowerrc file, remove any http_proxy / https_proxy settings that
you might have done earlier. This is important.
So final content of this file should look like :-
{
"directory": "app/bower_components"
}
Set environment variables in your pc - http_proxy and https_proxy to your corporate firewall proxy. In case, your corporate proxy requires authentication and if your password contains special characters, convert it to hex form as suggested by this link. As in my case escaping of characters with '\' did not help. Also I had to restart my system.
Note :
http_proxy and https_proxy should contain same proxy address as shown below
http_proxy = http://<user>:<password>#<your company proxy>:<port>
https_proxy= http://<user>:<password>#<your company proxy>:<port> ->Note no 's' in http://...
I am behind corporate firewall and I have to specify domain name too.
None of these answers worked for me. Here is what I did -
Downloaded CNTLM from http://cntlm.sourceforge.net/
Obviously installed it.
Open up cntml.ini and change the following
Domain your_domain_name
Username your_domain_username
Password your_domain_passowrd
PassLM 1AD35398BE6565DDB5C4EF70C0593492 (uncomment this)
PassNT 77B9081511704EE852F94227CF48A793 (uncomment this too)
Proxy http://localhost:8888
Go to services.msc and start the CNTLM Authentication service.
Download Fiddler 4/2 (whatever they call it).
Install this too. This will run in http://localhost:8888
Now whatever program you’re running forward(proxy) it to http://locahost:3128 ( that’s what CNTLM is running.)
In this case specify http.proxy and https.proxy as http://localhost:8888
This will work for other client programs. Just specify proxy as http://localhost:8888
its work for me to change in .bowerrc file
{
"directory": "client/lib",
"registry": "http://bower.herokuapp.com",
"proxy":"http://192.168.1.205:3228",
"https-proxy":"http://192.168.1.205:3228"
}
where client/lib is installation directory where do you want to install
and http://192.168.1.205:3228 is your proxy ip with port. corporate proxy can be different according to oraganization.
In addition to setting the below proxy in .bowerrc:
{
"directory": "app/bower_components",
"proxy":"http://<user>:<password>#proxy.company.com:<proxy-port>",
"https-proxy":"http://<user>:<password>#proxy.company.com:<proxy-port>",
"http-proxy":"http://<user>:<password>#proxy.company.com:<proxy-port>",
"strict-ssl": false,
"registry": "http://bower.herokuapp.com"
}
I am required to run the following commands to fix the issue:
npm cache clean
bower cache clean
bower install
The registry used in the the other answers is now deprecated. Please update it!
{
"proxy":"http://<user>:<password>#proxy.company.com:<proxy-port>",
"https-proxy":"http://<user>:<password>#proxy.company.com:<proxy-port>",
"registry": "https://registry.bower.io"
}
{
"directory": "library",
"registry": "http://bower.herokuapp.com",
"proxy":"http://<USERNAME>:<PASSWORD>#<PROXY_IP>:<PROXY_PORT>/",
"https-proxy":"http://<USERNAME>:<PASSWORD>#<PROXY_IP>:<PROXY_PORT>/"
}
This code worked for me. I am using Win 7 and chrome and also git bash. Here few things need to be cleared. This takes me huge time to find the actual data regarding the user name, password, proxy IP and Port. I will describe it step by step so that every learners can easily grasp the message:
Create a file in the notepad named .bowerrc in the login folder; You can go there by typing at Start>Run>%UserProfile% and press OK.
Type above code in the .bowerrc file with the following changes:
Replace <USERNAME> with your internet connection user ID or login ID
Replace <PASSWORD> with your internet connection password or login password.
Replace <PROXY_IP> and <PROXY_PORT> with the working proxy IP address and its port number.
**Note: There should be no angle brackets.**
Proxy IP should be different than your own IP.
Before using any proxy IP and port you should check it is working by changing your proxy IP and port.
You can go through this link to know the details of proxy settings at description here
From this proxy settings you will get Proxy IP and Port.
Recheck all the input so that all are correct and save and close the file.
Open git bash and change directory to the project file and type command and hit enter, in my case, git bash command:
a#a-PC MINGW32 /d/conFusion
$ bower install
It worked like magic.
In case it helps someone, I had a 'bower blocked by group policy' error.
Solution was to make an exception in CryptoPrevent, a application installed on our company computers to prevent crypto lockers.
For info, in your .bowerrc file you can add a no-proxy attribute. I don't know since when it is supported but it works on bower 1.7.4 and it solve the issue of bower behind a corporate proxy with an internal repository
.bowerrc :
{
"directory": "bower_components",
"proxy": "http://yourProxy:yourPort",
"https-proxy":"http://yourProxy:yourPort",
"no-proxy":"myserver.mydomain.com"
}
Regards
Please make sure there are no special characters in your proxy password. Convert it to hex. It works for me.