go get and AzureDevOps authentication not working - go

I am desperately trying to figure out why a go get command from a package residing inside a Git repo on AzureDevOps does not work.
Both golang projects (the project where the package resides as well as the consumer trying to import it) are located inside the same AzureDevOps project.
I am attempting to import a package from the one into the other but alas with no avail.
I followed the instructions found here: https://learn.microsoft.com/en-us/azure/devops/repos/git/go-get?view=azure-devops
I have created a PAT and tested that I am able to clone the repo from where I want to import the package from, and that works fine.
git clone https://me:78rt5u89dg635ef4fb30e4b0229e4329756986416673b4e331ag#dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
So I have in my mind (maybe incorrectly so) made the assumption that the PAT is able to authenticate correctly. That said, I unfortunately have to use cntlm proxy to get to the outside, but this works well in general.
In my golang project inside the terminal I have set : export GOPRIVATE=dev.azure.com
In my GitConfig file located at location ~/.gitconfig I have set the following values:
# This is Git's per-user configuration file.
[user]
email = me#my-comp.com
name = Morné Kruger
[core]
editor = vi
[url https://me:78rt5u89dg635ef4fb30e4b0229e4329756986416673b4e331a#dev.azure.com/my-comp/my-proj/_git/k8s-team-operator]
insteadOf = https://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
I then run the go get command with the verbose flag, and then get the following less positive response:
me#focalfossa:~/dev/my-comp/repos/k8s-teamapi-functions$ go get -x dev.azure.com/my-comp/my-proj/_git/k8s-team-operator.git
# get https://dev.azure.com/my-comp/my-proj/_git?go-get=1
# get https://dev.azure.com/?go-get=1
# get https://dev.azure.com/my-comp?go-get=1
cd .
git ls-remote https://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
# get https://dev.azure.com/my-comp/my-proj?go-get=1
cd .
git ls-remote git+ssh://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
# get https://dev.azure.com/?go-get=1: 403 Forbidden (0.274s)
# get https://dev.azure.com/my-comp/my-proj?go-get=1: 203 Non-Authoritative Information (0.307s)
# get https://dev.azure.com/my-comp?go-get=1: 203 Non-Authoritative Information (0.308s)
# get https://dev.azure.com/my-comp/my-proj/_git?go-get=1: 203 Non-Authoritative Information (0.311s)
cd .
git ls-remote ssh://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
mkdir -p /home/me/go/pkg/mod/cache/vcs # git3 git://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
# lock /home/me/go/pkg/mod/cache/vcs/540338d35ef4fb30e4b0229e4329756986416673b4e3318516900fca1efcee17.lock# /home/me/go/pkg/mod/cache/vcs/540338d35ef4fb485763tg30417801e630216673b4e3318516900fca1efcee17 for git3 git://dev.azure.com/my-comp/my-proj/_git/k8s-team-operator
cd /home/me/go/pkg/mod/cache/vcs/540338d35ef4fb30e4b02906874yh25630216673b4e3318516900fca1efcee17; git ls-remote -q origin
2.083s # cd /home/me/go/pkg/mod/cache/vcs/540338d35ef49059y87e329e417801e630216673b4e3318516900fca1efcee17; git ls-remote -q origin
go get: module dev.azure.com/my-comp/my-proj/_git/k8s-team-operator.git: git ls-remote -q origin in /home/me/go/pkg/mod/cache/vcs/540338d35ef4fb30e4b022125f7y879430216673b4e3318516900fca1efcee17: exit status 128:
fatal: unable to connect to dev.azure.com:
dev.azure.com[0: 13.107.42.20]: errno=Connection refused
dev.azure.com[1: 2620:1ec:21::20]: errno=Network is unreachable
Versions:
go1.16.5 linux/amd64
Ubuntu 20.04
GoLand 2021.1.2
git version 2.25.1
I would greatly appreciate any tips or troubleshooting ideas that could point out the error in my ways!

Related

cloud build trigger based on github commits does not work

i have the following build.yaml to build an image for my project
steps:
# Build the container image with the Python version of our choice.
- name: gcr.io/cloud-builders/docker
args:
[ 'build'
, '--tag=gcr.io/$PROJECT_ID/$_IMAGE_NAME'
, '--file=./Dockerfile_pipeline'
, '.'
]
images: ['gcr.io/$PROJECT_ID/$_IMAGE_NAME']
substitutions:
_IMAGE_NAME: pipeline:latestRun
This works fine when kicked off in gcp console via
gcloud beta builds submit --verbosity debug --config build.yaml
However, when i create a cloudbuild trigger (based on commits on my repo) that kicks off my build.yaml i am getting the following error
starting build "4cc47363-32f5-4862-8533-90810c972c5a"
FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/mmistroni/GCP_Experiments
* branch 450bfce1f17042d3163e39a3ecad89d06922ea47 -> FETCH_HEAD
HEAD is now at 450bfce adding marketloader trigger
BUILD
Already have image (with digest): gcr.io/cloud-builders/docker
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /workspace/Dockerfile_pipeline: no such file or directory
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1
What am i missing?
kind regards
marco
i found the solution after fiddling for a while
When the trigger is kicked off from github commit, the root directory is the root my repo, GCP_Experiments
the dockerfile instead resides in GCP_Experiments\dataflow\pipeline\Dockerfile_pipeline
So i had to do two fixes
1 - put the full path of the docker file as --file argument
2 - specify, as a PATH parameter, the directory where file resides relative to root repo, aka ./dataflow/pipeline
rgds

golang import github private repo

Completely confused.
go 1.13.5 using modules
I am trying to import a private repo from github. Searching is increasing the confusion not reducing it. I have tried a number of approaches including the keychain helper but I am going nowhere fast.
SSH is enabled and when tested establishes a connection successfully
My global gitconfig in C:/Users/me :
[user]
name = xxxxxxx
email = xxx.xxx#me.com
[core]
autocrlf = input
[alias]
st = status
[url "ssh://git#github.com/"]
insteadOf = https://github.com/
go get github.com/user/repo#master produces
go: finding github.com master
go: finding github.com/user master
go: finding github.com/user/repo master
go: finding github.com/user/repo master
go: downloading github.com/user/repo v0.0.0-20191211180807-ee1bcd94c84f
verifying github.com/user/repo#v0.0.0-20191211180807-ee1bcd94c84f:
github.com/user/repo#v0.0.0-20191211180807-ee1bcd94c84f: reading
https://sum.golang.org/lookup/github.com/!user/repo#v0.0.0-20191211180807-
ee1bcd94c84f:410 Gone
The link yields :
not found: github.com/user/repo#v0.0.0-20191211180807-ee1bcd94c84f: invalid version: git
fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in/tmp/gopath/pkg/mod/cache/vcs/9524fc42cfd4910346f55f112665f9a51df7c4b31085d50baa5e01453e55ca58:
exit status 128:
fatal: could not read Username for 'https://github.com': terminal prompts disabled
Doesn't matter what I do I get this "cannot read Username" but I thought it should be using SSH not HTTPS
What's going on??
Not feeling the love for Modules making my life easier so far...
After more searching and much experimentation, the issue is with the checksum failing on download of a private repo as by definition there is no checksum defined.
This was buried within a chain of responses on Github. There is an open issue to improve the documentation. Good idea.
This worked for me:
GONOSUMDB=github.com/username/*
go get github.com/username/repo

Unable to install package

Initially posted this on the Ethereum forum but feel that there might be more knowledge of the issue here as it is a Golang/git package installation issue
I am trying to compile smart contracts using the go-ethereum client.
I have go installed on my pc and installed ran the following commands:
go get -u github.com/ethereum/go-ethereum
cd $GOPATH/src/github.com/ethereum/go-ethereum/
make
Which all work fine.
When I run
make devtools
I get the following error:
Pros-MacBook-Pro:go-ethereum Santi$ make devtools
env GOBIN= go get -u golang.org/x/tools/cmd/stringer
# cd /Users/pro/go/src/golang.org/x/tools; git pull --ff-only
error: Your local changes to the following files would be overwritten by merge:
CONTRIBUTING.md
blog/blog.go
cmd/godoc/autocert.go
cmd/godoc/blog.go
cmd/godoc/godoc_test.go
cmd/godoc/handlers.go
cmd/godoc/main.go
cmd/godoc/x.go
cmd/goimports/goimports.go
cmd/gomvpkg/main.go
cmd/goyacc/yacc.go
cmd/guru/guru.go
cmd/guru/guru_test.go
cmd/guru/main.go
cmd/guru/referrers.go
cmd/guru/testdata/src/referrers/int_test.go
cmd/guru/testdata/src/referrers/main.go
cmd/guru/testdata/src/referrers/main.golden
cmd/present/static/styles.css
cmd/present/templates/slides.tmpl
cmd/stringer/endtoend_test.go
cmd/stringer/stringer.go
cmd/tip/tip.go
go/ast/astutil/imports.go
go/ast/astutil/imports_test.go
go/buildutil/allpackages_test.go
go/buildutil/fakecontext.go
go/buildutil/util_test.go
go/gcexportdata/example_test.go
go/gcexportdata/gcexportdata.go
go/gcexportdata/main.go
go/gcimporter15/bexport.go
go/gcimporter15/bexport19_test.go
go/gcimporter15/bexport_test.go
go/gcimporter15/bimport.go
go/gcimporter15/exportdata.go
go/gcimporter15/gcimporter.go
go/gcimporter15/gcimporter_test.go
go/gcimporter15/isAlias18.go
go/gcimporter15/isAlias19.go
go/gcimporter15/testdata/a.go
go/gcimporter15/testdata/b.go
go/gcimporter15/testdata/exports.go
go/gcimporter15/testdata/issue15920.go
go/gcimporter15/testdata/issue20046.go
go/gcimporter15/testdata/p.go
go/gcimporter15/testdata/versions/test.go
go/gcimporter15/testdata/versions/test_go1.7_0.a
go/gcimporter15/testdata/versions/test_go1.7_1.a
go/internal/gccgoimporter/gccgoinstallation_test.go
go/internal/gccgoimporter/importer_test.go
go/internal/gccgoimporter/parser.go
go/loader/loader_test.go
go/ssa/interp/external.go
go/ssa/ssa.go
go/vcs/vcs.go
go/vcs/vcs_test.go
godoc/cmdline.go
godoc/cmdline_test.go
godoc/dirtrees.go
godoc/dl/dl.go
godoc/godoc.go
godoc/index.go
godoc/pres.go
godoc/server.go
godoc/spec.go
godoc/static/gen.go
godoc/static/godoc.html
godoc/static/godocs.js
godoc/static/makestatic.go
godoc/static/package.html
godoc/static/playground.js
godoc/static/static.go
godoc/static/style.css
godoc/template.go
godoc/vfs/emptyvfs.go
godoc/vfs/gatefs/gatefs.go
godoc/vfs/mapfs/mapfs.go
godoc/vfs/namespace.go
godoc/vfs/os.go
godoc/vfs/vfs.go
godoc/vfs/zipfs/zipfs.go
imports/fastwalk.go
imports/fastwalk_dirent_fileno.go
imports/fastwalk_dirent_ino.go
imports/fastwalk_portable.go
imports/fastwalk_test.go
imports/fastwalk_unix.go
imports/fix.go
imports/fix_test.go
imports/imports.go
imports/sortimports.go
present/parse.go
refactor/eg/eg.go
refactor/eg/eg_test.go
refactor/eg/rewrite.go
refactor/eg/testdata/no_after_return.template
refactor/rename/rename.go
Please commit your changes or stash them before you merge.
Aborting
Updating 73e16cff..d74aaa1f
package golang.org/x/tools/cmd/stringer: exit status 1
make: *** [devtools] Error 1
Hoping that this wasnt too serious, I tries creating my go file with the following line:
abigen -abi=Store.abi --pkg=store --out=Store.go
bash: abigen: command not found
I suspect this has to do with my failed install and will appreciate any pointers on this
I was able to solve this by doing the following:
cd /Users/pro/go/src/golang.org/x/tools (This is the repo that had the issues)
Run git stash save && git merge origin/master && git stash pop
Run make devtools (The make file contains the instructions for installing abigen and placing it in usr/bin
I got help from this answer:
How do I ignore an error on 'git pull' about my local changes would be overwritten by merge?

Yocto build broken when setting a remote rpm repository with https

I have generated a Yocto image to be used on all my target devices. When that image is running on target devices, it must be able to be updated using a rpm remote repository through https protocol.
To try doing that, I have added a dnf bbappend to my custom layer:
$ cat recipes-devtools/dnf/dnf_%.bbappend
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += " \
file://yocto-adv-rpm.repo \
"
do_install_append () {
install -d ${D}/etc/yum.repos.d
install -m 0600 ${WORKDIR}/yocto-adv-rpm.repo ${D}/etc/yum.repos.d/yocto-adv-rpm.repo
}
FILES_${PN} += "/etc/yum.repos.d"
This is the content of repository configuration file included by dnf bbappend recipe:
$ cat recipes-devtools/dnf/files/yocto-adv-rpm.repo
[yocto-adv-rpm]
name=Rocko Yocto Repo
baseurl=https://storage.googleapis.com/my_repo/
gpgkey=https://storage.googleapis.com/my_repo/PACKAGEFEED-GPG-KEY-rocko
enabled=1
gpgcheck=1
This repository configuration breaks the build process of the image. When I try to build myimage recipe, I always get this error:
ERROR: myimage-1.0-r0 do_rootfs: [log_check] myimage: found 1 error message in the logfile:
[log_check] Failed to synchronize cache for repo 'yocto-adv-rpm', disabling.
ERROR: myimage-1.0-r0 do_rootfs: Function failed: do_rootfs
ERROR: Logfile of failure stored in: /home/yocto/yocto/build/tmp/work/machine-poky-linux/myimage/1.0-r0/temp/log.do_rootfs.731
ERROR: Task (/home/yocto/yocto/sources/meta-mylayer/recipes-images/myimage.bb:do_rootfs) failed with exit code '1'
However, when I replace the "https" by "http" in "baseurl" variable:
baseurl=http://storage.googleapis.com/my_repo/
Then the myimage recipe is built fine.
The host machine can download files from the https repository using wget:
$ wget https://storage.googleapis.com/my_repo/PACKAGEFEED-GPG-KEY-rocko
Previous commands works fine, so the problem is not related with the host machine, I think it must be something related with google certificates and yocto stuff.
I found some relevant information inside this file:
yocto/build/tmp/work/machine-poky-linux/myimage/1.0-r0/temp/dnf.librepo.log
The relevant part:
15:56:41 lr_download: Downloading started
15:56:41 check_transfer_statuses: Transfer finished: repodata/repomd.xml (Effective url: https://storage.googleapis.com/my_repo/repodata/repomd.xml)
15:56:41 check_finished_transfer_status: Fatal error - Curl code (77): Problem with the SSL CA cert (path? access rights?) for https://storage.googleapis.com/my_repo/repodata/repomd.xml [error setting certificate verify locations:
CAfile: /home/yocto/yocto/build/tmp/work/x86_64-linux/curl-native/7.54.1-r0/recipe-sysroot-native/etc/ssl/certs/ca-certificates.crt
CApath: none]
15:56:41 lr_yum_download_repomd: repomd.xml download was unsuccessful
Can some of you provide any useful advice to try to fix this?
Thank you in advance for your time! :-)
I finally fixed my issue removing completely my dnf bbappend recipe from my custom layer and adding this variable to my distro.conf file:
PACKAGE_FEED_URIS = "https://storage.googleapis.com/my_repo/"
After that, at the end of the build process the image contains a valid /etc/yum.d/oe-remote-repo file and all the necesary stuff to manage it. There is no need to copy "ca-certificates.crt" manually at all.
Also, it's important to execute this command after finishing the build of the image:
$ bitbake package-index
This command generates a "repodata" directory within the package feed needed by the target device once it uses the repo to update packages using dnf client.
I found a temporal hack to fix my issue:
$ cp /etc/ssl/certs/ca-certificates.crt /home/yocto/yocto/build/tmp/work/x86_64-linux/curl-native/7.54.1-r0/recipe-sysroot-native/etc/ssl/certs/
After that, I was finally able to build the image using the "https" repo.
Now I am in the process of fixing this issue in the right way. I'll come back with the final solution.

hg-git deploy to heroku from Lion

I have this application on my Lion machine that is in a mercurial repository so I'm using hg-git to deploy it to heroku.
~/.hgrc
[extensions]
hgext.bookmarks =
hggit =
.../project/.hg/hgrc
[paths]
default = https://validhgrepo.com
[alias]
push-heroku = push git+ssh://git#heroku.com:appname.git
Then when I run hg push-heroku it should deploy, but instead I get:
caseys-MacBook-Air:project casey$ hg push-heroku
pushing to git+ssh://git#heroku.com:appname.git/
creating and sending data
["git-receive-pack 'appname.git/'"]
! Invalid path.
! Syntax is: git#heroku.com:<app>.git where <app> is your app's name.
abort: git remote error: The remote server unexpectedly closed the connection.
This doesn't make any sense. I feel like the error message is misleading because that repository DOES exist.
Also this works perfect on my ubuntu machine with a similar setup.
Turns out this was related to this issue. I hadn't noticed the extra slash before. I applied a patch similar to this guy and it worked for me (on latest hg, hg-git and osx).
Full details of how to install the patch:
first uninstall it
sudo easy_install -m 'hg-git'
then delete the hg-git egg file in ~/Library/Python/2.7/site-packages
install as directory
sudo easy_install -Z 'hg-git'
open ~/Library/Python/2.7/site-packages/hg_git..../hggit/git_handler.py
apply patch by hand (mine was more like line 1118)
--- git_handler.py Thu Jul 28 22:05:45 2011
+++ patched.git_handler.py Thu Jul 28 22:11:44 2011
## -1066,6 +1066,8 ##
port = None
host, path = hostpath.split(hostpath_seper, 1)
+ if (host.find('heroku') > 0):
+ path = path.rstrip('/')
if hostpath_seper == '/':
transportpath = '/' + path
else:
Your git remote format is screwed.
In .git/config ensure that your remote takes the format of:
git#heroku.com:appname.git
where appname is your applications name on Heroku

Resources